Adaptive Loss Scaling for Mixed Precision Training
Authors: Ruizhe Zhao
Brian Vogel
Tanvir Ahmed
Venue:  (2020)
Keywords: Deep Learning
Mixed Precision Training
Loss Scaling
Backpropagation

Abstract: Mixed precision training (MPT) is becoming a practical technique to improve the speed and energy efficiency of training deep neural networks by leveraging the fast hardware support for IEEE half-precision floating point that is available in existing GPUs. MPT is typically used in combination with a technique called loss scaling, that works by scaling up the loss value up before the start of backpropagation in order to minimize the impact of numerical underflow on training. Unfortunately, existing methods make this loss scale value a hyperparameter that needs to be tuned per-model, and a single scale cannot be adapted to different layers at different training stages. We introduce a loss scaling-based training method called adaptive loss scaling that makes MPT easier and more practical to use, by removing the need to tune a model-specific loss scale hyperparameter. We achieve this by introducing layer-wise loss scale values which are automatically computed during training to deal with underflow more effectively than existing methods. We present experimental results on a variety of networks and tasks that show our approach can shorten the time to convergence and improve accuracy, compared with using the existing state-of-the-art MPT and single-precision floating point.

Under review as a conference paper at ICLR 2020
ADAPTIVE Loss SCALING FOR MIXED PRECISION TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Mixed precision training (MPT) is becoming practical technique to improve the speed and energy efficiency of training deep neural networks by leveraging the fast hardware support for IEEE half-precision floating point that is available in existing GPUs. MPT is typically used in combination with technique called loss scaling, that works by scaling up the loss value up before the start of back- propagation order to minimize the impact of numerical underflow on training. Unfortunately, existing methods make this loss scale value hyperparameter that needs to be tuned per- per-model, and a single scale cannot be adapted to different lay- ers at different training stages. We introduce loss scaling basedd training method called adaptive loss scaling that makes MPT easier and more practical to use, by removing the need this by introducing layer-wise loss scale values which are automatically computed during training to deal with underflow more effectively than existing methods. We present experimental results on a variety of networks and tasks that show our ap- proach can shorten the time to convergence and improve accuracy compared the existing state-of-the-art MPT and single-precision floating point.
model-specific loss scale hyperparameter. We achieve
INTRODUCTION
Training deep neural networks (DNNs) is well-known to be time and energy consuming, motivating the development of new methods and hardware make training more efficient. One way im- prove training efficiency is to use numerical representations that are more hardware-friendly. This the reason that the IEEE 754 32-bit single-precision floating point format (FP32) is more widely used for training DNNs than the more precise double precision format (FP64), which is commonly used other areas of high nigh-performance computing. In an effort to further improve hardware effi- ciency, there has been increasing interest in using data types with even lower precision than FP32 for
training (Micikevicius et al. 2018 Kuchaiev et al., 2018 Wang et al. 2018 Kalamkar et al. 2019
Mellempudi et al. 2019 Sakr et al. 2019). Of these, the IEEE half pref-precision floating-point (FP16) format is already well supported by modern GPU vendors (Choquette et al. 2018). Using FP16 for training reduce the memory footprint by half compared FP32 and significantly improve the runtime performance and power efficiency. Nevertheless, numerical issues like overflow, underflow, and rounding errors frequently occur when training low precision only.
Recent works propose various improvements, of which mixed precision training (MPT) (Micike- vicius et al., 2018 the ttate-of-the-art. Its core idea is use FP16 for the compute-intensive yet operations, such as matrix multiplication, for computational efficiency, while using FP32 for the operations that require high precision, such as batch normalization (loffe & Szegedy 2015) and gradient update accumulation. Activations and gradients, which largely con- tribute to memory consumption, are stored in FP16, while the weights are stored in FP32 for more accurate accumulation of gradient updates. Even though MPT seems promising and has wide sup- port from both hardware and software frameworks, still suffers from reliability issues, mainly due to the more limited dynamic range of FP16 being unable to adequately cover possible gradient val- ues during training. The most common issue is for small gradients to fall into the underflow gap and become zero, which makes training less effective.
precision-insesiine
Loss scaling (Micikevicius et al. 2018 Kuchaiev et al. 2018 Mellempudi et al., 2019) addresses the range limitation in FP16 by introducing hyperparameter scale the loss value before the

---

Under review as a conference paper at ICLR 2020
Underflow among activation gradients accinoss
Loss scalle expected by layer
underfoow (90
6.50
20
Aar-10000
50000
6.25
Rer- R2000
Rcr-130000
6.00
5.75
10
Ker= 10000
5.25
50000
5.00
Layer
Lay
(a) Underflow rate is calculated by counting absolute gradients helow 2 24 the smallest posi- tive FP16 number.
(b) Expected loss scale of each layer is calculated by over the (0.01 N- smallest absolute gradi- ent, where N is the size of each gradient and 0.01 permitted.
the largest underflow
Figure 1:
Statistics of activation gradients collected from training SSD (Liu et al. 2016) by FP32. Data are collected from different training iterations (120k in total). Layer ID are assigned in the topological order of backpropagation computation. Layers with higher II) are closer to the input.
start of the backward pass that the computed (scaled) gradients then be properly represented in FP16 without sigmificant underflow. For an appropriate choice of a, loss scaling can achieve state of the art results that are competitive with regular FP32 training. Unfortunately, there is no single value a that will work in arbitrary models, and so it often needs to be tuned per model. Its value must be chosen large enough prevent underflow issues from affecting training accuracy. However, if a is chosen too large, it could amplify rounding errors caused by swamping (Higham) 1993] even result in overflow. This observed sensitivity the particular choice of loss scale is also reported by Mellempudi et al. (2019), who find that different values can lead to very different ResNet-50 MPT convergence behavior. Furthermore, the data distribution of gradients can vary both between layers and between iterations (Figure []. which implies that a single scale is insufficient. For instance, gradients closer the input require higher loss scale that may cause overflow or severe rounding errors if the same value were used in layers closer to the output. Including the time spent tuning a, the total training time of MPT can even exceed regular FP32 training.
We introduce a loss baaee training method called adaptive loss scaling that makes MPT easier and more practical to use. We hope that this will help to utilize better existing hardware with support for fast FP16 operations. Our method improves the usability of MPT compared to existing methods by removing the need to tune a model-specific loss scale hyperparameter, while retaining (and in some cases surpassing) the accuracy of regular FP32 training. We achieve this by introducing layer- wise loss scale values which are automatically computed and dynamically updated during training deal with underflow more effectively than existing methods. Experimental results on several examples show that MPT with adaptive loss scaling can achieve the best model accuracy and the shortest overall training time, especially when training deep models on large datasets.
BACKGROUND
2.1 PRELIMINARY
As mentioned above. MPT Micikeviciuu et al.) 2018) uses FP16 for storing the activations and gra- dients and for the most compute-intensiig tasks, while FP32 is used only where increased precision is required. FP16 has three fewer exponent bits than FP32, limiting its dynamic range to magnitudes between Umin = 2 24 and Umaa = 65505. In practice, the gradients often have a larger range than this, resulting in numerical issues when using MPT. In FP16, if the absolute actual value of a gradi- ent [g] is smaller than Uman, it will become 0: and if it is larger than Umaz. it will be infinite. Also, even if value is in [umin: Umaz), the closer it comes to either bound, the less accurate its FP16 form is regarding absolute rounding error, e.g., 1024.1 is rounded to 1024. Underflow motivates loss scaling, while the overflow and rounding error are what loss scaling should be careful with.
2

---

Under review as a conference paper at ICLR 2020
Figure 2: Comparison between the standard backpropagation algorithm and the loss scaled one. Each layer has single output in this formulation. Output finds the output layer index, Gradw calculates the gradient update with respect the weights. Backprop calculates the activation gradient given the layer type OP and the input gradient.
5N+1 + initial error gradient; 8 8N+1 f- aδN+1;
uN+1 <- initial enror gradient:
for i +- layer indices in reversed
for i t layer indices in reversed
topological order do
topological order do
j - Output (i);
j -ー Cutput (i);
W Wi+ GradW (6,); 6, - Backprop (OP (i). 8,);
W; - Wi+ Gradw (6,) /0; 6, + Backprop (OP (i), 85); end
end
Algorithm 1: Standard backpropagation algorithm.
Algorithm 2: Standard loss scaling algo- rithm. is the loss scale.
FigureZshows the basic loss scaling algorithm (Algorithm2) compared to standard backpropagation without loss scaling (Algorithm []. Note that they differ only in that in Algorithm the initial error gradients - from the output layer are scaled by before the start of the backward pass, and that the weight gradient update GradW is then unscaled by the same a just before the weight update. Recall that a should be chosen large enough to prevent underflow issues from affecting training, while also being small enough to prevent overflow. Even when kept within this range, using larger value than necessary could introduce more absolute rounding error as aforementioned. Also, since loss scaling amplifies the ratio between the largest and the smallest elements within each gradient, swamping (Higham 1993), the phenomenon that summing small values with larger ones is inaccurate in floating prating-point, becomes more likely and may hinder training (Wang et al. 2018).
2.2 RELATED WORK
Many recent works focus on reducing rounding error to improve the training performance. Wang ctt al. (2018) devise chunk-based accumulation mechanism to mitigate the swamping issue. Sakr ctt al. (211)) improve the solution to the same problem by finding lower precision for accumulation by variance analysis. Alternatively. Hoffer et al. (2018) identify the numerical issues caused by batch normalization and propose to replace it by a more numerically stable and efficient alternative. These methods are orthogonal loss scaling, and we plan to study the effect of applying them together with adaptive loss scaling as future work.
Loss scaling that aims to improve mixed precision training by reducing the underflow rate in the computed gradients can be traced back to Micikevicius et al. (2018). They originally suggest to choose constant loss scale either empirically, or using factor that cannot scale the maximal absolute gradient value to overflow. Kuchaiev et al. (2018) propose two improved versions: one is called backoff, which simply makes the loss scale smaller if numerical error encountered during training; the other is logmax, which models the maximal absolute gradient value across training iterations by log-normal distribution, in order to estimate the proper loss scale value for the next iteration. We argue that these new solutions are still not ideal, since backoff is simply trial-and-error and can waste training workload; and logmax is risky to use when we do not have much gradient values to model that log normal distribution, or this assumption cannot apply. Mellempudi et al. (2019) further study the effect of the backoff method for 8-bit floating point.
ADAPTIVE LOss SCALING
As preview, Figure 3 shows a concrete example of our adaptive loss scaling approach in the back- ward pass of 3-layer Multi-Layer Perceptron (MLP). After the forward pass has completed, starting from the rightmost node, the gradients 8 are first propagated from loss L and are then scaled by scalar 04 before being consumed by the last linear layer. The weight gradients for this layer are then scaled by 1/a, just before the weight update for W, in order to prevent the particular choice of ar from affecting the computed gradient magnitudes. is at this point that our approach begins to differ from the standard loss scaling method. In the standard method, the same 04 would be used
3

---

Under review as a conference paper at ICLR 2020
W1
W2
W3
xx 82
y 8z
y (a40)
B2a3( = a2)
3za a3)
a
y1
1/2
y3
L
(31
(1)
(B2W2) 82
(33 W3) (3
Figure 3: An example of how the adaptive loss scaling method works based on 3-layer MLP (bias and activation functions are omitted). Black and red arrows represent forward and backward propa- gation respectively, and terms beside red arrows are gradients. Section 3.1 exxlains the symbols in this figure, specifically, $1. (3, and f3 denote the loss scale values calculated locally for each layer, and at,, 022, and 0g stand for accumulated scales. 0z is optional initial scale.
for all layers in the network. However, in our method, each layer calculates its own local loss scale value B3 based on the statistics of its output gradients and weights in the current iteration, in order to minimize underflow in its computed input gradients. This 8; is then used to scale weights W; before computing the scaled input activation gradients for layer i. Since the scaling effects from these local loss scales accumulate, when unscaling gradients for updating,
the product of all previous scale values, indicated by a; a4 IT, Pj. Thus, our approach attempts minimize underflow in every layer simultancously through the use of layer-local loss scales 3; which are also computed automatically based on the current layer statistics. Compared to the standard method, this removes the need perform model-specific hyperparameter tuning and enables layer-wise loss scaling.
3.1 Loss SCALED BACKPROPAGATION
We 2-tuple notation to denote the propagated entity from layer i: (as, 5,), in which o; is the loss scale value for layer and 6, is the gradient that has been scaled by Osy- To be more specific about this notation, case, layer i takes in (0641, 8,+1). 1), updates its weight by (v4-1ō++1) /Occ41 and produces (ax, 6.) for the previous layer We will elaborate more on how a; is calculated in the following sections.
take a N-layer MLP as example (see Figure the notation). In this
Algorithm 3: Backpropagatiion algorithm with adaptive loss scaling, assuming each layer has shows how multiple outputs work.
single output. Section shows
(aN+1 6x+1) < initial loss scale, and error gradient scaled by ag
for i + layer ID in a reversed topological order do
j + routaye (i)
W, + GittGeetteighhtraddeeee (8,) /aj
3, f Get (OP (4), (a, 6,))
(a,, 0.) t (a,Bi, (ackprop B,d,))
end
Algorithm 3 shows the pseudocode for adaptive loss scaled backpropagation for the case where each layer has single output (we describe how to handle the multiple-output case in Section 3.2.2p:
1. We start with the error gradients 5x41 computed from the output loss value for the last layer N + I. We may optionally scale this gradient by ON+1- Normally we keep it as 1.
2. As visiting each previous layer in reversed topological order of the computational graph, we retrieve the tuple (ay. و,) propagated to it from the next downstream layer that repre- sents the scaled loss for layer i's output. We calculate local loss scale value 3,, which will be used to scale 6, before we calculate the activation gradients for the previous layer.
3. We use ò, and other cached inputs (omitted) to compute the gradients for W. However, since these gradients have been scaled, we must unscale them using aj before performing the weight update.
4

---

Under review as a conference paper at ICLR 2020
4. Since 3, contributes to the magnitude of the gradient 8; we calculate the loss scale value ò, to be passed to the next previous layer as ay,S,.
3.2 Loss SCALE CALCULATION
This section describes how compute the layer-wise loss scales for the various operation types that are sufficient to support its implementation in general DNNs, which broadly consists of gen- eral matrix multiplication (GEMM) and element-wise operations. GEMM the basis of linear layers, while the element-wise category covers batch normalization (loffe & Szegedy (015), ac- tivation functions, and math operations such as the dlement-wise addition commonly used in skip connections (He et al. 2016).
3.2. GEMM
We start by considering linear layer with input activations X, weights W, and output activations Y. The GEMM computation for the forward-pass then given by Y WW (ignoring the bias term without loss of generality). Now consider the backward pass for this same layer, in which the "input" to the layer consists of the tuple (a, 5) received from the next downstream layer, where = a .8 Note that a represents the total loss scale (i.e., product of all downstream layer-wise scales). As shown earlier, the weight gradients for W are computed by (x76)/a.
We assume that both W and ò can be characterized by two i.i.d. normal random variables and g, respectively, so that w ~ ((ww: ob) and g ~ N( og)) We also assume that their product wg is another random variable with pero-mean normal distribution, i.e., p (O, o2) These assumptions are standard in the literature on weight initialization (He et al.) 2015 Glorot & Bengio, 2010) and low-precision foating-poin training (Sakr et al., 2019). Since w and g are uncorrelated, of is given by (o" + 222 (a² + (*)) based on product distribution rules, and can be computed from the corresponding empirical statisties.
P characterizes the distribution of the intermediate results before the final GEMM reduction happens, so that the output is the sum of N values sampled from p. where N is the number of columns. Intuitively, if the probability that experiences underflow is reduced, the final result will have less underflow rate as well. Let the upper bound for an underflow positive value be which can take the minimal subnormal value of the given low now-precision data type, e.g., u = Umin = 2 24 for FP16, then our objective corresponds to reducing P([pl u) by scaling or g-
P(alpl S a erf
ao, v S TTu + az
u
op op
2 erf e (Tis)
We introduce a new term, Tuf that specifies the threshold for the probability of underflow for in each layer, which can also be interpreted as the upper bound of underflow rate. Suppose either w or g is scaled by a before the multiplication, then P(l I u) < Tuf becomes the expected outcome of loss scaling. Since p assumed be N'0 az), it implies that [p) is a random variable with half-normal distribution, and P(a < u) = (at(u/ v2)). Therefore, we can deduce the lower bound of loss scale for each GEMM-basee layer by equation with Tuf and u.
Get.Branchoosscalle (f 6,) {a,) +- DescSort J.v for < 1....,NA. do i N.
GEM (W, aδ, Tuf) Hes Mg: Ow,, og
f E] E( W]. Var[aō]
of - (o_ + + (0 23)
0g max 6 umar then return (a,, (0./ on )ōx) L
return u/ (a, /2 erf - (Tw)))
Algorithm 4: Loss scaling for GEMM.
end Algorithm 5: Loss scaling for branches.
In practice, take this lower bound term as the loss scale value (Section 3.2.3 presents the details for the corresponding upper bound to prevent overflow). Algorithm iillustrates the steps the loss scale calculation for GEMM-based layer. Note that the computation of a, requires the statistics
5

---

Under review as a conference paper at ICLR 2020
of and g. These in turn require computing the sample mean and variance of W and 8, which is the main source of computational overhead, with around the same computation budget as batch normalization. In our current implementation, these statistics ferred to CPU to finish the loss scale calculation. There is potential to optimize the data transfer mechanism in the future, and for now simply reduce the frequency of loss scale update if the overhead is large. For FP16-based mixed precision training, we set u to 2-24 as required by the FPI6 representation range, Tuf is set 1.0 10 in all of our experiments, which corresponds to allowing an underflow rate of 0.1%.
calculated on GPU and then trans-
3.2.2 ELEMENT- AND BRANCHING OPERATIONS
Element-wise operations can take one input argument (unary), such as activation functions; or two operands (binary), e.g., element-wise multiplication and addition. Batch normalization also falls into this category. Heuristically, we do not update the loss scale for these operations, because nor- mally they will not significantly change the amount of underflow values, and there are no statistical properties that we can directly make use of without introducing much computational overhead.
One particular element-wise operation that requires special treatment is branching. it used mainly in networks that employ skip connections, such as ResNet (He et al. (2016), DenseNet (Huang et al., 2016), etc.; it also appears in object detection models that have multiple outputs, such as SSD (Liu et al. 2016). It copies its single input to multiple branches in the forward pass, and sums all received gradients during backpropagation. The special case that we should treat deliberately is when the gradients sum have different loss scales. If were to directly sum these gradients, would no longer be able to compute the output gradient's loss scale, preventing subsequent layers from restoring the correct gradient magnitude in their weight updates.
Our solution to this issue is to rescale input gradients before the summation happens, as shown in Algorithm 5 Suppose we have N input tuples {(a,, 6,)) and aj is the maximum loss scale among them, then the rescaling works by multiplying any gradient other than 8, by a factor to produce the gradient that has the same loss scale as j. which is calculated by aj/az. And if a; too large and may cause overflow in other gradients, which is decided by checking the scaled maximum absolute gradient, we will search in a descending order of all scales among inputs until we find one.
3.2.3 POST-PROCESSING
A raw loss scale value calculated from Equation Srnaatioonshooe should be post-processed by the following rules. Most importantly, the raw loss scale should be rounded down to the nearest power-of-two number. Otherwise, due the nature of floating-pott poinn,, scaled gradient cannot always be unscaled by the same scale, i.e., (ax)//a a F x: if a is not power of two (Muller et al. (0100).
The upper bound of layer-wist wise loss scale is determined by avoiding overflow. For the GEMM case, it can be simply calculated by choosing the maximal numbers from both operands, multiplying them together, and then taking Uman over that multiplication result as the largest possible loss scale, i.e. uman/ (one//mmxxW"> X ma). This is loose bound since these selected maximal numbers may not be multiplied together when calculating the output activation gradient. In practice, this upper bound is much larger than the lower bound, and simply choose the lower bound the loss scale value, and only switch to the upper bound only when the upper bound is smaller than the lower bound. Other operators will not update loss scale (except branching, which has been discussed in Section 3.2.2,, we assume they will not cause overflow.
EXPERIMENT
This section presents various experiments to show the benefit of using adaptive loss scaling over other loss scaling approaches. The models we focus on are basically for computer vision tasks, including image classification and object detection, and their topologies vary, ranging from quential architecture to skip connection based ones. We implemented our approach using Chainer 1.0 (Toki et al. 2019). deep learning framework that supports efficient automatic differenti- ation. For comparison, we choose FP32 training, loss scaling by a fixed value (Micikevicius et al. 2018), and dynamic loss scaling by backoff (Kuchaiev al. 2018) as baselines.

---

Under review as a conference paper at ICLR 2020
4.1 CIFAR
We first evaluate our method on CIFAR-10/100 (Krizhevsky 2009) image classification using ResNet models (He al. 2016) of depth 20, 56, and 110. We explore three different loss seal- ing options: without loss scaling, fixed loss scaling (Micikevicius et al. 2018) with scales selected from {16, 128, 1024, 4096, 8192, 16384), and adaptive loss scaling (T is set to 1e 3e= The other training settings are the same as specified in the original paper (He ct al. 2016).
Table 1:
Test accuracy results for ResNet models trained on 10010-10/100, each averaged from 4 runs with different random seeds. The best result for each combination of dataset and model is bolded. Fixed loss scaling is tied with ours for the best result on ResNet-20 (C100).
CIFAR
Depth
FP32
None
Fixed (best)
Fixed (worst)
Adaptive
20
91.25%
92.16%
9
92.16%
92.26%
C10
56
93.03%
92.79%
93.29%
92.78%
93.22%
110
93.57%
93.54%
93.85%
93.73%
93.90%
20
67.94%
68.31%
68.48%
68.18%
68.47%
C100
56
71.15%
71.17%
71.56%
71.26%
71.26%
110
71.14%
72.05%
72.46%
72.34%
72.66 %
Results are in Table [ First of all, ResNet models overfit on CIFAR (train accuracy reaches 100%), such that reducing arithmetic error may decrease its regularization effect and then result in worse test accuracy. That is why adaptive loss scaling is not very beneficial for ResNet-20 and 56. But regarding ResNet- 10, since it is much deeper than the others, gradients are harder to propagate and underflow is more harmful, and arithmetic errors hinder training rather than regularizing it. More importantly, in terms of the total training time, to find the best fixed loss scale we should train times to cover all candidates, while adaptive loss scaling only needs one round.
4.2 IMAGE CLASSIFICATION
We also compare loss scaling methods on ILSVRC2012 (Jia Deng et al. 2009 ResNet- 18 and 50 (He al. 2016) are baseline. Based on the training method proposed in the original paper, we set the data type of each layer by the default mixed precision training setting, and change only the loss scaling method. Due to the limitation of resources, we can only select 128 as the fixed loss scale. We also compare with dynamic loss scaling using the backoff strategy (Kuchaiev et al. 2018).
Table 2: Image classification evaluation for different loss scaling methods. Numbers showed here are top-1 test accuracy (%). None means no loss scaling applied.
Model
FP32
None
Fixed (128)
Dynamic
Adaptive
ResNet-18 18
69.76
71.24
71.39
71.39
71.44
5e
76.15
76.07
76.02
76.12
76.22
In general, adaptive loss scaling performs the best among all MPT and FP32 training results. Note that loss scaling with a fixed arbitrary scale (128) even reduces model test accuracy for ResNet-50 compared with no loss scaling, and on the contrary, there is no hassle in hyperparameter using adaptive loss scaling. Loss scale of each layer in 18 is listed in Figure 4a It shows that our calculated loss scale is much smaller than 128, which implies 128 is too large and may cause rounding error. We further compare the maximum standard deviation Omen of all gradients at different iterations between adaptive and fixed, and we observe that the ratio of a maz of fixed over adaptive is around 20 times, both for ResNet- 18/50, which can be the major cause for accuracy drop since that high variance can increase accumulation error (Sakr al. 2019).
Results are listed in Table 2
4.3 ОBJECT DETECTION
We select the Single-Shot Detector model (Liu et al. 2016) SSD512 (VGG-16 backbone, 512 input resolution) as our baseline for the object detection task. The basic training schedule stays the same as the original paper. SSD is rather challenging model for MPT: its VGG-16 backbone is not

---

Under review as a conference paper at ICLR 2020
adaptive
Loss scalos layers RasNet-18
ierr iter=10000
16
Iten-0
iter 50000
14
Itcr-100000
1bor550000
underfoow rate
100000
Iter-150000
Loos scale
30
ibc-200000
6
4
0
5
10
20 25 30
25
Layer ID
Layer
(b) The difference in underflow rate of each layer's activation gradient comparing fixed and adaptive loss scaling.
(a) Calculated loss scale across different layers and iterations during ResNet-18 18 training.
Figure 4: Examples that show the benefit from using adaptive loss scaling. Note that the underflow rate in Figure [4b collected by subtracting the percentage of zeros of the FP16 result and the cast- to-FP32 result. The higher A is, the more effective that adaptive loss scaling can mitigate underflow.
interleaved by batch normalization layers, which implies that gradients are not normalized, and their distribution can vary a lot across layers at different depth. It also has a multi-branched topology, in which each branch detects objects at different scale and passes different values. As seen in Figure SSD512 cannot be properly scaled by fixed loss scale.
Table 3: Test performance of SSD512 models trained by different loss scaling methods, measured in mAP (%). FP32 denotes the baseline results trained in FP32, and we select the golden value from (uu et al. 2017) for the case using 32 as the batch size. None means no loss scaling is used, "Fixed (best)" stands for the best performance we can find after trying out several fixed loss scale values (including {8, 128, 1024, 2048)), and "Dynamic" shows the results of dynamic loss scaling.
Batch
FP32
None diverged diverged
Fixed (best) Dynamic
Adaptive
8
78.94
79.11
75.04
79.24
32
79.50
80.01
80.17
80.31
Table shows the comparison result. We examine two scenarios with different batch sizes yet the same total number of iterations. Without loss scaling training diverges at very early stage. Loss scaling can make mixed precision training stabler, e.g., the best fixed loss scaling results perform similarly to the FP32 baseline. When the batch size is small, dynamic loss scaling performs poorly since NaN is more frequent and the current dynamic algorithm finds it hard to choose good scale.
Our adaptive method gives the best result even compared to the FP32 baseline. Figure 4b) shows the large amount of underflow rate reduced by using adaptive loss scaling, which can give hint about why it performs better. Regarding the speed overhead of calculating loss scale, computing the statistics takes around 27% of the overall training time if update loss scale per iteration. The update frequency of adaptive loss scaling results per 100 iterations, which reduces the overhead to 0.27%. Compared fixed loss scaling, which performs worse and requires 3 rounds of training to find the best scale, our approach seems appealing for the reduction in total training time it provides.
5 CONCLUSION
This paper presents adaptive loss scaling, a method that calculates layer-wise loss scale during run- time, improve the performance and usability of MPT. Empirically we find it works better than plain MPT, existing loss scaling methods, and even FP32 in some cases, regarding model accuracy and the time taken to converge. Future work includes evaluating adaptive loss scaling on other tasks and models, especially those for Natural Language Processing; and trying to find a tighter upper bound of loss scale for each layer, e.g., based on the variance analysis in (Sakr et al. 2019), such that each layer can be scaled more effectively; extending it to FP8 is also intriguing to try.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
J. Choquette, O, Giroux, and D. Foley. Volta: Performance and programmability. IEEE Micro, 38 (2):42-52, Mar 2018.
Cheng-Yang Fu, Wei Liu, Ananth Ranga, Ambrish Tyagi, and Alexander C. Berg. DSSD Decon- volutional Single Shot Detector. 2017. URL httttt///axxiii org. http://bs/1701.... 06659
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.. In ICCV, pp- 1026-1034, 2015. ISBN 9781467383912. doi: 10.1109/CCV22015.123.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, CVPR, 2016. htt.ttt:://arxiivoooo wttp://wwww..gg/b////sss/// 1538..338....
Nicholas J. Higham. The accuracy of floating point summation. SIAM J. Scientific Computing, 14 (4):783-799, 1993.
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. In NeurIPS, 2018. URL http://arxiv.
abs/ 3bs/1803.01814
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely Connected Convolutional Networks. CoRR, abs/1608.0, 2016. ISSN 0002-9645. doi: 10.1 10.1.0.7 243.
hbcp://bb//5555 06993
Sergey Ioffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML, pp. 148-456, 2015. URL htlp::/ttxxx...oo//
abs/ abs/1502. 03167
Jia Deng, Wei Dong, R. Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierar- chical image database. CVPR, 2009. ISBN 978-1-4244-388 1-9244-3999--8 doi: 109/C//22222 5206848.
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. A Study of BFLOATI6 org/ abs 1905. 12322
for Deep Learning Training. 5 2019. URL htRthhtp://arxttt
Alex Krizhevsky. Learning Multiple I Layers of Features from Tiny Images. Technical report, 2009.
Oleksii Kuchaiev, Boris Ginsburg, Igor Gitman, Vitaly Lavrukhin, Jason Li, Huyen Nguyen, Carl Case, and Paulius Micikevicius. OpenSeq2Seq. 2018.
Training for NLP and Speech Recognition with 111.0110001..7777
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yar Fu, and Alexander C Berg. SSD: Single Shot MultiBox Detector. In ECCV, 2016. URL https:
Naveen Mellempudi, Sudarshan Srinivasan, Dipankar Das, and Bharat Kaul. Mixed Precision Train- ing With 8-bit Floating Point. 2019. URL URL http:://atxxxii org/ abs 1905. 12334
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg. Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed Precision Training. In ICLR, 2018. htt.:tt:p/axxxi.ooog...
wttp:o/g/aabssasaaa 03740
Jean-michel Muller, Nicolas Brisebarre, Florent De Dinechin, Claude-piers Jeannerod, Guillaume Melquiond, Nathalie Revol, and Serge Torres. Handbook of Floating- PPint Arithmetic. 2010.
ISBN 97808 76470449.

---

Under review as a conference paper at ICLR 2020
Charbel Sakr, Naigang Wang. Chia- Yu Chen, Jungwook Choi, Ankur Agrawal, Naresh Shanbhag, and Kailash Gopalakrishnan. Accumulation Bit- Width Scaling For Llw Precision Training Of Deep Networks. In ICLR, 2019. URLhttp: http::://axxvvvoo org/ abs /10..191..... 06588
Seiya Tokui, Ryosuke Okuta, Takuya Akiba, Yusuke Nitani, Toru Ogawa, Shunta Saito, Shuji Suzuki, Kota Uenishi, Brian Vogel, and Hiroyuki Yamazaki Vincent. Chainer: A deep learn- ing framework for accelerating the research cycle. Proceedings of the 25th ACM SICKDD International Conference on Knowledge Discovery & Data Mining, pp. 2002-2011 ACM, 2019.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Ye Yu Chen, and Kailash Gopalakrishnan. Train- ing Deep Neural Networks with 8-bit Floating Point Numbers. In NeurIPS, 2018. URL
http:/ /arpvvaxvvvv..vv////v//eae/aaaiv org/ abs 181. 08011
10

---

Under review as a conference paper at ICLR 2020
A DETAILED ANALYSIS ON CIFAR RESULTS
Table shows that adaptive loss scaling is beneficial for training 1ess while less advanta- geous for ResNet-20 and ResNet-56. We hypothesize the reason behind is that underflow causes more numerical problems when the model is deeper. For shallower models, the difference between the oracle gradient values and the underflowing ones is moderate and can even be viewed as form of regularization. This argument is supported by the fact that the training accuracy of ResNet mod- els on CIFAR can always reach 100%. In this way, even though adaptive loss scaling can improve the accuracy of the computed gradients, this does not necessarily always translate to improved test accuracy.
Table 4: The effect of different fixed loss scales on the test accuracy, which is measured for ResNet- 20 and ResNet-56 on CIFAR- 10. Numbers on the first row give the fixed loss scales.
Model
16
128
1024
4096
8192
16384
ResNet-20
92. 92.16%
92.19%
92.16%
92.20%
92.24%
92.24%
92.24%
ResNet-56 92.80%
93.28%
92.79%
93.08%
93.19%
93.19%
93.19%
We dive deeper into this argument by reviewing Table which shows the test accuracy of the two shallower ResNet models on CIFAR-10. For both models, the test accuracy first increases to maxima at 16, then there is sudden drop 128, and finally it climbs up to plateau. Our hypothetical interpretation is as follows:
1. Initially the test accuracy is low. Here the underflow rate is expected to be at its highest, and it is the major cause for the low test accuracy.
2. The test accuracy then increases with loss scale, mainly due to the mitigation of underflow by loss scaling. However, the gradients become more accurate, the regularizing effect from underflow is also reduced and the test accuracy will drop, until the loss scale reaches around 128.
3. If the loss scale continues increase, the high rounding error and swamping problem caused by large scales will arise. It adds another kind of regularization, which is relatively more harmful than what underflow may cause, and the test accuracy cannot improve much.
Even though this interpretation is hypothetical, this empirical evaluation in Table 4 Isoows that the relationship between the goodness of loss scaling scheme and test accuracy is complicated when the model tends to overfit.
B EFFECT FROM DIFFERENT LOSS SCALES ON SSD
Here we present how changing fixed loss scales will affect the SSD training result, in order to understand the benefits of both training time and model accuracy from using adaptive loss scaling.
Table 5: Changes in mAP after changing the fixed loss scales. Batch size 8.
Loss scale
8
16
128
1024
2048
mAP (%)
diverged 24.62 diverged
79.04
79.04
79.11
Table 5) gives all the current empirical results. No loss scaling and fixed loss scaling with scale as 16 are both diverged. 8 almost does not improve during training. { (128, 1024, 2048) all give good results compared to other scale candidates.
11


Reviews:
[{"review_id": "rJgz741FFr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an adaptive loss scaling method during the backpropagation stage for the mix precision training to reduce the underflow. Compared with the previous work, which scales the loss by human design, and needs to be consistent in all layers. The authors state that they can decide the scale rate layer by layer automatically to reduce the underflow in a low precision situation. \n\nThey calculate the scale rate using the statistic information of the layer weight and gradient. By adaptively scale each layer’s loss and gradient, this method can reduce the underflow rate better than the previous work. Additionally, the authors claim that the computation overhead is not significant, so it is efficient to use rather than searching from a set of fix scale rates. \n\nThe experiments present on image classification and objective detection benchmarks. From the result, we can see that the adaptive loss scale reaches a relatively high point on all the tasks.\n \nPros:\n \n- The method is straight forward and easy to understand. The motivation is good. They get some impressive results on ResNet110 and SSD512 comparing with the fixed scaling method. Besides, they give some analysis of their advantages and disadvantages in different networks, which looks promising to me.\n \nCons:\n\n- One question in Section 3.2.1, the assumption that w, g, p can be treated as the random variable with Gaussian distribution seems not natural in the training process. Especially p is a zero-mean distribution. The cited paper uses this assumption in a more convincing case, such as the weight initialization task. Notice that He et al., 2015 claim that the product of weight and gradient can be a zero-mean normal distribution is based on the weight is a symmetric distribution around zero, which is not true in neither this paper’s assumption nor the real training situation. \n\n- In the objective detection part, I can not find which dataset the authors use. Though the author state that they follow Liu et al., 2016 ’s work, there are also several tasks in Liu’s paper, and I can not directly match the resulting point with any of those tasks, which makes me hard to confirm the experiment result.\n\n- The experiment setting is unclear. Here are two questions. 1, What is the initial scale at the last layer? It should be manually designed in the experiment, and I think this value may affect the other layer’s scale as well. If the algorithm is robust for this scale, it is better to show some study on that. 2, What update frequency is used in the experiment? The authors say that the overhead can be reduced by reducing the frequency, but they do not clearly show which frequency they use in their experiment, if the frequency does not affect the performance, it is also better to claim or show some study on that.\n\nMinor comments:\n- Figures can use a larger font.  Figures 4a and 4b can be aligned better. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "B1xU9sLaKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a method to train models in FP16 precision. The authors show that the key reason of training performance drop is the overflow or underflow of back propagation information. Instead of using a fixed value or dynamic value proposed by a previous work, this paper adopts a more elaborate way to minimize underflow\nin every layer simultaneously and automatically based on the current layer statistics. Experiment results on CIFAR10, ImageNet and Object Detection models are conducted to demonstrate the effectiveness of the proposed method.\n\nThere are some concerns about this paper:\n1. This paper tries to solve a very practical problem which is good, however, the stability of this method which is very important for real applications remains unclear. More networks such as VGG/ResNet/depthwise-conv based networks, more initialization methods (such as gaussian, xavier, kaiming), w/o bn layers,  and more tasks such as segmentation and detection with different batch sizes are strongly recommended to make this work more solid.\n2. In the experiments, it seems that dynamic loss scaling method works well too except on the configuration of SSD batchsize=8. Why dynamic loss scaling fails on this case? More detailed analysis are recommended to show the advantage of the proposed method.\n3. In  many experiments, it seems adaptive loss scaling with FP16 is even better than FP32, is this stable? Could we further improve the FP32 results if using dynamic / adaptive loss scaling? ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1giC2t7cH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper mostly reads well. It proposes to use statistics from previous activations to compute and adaptive scaling of the loss such that the amount of underflow is minimized. The scaling is defined per layer. Experiments are carried for various model sizes and datasets. \n\nIf anything I think the paper can do a better job at centralizing (maybe in an appendix) the gritty details (e.g. how the stats are computed etc). Unfortunately, the best way of doing this might be in the form of code or maybe pseudocode, but being quite explicit in all technical details. \n\nRight now this is mentioned in the text (same way as batch norm stats if I understood correctly, based on the current minibatch). Though is not clear how the variance on w is treated. How is the variance on dirac delta (backpropagated error) is converted into a scalar (that will be for the entire loss).", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes to improve mixed precision training by adaptively scaling the loss based on statistics from previous activations to minimize underflow during training. However, the method is designed rather heuristically and can be improved with stronger theoretical support and improved representation of the paper. 

Decision: Reject