Revisiting BFfloat16 Training
Authors: Pedram Zamirai
Jian Zhang
Christopher R Aberger
Christopher De Sa
Venue:  (2021)
Keywords: 16-bit training
Low precision training
Deep learning hardware

Abstract: State-of-the-art generic low-precision training algorithms use a mix of 16-bit and 32-bit precision, creating the folklore that 16-bit precision alone is not enough to maximize model accuracy. As a result, deep learning accelerators are forced to support both 16-bit and 32-bit compute units which is more costly than only using 16-bit units for hardware design. We ask can we do pure 16-bit training which requires only 16-bit compute units, while still matching the model accuracy attained by 32-bit training. Towards this end, we study pure 16-bit training algorithms on the widely adopted BFloat16 compute unit. While these units conventionally use nearest rounding to cast output to 16-bit precision, we show that nearest rounding for model weight updates can often cancel small updates, which degrades the convergence and model accuracy. Motivated by this, we identify two simple existing techniques, stochastic rounding and the Kahan accumulation, to remedy the model accuracy degradation in pure 16-bit training. We empirically show that these two techniques can enable up to 7% absolute validation accuracy gain in pure 16-bit training. This leads to 0.1% lower to 0.2% higher matching validation accuracy compared to 32-bit precision training across seven deep learning applications. 

Under review as a conference paper at ICLR 2021
REVISITING BFLOAT16 TRAINING
Anonymous authors
Paper under double-blind review
ABSTRACT
State-of-the-an generic low- uww-prcission training algorithms use mix of 16-bit and 32-bit precision, creating the folklore that 16-bit precision alone is not enough to maximize model accuracy. As a result, deep learning accelerators are forced to support both 16-bit and 32-bit compute units which is more costly than only using 16-bit units for hardware design. We ask can we do pure 16-bit training which re- quires only 16-bit compute units, while still matching the model accuracy attained by 32-bit training. Towards this end, study pure 16-bit training algorithms on the widely adopted BFloat16 compute unit. While these units conventionally use nearest rounding to cast output to 16-bit precision, we show that nearest round- ing for model weight updates can often cancel small updates, which degrades the convergence and model accuracy. Motivated by this, we identify two simple exist- ing techniques, stochastic rounding and Kahan summation, to remedy the model accuracy degradation in pure 16-bit training. We empirically show that these two techniques can enable up to 7% absolute validation accuracy gain in pure 16-bit training. This leads to 0.1 1 lower 0.2% higher matching validation accuracy compared to 32-bit precision training across seven deep learning applications.
INTRODUCTION
Recently there has been an explosion in the compute resources required for training deep learning models (Shoeybi et al., 2019
Rajbhandari al. 2019: Real al. 2019). As result, there has
been broad interest in leveraging low- ow-precision (< 32- training a algorithms to reduce the required
compute resources (De Sa et al. 2017 Hubara et al. 2017 Gupta al. 2015). Among these algo-
rithms, mixed-precision training in which model activations and gradients are stored using a 16-bit floating point format while model weights and optimizer states use 32-bit precision-is commonly used when training generic deep learning models (Micikevicius et al. 2017 Kalamkar et al. 2019).
While there wide body of literature showing that low-precision training can minimally impact accuracy on specific models (Wang et al. 2018b De Sa et al. 2015 Zhang et al. (017)) conven- tional wisdom suggests that least some 32-bit computation required as fail-safe in generic deep learning training. As such, new accelerator architectures for deep learning are forced to support both 32-bit and 16- compute units. This much more costly in terms of area, power, and speed when compared to hardware with only 16-bit compute units (Horowitz) 2014 Galal et al. 2013).
In this paper we question if 32-bit compute units are truly needed for new deep learning hardware accelerators. Namely, can we match the model accuracy of 32-bit-precision algorithms while lever- aging only 16-bit compute units? To answer this question, we study pure 16-bit training algorithms, ones which use only 16-bit compute units and which store activations, gradients, model weights, and optimizer states all in 16-bit precision. Specifically, we focus on training with the 1 com- pute unit which is widely adopted in modern deep learning accelerators (Jouppi et al. 2017 Burgess al. 2019). Such units take 16-bit inputs, perform computation, and then round the results to 16- bit output. BFloat 16 compute units can provide 3x higher power efficiency, 1.5x lower latency, and 1.5x less chip area than 32-bit units (Horowitz) 2014; Galal et al. 2013). In addition, pure 16-bit training algorithms can reduce the memory footprint and bandwidth consumption of model weights and optimizers by 2x compared to mixed precision or 32-bit precision training, especially for large models with billions of weights (Shoeybi et al. 2019 Rajbhandari et al.] 2019). Developing reliable pure 16-bit training algorithms will enable hardware designers to realize these advantages.
The simplest approach to pure 16-bit training is to take 32-bit baseline and "make it oow-precision" by replacing all the bi- numbers with 16-bit numbers and replacing each 32-bit floating-poii op-

---

Under review as a conference paper at ICLR 2021
eration with its 16-bit analog, using nearest rounding to quantize necessary: call this approach the standard algorithm. Unfortunaelly, we show empirically that standard pure 16-bit training does not match 32-bit training on model accuracy across deep learning models. For example. the stan- dard pure 16-bit training algorithm one would run on conventional hardware attains 16% and 7% lower training and validation accuracies than 32-bit baseline. Motivated by this observation, start by analyzing what factors limit the model accuracy of this standard pure 16-bit algorithm.
The goal of our analysis is to inspire clean, minimal set of simple techniques that allow pure 16-bit training to attain strong model accuracy for state-of-the-art deep learning models across applica- tion domains. Towards this end, we derive insights from a simple seast-squares regression model in Section g Using this least-squares regression model, we reveal that nearest rounding of compute unit outputs causes significant convergence degradation and consequent model accuracy loss. More concretely, we show a key theoretical insight hidden in existing work: when running stochastic gra- dient descent on least-squares regression model, nearest rounding while updating model weights ignores small updates. This phenomenon significantly degrades the convergence of stochastic gra- dient descent when model updates become small relative model weights, which is also what observe when training deep learning models. In comparison, nearest rounding in the forward and backward pass of backpropagation has a negligible impact on convergence. These insights lead us to consider two simple existing techniques to achieve high-accuracy pure 16-bit training. First, we can use stochastic rounding instead of nearest rounding for the model weight updates. Here, the rounded weights become an unbiased estimate of the precise weights without rounding: thus, regardless of the magnitude of updates, the expectation of rounded weights converges at the same speed as the precise weights. Second, we can use the well-known Kahan summation algorithm (Kahan 1965) to accumulate model updates while still keeping nearest rounding for all operations. This method tracks and compensates weight rounding errors across iterations with auxiliary 16-bit values, which avoids catastrophic cancellation of many small model weight updates.
Empirically, in Section we first validate that, as suggested by our theory, nearest rounding for model weight updates is the sole bottleneck for convergence and model accuracy several deep learning models. We then demonstrate that pure 16-bit training using stochastic rounding or Kahan summation model weight updates can match 32-bit training in model accuracy across wide range of applications (He et al. 2016 Amodei et al. 2016 Devlin et al.) 2018 Naumov er al., 2019). To validate that nearest rounding for model weight updates is the cause of the accuracy degradation, we show that if we store model weights in 32-bit precision without rounding during weight updates, and we keep using 16-bits and nearest rounding for all other operations. then the attained model accuracy matches full 32-bit precision training. Next, we demonstrate that 16-bit training with stochastic rounding for weight updates attains model accuracy matching 32-bit training for five out of seven applications in our study. Note that while it works most of the time, this is not silver bullet, as using stochastic rounding alone could not fully match bi- training on all models. To address this, we show that Kahan summation for model weight updates closes remaining gaps on all the models we consider; this Kahan summation comes with a trade off, as it requires 2x weight memory, but achieves up to 0.2% higher validation accuracy than stochastic rounding. Our results suggest that deep learning accelerators using only 16-bit compute units are feasible if stochastic rounding and Kahan summation are supported respectively by the hardware and the software stack.
2 PRELIMINARY
In this section we establish the background and notation for our study and present the prelimi- nary observations that motivate work. We focus on the case of stochastic gradient descent (SGD), which is the primary workhorse used to train deep learning models. SGD computes gradi- ents from subset of training samples, and uses them to update the model weights as to decrease the loss in expectation. In the classic supervised learning setting, let (X, y) be a dataset where X [01,.2 T2, ..., xx,] E Raxd and y = (V1, .... ..... Vno) E " On this dataset, we use stochastic gradient descent to optimize loss function f(w)= = 1/n Σ -= fi( Z,, defined by the model. At the f-th iteration, we sample an index subset (t)) C {1, n and compute a sample gradient V fa (w,) as an unbiased estimate of the full gradient V f(w). In deep learning, model training can be described as a compute graph where the compute graph operators such as addition and ma-
This nearest rounding is the standard rounding mode for compute unit output commonly supported across hardware platforms (Intel 2018. Nvidia 2020
2

---

Under review as a conference paper at ICLR 2021
trix multiplication are the nodes. For example, the model weight update operator is defined as the subtraction in the operation W1+1 = w - a fa(t) (a(e((w)) (w,) which updates the model weight w.
16-bit Compute Units On modern hardware, numerical operators in the compute graph are sup- (FMAC) compute units. These units work by computing a +- a + (ar x y), where x and y are input floating point numbers, and a is an accumulator that is part of the FMAC unit. Importantly, for a 16-bit FMAC unit, the accumulator has higher-than- 16- bit precision. This higher precision accumulator is inexpensive compared to the multiply in terms of chip area and energy consumption in FMAC units, but is critical to the numerical accuracy of operations such as matrix multiplication and convolution. Thus 16-bit FMAC units with higher precision accumulator is standard for modern hardware including TPUs and GPUs (Chao & Saeta 2019 Markidis et al. 2018 Stephens 2 this will likely continue to be standard in emerging accelerators. Because of the higher precision accumulator, the result in the accumulator then needs to be rounded to 16-bits before it is output from the FMAC unit (e.g. before writing to memory). FMAC units use the same hardware implementation to support all operators from simple additions to computationally intensive convolutions, so this output-roundin step happens for all the operators in a compute graph.
ported by fused
Nearest Rounding FMAC output rounding widely implemented with nearest rounding as the standard mode, due to its efficient support across hardware platforms. "Nearest rounding" means rounding a higher precision floating point number to the closest low-precision representable value. Given that the add step already uses accurate higher preicision accumulators, this nearest rounding is the primary source of numerical ervors for training using 16-bit FMAC units. In this context of 16 bit FMAC units and nearest rounding, we discuss the following training-precision approaches.
In 32-bit precision training, all the compute graph operators read and write memory using 32- bit precision. These operators require 32-bit compute units, which constrains the compute and memory efficiency.
In mixed precision training, model weights are stored in 32-bit precision while activations and gradients use 16-bit precision. Thus, new accelerators customized to maximize efficiency for mixed precision training still require 32-bit compute units for operators involving 32-bit weights as the input; this has lower efficiency in power, speed and chip area than only using 16-bit units.
In pure 16-bit training, activation, gradients and model weights are all stored in 16-bit precision. All operators use pure 16-bit input and write out 16-bit output after rounding. Thus, aside from just saving memory, pure 16-bit training can eliminate the requirement for 32-bit compute units. This opens the possibility for highly-efficient accelerators using only 16-bit compute units. In spite of this favorable efficiency, we now show that it can be surprisingly challenging for standard pure 16-bit training to match 32-bit training on model accuracy.
Motivating Observations Although recent works have shown that certain models are robust to numerical error during train- ing (Wang et al., 2018b De Sa al. 2015: Zhang al. 2017 surprisingly, we observe that is challenging for pure 16-bit train- ing to attain the same accuracy as bi- precision training on sev- eral state-of-the-art deep learning models. To demonstrate this, we compare 32-bit precision training and standard pure 16-bit train- ing (using nearest rounding for all its operator outputs). For ex- ample, Figure [ iillsstrates that for a BERT- BERT-Base model for natu- ral language inference, the standard pure 16-bit training algorithm demonstrates 16% and 7% lower training and validation accura- cies than 32-bit precision training This gap suggests that nearest rounding is the primary source of numerical error in pure 16-bit algorithms, significantly degrading the convergence and model accuracy. To alleviate this problem, in Section we study how nearest rounding impacts convergence, and we expose insights which lead to simple techniques to improve the model accuracy in pure 16-bit training.
Assuncyy
80
32-bic
Standard 16-bit
5K
10K
15K
Iterations
Figure 1: Standard pure 16- bit training shows lower train- ing accuracy compared to 32- bit training on a BERT model.
We note that 32-bit and standard 16- bit training in Figure I start from the same initialization with the same model accuracy. The gap at the beginning of the two curves is due to curve smoothing for visualization clarity. We refer Appendix DJ for the unsmoothed curves which start from the same model accuracy.
3

---

Under review as a conference paper at ICLR 2021
3 PRECISE MODEL WEIGHT UPDATES ARE ALL YOU NEED
To understand how to improve the model accuracy attained by pure 16-bit training, in this section we analyze the impact of nearest rounding, the primary source of numerical error in the standard 16-bit training algorithm. In Section 3.T we show that when model updates are small relative to the model weights, nearest rounding for model weight updates ignores small updates and consequently impedes convergence of stochastic gradient descent. In contrast, we show that nearest rounding in the forward and backward compute can have a much weaker impact on the convergence throughout training. These insights emphasize the importance of more precise model weight updates for pure 16-bit training. To achieve such precise updates using pure 16-bit training. in Section 322 we con- sider using two simple existing numerical techniques, stochastic rounding and Kahan summation, for model weight updates. Although we use a least-squares regression model for its simplicity in exposing insights, will show in Section 3 that our insights transfer empirically representative deep learning models.
3.1 UNDERSTANDING THE IMPACT OF NEAREST ROUNDING
In this section, we use simple least-squares regression model 2n Σ ||o y4 11 with batch size I as a proxy to expose the impact of numerical errors due to nearest rounding. First, we discuss the impact of nearest rounding for model weight updates. Then, we show that nearest rounding for forward and backward compute has comparatively much weaker influence on accuracy. More concretely, focus underdetermined least squares regression problems with high dimensional model weights and input features; this is intended to reflect the overparameterized setting in modern deep learning models (Li et al., 2018 Jacot al. 2018). In this setting, the model has the capacity to perfectly fit the dataset with yi 2. w² where wt is the minimizer of f(w) (A1). To ensure convergence with a bounded gradient variance, we assume bounded input data ||>; 1 s L (A2). We let denote the machine epsilon of our floating point format, such that if u and v are adjacent representable numbers in our floating point format, e|ul S Iu - vl S 2e lu] Under this stan- dard assumption for numerical analysis on floating point numbers (Stoer & Bulirsch 2013). nearest rounding Q (-) will have a bounded error of Q(u)) u S c/ul for any in range. To simplify the presentation, we ignore overflow and underflow in our analysis here, and disregard factors of e² (as standard in analyses of floating point error).
Nearest Rounding for Model Weight Updates When stochastic gradient descent updates model weights with nearest rounding, the model weights evolve as w = Q (w, - a (o()) (ur)) For a weight dimension i, if the model update SaVi (w:),, is smaller than half of the difference between (ro.]: and its neighboring representable value in a certain precision format, nearest rounding cancels this model update. This often emerges in the mid-to-late training stage when the magnitude of gradient becomes small or learning rate decays small. Formally, we show that nearest rounding can cancel updates across all dimensions for least-squares regression model when approaching the optimal weights in Theorem D we defer the proof to Appendix A
Theorem 1. Consider running one step of SGD on a least-squares regression model under assump- tions Al and A2. The model weight update will be entirely canceled by nearest rounding if
w w" iI aL+E
min [w]
where w denotes the j-th dimension of the optimal solution w = arg min f(w). Addition- ally, if we run multiple steps of SGD using nearest-rounded weight updates and fixed learning rate S L,L, then the distance of the weights w, at any timestep f.rrom the oplimum is
bounded by
<c(1 aL) min u3l 1.l.||l lwwo w0
[w w01l min
aL +
Theorem reveals that for the least-squares regression model, nearest rounding cancels the entire model updates when the distance towards the optimal solution w" is small relative to the magni- tude of w² Thus in the late stage of training, the model weights can halt in a region with radius azi+c min, w around w". Our lower bound shows that this region limits the convergence of SGD with nearest rounding on model weight updates for least-squares regression models. In this bound, the key property is the dependency on the step size: as the step size becomes small, this error lower bound becomes worse, which is the opposite of the usual effect of diminishing the step size in SGD.
4

---

Under review as a conference paper at ICLR 2021
Given that w" can be arbitrarily far from the zero vector, our lower bound also reveals substantial convergence degradation for stochastic gradient descent using floating point numbers. This specific degradation, which depends on the magnitude of w², does not emerge in training with fixed point numbers which is a setting widely used for analyzing the impact of rounding (Hou et al. 2018 Li etal. 2017). Because the lower bound on || - =w'l is also in the order of O (c), this bound is worse for lower precision formats with a large c value. In Section we will empirically show that these insights from the least-squares regression model can also generalize to deep learning models, which explains the convergence and model accuracy degradation due to the small updates cancellation in model weight updates.
Nearest Rounding for Forward and Backward Compute In contrast to the significant conver- gence degradation imposed by nearest rounding on model weight updates, we show that the nearest rounding in the gradient computation (in the forward and backward passes of backpropagation) can impact convergence minimally. To show this, we consider stochastic gradient descent with nearest rounding only for compute operations which generate activations and gradients. Here to compute the gradient for caat-squares regression models, the linear layer passes the rounded activation a; = Q (ar" uo 34) to the loss layer. (We see no quantization error within the dot product ar. w itself, as all accumulation here is done with the petterr-reciiion accumulator of the FMAC.) In the backward stage, the loss layer feeds the rounded activation gradients Ja. Q (a:) back to the linear layer. The weight gradient is then computed as VQli(w) =- Q (9,,, Q ( (Q (at w, y/) æ=)) To isolate the impact of nearest rounding for activations and gradients, we do not round model weights in this setting. Formally, upper bound on l|w, - w=1 which can be substantially smaller than the lower bound in Theorem [
show that SGD with activation and gradient rounding allows for an
Theorem 2. Consider running multiple steps of SGD on least- masstssquarss regression model under assumptions Al and A2, using nearest rounding for only forward and backward compute, but exact arithmetic for model weight updates. Then if the step size is small enough that o S 1/T,, i, the distance of the weights we at any timestep t. f.rrm the optimum will be bounded by
E [[l - =011 [12] S exp (-sppl (1-2))
-12000 - ==
where μ is the smallest eigenvalue of the data covariance matrix E/= x,a""
As can be made arbitrarily large, this bound guarantees us substantially more accurate solutions than the lower bound attained by using nearest rounding only for model weight updates in Theo- rem M This shows that rounding for the weight updates is the primary source of error, as even with all other operations quantized, the algorithm is guaranteed to converge closer to the optimum than is even possible with just the weight updates rounded with nearest rounding. Note that the bound in Theorem 2 is able to guarantee arbitrarily accurate solutions because ignore underflow here. In practice, precision would eventually be limited by underflow even in the setting of Theorem 2 however, the underflow threshold for B is small enough that this represents level of error that deep learning applications are generally able to tolerate. We refer to Appendix A for the proof.
Theory Validation To validate our insights, we compare the impact of nearest rounding for model weight updates against that of nearest rounding in forward and backward compute on a synthetic 10-dimensional least-squares regression problem. Specifically, the input data are sampled from a zero-mean unit- variance normal distribution while the model weight is gener- ated from a uniform distribution in the range of [0, 100). We perturb the label with a zero- mero-mean normal distribution with standard deviation 0.5. As shown in Figure 2 when using a learning rate 0.01 and 16-bit nearest rounding for model weight updates, we observe that the training loss saturates a magnitudes higher level than stochastic gradient descent with- out rounding because of updates cancellation. Meanwhile, when using nearest rounding only for forward and backward compute, the loss saturates at a level close to that attained by training without rounding. These observations align with insights on the relative impact of nearest rounding for model weight updates and for forward and backward compute.
16sht wright
1 gra aasosst mun ing
sxotred rounlagg
Training loss
10
10*
500
1000
1500
2000
Iterations
Figure 2: Theory validation. On a least square regression model, (smoothed) training losses with 16- bit nearest rounding for weight up- dates saturate at higher level than 32-bit training. With only using nearest rounding for forward and backward compute, the losses satu- rate much closer to 32-bit training.
5

---

Under review as a conference paper at ICLR 2021
3.2 HIGH-ACCURACY PURE 16-BIT TRAINING
In Section 3.1 we showed that nearest rounding for model weight updates is the bottleneck for convergence in the standard pure 16-bit training algorithm; this is because it cancels small model updates which degrades the model weight precision. This motivates us to consider two existing techniques, stochastic rounding and Kahan summatton (Kahan 1965) for improving weight updates. These techniques have been reliably applied in different numerical domains (Hopkins et al., 2020] Antonana et al., 2017) and can hypothetically enable high-accuracy pure 16-bit training. We present details on how to integrate these techniques into SGD and AdamW optimizers with pure 16-bit model weights and optimizer states such as momentum in Appendix
Stochastic Rounding Stochastic rounding for floating point numbers has been used in training certain model components (Zhang al. 2018) and can potentially improve the model accuracy of pure 16-bit training for general models. Specifically, let S be the set of all values representable by a limited precision format: the upper and lower neighboring values for a E R are au ming2a, and at = max+ I. Stochastic rounding randomly rounds a up to au with probability (a ai)/ - ai) and otherwise rounds down at We consider pure 16-bit training using stochastic rounding only for the subtraction output in the model update w; a (ott)(( (wz). We keep nearest rounding for all the other compute operations. Here, the rounded model weight is an unbiased estimate of the precise value, so it will still make progress in expectation; this prevents the halting effect from nearest rounding on model weight updates. We note that modern hardware, stochastic rounding can be implemented without any expensive multiplication
division arithmetic (De Sa et al. 2017 p. Thus using stochastic rounding for model weight updates adds minimal overhead when training modern deep learning models; we discuss explicitly how to achieve this in Appendix B.
Kahan Summation The Kahan summation algorithm uses an auxiliary variable to track numerical errors and to compensate the accu- mulation results. In the context of pure 16- bit training, we use a 16-bit auxiliary vari-
Algorithm 1 SGD updates with Kahan summation
1: Auxiliary value co <- 0 at initialization
2: Input: Model updates -a (o(a)(1))) (w,) at iter. t
3: uc+1 + --aV -a))a()(())) (w))
able C2 E Rd to track the error in model
4: yi+1 + -uc+1 Ct
> Compensate updates
weights. To ensure pure 16-bit data paths,
5: 8t-1 t- -w; + yt+1
b Accumulate updates
we keep nearest rounding for all operators in compute graphs, including those during Ka- han accumulation in Algorithm I At itera-
6: C++1 +- (8t+1 w.)
yt-1 D Measure errors
7: W[+1 +- 8241
Return: W
tion f, we first compensate the current model
update u4+1 by subtracting the previous error q. We then compute the new model weights by adding the compensated updates yt++1 to the current weights wt. We reversely subtract previous model weights W, and the compensated updates yr+1 to acquire the new numerical error C++1 in the updated weights 1r+1. For small updates u, which would cause no change in the weights after nearest rounding, this reverse subtraction records the canceled updates in the eгror Ct+1. Across it- erations, small updates can be accumulated in c++1 until ct+1 grow large enough to affect the model weights; this allows convergence to continue when it would otherwise halt due to nearest-rounding effects. In spite of the additional auxiliary value, pure 16-bit training with Kahan summation for model weight updates can still have advantages in terms of throughput and memory consumption compared to 32-bit and mixed precision training; we refer to Appendix B.2 for details.
4 EXPERIMENTS IN DEEP LEARNING
Our theory in Section reveals that nearest rounding on model weight updates is the primary source of numerical error during training. This motivates us to suggest using stochastic rounding and Kahan summation in pure 16-bit training for improved model accuracy. To first validate our theory, in this section we start by demonstrating that by ablating nearest rounding on model weight updates from the standard 16-bit training algorithm, the model accuracy gap compared 32-bit precision training can be closed on deep learning models. Next, we show empirically that with either stochastic round- ing or Kahan summation on model weight updates, pure 16-bit training can match the accuracy of 32-bit bit precision training across representative deep learning applications.
To validate the accuracy bottleneck, we consider three representative models: ResNet- 18 (He et al., 2016) on the CIFAR10 image classification (Krizhevsky al. 2009). BERT-
Experiment Setup

---

Under review as a conference paper at ICLR 2021
Table 1: Model accuracy bottleneck for the standard pure 16-bit training algorithm. This algorithm shows validation accuracy gap compared to 32-bit training. In an ablation of this algo- rithm, we use 32-bit model weights and turn off nearest rounding only on model weight updates. This eliminates the gap, suggesting that nearest rounding on model weight updates is the accuracy bottleneck.
Model
Dateset (Metrie)
32-bit
Standard 16-bit
Standard 16-bit & 32-bit weights
ResNet-18 18
CI (Ace%)
95.45 + 0.07
94.23 + 0.12
95.40 + 0.05
DLRM
Kaggle (AUC%)
80.27 + 0.01
78.49 + 0.08
80,26 + 0.01
BERT-Base
MNLI (Ace%)
84.26 1 0.08
77.53 1 0.07
84.34 + 0.04
32-bit
Standard 16-bit
Standard 16-bit w/ 32-bit weights
1
DaRMe
BERT-MNLI
100
80
Aveuracy
Accuracy
AUC
80
90
76
60
43K 8GK
102K 20HK
6K 12K
Iterations
Iterations
Iterations
Figure 3: Training accuracy gap imposed by the standard pure 16-bit training. The standard algorithm fails to match the training accuracy of 32-bit training, especially in the middle-to-Eee latee stage. We close this accuracy gap by ablating nearest rounding for weight updates from the standard algorithm. This indicates that nearest rounding for model weight update is the accuracy bottleneck.
Base (Devlin et al. 2018) on the MNLI natural language inference (Wang et al. 201 Sa). and DLRM model (Naumov et al.) 2019) on the Kaggle Advertising Challenge (CriteoLabs 2014]. To exten- sively evaluate pure 16-bit training with stochastic rounding and Kahan summation, we additionally consider larger datasets and more applications: ResNet-50 on the ImageNet (Deng et al. 2009). BERT-Base on the Wikil03 language mode I Meeity et al. 2016). DLRM model on the Criteo Terabyte dataset (CriteoLabs 2018), and Deepspeech2 (Amodei et al., 2016) on the LibriSpeech datasets (Panayotov et al. 2015p. As there is no publicly available accelerator with the software and hardware support necessary for our study, we simulate pure 16-bit training using the QPyTorch simulator (Zhang et al. 2019). QPyTorch models PyTorch kernels such as matrix multiplication as compute graph operators, and effectively simulates FMAC units with 32-bit accumulator El For all training algorithms, we use the same hyperparameters as their original papers code repositories. We report statistically meaningful results with averaged metrics and standard deviations across runs with 3 random seeds. We refer to Appendices C and for experiment details and extended results.
The Model Accuracy Bottleneck To validate our insights from Section 3 we first show empiri- cally that nearest rounding on the model weights is the primary model accuracy bottleneck on several deep learning models. To do this, we keep the model weights in 32-bit precision and turn off nearest rounding on the model weight updates while keeping nearest rounding for all other operators in the compute graph. Figure 3 shows that the standard pure 16-bit training algorithm with nearest round- ing on all operators has up to 16% training accuracy gap compared to 32- training. Although this gap can be small in the early training phase, it grows larger in later stages. In contrast, by ablating nearest rounding on model weight updates, the standard algorithm can fully match the training ac- euracy attained by 32-bit training. We notice in Table | that this ablation can also close the 1.2% to 6.7% validation accuracy gap when comparing the standard pure 16-bit training to 32-bit train- ing. These observations validate our insights from Section 3.T and motivate the use of stochastic rounding and Kahan summation on model weight updates.
We suhsample 25% of the Wikil03 and 100 hours of Librispeech training set because of the training time. 'Following the convention in mixed precision training (Micikevicius et al. (017). our simulator uses fused operators for computationally inexpensive activation and normalization layers.
7

---

Under review as a conference paper at ICLR 2021
Table 2: Pure 16-bit training can match 32-bit training on model accuracy. With stochastic rounding or Kahan summation for model weight updates, pure 16-bit training can attain 0.1% lower to 0.2% higher absolute value for validation accuracy metrics across applications.
Model
Datesct (Metric)
32-bit
16-bit Stochastic
16-bit Kahan
Standard 16-bit
ResNet-18
CIFARIO (Ace%)
95.45 + 0.07
95.33 0.08
95.36 + 0.07
94.23 + 0.12
ResNet-50
ImageNet (Ace%)
75.70 = 0.05
75.45 0.03
75.61 = 0.14
67.10 + 0.24
Kaggle (AUC%)
80.27 + 0.01
80.18 + 0.02
80.26 + 0.01
78.49 1 0.08
DLRM
Terabyte (AUC%)
80.32 + 0.00
80.25 + 0.00
80.32 0.00
78.79 + 0.02
MNLI (Acc%)
84.26 = 0.08
84.35 1
84.45
77.53 + 0.07
BERT
Wikil03 (PPL)
5.50 = 0.50
5.84 0.58
5.45 = 0.51
56.88 + 1.77
DeepSpeech2
Librispeech (WER)
62.71 = 0.07
62.85 0.07
62.87 + 0.18
69.42 0.22
32-bit.
16-bit stochastic
16-bit Kahan
Standard 16-bit
ResNet-50 50-ImageNet
MLRM-Teraaytee
BERT-Wik103
DSS2- Liri-piiis
75
80
100
Acerrey
AUC
PPL
WER
50
60
25
70
145K 290K Iterations
42K 85K Iterations
10K 21K
12K
Iterations
Iterstions
Figure 5: Training accuracy for pure 16-bit training. With stochastic rounding or Kahan sum- mation enabled for model weight updates, pure 16-bit training matches 32-bit precision training in terms of training accuracy with negligible differences across the applications in our experiments.
High-accuracy Pure 16-bit Training Next, we validate empirically that enabling stochastic rounding or Kahan summation for model weight updates allows pure 16-bit training attain match- ing model accuracy as 32-bit training. In Table 2 we first show that by using stochastic rounding for model weight updates, pure 16-bit training matches the validation accuracy of 32-bit training with at most 0.1% difference on the CIFARIO I0, Kaggle, Terabyte, MNLI and Librispeech datasets, a major- ity of the applications in our experiments. For applications where stochastic rounding still shows non-negiglable accuracy gap with more than 0.1% 1% discrepancy, we show that Kahan summation for model weight updates can enable pure 16-bit training to match the model accuracy of 32-bit train- ing algorithms. We show that Kahan summation for model weight updates can boost pure 16-bit training to higher validation accuracy than using stochastic rounding. More concretely, the Kahan summation for model weight updates shows 0.2% higher top-1 accuracy and 0.1% higher AUC re- spectively for ResNet-50 on ImageNet and for recommendation on Terabyte than using stochastic rounding. Consequently shown in Table by using Kahan summation for model weight updates, pure 16-bit training match the model accuracy attained by 32-bit precision training across all the applications in our experiments. This validates that stochastic rounding and Kahan summation can enable pure 16-bit training algorithms to match the model accuracy of 32-bit training.
Memory efficiency and model accuracy trade-off Addi- tionally, we show that stochastic rounding and Kahan sum- mation can be combined for pure 16-bit training, which exposes a memory efficiency and model accuracy trade- off for practitioners to exploit. As an example, in Fig- ure we demonstrate this trade-off by incrementally replac- ing stochastie rounding with Kahan summation on various model weights in the DLRM model on the Kaggle dataset. As we apply Kahan summation to more model weights, the weight memory cost increases by up to 2x. As this cost increases we also observe up to 0.04% improvement in AUC. This exploits memory efficiency and model ac- curacy trade-off that users should consider when deciding which technique to leverage.
$0.25
AUC
80.20
1.0
1.5
Weight memory (relative)
Figure 4: Efficiency and accuracy trade-off. With stochastic rounding and Kahan summation on different parts of the DLRM- DLRMeKaggle, it attains higher model accuracy at the cost of more weight memory.
8

---

Under review as a conference paper at ICLR 2021
5 RELATED WORK
There is a plethora of research work on low-precision training for deep learning models. On cer- tain specific models such as convolutional or recurrent neural networks, training with fixed point or floating point precisions lower than 16-bit has been shown to be feasible when using customized techniques (Wang et al.) 2018b; Zhou et al., 2016 Hubara et al., 2017 Courbariaux et al. 2014 Out et al. 2016 [un et al. 2019p- Instead of proposing new techniques for specific model types using lower than 16-bit precision, we focus on finding the minimal set of simple techniques re- quired by generic mode training future general purposs deep learning accelerators requiring only modern 16-bit compute units, Such emerging accelerators have the potential to unlock substantially improved hardware efficiency compare to those still requiring 32-bit compute units.
Recent work shows that in low precision training, stochastic gradient descent with stochastic round- ing for model weight updates only degrades worst-case upper bounds on convergence minimally; this shows stochastic rounding is an effective technique to attain strong model accuracy low pre- cision training (L. et al. 2017 Hou et al. 2018). Our analysis is orthogonal and complementary to these upper bounds. Specifically, we prove a lower bound on convergence when using standard near- est rounding for model weight updates. This lower bound shows that nearest rounding for weight updates can substantially degrade the convergence even in the most optimistic case no matter how learning rates are tuned. We use this lower bound together with the previous upper bounds in- form future accelerator designers that only supporting nearest rounding is not enough to maximize model accuracy; to alleviate this problem, stochastic rounding for model weight updates is of the minimal supports required in future training accelerators.
6 CONCLUSION
In this paper we study pure 16-bit training algorithms that require only 16-bit compute units. We show that nearest rounding on model weight updates is the primary cause of convergence and model accuracy degradation in standard pure 16-bit training. To alleviate this issue, we apply two existing techniques: stochastic rounding and Kahan summation. With these techniques, we demonstrate that pure 16-bit training can match the model accuracy of 32-bit precision training across many deep learning models. Our study suggests that it is feasible to design high-accuracy deep learning accel- crators using only 16-bit compute units if stochastic rounding and Kahan algorithm are supported.
REFERENCES
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anuhhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End-to- end speech recognition in english and mandarin. In International conference on machine learning, pp. 173-1822 182, 2016.
Mikel Antonana, Joseba Makazaga, and Ander Murua. Reducing and monitoring round-off error propagation for symplectic implicit runge-kutta schemes. Numerical Algorithms, 76(4):861-880, 2017.
Neil Burgess, Jelena Milanovic, Nigel Stephens, Konstantinos Monachopoulos, and David Mansell. Bfloat16 processing for neural networks. In 2019 IEEE 26th Symposium on Computer Arithmetic (ARITH), pp. 88-91. IEEE, 2019.
Chao and B Sacta. Cloud tpu: Codesigning architecture and infrastructure. In Hot Chips, vol- ume 31, 2019.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David, Training deep neural networks with low precision multiplications. arXiv preprint 1rXiv: 2014.
CriseoLabs. Display advertising challenge, 2014. http:///g.....ll.....//..ooo.... 02 hhhp:poleeeaiippp--aaeertaaaaaa-
CriteoLabs.
Criteo 1tb click logs dataset, 2018. wow.icck-ooosdaaeeee
URL

---

Under review as a conference paper at ICLR 2021
Christopher De Sa, Matthew Feldman, Christopher Ré, and Kunle Olukotun. Understanding and optimizing asynchronous tow-precision stochastic gradient descent. In Proceedings of the 44th Anmual International Symposium Computer Architecture, pp. 561-574, 2017.
Christopher M De Sa, Ce Zhang, Kunle Olukotun, and Christopher Ré. Taming the wild: A unified analysis of hogwild-style algorithms. In Advances in neural information processing systems, pp. 2674-2682, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /8..vv:8//0.0 1810004800 2018.
Sameh Galal, Ofer Shacham, John S Brunhaver II, Jing Pu, Artem Vassiliev, and Mark Horowitz. Fpu generator for design space exploration. In 2013 IEEE 2 Symposium on Computer Arith- metic, pp. 25-34. IEEE, 2013.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning, pp. 1737-1746, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pailern recognition, pp- 770-778, 2016.
Michael Hopkins, Mantas Mikaitis, Dave R Lester, and Steve Furber. Stochastic rounding and reducce--recission fixed-point arithmetic for solving neural ordinary differential equations. Philo- sophical Transactions of the Royal Sociery 3782116)):0010052, 2020.
Mark Horowitz. 1.1 computing's energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pp. 10-14. IEEE, 2014.
Lu Hou, Ruiliang Zhang, and James Kwok. Analysis of quantized models. In International Conference on Learning Representations, 2018.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Jour- nal of Machine Learning Research, 18(1):686968888 2017.
Intel.
Bfloat16-hardware numerics definition, 2018.
URL
https: //
Boftware wa www.wwreee.m httt/eewrr..itaeecccc////oo//tte html
lop /own/dwwn..../ dodwwwwwdeee
oat aat.::/.eaaarrteeeee-eeerrrreree
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel:
Convergence and gen-
eralization in neural networks. In Advances in neural information processing systems, pp. 8571- 1-
8580, 2018
Norman P Jouppi, Cliff Young. Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analy- sis of tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, pp. 1-12, 2017.
W Kahan. Further remarks on reducing truncation errors, commun. Assoc. Comput, Mach, 8:40, 1965.
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al. A study of 6 for deep learning training. arXiv preprint arXiv: 190e:/222.2222 12322, 2019.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 dataset, 2009. //www. cs. wwtrontto.ed//krri//// html
URL https:
10

---

Under review as a conference paper at ICLR 2021
Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, and Tom Goldstein. Training quantized nets: A deeper understanding. In Advances in Neural Information Processing Systems, pp. 1015581111222 2017. 2017.
Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in werrparameterizzd matrix sensing and neural networks with quadratic activations. In Conference On Learning The- ory, pp. 2-47. PMLR, 2018.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.0101,, 2017.
Stefano Markidis, Steven Wei Der Chien, Erwin Laure, Ivy Bo Peng, and Jeffrey S Vetter. Nvidia tensor core programmability, performance & precision. 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 522-531. IEEE, 2018
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXivv1609.078433 2016.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg. Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv: 2017.
Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sun- daraman, Jongsoo Park, Xiaodong Wang. Udit Gupta, Carole-Jcan Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavski,, Yinghai Lu, Raghuraman Krish- namoorthi, Ansha Yu, Volodymyr Kondratenko, Stephanie Pereira, Xianjie Chen, Wenlin Chen, Vijay Rao, Bill Jia, Liang Xiong, and Misha Smelyanskiy. Deep learning recommendation model for personalization and recommendation systems. CoRR, abs/1906.00099 2019. URL
attps::/arxxiae org/ abs www. 00091
Nvidia. Floating point and ieee 754 compliance for nvidia gpus, 2020. URL https://docs. wow.iiginooooiiinnnnnnniiii html
nvidia www.c///cucce cane://cda/// wtteeiii......eaeeeeeeeeeeeeeeee
Joachim Ott, Zhouhan Lin, Ying Zhang, Shih-Chii Liu, and Yoshua Bengio. Recurrent neural net-
works with limited numerical precision. arXiv preprint 1608 2016.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp. 5206 52100 IEEE, 2015.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint 1rXiv:1910.02054, 2019.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780 4789, 2019.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron- Training multi-billion parameter language models using gpu model parallelism. arXiv preprint urXiv:1909.000
2019.
N Stephens. Bfloat16 processing for neural networks anmv8-a. See https://communiry arm. cow/deveoperii-ppddtt/ppeessssoo producctsppoocessos////ii http:///wwsss..oob.ooooo/pssphpp networks-on-armmv8 2d00 (accessed 14 October 2019), 2019.
nor-uualll
Josef Stoer and Roland Bulirsch. Introduction to numerical analysis, volume 12. Springer Science
& Business Media, 2013.
Xiao Sun, Jungwook Choi, Chia-Y Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalak- shmi Viji Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In Advances in Neural Information Processing Systems, pp. 1900049000 2019.

---

Under review as a conference paper at ICLR 2021
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 2018a.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Ye Yu Chen, and Kailash Gopalakrishnan. Train- ing deep neural networks with 8-bit floating point numbers. In Advances in neural information processing systems, pp. 7675-7684, 2018b.
Hantian Zhang, Jerry Li, Kaan Kara, Dan Alistarh, Liu, and Ce Zhang. Zipml: Training linear models with end-to-end low precision, and a little bit of deep learning, In International Confer- ence on Machine Learning, pp. 4035-4043, 2017.
Jian Zhang. Jiyan Yang, and Hector Yuen. Training with low-precision embedding tables. In Systems for Machine Learning Workshop NewrIPS, volume 2018, 2018.
Tianyi Zhang, Zhiqiu Lin, Guandao Yang, and Christopher De Sa. QPyTorch: A low-precision arithmetic simulation framework. arXiv preprint arXiv:1910.045: 2019.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: 1606.06/60, 2016.
12


Reviews:
[{"review_id": "PiMZzwLC_9a", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I think the use of QPyTorch for the experiments here invalidates the results since the intermediate matrix multiplies are done in single precision (FP32), and so are more optimistic than a pure 16-bit implementation. (This is both according to the authors Sec 4, experiment setup; and according to the QPyTorch paper arxiv:1910.04540, Sec 3 intro.) For these kinds of experiments to be meaningful, they have to be done on native 16-bit hardware which luckily is becoming more common, e.g., Google's TPUs or the newer NVIDIA GPUs.\n\nThere are two other problems. First, it is not clear how stochastic rounding would be implemented in hardware. Doing it for every MAC operation could likely be even more expensive than just doing 32-bit MAC operations, since it involves the generation of random numbers, division, etc. Second, Kahan summation takes up twice the weight storage, so a more detailed calculation is needed to compare any hardware/energy savings to use that instead of just 32-bit.\n\nAs an aside, it may be interesting in Figure 1 to zoom in on the initial part of training to understand where the difference between 32-bit and standard 16-bit comes from in early training since at that point, the gradients are generally larger than later on in training.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AHYkIuHCjdF", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "During the rebuttal, I concluded that this submission is highly confusing, rather misleading.  I was led to believe the authors are in fact talking 'pure 16b MAC' - meaning 16b FP multiplies and 16b accumulate. After reading their responses to R4, I now learnt that they in fact are using 32b accumulates as is already standard practice in BF16 - If so, they have little or nothing new to offer. Rounding discussions the paper focuses on become highly secondary.  BF16 is already well-understood and accepted. Their writeup was highly misleading to say the least. I change my rating based on this.  \n\nThey use a representative set of benchmarks which include, CNN-based Resnet, recommendation proxy DLRM, and NLP proxy BERT. \n\nNovelty is limited, but critical: They reiterate prior observations that accurate weight updates are more critical for maintaining accuracy than forward-backward gradient updates. Former is what suffers when round-to-nearest cancels out small updates. Stochastic rounding has also been published before and shown to still miss the accuracy mark with >0.,1% accuracy gap for some important benchmarks. Novel part of this work is suggesting a Kahan summation based software fix on top of stochastic rounding to overcome the accuracy loss.\n\nPrimary issue: This is a very confusing and misleading writeup.  Authors should have clearly spelt out that they are in fact talking about BF16 - which is well-publicized for years now as 16b mule and 32b accumulate - and not labelled it as 'pure 16b MAC' - which is already known to exist as well, and proven to be not sufficient for DL training.  In light of this, this work has very little value-add. I change my rating now to 'clear reject' for their misleading writing style.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7frBEYS7OlR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the possibilities of reducing the precision of the weight update operation (i.e. AXPY ops) from 32 bit to 16 bit in today’s BFloat16 training framework. To enable 16 bit update, the authors proposed two techniques, i.e. stochastic rounding and Kahan summation. The authors use a simple least-squares regression model to theoretically explain the model accuracy degradation due to nearest rounding, then experimentally demonstrate the effectiveness of two techniques on a range of deep learning models and dataset. \n\nStrong points:\n\nThis paper is very well written. The problem is well addressed, and the solutions are well explained. The authors also provided both theoretical analysis and comprehensive experimental results. \n\n\nWeek points:\n\n1). Novelty: Both the problem addressed, and the solutions proposed in this paper have been reported in recent publications. In particular, both (https://papers.nips.cc/paper/7994-training-deep-neural-networks-with-8-bit-floating-point-numbers) and (https://papers.nips.cc/paper/8736-hybrid-8-bit-floating-point-hfp8-training-and-inference-for-deep-neural-networks) investigated the reduced precision of weight update from 32 bit to 16 bit or less. Moreover, the former introduced stochastic rounding and the later introduced round-off residue which basically the same as the Kahan accumulation technique discussed in this paper. Although, both papers discussed this topic in the FP8 training frameworks, while this paper in the BFloat16 framework, the basic concepts are the same. The authors of this paper summarized the techniques nicely, however, the novelty limited.\n\n2). On the same note, theoretical analysis on the impact of rounding mode on quantized weight update were also discussed in recent publications, such as (https://arxiv.org/abs/1706.02379 and https://openreview.net/forum?id=ryM_IoAqYX). It would have been nice to include these discussions as background knowledge and to distinguish this work from others. \n\nTo clarify:\n\n1). The proposed Kahan Summation method created a second tensor to store/accumulate the quantization error. Both weight and rounding error tensors are in 16 bit which in total, effectively, is 32 bit. Since AXPY is a very fast operation, this method does not seem to save much in terms of memory or speed. \n\n2). Today, SGD is often used with momentum, could the authors comment on the precision of momentum accumulation. And how about other popular optimizers, such as Adam?\n\n3). The analysis, i.e. Theroem 1 and 2, is based on a simple least-square regression model. Can this theory generalize to deep learning models?\n\n4).  On Table 2, last two rows of last column, the value seems to be inconsistent with other data in the same row. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6T1y9Alj2t", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis work reinvigorates half precision training as an alternative to either full single precision or mixed half and single precision. The authors demonstrate that the nearest rounding is the culprit for the worse performance of half precision training compared to single precision, due to cancelling small updates. They then propose two known techniques that can mitigate this effect, stochastic rounding and Kahan summation. Empirically, they demonstrate on an extensive suite of tasks that the proposed adjustments lead to half precision performance that is almost on par to single precision. \n\n### Pros\n- Solid experimental evaluation and results on a large variety of tasks\n- Both modifications that need to done are simple and straightforward\n\n### Cons\n- Limited novelty\n\n### Recommendation \nOverall, even though the paper does not have a lot of novel content, the experimental evaluation is thorough and demonstrates that the improvements close the gap to single precision training. Therefore I am keen on accepting this work, although admittedly not by much. \n\n### Detailed feedback\nThe paper is relatively well written and easy to follow. The authors nicely motivate their modifications which clearly show that the gap between half and single precision training is almost closed. My main point of criticism is that the novelty of this paper is relatively small, as the techniques proposed for performing the quantized weight update are, firstly, not new and, secondly, have been used in previous works for similar reasons. More specifically, performing stochastic rounding for the weight update in order to make progress when the magnitude of the update is small (thus rounded to zero) was also proposed at [1], which also showed successful 16 bit training (albeit for outdated tasks). As for Kahan summation; that has been proposed before for training quantised neural networks at [2] (again as a means to avoid making no progress when the parameter update is small). As a result, the true contributions of this work lie on the extensive empirical evaluation along with the theoretical analysis of rounding in a simple linear model. \n\nAs for other feedback and questions\n- At section 3.1 you first assumption A1 seems peculiar, as you state that you work on a least squares regression problem, but you assume that the model is overparametrized. How can this be the case? In a linear model, the amount of parameters is bounded by the dimensionality of the input (and it doesn’t seem that you perform any feature expansion for x). I believe that a more reasonable statement would be to assume that the actual data are generated by a linear model, hence there exists a true solution w^*.\n- Stochastic rounding requires the generation of random numbers; can this step also be done accurately in bfloat16?\n- From the evaluation it seems that Kahan summation performs better but it increases the memory size by a factor of 2; how does this fare memory wise to having the weights in single precision and is it a worthy tradeoff?\n- I believe that comparisons against other methods for quantised training that use fixed point instead of floating point would be interesting, e.g., [3, 4]. This would highlight the differences between these two formats and show whether the more expensive floating point operations are necessary.\n\n[1] Deep Learning with Limited Numerical Precision, S. Gupta, A. Agrawal, K. Gopalakrishnan, P. Narayanan, 2015\n[2] Training Deep Neural Networks in Limited Precision, H. Park, J. H. Lee, Y. Oh, S. Ha, S. Lee, 2018\n[3] Training and Inference with Integers in Deep Neural Networks, S. Wu, G. Li, F. Chen, L. She, 2018\n[4] Per-Tensor Fixed-Point Quantization of the Backpropagation Algorithm, C. Sakr, N. Shanbhag, 2018", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After reading the paper, reviews and authors’ feedback. The meta-reviewer agrees with reviewers that the paper has limited novelty and could be more clear about mix precision training. Therefore this paper is rejected.

Thank you for submitting the paper to ICLR. 

Decision: Reject