Training with Mixed-Precision Floating-Point Assignments
Authors: Wonyeol Lee
Rahul Sharma
Alex Aiken
Venue: Accepted by TMLR ()

Abstract: When training deep neural networks, keeping all tensors in high precision (e.g., 32-bit or even 16-bit floats) is often wasteful. However, keeping all tensors in low precision (e.g., 8-bit floats) can lead to unacceptable accuracy loss. Hence, it is important to use a precision assignment—a mapping from all tensors (arising in training) to precision levels (high or low)—that keeps most of the tensors in low precision and leads to sufficiently accurate models. We provide a technique that explores this memory-accuracy tradeoff by generating precision assignments for convolutional neural networks that (i) use less memory and (ii) lead to more accurate convolutional networks at the same time, compared to the precision assignments considered by prior work in low-precision floating-point training. We evaluate our technique on image classiﬁcation tasks by training convolutional networks on CIFAR-10, CIFAR-100, and ImageNet. Our method typically provides > 2× memory reduction over a baseline precision assignment while preserving training accuracy, and gives further reductions by trading off accuracy. Compared to other baselines which sometimes cause training to diverge, our method provides similar or better memory reduction while avoiding divergence.


Published in Transactions on Machine Learning Research (06/2023)
Training with Mixed-Precision Peaaing-PPoint Assignments
Wonycol Lee Stanford University, USA
worgeol..es.css@mmail
Rahul Sharma Microsoft Research, India
rulshha@microooff..ct
Alex Aiken Stanford University, USA
uakeenSslanfor edu
Reviewed on OpenReview: https: openreview. net/ forun? id= 2oX475408
Abstract
When training deep neural networks, keeping all tensors in high precision (e.g., 32-bit or even 16-bit floats) often wasteful. However, keeping all tensors in low precision (e.g., 8-bit floats) can lead to unacoeptable accuracy loss. Hence, it is important to precision assignment mapping from all tensors (arising in training) to precision levels (high or low) that keeps most of the tensors in low precision and leads to sufficiently accurate models. We provide a technique that explores this memory-accuracy tradeoff by generating precision assigmments for convolutional neural networks that (i) use less memory and (ii) lead to more accurate convolutional networks at the same time, compared to the precision assignments considered by prior work in low-precision floating-point training. We evaluate our technique on image classification tasks by training convolutional networks on CIFAR-10, CIFAR-100, and ImageNet. Our method typically provides 2x memory reduction over baseline precision assigument while preserving training accuracy, and gives further reductions by trading off accuracy. Compared to other baselines which sometimes cause training to diverge, our method provides similar or better memory reduction while avoiding divergence.
1 Introduction
In deep neural network training, floating-point formats are usually used to represent tensors and it is worthwhile to use the smallest bitwidth format that gives acceptable results. For example, it is common to replace tensors using 32-bit floats with tensors that use 16-bit floats (Kalamkar al., 2019; Micikevicius et al., 2018). The benefits are easy understand: computations using lower-precision floats not only use less memory but are also faster (due to improved vector parallelism, locality, and reduced data movement). The downside is that there is generally some loss of training accuracy, and in the worst case training may not even converge.
For such low-precision floating-point training, the most common approaches use two floating-point formats-on one for lower-precision floats (e.g., 8-bit floats) and the other for higher-precisian floats (e.g., 16-bit floats)-and assign one of the two formats each tensor (including weights, activations, and their gradients). The precision assignments studied in previous work fall into one of two assignment schemes (which both have several variants): the uniform assignment uses low precision for almost all tensors (often excepting those in the first and/or last few layers) (Micikevicius et al., 2018), while the operator-based based assignment limits low precision to the input tensors of certain operators (e.g., convolutions) (Sun et al., 2019). Prior work has shown that both precision assignment schemes (with well-chosen low-bitwidth floating-point formats) can match the accuracy of 32-bit-float training (Cambier et al., 2020; Chmiel et al., 2021; Drumond et al., 2018; Fox et al., 2021; Kalamkar et al., 2019; Micikevicius et al., 2018; Sun et al., 2019; Wang et al., 2018).
There is an important limitation in all prior approaches low-precision doating-point training: they use very few precision assignments (most often just one) for given set of models, but there are some other

---

Published in Transactions on Machine Learning Research (06/2023)
75
75
test (%5
test accuracy (96)
test accurary
65
60
55
55
40-
50
50
45
45
100
150
200
50
100
150
200
50
100
150
200
epoch
epoch
epach
(a) SqueezeNet
(b) ShuffleNe-v22
(c) MobileNet-v2
Figure 1: Training trajectory of various models on CIFAR-100. Colors denote precision assignments: all-32-bit bit Tip32 (red), uniform Tunif (yellow), and operator- based Tap (blue) (see $3.1); the latter two use the 8-bit (and 16-bit) floats in Sun et al. (2019) as low (and high) precision numbers. Markers denote the "width multiplier" of a model, which controls the capacity of the model (see $5.3): 1.0 0.5 0.25 (A), and 0.1 (+). Some lines of Tunif are missing as they converge to small values diverge. Observe that neither Tunif nor Top works best for all models: in some models, Top has similar accuracy to TTp32 but in other (and all) models, the accuracy drop of Top (and unit)) from Tfp32 are noticeably large (i.e., >1%).
models and inputs where the chosen precision assignment (i) results in noticcably worse accuracy than 32-bit-float training, (ii) causes training to even diverge, (iii) admits more efficient assignment that achieves similar training accuracy (see Figures 1, 3, and 4).
In this paper, we present a new, automated mothod for choosing precision assignments that removes the limitations described above. To do so, we formally introduce the memorg-acewraey trudeoff problem ($3.2): given a dataset, a model, and two seating-point precision levels (i..,, bitwidths; high and low), find a mixed precision assignment (a mapping from all tensors arising in training to high/low precision) for the model that maximizes training accuracy subject to given upper bound on the model aggregate (i.e., the total number of bits of all tensors appearing in training). The model aggregate is a proxy for the memory and time required for training, as it is roughly proportional to memory footprint and also well-correlated with training time (since training is often dominated by data movement) (Micikevicius et al., 2018).
We prove that the memory-accuracy tradeoff problem theoretically difficult (namely NP-hard) partly due to the exponential number of possible mixed precision assignments (which we often refer to simply as precision assignments for brevity) ($3.3). The large number of possible assignments makes the problem difficult in practice as well: there is no known analytical method for predicting the training accuracy of a given precision assignment, and for any practical model there are far too many precision assignments to simply test them all.
We propose a simple (heuristic) approach to the tradeoff problem that prioritizes tensors for low-precision formats based on the tensor's size (with an additional step described below) ($4.1). More specifically, our algo- rithm takes as input a single parameter giving a desired upper bound on the model aggregate. Starting with the largest lensor in the model, tensors are assigned low precision in size order (from largest. to smallest) until the model aggregate falls below the given upper bound; all remaining tensors are assigned high precision. Our main result is that this method discovers mixed precision assiguments that use less memory while achieving higher training accuracy than previous approaches. While we cannot show that our method finds Pareto-optimal memory-accuragy tradeofTs, we do show that our results are closer to Pareto-optimal than prior methods.
Some precision assignments initially generated by our algorithm cause training to diverge due to an excessive number of overflows. To address this issue, we propose an overflow handling technique that promotes tensors causing too many overflows from low precision to high precision during training ($4.2). In our experiments, these promotions consume only a small amount of additional memory (< of the maximum model aggregate) and prevent training from diverging. The overflow handling technique is not specific to our algorithm and can be applied to other precision assignment methods as well.
We ovaluate a PyTorch implementation of our mothod on standard image classification tasks by training four convolutional works (and their variants) on CIFAR-10, CIFAR-100, and ImageNet (65). We first. demonstrate that the precision assignments computed by our method alleviate the limitations of existing methods: they indeed explore the tradeoff between memory and accuracy and exhibit better tradeoff than
2

---

Published in Transactions on Machine Learning Research (06/2023)
the uniform and operator-based assignments. We then show the two main components of our method (i... precision demotion of larger lensors and precision promotion of overflowing tensors) are both important to produce competitive precision assignments. We also provide some guidance on how users may apply our method to navigate the memory-accuracy tradcoff.
To summarize, this work makes four main contributions:
We formally introduce the memory-accuracy tradeoff problem explore better mixed precision assignments for low-precision floating-point training and prove the NP-hardness of the problem.
We present novel precision assignment technique, heuristic solution to the tradeoff problem, that proposes assignments based on a single parameter denoting desired upper bound on the model aggregate.
We present a novel technique that handles an excessive number of overflows arising in training while using a small amount of additional memory. The technique is applicable to any (not just our) precision assignments. We remonstrate that the mixed precision assignments found by our method do explore the tradeoff between memory and training accuracy, and outperform existing precision assignment methods.
We remark that this work focuses on low- sww-prccssonn flouting-point training, not fired-poing preddppoint training (which uses fixed-point. formats), since we want to larget upcoming (or very recent) hardware with nalive support for low-precision floats (e.g., 8-bit floats) and their operations (e.g., Andersch et al. (2022)). Also, this work focuses on low-prccision training (which trains a model from scratch), not inference (which assumes a pre-trained model). More discussion is in $2. We further remark that in our experiments we simulate low-precision formats (e.g., 8-bit. floats) with 32-bit. floats as in prior works, since a hardware and software ecosystem that natively supports these formats does not yet exist. Similarly, we do not include other models (e.g., language models) in the experiments, since no current software stacks support per-tensor precision assignments for certain operators that those models use. More details are in $5.1 1 and $5.2.
For image classification tasks and convolutional networks, our precision assignment method typically provides > 2x memory reduction over the operator- based assignment while maintaining similar training accuracy and gives further reductions by trading off accuracy. Our method also provides similar memory reduction to the uniform assignment, while avoiding the divergence of training often caused by a uniform assignment.
The paper is organized follows. After discussing related work ($2), we define the memory-accuracy tradeoff problem and study its hardness ($3). We then describe our algorithm for the problem ($4) and our evaluation ($5). We conclude with limitations and future work ($6).
2 Related Work
Low-precision floating-point training has been extensively studied since the work of Micikevicius et al. (2018). One active research direction is to select appropriate floating-poit point formats (or their variants) for low- and high-precision numbers in training. Various floating-point formats have been proposed, including FP16 (Micikevicius et al., 2018), BF16 (Kalamkar al., 2019), FP8 (Micikevicius et al., 2022; Wang et al., 2018), HFP8 (Sun et al., 2019), and FP6 (Chmiel et al., 2021), along with some variants such as HBFP (Drumond et al., 2018), S2FP8 (Cambier et al., 2020), and BM (Fox et al., 2021). Recently, the problem of automatically selecting such floating-poin formats has been considered (Yang et al., 2022). Another research direction is to develop algorithmic techniques that improve training accuracy under low precision: e.g., (Björck et al., 2021; Sa et al., 2018; Yang et al., 2019a; Zamirai et al., 2020). Our work is orthogonal and complementary to all these prior works: they consider various floating-point formats or training algorithms but use a fixed precision assignment, which is either the uniform or operator-basse basod assignment (or their variants): our work explores various precision assignments once floating-por point formats and training algorithms are fixed (e.g., based on the prior works). The tradeoff between memory and accuracy in training is also considered in Yang et al. (2022), but the work differs from ours: they vary flonting-poin formats when precision assignment is fixed, while we vary precision assignments when floating-poin formats are fixed.
Low-precision fixed-point training uses fixed-point formats as a low- precisioi representation instead of a floating- point format. Some works use fixod-point formats for forward tensors and floating-pols formats for backward tensors: e.g., (Choi et al., 2018; Courbariaux et al., 2015; Jacob al., 2018; Sun et al., 2020; Yang et al., 2019b). Other works use only fixed-point formats for all tensors: e.g., (Banner et al., 2018; Das et al.,
3

---

Published in Transactions on Machine Learning Research (06/2023)
2018; Gupta et al., 2015; Rajagopal et al., 2020; Sakr & Shanbhag, 2019; Wu et al., 2018; Zhang et al., 2020; Zhou et al., 2016). Among all these works, some consider various mixed precision assignments with different. bitwidth (fixed-ointt) formats: (Sakr & Shanbhag, 2019; Zhang et al., 2020); but they are not applicable to our context (i.c., floating-pont point training) since they rely on some properties of fized- point formats that do not hold for flouting-poin formats (e.g., all numbers in a given format are equally distributed). The approach taken in Rajagopal ct al. (2020) is orthogonal and complementary to ours: they use only the uniform procision assignment, but change the underlying low-precision formats during training; we consider various mized precision assignments, but fix the underlying low- www-priiiionn formats during training.
Low-precision inference, often called neural network quantization (in a narrow sense), aims at reducing the latency or memory of neural network inference (instead of training) by using low-precision numbers (Nagel et al., 2021). Existing approaches typically assume a pre-trained model and try to find low-precision formats for each part of the inference computation, either by retraining the model (called quantization-aware training) or without any retraining (called post-training quantization); see, e.g., Gholami ct al. (2022); Qin et al. (2022) for surveys. Some works on inference consider various mixed precision assignments, but they are not applicable to our context: they focus on making inference more efficient and usually assume a pre-trained model; we focus on making training more efficient and aim at learning a model from scratch.
Floating-point tuning is another related topic, which considers the following problem: given a program, assign appropriate formats (among given candidates) to the program's floating-point variables such that the program's 's output has an error smaller than a given threshold for all given inputs, while also maximizing performance (Chiang et al., 2017: Guo & Rubio- Gonzálzz, 2018; Menon et al., 2018; Rubio- Gonzz-zzzz et al., 2013; 2016). This problem is different from the problem we focus on: the former considers the floating-pots poinn error after a single run of a program, while consider the training accuracy after large number of runs of program (i.e., a gradient computation) where each run affects the next run; further, the former considers generul-purpose programs, while we consider deep learning programs and exploit their unique features.
3 Problem
In this section, we first provide background on low-precision ftoating-point training ($3.1), based on which the memory-accuracy tradeoff problem is introduced ($3.2). We then prove the NP-hardness of the problem ($3.3). Our approach in $3 4 is more formal than most related works for two reasons: (i) we show the problem is NP-hard, which has not been considered in prior work; and (ii) to clearly describe the precision assignments to be considered.
3.1 Background: Low-Precision Floating- Ploiting-Point Training
Let T be the set of real-valued tensors and let [n] denote the set (I....,n). For supervised learning task, we usually consider model network M = -f1,..., fn) parameterized by 0 = (f1,..., 0n) E T", and a loss network C= (fath...,, fm), where fi 12 - T is a primitive operator on tensors (e.g., convolution, batch normalization, maxpool, and softmax). Given an input-output pair (x, v) E T2, the model M computes a predicted output y' of x by iteratively applying (.(,0,) to x (i E [m]), and L computes a loss from y by iteratively applying fr(:, y) to yf (i' E [m]) A standard way to train M is to minimize the loss value using the gradient descent algorithm: iteratively update 0 by following the gradient of the loss with respect to O.
Floating-point training. In practice, we perform gradient computation usually with tensors represented in floating-point formats. Let π TS- + FP be precision assignment giving the floating-point format of each lensor, where TS A {v;, du;, 0j. d0, | E [m + 11,,j E [m]} is the set of tensors arising in a gradient computation (explained below), and FP é {fp(e, 6) e, m E N, b E Z} is the set of floating-pon point formats. Here fp(e, m. b) denotes floating-point format that consists of 1-bit sign, an e-bit exponent, and an m-bit mantissa, and has an (additional) exponent bias of b Z. A common choice of a is Ti3s2 (t) e fp32 for all E TS, where fp32 e fp(8, 23,00 0) is the standard 32-bit floating-point format.
4

---

Published in Transactions on Machine Learning Research (06/2023)
0
0n
Fm
fm
8
02
B B
th
0
0
m=11
d/1.1
fm,1
dv1
dv2
d
dv
dm
dD
df1.2
forward compulation
de
de
backward computation
Figure 2: A diagram showing the tensors and operators used in a gradient computation; see Eq. (1) for details. For brevity, rounding functions rnd(() are omitted.
Given precision assignment TT, gradient computation is typically performed by the backpropagation algorithm: with 01 = (x) and dum+1 = rnda(du. 2) (1), compute
8, rnda(w 1) (fi(0, i:)),
8 (o)
(1)
dv. rnda (df a,1 (dvi+1, (du:41,,,,00 0,, 0.)).
d0; madx(a))) a0,) (df 5.2 (du++1, 0g 0;)),
for E [m]] and j E [n]] see Figure 2 for a diagram. Here rnd : FP x T + is a function rounding a given input to given Moating-point format, df d/i. : T + T are the backward operators of fa with respect to its first and second arguments, respectively, and â; 0 if i E [n] and y otherwise. We call is and 0j the forward tensors, and du; and d0, the backward tensors. We put hat over each tensor to emphasize that its value is the outpul of rounding function to possibly low-precision format; remark that such a rounding function is not used within fi, df.,1, and df since they typically use large bitwidth floats (e-g., fp32) and no fow-precision floats internally (Cambier et al., 2020; Kalamkar et al., 2019). After the computation, d0, stores the gradient of the loss value with respect to 8,.
The overall picture of floating-point training is now described as follows. In each iteration of the gradient descent algorithm, we compute d0, via Eq. (1) using given precision assignment म, training data (x, y), and current weights O. We then update each 8, by 8, t rnd(p=2(0) (0 di,) given learning rate 1) > 0, and proceed to the next iteration until the training ends. Here we use fp32 to represent. 0j by following the convention in low-precision floating-point training: a "master copy" of weights (i.c., 0z) is stored separately from the weight values (i.e., 02) used in a gradient computation, and is usually represented by fp32 (Cambier et al., 2020; Kalamkar et al., 2019; Micikevicius et al., 2018). The memory overhead of this master copy is very small compared the memory required to store other tensors (o.g., activation tensors vt) (Micikevicius et al., 2018).
Low-precision floating-point training. In low-precision training, we use precision assignment æ where some tensors have a smaller bitwidth than fp32. Particularly well-studied are that use two predetermined floating-point bitwidths (which are different) and optionally vary the rest of the format from tensor to tensor. We call C: TS x flo, hi} + FP a precision-eandidaat assignment if C(t, has the same bitwidth for all / E TS, the same holds for hi, and the bitwidth for lo is smaller than that for hi. We define ITC)) e {T: TS + FP Vt E TS. (() E (C(t, C(t,hi))} as the set precision assignments that conform to C.
Among various precision assignments in IIC), two have received the most attention: the uniform assignment Wunif, (Micikevicius et al., 2018) and the operator- based assignment Tap.c (Sun al., 2019). The former assigns low-precision formats to all tensors uniformly", and the latter to (most of) the input tensors of GEMM operators (in both forward and backward passes):
(tu A C(t, for all E TS,
C(t, if E (v,, O,, d for some
(tp. A
and fi is a GEMM operator (but not the first/last one) C(t,hi) hi) otherwise,
(2)
where GEMM operator refers to general matrix multiplication operator which arises in, e.g., fully-connected or convolutional layers. A particular variant Top'.c of Top.c has received much attention as well (Kalamkar et al., 2019), which assigns low-precision formats to (most of) the input and output tensors of GEMM operators: it is defined as opp. except that {v, f. dv in Eq. (2) is replaced by {u, G, U:+1, du, d8, du,+
For simplicity we define Fwiif, without the common exceptions for tensors near v1 and Um-11
5

---

Published in Transactions on Machine Learning Research (06/2023)
We note that the precision assignments used in apex. amp and torch, amp (Nvidia, 2019; PyTorch, 2022) correspond to Top.Cc and Top',C; respectively. For several choices of C, these assignments have been shown to produce training accuracy similar to that by Tip32 on many datasets and models (see $1 $1-2).
3.2 Memory-Accuracy Tradeoff Problem
We now introduce the following problem based on $3.1, to address the limitation of cxisting approaches for low-precision floating-point training discussed in $1:
Problem 3.1 (Memory-aceuracy tradcoff). Given training data {(r,, 9:)}, a model and loss network M and C, a precision-candidan candidate assignment C, and a lower bound r E [0. 1] on the low- cow-precision ratio, find a precision assignment æ E (C()) that maximizes acc(I) subject to ratio)((*) 2
Here acc(T) denotes the accuracy of the model M when trained with π on {(Ta, vi)}, and (atio denotes the low-precision ratio of π,, ie., the portion of the tensors represented in low- cowpprecision under n,, among all tensors appearing in gradient computation:
ration(TT) size((1 E TS m(() C(1, E [0.1]
size(TS)
where STZ) A Eier size(t) denotes the total size (i.e., number of elements) of all tensors in T' TS?² For instance, rationo(mm) = 0 and (alio))(Taa) = 1 for the all-bighh-reeciionn assignment mhi and the all-low-precision assignment Th. The problem asks for a precision assignment that maximizes training accuracy under a memory constraint, which expressod as a fraction of the memory required train the model using Thi-
3.3 NP-Hardness of the Problem
We prove that the memory-accuracy tradeoff problem from $3.2 is NP-hard by showing that there is a polynomial-time reduction from the knapsack problem to this problem:
Theorem 3.2. Problem 3.1 is NP-hard.
Proof sketch. Recall the knapsack problem: given items with weights wy N and profits pi E D (i e [m]), find subset of the items that maximizes the total profit while its total weight docs not exceed given threshold W E N.
Given an instance (w,p. W) of the knapsack problem, we construct an instance of Problem 3.1 such that we get the following (informal) correspondence between the two: w; corresponds to the size of the parameter tensor 0g p. to the i-th component of the input data; W the lower bound on the low-precision ratio (in an inverse way); and selecting the i-th item corresponds to assigning high-precision format to the tensor 0g (and related tensors), which roughly decreases the low-precision ratio by w, while increasing the accuracy of the model (after training) by ps. Based on this informal correspondence, we formally prove that an optimal solution to the above instance of Problem 3.1 can be converted in linear time to an optimal solution to the given knapsack problem (w.p. W). That is, we have a lincar-time roduction from the knapsack problem (which is NP-hard (Karp, 1972)) to Problem 3.1 which is therefore NP-hard. For a detailed proof, see Appendix A. J
Intuitively, the proof relies on two aspects of Problem 3.1: the size of the search space (i.e., [I(())) is exponential in the size of the problem (especially ITSI), and some values representable in high-precision format underflow to 0 in a lower-precision format. Note that underflows are relevant in low-precision training: they frequently arise in practice, degrading the results of training (Micikevicius et al., 2018). The NP-hardness result indicates that it is unlikely any polynomial- algorithm solves the problem exactly.
4 Algorithm
In this section, we propose a novel (heuristic) algorithm for the memory-accuraey tradcoff problem ($4.1), and a new technique to handle overflows arising in training (64.2). We point out that the former algorithm
2 As explained $1, the low-precision ratio a proxy for the reduction in memory well training time (because the low-pierision ratio increases the model aggregate decreases).
6

---

Published in Transactions on Machine Learning Research (06/2023)
finds an initial precision assignment before training starts, whereas the latter technique updates the current precision assignment while training proceeds.
4.1 Precision Demotion for Saving Memory
Consider imput to the memory-accuracy trade-off problem (Problem 3.1): M (f, and C (fat1...., fm), a precision-candidatt assignment C, and a lower bound r the low-precision ratio. Given the input, our algorithm finds a precision assignment π in two steps (Algorithm 1).
model and loss network
Tensor grouping. We first group tensors in TS such that each group consists of all the tensors between two "adjacent" GEMM operators (see below for details). This grouping reduces the search space over precision assignments, from all of II(C) a subset in which the same precision assigned to the tensors in the same group. This specific grouping strategy is based two observations: majority of saating-point operations are carried out in GEMM operators, and it is standard (c.g., in PyTorch) to use the same precision for a forward lensor and its corresponding backward lensor.
Formally, we group tensors as follows. Let fa and for (k < K') be GEMM operators that are "adjacent", i.c., there is no GEMM operator in {fR41....,, fx-1)- For each such (fk: fie), we create a group {ve, du; 0z. d0, I i E (k; k] n [m + +1,5 E [k,k] n [n]]- After that, we create two more groups for the remaining tensors: one for the tensors near V1 and the other for tensors near Um+11 As result, we obtain set of disjoint groups of tensors (Ti, .2....) . 2T.
Precision demotion. Given the groups of tensors, Ti, T1,72,..., we construct a precision assignment a follows: initialize π to the all-high-precision assignment and update π by demoting the precision of all tensors in a group to low precision, one group at a time, until the low-precision ratio of x becomes greater than r. We demote the precision of groups in decreasing order of their sizes (i.e., the total number of elements in tensors); that is, the precision of a larger size group is demoted earlier. Formally, let Ty........ be the reordering of (Ti, T2,...... such that size(TT) 2 size(TT)) 2 ..... After initializing π by x(t) C(t, hi) for all t, we iterate over i E N and update π to :tt) = C(t, for all E Ty, until (atio 2 r is first satisfied. The resulting π is the output of our algorithm.
The intuition behind using group size as the priority order for precision demotion is based on the fact that it is actually oplimal in a very simplified setting. Suppose that an input & to the model M stores a quantity of information I and the forward computation of M is nothing but a process of extracting the information in the input into a small number of values, i.c., the tensor U241- Assume that passing through each group O = [[K+1,-.., Swr } of operators (corresponding to the group T. of vensors) reduces the amount of information by factor a, E (0, 1), and using low precision on the group T, further reduces the amount of information by constant factor 3 E (0, for all i. Then, the amount of information left in uo+1 becomes I x (a102 ...) x 8², where l is the number of groups in low precision. In this simplified setting, maximizing the amount of information in ca+1 is equivalent to minimizing the number of groups in low precision, which is achieved precisely by demoting the largest groups first (when there is a constraint on the low preciiiiioon ratio). We show empirically ($5.4) that using the decreasing size order in precision demotion indeed produces better precision assignments than using other orders.
4.2 Precision Promotion for Handling Overflows
While our algorithm in $4.1 exerts constraint on memory usage, places no explicit constraint on training accuracy, and so not surprisingly for some models and datasets the resulting precision assignment causes training to diverye accuracy decreases significantly and remains low after some point. We observe that when training begins to diverge (and bit before that), many overflows occur in the rounding function of some tensors 0;, i.c., an input tensor to the function (nda(e))()) in Eq. (1) contains many elements whose magnitude is larger than the maximum representable nunber of the format TT;;) (Figure 6(a-b); $5.4). This rapid increase in overflows in individual tensors is a signal that training may diverge.
Precision promotion. Based on this observation, after each gradient computation we update the current precision assignment и by promoting to high precision (i.e., C(t, hi)) any forward tensor whose overflow ratio is greater than given threshold O E (0, 1); this updated precision assignment is used in the next gradient.
7

---

Published in Transactions on Machine Learning Research (06/2023)
Algorithm 1: Computing x with precision demotion
Algorithm 2: Training with precision promotion
Input: (fi,..., (fn) C,
Input: ह, O,
/* Tensor grouping */
/ Training loop // Initialize weights O
k 1; Tk- =
for i to do
while training not finished do
Tj Th U {U, de:) if k S n then { TT = Ta U (0,, d0;
Compute the current gradient d0 using Update O using d6 /* Precision promotion */ for forward TS do if overflow ratio(t) θ then T(t) h(t
if ʃa is GEMM then { k = k + 1; Ts = 0
end
/ Precision demotion */
(T....., TT) = sort(Ti...., Ts)) by decreasing size
(t) = Ct,, hi) for all t TS
end end end return 0
for j = 1 to do
if ration. (T) then (break) }
r(t) Ctt, lo) for all t e TJ
end return
computation (Algorithm 2). Here the overflow ratio of t TS denotes the number of overflows arising in the rounding function of in Eq. (1), divided by the number of elements in f. We show empirically ($5.4) that training always converges using this technique and the additional memory cost of promotion is small (in our experiments, < 3% the maximum model aggregate"). For the experiments, we use θ = 0.01; in fact we found that a wide range of values for @ (0.1, 0.01, and 0.001) all work well. Note that this technique not. specific our algorithm and can also be applied to other precision assignment methods.
We apply precision promotion only to forward tensors for two reasons. First, dynamic loss scaling (Micikevicius et al., 2018; Nvidia, 2019; Py 2022; Sun et al., 2019) already handles overflows in backward tensors, but not in forward tensors: loss scaling multiplics the backward loss tensor dum-1 by a constant before performing backward computation, to scale up all backward tensors; the dynamic version adjusts the constant during training in a way that avoids overflows in backward tensors. Note that dynamic loss scaling does not affect forward tensors at all. Second, we cannot use similar idea to handle overflows in forward tensors, because forward tensors are not linear in the input tensor 21 whereas backward tensors are linear in the backward loss tensor dim+1 (by the linearity of differentiation).
Precision promotion incurs little if any computational overhead: checking whether single rounding operation overflows is cheap, and we only apply rounding functions to the output tensor of an cechhmmtii-inttnniiii operator (e.g., convolution and batch normalization), amortizing the cost of the overflow checks over a large number of other operations.
5 Experiments
In this section, we evaluate our precision assignment technique (developed in $4) on standard training tasks answer three rescarch questions:
Does our technique explore the tradeoff between memory and accuracy and achieve a better tradeoff
than existing (fixed) precision assignments ($5.3)?
Are the two main components of our technique, precision
of larger/overflowing
tensors, important for good performance ($5.4)?
How can we choose the parameter r in our technique (i.e., a lower bound on the low-precision ratio) ($5.5)?
FFor each training where methods (presented in $4) used, measure the model aggregate when the training starts and when ends. We observe that the difference between the values (averaged over four different runs) is 3% of the maximum model aggregate (t.e., the model aggregate when all tensors in high precision).
8

---

Published in Transactions on Machine Learning Research (06/2023)
5.1 Implementation
We have implemented our precision assignment technique using PyTorch (Pasıke et al., 2019). Given a model and loss network, and a datasct, our implementation takes as parameters precision-candidate assignment C and a lower bound on the low-precision ratio; it then automatically assigns precisions to tensors (appearing in training) according to our technique and uses those assigned precisions in gradient computations. To make these procedures automatic, our implementation works as follows:
For each primitive operator in PyTorch (e.g,, torch.nn.Conv20l nn.vvdd), our implementattin provides its wrapped version (e.g., ext3. nn. 113...n.Coom Conv2d) which records auxiliary information for our technique (e.g., floating Coating-point format of input/outpin tensors) and applies proper rounding functions in forward/ forward/backward /backaard computations based on the auxiliary information. Models should now use the wrapped classes instead of the original ones. Our implementation first constructs a computation graph (of a given model and loss network) dynamically by running a forward computation on a minibatch of input data, The computation graph and other information (c.g,, each tensor's size) are recorded in the wrapped classes.
Using the auxiliary information just recorded, our implementation then constructs an initial precision assignment according to $4.1, and starts training with this assignment. During the training, our implementation uses the current precision in gradient computations, and updates it after each gradient computation according to $4.2. We record the precision assignment also in the wrapped classes to automatically apply proper rounding functions in gradient computations.
We simulate low-precision formats used in the experiments (e.g., 8-bit floats) and their operations, with 32-bit floats and 32-bit operations followed by rounding functions as described in Eq. (1); simulating low- cow-precision formats is the standard methodology set by prior works on low-precision training (Cambier et al., 2020; Fox et al., 2021; Kalamkar et al., 2019; Micikevicius et al., 2022) and we simply follow this. 4 We implement the rounding functions based on the QPyTorch library (Zhang et al., 2019), but a few extensions are required, e.g., to support exponent bias and signal overflows for dynamic loss scaling. We automatically apply these rounding functions after each primitive operator, by using PyTorch's 's hook fcature (c.g., nn. Module. register_ registerr+hookk).
5.2 Experiment Setups
Datasets and models. As benchmarks for our experiments, we the image classification task and three datasets for the task: CIFAR-10 and CIFAR-100 (Krizhevsky, 2009), and ImageNet (Russakovsky et al., 2015); these task and datasets have been widely used in recent works on low-precision training as a standard choice (Chmiel et al., 2021; Rajagopal et al., 2020; Sakr & Shanbhag, 2019; Wang et al., 2018) and we simply follow this. For the task and datasets, we use four well-known models: SqueezeNet (landola et al., 2016), ShuffleNet-v2 (Ma et al., 2018), MobileNet-v2 (Sandler et al., 2018), and ResNet-18 (He et al., 2016); they are chosen since models with relatively few weights, such as these, are generally known to be more difficult to train with low precision than those with more weights (Sun et al., 2019). We considered other tasks (c.g., language modeling) and related models (e.g., WwW./trranffommr-beaee experiments because substantial additional implementation effort orthogonal to our main contributions would be required: these models use some PyTorrc operators that do not support per-tensor precision assignments, so applying our technique to these models requires significant modifications to PyTrrch internals.
models) but did not include them in our
Precision-candidaae and precision assignments. For the experiments, we use the precision-candidatte assignment C studied in Sun al. (2019), which uses 16-bit (and 8-bit) floats for high (and low) precision; in particular, C(t, hi) f6 0) for all baaaard//bawwrr) tensors f, and C(t, lo) fp(4, for all forward tensors L and fp(5, 2, 0) otherwise. We choose this particular C since it uses sub-32-bit Koating-poinn formats for both low and high precision and the precision assignment Top.C was shown to achieve accuracy comparable to bit training (Sun et al., 2019). The three floating-point formats used in C have subnormals but no
4Thh S-bit formats used in our experiments (see $5.2) began to be supported natively in hardware very recently (by NVIDIA H100 GPU). But access to such hardware is still very limited (e.g,, no major cloud services including AWS, Azure, and Gougle Cloud provide it), and these formats are not yet supported natively in software (e.g., PyTorch, Teusorflow, and cuDNN). Due to such lack of hardware and software ccosystem natively supporting these formats, we chose simulate them in prior works. For instance, the PyTorc functions nn. RNN and nn. MultiheedAttention do allow to change the precision intermediate tensors (e.g.; input//utput tensors of each GFMM operator used the functions) deer-def formats (e.g., fp(4,3,4) 4)).
9

---

Published in Transactions on Machine Learning Research (06/2023)
infinities and NaNs, which are rounded to the largest or smallest representable mumbers. Since our technique is parameterized by a precision-candidate assignment, it is easily applied to other assignments as well.
We evaluate our technique by varying its parameter r (i.e,, a lower bound on low-precision ratio) over deciles We write Tours,r to denote the precision assignment chosen by our technique (described
r E 00,,0.,,222 0.2..... 1}. in $4) for a given r; e.g., Tours,0 is the all-high-preccision assignment, and Tours,1 the all-low-pre all-low-precision assignment equipped with our precision promotion technique ($4.2). Following Sun et al. (2019), all precision assignments (inchuding Tours,)) in experiments use high precision (i.e., 16 bits) for all backward weight tensors (i.e., dd,). For each precision assignment π, its low-precision ratio can change during training due to our precision promotion technique (when applicd), so we compute the average of the ratio over all epochs and report this value as the low-precision ratio of x.
Other setups and compute time. All experiments were performed on NVIDIA V100 GPUs; total compute time for all experiments was 1,081 GPU days. We train all models in a standard way: we apply dynamic loss scaling (a standard technique used in low-precision floating-poinl training; see $4.2 for details) except. for bi- training, and use standard settings (e.g., learning rate); see Appendix B for details. Due to random variations in training, we perform four runs of training for each configuration and report the average and the range of measured quantities.
5.3 Comparison with Existing Precision Assignments
To compare our technique with existing precision assignments for floating- training, we train each model with the following precision assignments: all-32-bit Tha2, uniform Tunit (Micikevicius ct al., 2018), operator- based Top (Nvidia, 2019; Sun et al., 2019), its variant Top' (Kalamkar al., 2019; PyTorch, 2022), and ours Tours,r (see $3.1 and $5.2 for their definitions). We choose Tunif, Tap, and Top' as baselines because existing pre- cision assignments for floating-poit training fall into one of the three assignments (or their variants) (see $1 2).
We train the four models mentioned in $5.2 on CIFAR-10 and CIFAR-100, and ShuffleNet-v2 on ImageNet. We also train smaller variants of the four models (which are more difficult to train with low precision) on CIFAR- 100, We obtain these variant models by following Sun et al. (2019), i.e., by applying a well-known approach for model reduction that uses a parameter called the width multiplier (Howard et al., 2017): each variant model reduces the number of channels in most tensors by a width multiplier; we use three values 0.25, 0.1} for the width multiplier. We train just one model on ImageNet due to the large amount of computation involved: for each model, 44 training runs (11 choices for r and 4 runs for each choice) are required for Tours, and each run on ImageNet takes nearly a half day with 16 GPUs. We use ShuffleNet-v2 for ImageNet since the model shows interesting memory-accuracy tradeofis when trained on the (smaller) CIFAR datasets.
ImageNeet Figure 3 presents training results of ShuffleNet-v2 on ImageNet: its left graph plots the average training trajectory for each precision assignment, and its right graph shows how each precision assignment. trades off between memory and accuracy, where memory is represented (inversely) by the low-precision ratio of the assignment (which is averaged over all epochs; see $5.2) and accuracy is the best test accuracy of the model during training. Each point in the right graph shows the average accuracy of four runs of training, while the shaded area shows the variation in accuracy among those four training runs.
Figure 3 shows three points. First, as the parameter r increases, the average accuracy drop of Tours, from Tfpa2 increases (up to 5%). In contrast, Tunif and Top' have a much larger average accuracy drop (more than 30%), as some training runs diverge when Tunif and Tap' are used. Second, the tradcoff given by Toursrr is better (i.e., closer to Parelo-optimal) than by Top: Tours,r for E (0.3, 0.43 has both higher accuracy and larger low-precision ratio (i.e., memory reduction) than Top- In particular, Tours,0.1 has 1.6x the memory reduction of Top- Third, Tours, provides options that Top cannot (which has an accuracy drop of >1%). If we want accuracy closer to Tip32, say within 0.5%, we can use Tours,.2 with 2.6% more memory than Top If we can tolerate a larger accuracy loss, say x 3%, then we can use Tours,0.7 with 2.9x the memory reduction of Tap-
CIFAR-10/ 1000. Figure 4 presents the memory-accurase tradcoffs of precision assignments for the four models on CIFAR-10 and CIFAR-100, and their smaller variants (with width multiplier 0.25) on CIFAR-100. The results for other smaller variants are similar and included in Figure 10 (see Appendix C.1).
10

---

Published in Transactions on Machine Learning Research (06/2023)
The conclusions from Figure 3 hold for Figure 4: Tours, provides range of options by varying r and exhibits better tradeoff than Tunif, Top; and Top' in almost all cases. We give detailed comparison as follows. First, in half of all 12 plots, Tunif shows a similar tradeoff to Tours,1- But in the remaining half, Tunif has an accuracy drop much larger than all other precision assignments including Tours,r; since using uuiff often makes training diverge while using, e.g., Toure,1 does not do so. Second, in all but two plots, Foura,r shows a strictly better tradeoff than Top- Toursr has noticcably larger (> 2x) memory reduction than Top while maintaining similar accuracy. Even in the two plots, Fours,r has tradeoff very close to Top. Note that in three plots, Top has an accuracy drop of > >1% while Toussr provides several options that have smaller accuracy drops and more memory savings at the same time. Third, Yours,r shows a strictly better (or similar) tradeoff than Tap' in all but two (or two) plots. Note that Top' has accuracy smaller than Top in all but one plots. Also it has an accuracy drop of > of all plots, and sometimes makes training even diverge (in one plot here and three other plots in Figure 10).
11% in half
Additional results. To isolate the effect of our precision promotion technique on the above results (Figures 3, and 10), we compare our precision assignments and the baseline assignments while applying the prccision promotion to all of them, and present the results in Appendix C.1 (Figures 13 13-. Two observations can be made in these results: for each precision assignment, (i) if training diverged without the precision promotion, applying the precision promotion prevents such divergence and produces much higher accuracy; (ii) otherwise, the accuracy and the low-precision ratio remain similar regardless of using the precision promotion. These observations lead to the same conclusion as in the above: our assignments provide similar or better tradeoff between memory and accuracy than the baseline assignments, even when the latter are equipped with our precision promotion technique. In addition, these observations also indicate that our precision promotion technique can effectively handle divergence in training (see $5.4 for more results on this).
5.4 Ablation Study: Precision Demotion and Promotion
Precision demotion. To evaluate the decision to use precision demotion in decreasing-size order, we train the four models on CIFAR-100 with Tours,r, Tours(ine), (which demotes tensor groups in increasing-size order) and Tours|rand].r (which demotes tensor groups in random order). For Fours(rand]- three different random orders are used in each case. The results, presented in Figure 5 (and Appendix C.2), show that the order of precision demotion has a significant impact on the resulting memory-accuracy tradcoff, and that decrcasing order provides the best results in nearly all cases. Increasing order consistently shows the worst results, suggesting our intuition (given in $4.1) for choosing decreasing order has some basis in reality.
Precision promotion. To understand whether precision promotion of overflowing tensors is important to our technique, we train ShufleNet-v2 on ImageNet using Toursjno-promojrr which does not promote lensors. The results, presented in Figure 6(a), show that several training trajectories diverge in early epochs and fail to recover afterwards. Figure 6(b) plots the top-5 tensor overflow ratios for the highlighted trajectory in Figure 6(a). The overflow ratios first spike about when divergence occurs around epoch 11. A closer look shows that the spike in overflow ratio occurs shortly before divergence, and starts first in a few tensors and then propagates to others. These observations indicate that an excessive number of overflows in a few tensors are the cause of the training divergence.
Finally, Figure 6(c-d) shows that precision promotion is effective at preventing the divergence of training while sacrificing only small amount of memory reduction. The figure shows ShuffleNet-v2 on ImageNet trained using our technique with and without precision promotion. Figure 6(c) shows that without precision promotion large accuracy drops occur due to divergence, whereas with precision promotion training converges. Figure 6(d) shows that the total size of tensors promoted to high precision is small for all r values. See Appendix C.2 for similar results for CIFAR-10.
5.5 Choasing the Value of
The time and space savings of our method are most significant. when a model is regularly retrained, which commonly occurs when new data is periodically incorporated into an existing model. Assuming that new data has a similar distribution to existing data, we can choose a single r (a parameter in our method) by conducting one set of experiments where we train with Tip32 and Tours,r for different r and then choose the value that maximizes model aggregate savings while still having an acceptable drop in accuracy.
11

---

Published in Transactions on Machine Learning Research (06/2023)
test accuracy (%)
test accuracy (%)
66
64
40
62
fp322
fp33
20-
op
0p
60
00'
00
unif
32
20
40
60
80
0.00
0.25
0.50
0.75
1.00
epoch
low-prec. ratio
Figure 3: Results of training ShuffleNet-v2 on ImageNet with Tfp32; Tunif (Micikevicius et al., 2018), Rap (Sun et al., 2019), Top' (Kalamkar et al., 2019), and Tours, Left: Each line shows the average training trajectory for each precision assignment; Tours,r is colored from navy to yellow (darker for smaller r). A zoomed-in version of this plot can be found in Appendix C.1. Right: Each point shows the memory-accurae each procision assignment; a red-dashed line shows the accuracy of Tipa2; and shaded areas show the variation among four training runs. In the right figure, top-right points are better than bottom-left ones. Observe that there are @ above and the right of and +, respectively. is missing as its y-value is too small.
tradeoff of
94
test accuracy (98)
92
70
test accuracy 198
30
90
test accuracy
68
54
52
66
to32
op"
-00
unif
unif
uni
nans
ours
ours
70
0-
0.00
0.25
1.00
0.00
0.50
1.00
0.50 low-prec.
low-prec.
low-prec.
(a) CIFAR-10, SqueezeNet
(b) CIFAR-100, SqueezeNet
(c) CIFAR-100, SqueozeNet 1
66
test accuracy (96)
92
72
test accuracy (9)
64
90
accuracy
70-
62
68
fp32
fp32
60
fg32
0p
66
unif
unif
N
uni
ours
ours
ours
1.00
0.75
1.00
0.50
low-prec. retio
ratio
low.prec, ratio
(d) C- ShuffleNet-v2
(e) CIFAR-100, ShuffleNet-r2
(f) CIFAR-100, ShuffleNet-v2
6B
test accuracy (1%)
accuracy 196)
66
text accuracy
64
1p32
fosz ap
1p32
89
62
op
op"
op'
80
40
0.00
0.75
0.00
0.50
0.75
0.50
low-prec. ratio
low-prec, ratio
low-prec. ratio
(g) CIFAR-10, MobileNet-v2
(h) CIFAR-100, MobileNet-v2
(i) CIFAR-100, MobileNet-v2
Test acturacy (%)
test accuracy (%)
test accuracy (%4
70
94.
68
76
92
fp32
fp32
66
fp32
90
00
80
na'
76
52
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
low prec. ratio
low prec. ratio
low-prec, ratio
(i) CIFAR-10, ResNet-11
(k) CIFAR-100, 18
(1) CIFAR-100, ResNet-18f
Figure 4: Memory-accuracy tradeoffis of Tunif (Micikevicius al., 2018), Top (Sun et al., 2019), Top (Kalamkar et al., 2019), and Tours.r for four models and their smaller variants CIFAR-10 and CIFAR- 100. The variant models have width multiplier 0.25 and are marked by t. Top-right points are better than bottom-left. ones. In all but three plots, there are @s above and to the right of X and +, respectively; even in the three plots (g,h,k), os have almost the same tradeofTs to X and +. In half of all plots, * has much smaller y-values than other points, The training trajectories for the above plots and the results of other smaller models are in Appendix C.I.
12

---

Published in Transactions on Machine Learning Research (06/2023)
74
72
test accuracy (%)
test accuracy (86)
72
test accuracy
70
72
70
68-
ourz[inc] oustrand]
88
ounslinc]
ours/rand]
ours]inc]
curs[aad]]
66
88
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.75
0.50
0.75
1.00
low-prec. ratio
low-prec. retio
low-prec, retio
(a) SqueezeNet
(b) ShuffleNet-v2
(c) MobileNet-v2
Figure 5: Memory-accuracy tradeoffis of Tours fours/ine))rr: and Truns for three models on CIFAR-100. Observe that s are above and to the right of other points in nearly all cases. The results of ResNet-18 are in Appendix C.2.
1.0
test accuracy (96
test eccuracy (%)
0.8
60
60
0.6
40
too-
0.4
20
100-2
20
0.2
oursEno promo]
oursino promo]
109-5
0.0
0.0
20
60
20
40
80
0.00
0.25
0.50
0.75
1.00
0.0
0.2
0.4
0.6
0.8
1.0
epoch
epoch
low-prec. retio (c)
low-prec, ratio (start)
(a)
(b)
(d)
Figure 6: Training ShuffleNet-v2 on ImageNet with Tours,r and Tours Coursfno-pret pnom-oomoo (a) Training trajectories
of Toursjoo-rromo), for different colors denote values (darker for smaller r)- (b) Top-5 overflow ratios of tensors at each epoch, for the highlighted trajectory in (a); the largest ratio is blue and the fifth largest red. (c) Memory-accuracy tradcoffs of Tours.r and Tours(no-promoo.-- (d) Low-precision ratio when training ends vs. when training starts, for Tours,r and Jouussno-promool.- The results on CIFAR-10 are Appendix C.2.
To simulate this scenario, we create five datasets ImageNeet220-ii (i E [5]) as follows, so that each of them contains different but similar data: randomly select 1/5 of the classes in ImageNet (which has 1000 classes in total), and split the training data of each class evenly into five new datasets.
For each Ima2eNet-200-i, we train ShuffleNet-v2 with Tip32 and Tours,r and present the results in Figure 7. Based the tradeoff results of Woursrr, can choose r = 0.4 if we desire an average of < 1% accuracy drop from Tipa2, and we can choose r = 0.9 if an average as 3% accuracy drop is tolerable. We make two more observations: the tradeoff result. of Tours,r is similar across all five datasets even though each dataset is different, and for cach the variance the accuracy of Nours.r from different datasets and runs of training is similar to that of Tfp32- Thus expect that accuracy drop similar to Figure? 7 with acceptable variance.
tast accuracy (9))
ours
0.00
0.25
0.75
1.00
low tow.prec.
new but similar dataset, Tours,r would have
Memory-accuraccy tradeoffis of Tours, for Shufleeet- v2 on ImageNet-200-i (i [5)).
Figure 7:
6 Limitations and Future Work
Our work has the same limitation present in prior works on cow-precision floating point training: low-precision floats and operations are simulated in software (instoad of being handled natively in hardware) and so the potential speedup of method is not directly measured, though do expect speedups to be proportional to the reduction in the model aggregate. We leave it as future work to perform such experiments very recent or future hardware (e.g., NVIDIA H100 GPU) that natively supports more low-precision formats. Another direction for future work is to integrate our method into systems for automatically optimizing deep learning computations (e.g., Jia et al. (2019); Unger et al. (2022)) to accelerate training.
Acknowledgments
We thank Thiago S. F. X. Teixcira for helping us run the experiments on cluster machines, and Colin Unger for providing helpful comments earlier version of this paper. This work supported in part by the
13

---

Published in Transactions on Machine Learning Research (06/2023)
Advanced Simulation and Computing (ASC) program of the US Department of Energy's National Nuclear Security Administration (NNSA) via the PSAAP-III Center at Stanford, Grant No. DE-NA0002373.
References
Michael Andersch, Greg Palmer, Ronny Krashinsky, Nick Stam, Vishal Mehta, Gonzalo Brito, and Sridhar Ramaswamy, NVIDIA Hopper Architecture In-Depth. https://developerr http:://ddeeloooe..iiiiiee.ee www.iiiiiii..... com/bloc aataa--boppr--rrcttttttreee depth/, 2022.
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In NewrIPS, pp. 5151-5159, 2018.
Johan Björck, Xiangyu Chen, Christopher De Sa, Carla P Gomes, and Kilian Weinberger. Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision. In ICML, pp. 980 991, 2021.
Leopold Cambier, Anahita Bhiwandiwalla, Ting Gong, Oguz H. Elibol, Mehran Nekuii, and Hanlin Tang. Shifted and Squeezed 8-bit Floating Point format. for Low-Precision Training of Deep Neural Networks. In ICLR, 2020.
Wei-Fan Chiang, Mark Baranowski, Ian Briggs, Alexey Solovyev, Ganesh Gopalakrishnan, and Zvonimir Rakamaric. Rigorous floating-point mixed-precision tuning. In POPL, pp. 300-315, 315, 2017.
Brian Chumiel, Liad Ben-Uri, Moran Shkolnik, Elad Hoffer, Ron Banner, and Daniel Soudry. Neural gradients are near-lognormal: improved quantized and sparse training. In ICLR, 2021.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce T-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. PACT: Parameterized Clipping Activation for Quantized Neural Networks. 3rXiv:1805.0600 2018.
Matthicu Courbariaux, Yoshua Bengio, and Jier-e David. BinaryConnect: Training Decp Neural Networks with binary weights during propagations. In NeurIPS, pp. 1123-3131 2015.
Dipankar Das, Naveen Mellempudi, Dheevatsa Mudigere, Dhiraj Kalamkar, Sasikanth Avancha, Kunal Baner- jee, Srinivas Sridharan, Karthik Vaidyanathan, Bharat Kaul, Evangelos Georganas, Alexander Heinecke, Pradeep Dubey, Jesus Corbal, Nikita Shustrov, Roma Dubtsov, Evarist Fomenko, and Vadim Pirogov. Mixed Precision Training of Convolutional Neural Networks using Integer Operations. In ICLR, 2018.
Mario Drumond, Tao LIN, Martin Jaggi, and Babak Falsafi. Training DNNs with Hybrid Block Floating Point. In NeurIPS, pp. 451-461, 2018.
Sean Fox, Seyedramin Rasoulinezhad, Julian Faraone, david boland, and Philip Leong, A Block Minifloat. Representation for Training Deep Neural Networks. In ICLR, 2021.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. A Survey of Quantization Methods for Ellicient Neural Network Inference. In Pow-Power Computer Vision: Improving the Efficiency of Artificial Intelligence, pp. 291-326. CRC Press, 2022. URL http:///axxvv..rgaaa/////a/
13630.
Hui Guo and Cindy Rubio-González. Exploiting Community Structure for Floating-Poini Precision Tuning. In ISSTA, pp. 333 343, 2018.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep Learning with Limited Numerical Precision. In ICML, pp. 1737 1746, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, pp. 770-778, 2016.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXio:1704 744604861, 2017.
14

---

Published in Transactions on Machine Learning Research (06/2023)
Forrest N. landola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size. ar arv: 100.000077000 2016.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and Training of Neural Networks for Efficient. Arieger-rithmmetic-- Inference. In CVPR, pp. 2704-2713, 2713, 2018.
Zhihao Jia, Matei Zaharia, and Alex Aiken. Beyond Data and Model Parallelism for Deep Neural Networks. MLSys, pp. 1 13, 2019.
Dhiraj D. Kalamkar, Dheevalsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. A Study of BFLOAT16 for Deep Learning Training. arXiu: 12322, 2019.
Richard M. Karp. Reducibility Among Combinatorial Problems. In Complerity of Computer Computations, pp. 85-103, 1972.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009. URL http://wwww...oooooooooooooooot. www.tto...e.ttttiieeeeeeeeeeeeee wdt.-ttrizzeearningg fewtures-2000-------
https://www.com
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. ShuffleNet V2: Practical Guidelines for Efficient CNN Archilecture Design. In ECCV, pp. 122 138, 2018.
Harshitha Menon, Michael O. Lam, Daniel Osei-Kuffuor, Markus Schordan, Scott Lloyd, Kathryn Mohror, and Jeffrey Hittinger. ADAPT: Algorithmic Differentiation Applied to Floating-Ponn Point Precision Tuning. SC, pp. 48:1 48:13, 2018.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed Precision Training. In ICLR, 2018.
Paulius Micikevicius, Dusan Stosie, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, Naveen Mellempudi, Stuart F. Oberman, Mohammad Shoeybi, Michael Y. Siu, and Hao Wu. FP8 formats for deep learning, arXXv:2209.05.. 2022.
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. A White Paper on Neural Network Qnantization. arXiv:2106.08200 2021.
Nvidia. Documentation of apex.amg amp. https://nvidia. github. wwt.abbba//ppea/aap..... html, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilankurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, Peigh-eerforance Deep Learning Library, In NewrIPS, pp. 8024-8035, 2019.
PyTorch. Documentation of torch. amp. https://pytorch wrgg//ccs/staale/mm...tt 2022.
Haotong Qin, Yifu Ding, Wei Fan, Chris Leff, Mehdi Bahri, and Eric Shaw. Awesome Model Quantization. https://githin
-uadtillqaantizztiion 2022.
Aditya Rajagopal, Diederik Adriaan Vink, Stylianos Venieris, and Christos-Savaa Savvas Bouganis. Multi-Procisionn Policy Enforced Training (MиPPET): A Procision-Switching Strategy for Quantised Fixed-Point Training of CNNs. In ICML, pp. 7943 00327952211 2020.
15

---

Published in Transactions on Machine Learning Research (06/2023)
Cindy Eibio-Goozáile Cuong Nguyen, Hong Diep Nguyen, James Demmel, William Kahan, Koushik Sen, David H. Bailey, Costin lancu, and David Hough. Precimonious: Tuning Assistant for Floating-Point Precision. In SC, pp. 27:1 27:12, 2013.
Cindy
Cuong Nguyen, Benjamin Mehne, Koushik Sen, James Demmel, William Kahan, Costin Iancu, Wim Lavrijsen, David H. Bailey, and David Hough. Floating-Point Precision Tuning Using Blame Analysis. In ICSE, pp. 1074-1085, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjoev Satheesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):2111225 222, 2015.
Christopher De Sa, Megan Leszezynski, Jian Zhang, Alana Marzoev, Christopher R. Aberger, Kunle Olukotun, and Christopher Ré. High-Accuracy Low-Precision Training. arXiv: 100::8880038,, 2018.
Charbel Sakr and Naresh Shanbhag. Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm. In ICLR, 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chen. MobileNet Inverted Residuals and Linear Bottlenecks. In CVPR, pp. 4510 4520, 2018.
Xiao Sun, Jungwook Choi, Chia- Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalakshmi (Viji) Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit Floating Point (HFP8)
Training and Inference for Deep Neural Networks. In NeurIPS, pp. 4901-4910, 2019.
Xiao Sun, Naigang Wang, Chia- Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi (Viji) Srinivasan, and Kailash Gopalakrishnan. Ultra-Low Precision 4-bit Training of Deep Neural Networks. In NeurIPS, pp. 1796 1807, 2020.
Colin Unger, Zhihao Jia, Wei Wu, Sina Lin, Mandeep Baines, Carlos Efrain Quintero Narvaez, Vinay Ramakrishnaiah, Nirmal Prajapati, Pat McCormick, Jamaludin Mohd- Xi Luo, Dheevatsa Mudigere, Jongsoo Park, Misha Smelyanskiy, and Alex Aiken. Unity: Accolerating DNN Training Through Joint Optimization of Algebraic transformations and Parallelization. In OSDI, pp. 267 284, 2022.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Y Yu Chen, and Kailash Gopalakrishnan. Training Deep Neural Networks with 8-bit Floating Point Numbers. In NewrIPS, pp. 7686-7695, 2018
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and Inference with Integers in Deep Neural Networks. In ICLR, 2018.
Chengrun Yang, Ziyang Wu, Jerry Chee, Christopher De Sa, and Madelcine Udell. How Low Can We Go: Trading Memory for Error in Low- -ow-Precision Training. In ICLR, 2022.
Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, and Chris De Sa. SWALP: Stochastic Weight Averaging in Low Precision Training, In 1CML, pp. 7015-7024, 2019a.
Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian-sheng Hua. Quantization Networks. In CVPR, pp. 7308-7311 7316, 2019b.
Pedram Zamirai, Jian Zhang, Christopher R. Aberger, and Christopher De Sa. Revisiting BFloat16 Training. NV1:2010.06110 2020.
Tianyi Zhang, Zhiqiu Lin, Guandao Yang, and Christopher De Sa. QPyTorch: A Tow-Precision Arithmetic Simulation Framework. XrXi::1910.0455 04540, 2019.
Xishan Zhang, Shaoli Liu, Rui Zhang, Chang Liu, Di Huang, Shiyi Zhou, Jiaming Guo, Qi Guo, Zidong Du, Tian Zhi, and Yunji Chen. Fixod-Point Baoo-Poronn Training. In CVPR, pp. 2327-2335, 2020.
Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. DoReFa-Nett Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients. arXiv:1606.066 2016.
16

---

Published in Transactions on Machine Learning Research (06/2023)
A Problem: Deferred Proof
Theorem 3.2. Problem 3.1 is NP-hard.
Proof. We prove the NP-hardness of Problem 3.1 (the memory-accurac, tradeoff problem) by reducing the knapsack problem (which is NP-hard) to the tradeoff problem. More precisely, we prove that the knapsack problem can be solved in polynomial time if we assume an oracle for the tradeoff problem.
Recall the knapsack problem: given items with weights w, E N and profits ps N (i [n]), and given threshold W E IN, decide which items to choose such that the total profit of the chosen items is maximized while their total weight does not exceed W. That is, find a E 1]" that maximizes Eigno apps subject to Eie(n a < W. This problem is well-known to be NP-hard (Karp, 1972).
Given an instance of the knapsack problem (w,p, W), we construct an instance of the tradeoff problem follows.
Notations. The following construct uses constant & E N and floating poiti formats fPn fPlo E FP (one for high precision and the other for low precision). Below we will specify the conditions they should satisfy, and show that some k, fphi and fpi indeed satisfy the conditions. We write rndbi(-) (.) and r.d)(.) as shorthand for rndsp,,) (-) and (.).
Training setups. We consider very simple setting for training: the gradient descent algorithm with a learning rate 17 = 2-
3- (I N) is applied for just one epoch; all parameters are initialized to 0 and their master
copies are represented in fpm: and the negative loss of a model on training data (i.c., - (x)) y) using notations to be described below) is used as the accuracy of the model. Here / E N can be any natural number. Model and loss networks. A model network M and a loss network L are given as Figure 8, where M has n parameter tensors 0, E RR of size w; (i E [n]). For an input-output pair (x.y) E RD x R, M and L compute a predicted output fo(r)= E R and a loss L(fo(x), v) E as follows (assuming that no rounding functions are applied):
fo(x) 2 Σ
Σ 01
L) 2 (f(x)) vl.
xE(m] JE(ws)
Roughly speaking, M is (a variant of) a linear classifier and L is a êy-loss (scaled by 2-*).
Training data. Training data consists of single input-output pair (x, E Ra x IR that satisfies the following:
I; rndio (/VVV/PP// pt/wa),
y < (2+()
Σ wasz
iE(m]
for all E [n]. Here y take any value long as it satisfies the above inequality. Note that v; be different from Vp./w, since the latter value may not be representable in fPlo.
Precision-candidate assignment. A precision-candidate assignment C: TS x {hi, lo} - FP is given as:
C(t, hi) è fpmr C(t, e fpio for all t TS.
That is, for all tensors, fpni is used as a high-precision format and fpm as a low-precision format. Here fp and fp should satisfy the following:
Chi 2 Glo mhi 2 Mlo+
(3)
(m) s < |s| err
for all E S₁, for all E S2, for all e 52 U3.
(4)
rndio (s) 0
(5)
rndai (8) =
(6)
Here ehi and mhi (and eo and mlo denote the number of exponent bits and mantissa bits of fpm
(and fpi.). and err and S, are defined as: err a 1/(6n max:c(m]Pi), Si & {Vpi/w, [n]]- Eq. (4) says that the relative error of
E [n]],
S₂ a ( (2- i E [m]], and S3 A (2-(+)
representing each E S in fpip should be less than err. Eq. (5) says that each s Sy should underflow to 0 when represented in fp Eq. (6) says that each 8 S2 U S3 should be representable in fpai-
17

---

Published in Transactions on Machine Learning Research (06/2023)
0 E w
y IR y loss
(01.) je(w]]
Po R (x:)ie(n)
11 R
Un+1 E R
P2n+1 R Lie(n]
12n+ E
split
conv
sum
Vo.1
(01.) V2 jE(ws]
jee(w]] Pnti.j
[V22+1 - yl
8 Wn
(0n,1) jc(wn]
2/2 R
V2n
Wm
conv
Pan
(On.j Vn)
jelwal
M
L
Figure 8: The model network M and the loss network L used in the proof of Theorem 3.2.
Low-precision ratio. A lower bound E [0, 1] on the low-precision ratio is given as:
max (0.1- 2W +
E 10.
size (S)
So r decreases linearly as W increases.
We make three points on the above construction.
First, each part the knapsack problem (w,p, W)) is used in the following parts of the construction: we is used mainly in the size of the parameter tensor 0, p3 in the input ø; and W in the lower bound Second, there exist k E N and fpm: fPt FP that satisfy Eqs. (3) (6). This can be shown as follows: first, by taking sufficiently many exponent and mantissa bits for fP
make Eq. (4) satisfied; next, by taking sufficiently large k, we can make Eq. (5) satisfied; finally, by taking sufficiently many exponent. and mantissa bits for fPhir we can make Eq. (3) and Eq. (6) satisfied (since æ, is representable in fpl. and (k++)) is a power of two).
Third, some well-known models (e.g., ShuffleNet-v2) have a similar structure to M in that they apply the following operations as subroutine: split tensor into multiple tensors, apply some operators each split tensor, and combine the resulting tensors into a single tensor.
We now prove that the knapsack problem (w, p, W)) can be solved in polynomial time, if an oracle to the above tradcoff problem is given. Suppose that π E II(C) is an optimal solution to the above tradcoff problem (given by the oracle). Define an item selection a E {0, 1)"h for the knapsack problem as:
{1 if ((du4+s) = ((du2++1) = fp a; $
otherwise
for each i E [n]. Note that can be constructed from in linear time. Thus, it suffices to show that is an optimal solution to the knapsack problem (w,p, W), which is equivalent to the following two claims:
Claim 1: We have Σig o:w; < W.
Claim 2: For any a' E 10, with EiE(n) aww, W, we have Σieln) dpc. S Eie(n) a.p.
We now prove each claim as follows.
Proof of Claim 1. If a = (O,...,0), then the claim clearly holds. Suppose that a * (O,...,O). Then,
1 2 2 a,w; size( (TS)
2 rationo 2 1 2W
size(TS)
Here the first incquality uses a + (O...., 0) and the definition of a and M; the second inequality uses the fact that is a valid solution to the above tradcoff problem; and the third incquality uses the definition of Hence, the claim holds.
18

---

Published in Transactions on Machine Learning Research (06/2023)
Proof of Claim 2. Suppose that the claim does not hold. Then, there exists a' E {0, 13"h such that
Σ
aiw, S W
Σ kE(n
dep
a:P:
iE/m]
Define precision assignment n' II((C) as:
ा| (dd2n+1) fphi:
x' (t) fpio x' <'(0))) 2 n' (dun+i) fpni
for all i E [x]] with af 1,
for all other t E TS.
Then, we have ration((T)) 2 by Eie(n) afw; S W and the definition of n', M, and r. Hence, it suffices to show acc(T) < acc(T'), because this would contradict to the fact that is optimal solution.
To show acc( < acc(x'), we prove the following two lemmas: the first lemma gives a closed form of acc(T) and acc(=²), and the second lemma shows that Eiejn) (rea, is close to Σiejn) B.p: (where the former summation appears in acc(TT) and acc(T²)).
Lemma A.1. The following hold:
acc(TT) 2- + (28-H)
Σ arwaa
acc(TT) 2- 2-(2++
Σ dwwa
iE(n]
iE(m)
Proof. We prove the equation for aco(T) only, since the equation for acc(T²) can be proved similarly. First, we show that for all i E [n] and E [w,].
do 0c 2- R,g..
(7)
Pick any i E [n]] and j E [ws]- Note that by the definition of M, we have
doe,5 = rdd,( (rud (rudd (uzn (2 ()) rndg (rudeo (1:)))
rndn(d0,)) Ind (rnd du (2) (2-*)) xz:)
where the second equality uses Eq. (3) and that o; is representable in fP We prove Eq. (7) by case analysis on ag. Suppose a; Then, by the definition of at, T(d0,) ((dun+s) T (dd2n++)) fpmi From this, get the desired equation:
d0., = rndai (rndaardddaa?
2ra) = rndni(2 xzi) = 2 2
where the last two equalities use Eq. (6). Suppose now a = 0. Then, by the definition of an, at least one of x (d0;),, I +s) and T (dv2n++) is fplo- If T( (dvu+++) = fpi. or π ((du2n-1) = fp we get the desired equation:
do., = rudd (r (1)22)))) xx (ee) (0 x;)) = 0,
where the first equality uses Eq. (3) and Eq. (6), and the sccond equality uses Eq. (5). The remaining case is when (dvn+::) 7 ((dv2++1) fpm and x(d0,) fp We get the desired equation in this case as well:
d0..; rndio (rnd, (2- x.
rnddo xx)) O.
where the second equality uses Fq. (6), and the last equality uses Eq. (5). Hence, we have proved Eq. (7). Next, let 0g be the i-th parameter tensor before training starts, and 0f be the corresponding tensor after training ends (i E [n)). Then, by the definition of the tradeoff problem constructed above, we have 0G, = 0 and
8',5 = 0z. rndni(2 .do.,) = 0 - r2dmi22* (a. 2 2-xx))) as (-2-
19

---

Published in Transactions on Machine Learning Research (06/2023)
where the second equality uses Eq. (7) and the third equality uses Eq. (6). Using this equation, finally obtain the conclusion of this lemma:
acc(T) = --fo (x),
- k
Σ
Σ
iE|ml jE[wn
-2 A
Σ 04
xz)
de(n),jE[es
= 2 k +
Σ
aj 2 (1+1)) waa?
iE|n]
= 2 h y
02 2 (k++1) mer??
VE(n]
2 ky + 2 (26++1)
iE|n]
where the first two equalities use the dcfinition of accuracy, and the second last equality uses the definition of y. This concludes the proof of the lemmaa.
Lemma A.2. For any 3 E 10,1)"
Σ sEin]
Brwast Σ
Biyss
<
1
2
iE|m]
Proof. We first show that for any i E [n],
|eia? Pel 2n
Pick any i e [n]. By Eq. (4) and the definition of Fi, we have
Ps
Ps
Pr
6n
maxjelm] Pj
wi
6np,
From this, we have
V
w/
Pi (1 1
6mp:
x4
V
Pi
1+
1 Grape
1
Wi
Pi
(1
6np,
1
2
x
W/
Pi
(1 +
6mp,
2
From this, we obtain the desired result:
| pa < ps ((1 Graps 1) Pi 3mp: + (6mp:)3 3
1
<
=
pi
3np;
6mp:
2mpt
2n
where the second incquality uses 6np; > 1 (as n, Ps E N).
Using this result, we can show the conclusion as follows:
Σ 3
Σ
Baps
Σ <E(M)
3,wwar? - pi) S Σ |&d
|wa - -pal <
Σ
1 1
2n 2'
VE(M)
(E(m]
*E(n)
iE(m)
where the last inequality uses |3&|| S
This completes the proof of the lemma.
Using the two lemmas, we now prove acc(TT) acc(TT). By Lemma A.2 and Exe(s) aip; < Eie(n) app, we have
Σ a
< Σ alp.
a&Pi
EEin]
(x)
sE|n
RE(N]
where the second incquality comes from as of (0, 1) and ps E N. From this, and by Lemma A.1, we obtain aco(T)) < aco(m') ) as desired. This concludes the proof of Claim 2, thereby finishing the proof of the theorem. 7
20

---

Published in Transactions on Machine Learning Research (06/2023)
Remark A.3. In the proof of Theorem 3.2, proved the NP-hardness of Problem 3.1 by making use of only few limited aspects of the problem. For instance, we used the fact that some values representable in high- precision format round to zero in a low-precision format; on the other hand, many other values representable in a high-precision format round to non-zero values in a low-precision format, and this indeed occurs in practical training (even more frequently than underflows). Also, we used a simple setting for training which a gradient descent algorithm applicd for one epoch, training data consist of one inpul-output pair, and test data is the same as training data; on the other hand, in practical training, gradient descent algorithm is applied for many epochs, training data consists of many input-output pairs, and test data is different from training data.
Problem 3.1 is general enough so that it embraces all the aforementioned aspects of floating-poiin pointss and training, including those that are not considered in the proof of Theorem 3.2. Since those aspects are likely to make the problem even more difficult, we conjecture that the problem would be more intractable than being NP-hard. 7
B Experiments: Deferred Details
The datasets we use have the following licenses:
CIFAR-10 and CIFAR-100: These datasets are under the MIT license.
ImageNet: This dataset can be used "only for non-commercial research and educational purposes." For more dotails, see its webpage (Stanford Vision Lab, 2020).
The implementations of models we use have the following licenses:
Nqu for CIFAR-10 and CIFAR-100: We adapt an implementation of the model in a public GitHub repository (Pathak, 2020), whose license information is not available.
ShuffleNet-v2, MobileNet-v2, and ResNet-18 for CIFAR-10 and CIFAR-100: We adapt an implementation of these models in a public GitHub repository (kuangliu, 2021), which is under the MIT license. ShuffleNet-v2 for ImageNet and ImageNet-200-i: We adapt an implementation of the model in the torchvision library (PyTorch, 2022b), which is under the BSD 3-Clause license.
The details of how we train models are as follows:
Four models on CIFAR-10 and CIFAR-100: We train the four models with a standard setup (kuangliu, 2021). In particular, we run the (non-Nesterov) SGD optimizer for 200 epochs with minibatch size of 128 (over GPU), learning rate of 0.1, momentum of 0.9, weight decay of x 10-4, and the cosine annealing scheduler for learning rate. For dynamic loss scaling, we use initial scale of 2 growth factor of 2, back-off factor of 0.5, and growth interval of epoch, as suggested in PyTorch (PyTorch, 2022a). ShuffleNet-v2 ImageNet: We train the model with the default setup given in PyTorch's GitHub repository (PyTorch, 2022c), except that use larger minibatch size and learning rale in (Goyal et al., 2017; Kalamkar et al., 2019; Krizhevsky, 2014; PyTorch, 2022d) to reduce the wall-clock time of training. In particular, we run the (non-Nesterov) SGD optimizer for 90 epochs with minibatch size of 1024 (over 16 GPUs), learning rate 0.4, momentum of 0.9, weight decay of 10-4, and the cosine annealing scheduler for learning rate. For dynamic loss scale, we use initial scale of 216 growth factor of 2, back-off factor of 0.5, and growth interval of 0.5 epoch, as suggested in PyTorch (PyTorrh, 2022a). ShuffleNet-v2 on ImageNet-200-i: We train the model with the same settings for ImageNet except that use the default values for minibatch size and learning rate given in (PyTorrc,, 2022c), i.e., minibatch size of 256 (over 4 GPUs) and learning rate of 0.1.
C Experiments: Deferred Results
C.1 Comparison with Existing Precision Assignments
Figure 9 presents a zoomed-in version of Figure 3 (left).
Figure 10 presents results omitted in Figure 4: training results of smaller variant models (which have width multiplier 0.5 or 0.1) CIFAR-100 with Tp32; Tunif, Tops Top', and Tours, The figure shows similar results
21

---

Published in Transactions on Machine Learning Research (06/2023)
to Figure 4: the results for the variant models with width multiplier 0.5 (and 0.1) similar to those for the original models (and the variant models with width multiplier 0.25).
Figures 11 and 12 show the average training trajectories for the configurations presented in Figures and 10. Figures 13 and present the same results as Figures and except the lollowing: the former, Top and "op' are equipped with our precision promotion technique, whereas in the latter they do not so. Figures and 14 do not include Tunif because this assignment with the precision promotion is identical to Tours,1-
C.2 Ablation Study: Precision Demotion and Promotion
Figure 16 presents results omitted in Figure 5: training results of ResNet-18 on CIFAR-100 with Tours.r Nours(ine).r: and Rounsfrad)).- The figure shows similar results to Figure except that it shows smaller differences in memory-aceuracy tradeoff between the three precision assignments.
Figure 17 presents results omilted in Figure 6: training results of four models on CIPAR-10 with Tours.r and prommj.-- The figure shows similar results to Figure 6 except that the training of ResNet-18 on CIFAR-10 does not diverge even with (oorroo--promo), for all r values.
References for Appendix
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, Large Minibatch SGD: Training ImageNet. in Hour. arXiv:1706 02677, 2017.
Dhiraj D. Kalamkar, Dheevalsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. Study of BFLOATI6 for Deep Learning Training. arXiu: 12322, 2019.
Richard M. Karp. Reducibility Among Combinatorial Problems. In Complexity of Computer Computations, pp. 85-103, 1972.
Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXin:1404 2014
kuangliu. httpss/gggttb...cc///mmaaa////rd 2021.
Gaurav Pathak. https://githen. http://tttuuu..mgspss...so
http:://ggtthhoooooeege222/g//gs 2020.
PyTorch. Documentation of torch torch.amp amp. http::/p////pooo....oc/////////a 2022a.
Py'Torch. http:///w.....ooaoocac..cc///a/t 2022b.
PyTorch. http://hhh.hcoctoooc///////////a 2022c.
PyTorch. 2022d.
Stanford Vision Lab. https: https://mmmae-net.. orgpettooodddoooddddo .hp, 2020.
httpp:///mageenne..oo///m/oolll
22

---

Published in Transactions on Machine Learning Research (06/2023)
70
fp32
op
65
unif
ours
test accuracy (%)
60
55
50
45
40
30
70
90
epach
Figure 9: A zoomed-in version of Figure (left). Results of training ShufleNet-v2 on ImageNet with Tipa2, Tunif (Micikevicius et al., 2018), Top (Sun et al., 2019), Top' (Kalamkar et al., 2019), and Tours,- Each line shows the average training trajectory for each precision assignment; Tours, is colored from navy to yellow (darker for smaller r).
23

---

Published in Transactions on Machine Learning Research (06/2023)
66
test accuracy 196)
64
test accuracy 196
40
X
62
36
8D
1p32
32
op'
03'
und
unif
ours
24
0urs
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
low-prec. ratio
low-prec, ratio
(a) CTFAR-100, SqueezeNet!
(b) CTFAR-100, SqueezeNet
96
70
test accuracy (96
test accuracy 1%)
52
68
48
86
1p32
44
60
00'
unf
unif
0.00
0:25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
low-prec. ratio
low-prec. ratio
(c) CIFAR-100, ShufleNet-v22
(d) CIFAR-100, ShuffleNet-v2 2
72
test accuracy (S)
70
test accuracy (%)
52
68-
48
66
fp32
44
fp32
00
00
00'
32
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
low-prec. ratio
low-prec. ratio
(e) CIFAR-100, Mob
(f) CIFAR-100, MobileNet-ve v
best accuracy (96)
test accuracy (9)
74
72
52
1p32
fp32
70
op
00/
48
60
03'
10
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
low-prec. ratio
low-prec. ratio
(g) CIFAR-100 100, ResNet-181
(h) CIFAR-100, 18
Figure 10: Continued from Figure 4. Memoryyaaccracy IradeofIs of Tunif (Micikevicius et al., 2018), Top (Sun et al., 2019), Top' (Kalamkar et al., 2019), and Tours,r for smaller variants of four models on CIFAR-100. The variant models have width multiplier 0.5 (marked by 1) or 0.1 (marked by 9). Top-right points are better than bottom-left ones. In all but one plots, there are @s above and to the right of X and +, respectively; even in the one plot (g), os have almost the same tradeoffs to and +. In three of all plots, * has much smaller y-values than other points; * is missing in (h) as its y-value is small.
24

---

Published in Transactions on Machine Learning Research (06/2023)
100
60
bett accuraty (96)
60
80
40
p.32
50
test
best
0g
uni
unf
unif
0U.
06.5%
0015
60
20
50
100 opoch
150
200
50
100
150
200
300
150
200
epoch
opoch
(a) CIFAR-10, SqueezeNet
(b) CIFAR-100, SqueezeNet
(c) CIFAR-100, SqueezeNet
100
70
test accurety (%)
90
70
60
eccuracy
60
50
1932
1p32
fp32
09
0p
0p'
50
00
test
40
0p
10
30
50
100
150
200
100
150
200
100
150
200
epoch
opach
opoch
(d) CIFAR-10, ShuffleNet-v2
(e) C1FAR-100, ShuffleNet-v2
(f) CIFAR-100, ShuffleNet-v2f
100
70
test accuracy (%)
(%)
90
70
60
1p32
1p32
1p32
09
0p
70
50
40
op
unit
0U7S
0urs
opoch
opoch
opach
(g) CIFAR-10, MobileNet-v2
(h) CIFAR-ION 100, MobileNet-v2
(i) 1-FAR-100, MobileNet-v2'
100
80
70
test accuracy (96)
test accuracy (9)
accurscy
80
1p32
fp32
60
50
op
OUTS
uni
0075
uni
40
unif
0U15
60
50
100
150
200
100
150
200
100
150
200
epoch
(j) C-FAR-10, ResNet-18
(k) CIFAR-100, ResNet-18
(1) CIFAR-100, ResNet-18"
Figure 11: Training trajectories for the configurations shown in Figure 4. Each line shows the average training trajectory for cach precision assignment. Toursr is colored from navy to yellow (darker for smaller r).
25

---

Published in Transactions on Machine Learning Research (06/2023)
70
50
02st accurcy (50)
best accuracy (56)
50
30
op
op
unif
unif
30
10
50
100
150
200
50
100
150
200
opoch
opoch
(a) CIFAR-100, SqueezeNet*
(b) CIFAR-100, SqueezeNet
1est sccuracy (%)
test accuracy (%)
60
40
f:32
fo32
0p
op
30
Op
40
50
100
150
200
50
100
150
200
epoch
epoch
(c) CIFAR-100 100, ShuffleNet-v2'
(d) CIFAR-100, 2
accuracy (%6)
70-
test accuracy (%)
50
60
fp32
fo32
50
0p
op'
30
opoch
opoch
(e) CIFAR- 100, MobileNet-v2+
(f) CIFAR-100, MobileNet-v22
80
70
test accuracy (S6)
test accuracy (%)
70
60
50
50
op'
unif
unif
ours
ours
40
30
50
100
150
200
50
100
150
200
epoch
epoch
(g) CTFAR-100, Res
(h) CIFAR-100, ResNet-181
Figure 12: Training trajectories for the configurations shown in Figure 10. Fach line shows the average training trajectory for cach precision assignment. Toursr is colored from navy to yellow (darker for smaller r).
26

---

Published in Transactions on Machine Learning Research (06/2023)
test accuracy (%)
66
64
62
oolpromol opppromo] nurs
32
0.00
0.25
0.50
1.00
fow-prec.
Figure 13: Results corresponding to Figure 3. The only difference from Figure 3 is that Top and Top' here are equipped with our precision promotion technique, whereas Top and Top' in the previous figure do not so.
test accurary (86)
accuracy (54
test accuracy (66)
56
54
52
66
oplproma] npTppommo] burs
oploroma]
50
opfarome] oplpprmm) ours
72
apl[promo] ours
0.00
0.25
1.00
0.00
0.50
0.75
1.00
0.50
0.75
low-prec.
low-orec.
low-prec.
(a) CIFAR-10, 10, SqueezeNet
(b) CIFAR-100, SqueezeNet
(c) 100, SqueezeNet" f
66
test test accuracy (95
test accurscy (%)
accuracy (96)
64
62
60
opipromol
oploromo]
promo]
80
25
0.00
0.75
0.50
low-prec. ratio
low-arec. ratio
low-prec ratio
(d) CIFAR-10, ShuffleNet-v2
(e) CTFAR-100, ShuffleNet-r2
(f) CIFAR-100, ShuffleNet-v2"
test accuracy (%)
66
test accuracy
test accuracy
64
88
Ip32
69
fos2
62
1p32
ualpromo] uplppromo
optarama]
op(aroma]
80
40
[prooo)
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
low-pret. retio
low.prec, ratio
low-prec, ratio
(g) CIFAR-10, MobileNet-v2
(h) CIFAR-100, MobileNet-v2
(i) CIFAR-100, v
test accuracy (%6)
test eccuracy (%)
78
test sccuracy (%)
91
76
92
90
fp32
propmol [oromo]
74
fp32
aolpromol oo [promoo]
fa32
octpromo]
76
52
op [elprroo)
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
low-prec. ratio
low-prec, ratio
low-prec, ratio
(i) CIFAR-10, 10, 18
(k)) CIFAR-100, ResNet-18
(1) CIFAR-100, ResNet-18
Figure 14: Results corresponding to Figure 4. The only difference from Figure 4 is that Top and Top' here are equipped with precision promotion technique, whereas Top and Top' in the previous figure do not so.
27

---

Published in Transactions on Machine Learning Research (06/2023)
64
test accuracy (%6)
40
X
best accuracy
82
36
60
fp32
32
fp32
ocl promol
expromoo osppromo]
[promo]
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
low-prec. ratio
low-prec, ratio
(a)
CIFAR-100, SqueezeNet
(b) CIFAR-100, SqueezeNet
test accuracy 18)
test accuracy (%)
68
48
66
64
1p32
1p32
op(promo) oprmmo
azépromo)
ooppromo)
24
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
1.00
low prec. ratio
low-prec. ratio
(c) CIFAR-100, ShuffleNet-22
(d) CIFAR-100, 2
test accuracy (%
test accuracy (%)
66
1p32
1p32
oc(promo]
exppromo]
21
36
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
low prec. ratio
low prec. ratio
(e) CIFAR- 100, MobileNet-v2
(f) CIFAR-100, MobileNet-v2 v
test sccuracy (86)
76
test accuracy (%)
72
70
potpromo]
fp32
exppromol
fp32
a0
0.50 low-prec. ratio
1.00
low-prec. ratio
(g) CIFAR-100, ResNet-188
(h) CIFAR-100, ResNet-18
Figure 15: Results corresponding to Figure 10. The only difference from Figure 10 that Top and Top" here are equipped with our precision promotion technique, whereas Top and Top in the previous figure do not so.
28

---

Published in Transactions on Machine Learning Research (06/2023)
82
accuracy (36)
80
78
76
cursfinc]
ours]rand] ours
0.00
0.50
0.75
low-prec.
(a)
ResNet-18
Figure 16: Continued from Figure 5. Memory-accurant tradeoffs of Tours, Toursfine), and Toursfrand),r for
ResNet-18 18 on CIFAR-100. Observe that @s are above and to the right of other points in nearly all cases.
1.0
60
test accuracy (9
80
0.8
accuracy
0.6
0.6
40
40
0.4
top-2
20
0.2
tpp
oursEno promo]
oursino promo]
top-
0.0
0.0
50
100
150
200
50
100
150
200
0.00
0.25
0.50
0.75
1.00
0.0
0.2
0.4
0.6
0.8
1.0
epoch
opach
low-prec. retio
low-prec. ratio (start)
(a) CIFAR-10, SqueezeNet
100
1.0
80
test accuracy (%)
80
0.8
test accuracy
60
0.6
40
40
0..4
top-2
0.2
10p-3
ours/no promo]
0.2
oursino promo]
0.0
50
100
150
200
200
0.00
0.25
0.50
0.75
1.00
0.0
0.2
0.4
0.6
0.8
1.0
epach
epach
low-prec. retio
low-prec. ratio (start)
(b) CIFAR-10 10, ShuffleNet-v2
100
100
1.0
(end)
test accuracy (%)
overfoow ratio
0.8
best accuracy (%6
0.8
60
60
0.66
40
0.4
20
0.2
tsp
1op-4
20
oursino promo]
0.2
ourstno promo] ours
ours
0.0
50
100
150
200
0.00
0.75
1.00
0.6
0.8
1.0
epoch
low-prec.
low-prec.
(c) CIFAR-10, MobileNet-v2
100
0.10
100
1.0
accuracy (%6)
80
0.08
test eccuracy (86)
60
overtoowss
60
0.6
0.04
top-1
20
0.02
lop-4
20
ours(ne-promo)
80.
ours(ne-promol ours
top
ours
0.00
50
100
150
200
epoch
low prec.
wwwpprec.om (start)
(d) CIFAR-10, ResNet-18
Figure 17: Continued from Figure Training four models on CIFAR-10 with Tours.r and Tours promoo))).
Column 1; Training trajectories of prumsjno-promoo.r for different r; colors denote values (darker for smaller r). Column 2: Top-5 overflow ratios of tensors each epoch, for the highlighted trajectory in (a); the largest ratio is blue and the fifth largest red. Column 3: Memory-accuracy tradeoffs of Tours, and Tours Tours/no-promo).. Column 4: Low-prccision ratio when training ends vs. when training starts, for Tours, and Tours(no-rromo).r
29


Reviews:
[{"review_id": "rTJqBkEPm3", "reviewer": "Reviewer_NKVw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed method is simple enough, thus easy to implement in different hardware platforms and deep learning frameworks, which is very important for its usability.\n2. The proposed method makes sense and theoretical analysis is provided.\n3. Empirical performance is impressive. Extensive empirical results are provided with convincing ablation studies.\n\nWeaknesses:\n\n1. The authors mention several times in the paper that the precision promotion technique that handles the overflow issue can be applied to other precision assignment methods. However, related empirical results are not provided. I think this is not just a \"bonus\" to provide, but an essential step to prove that the proposed assignment strategy really has practical contributions standing alone. If the existing assignment methods are very competitive with the precision promotion technique, and if the overall low-precision ratio is not influenced much for sure, then the contribution of the proposed assignment strategy is undermined in my opinion. I hope the authors could provide some comparison in this perspective.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work provides a technique that explores this memory-accuracy tradeoff by generating precision assignments that use less memory and at the same time lead to more accurate models, compared to the precision assignments considered by prior work in low-precision floating-point training. In one sentence, the proposed method is to assign low-precision to tensors in a decreasing order of the tensor size. Ablation studies show that this ordering is better than the increasing order and the random order. When combined with special precision promotion for those tensors encountered with overflow issues, the proposed method is shown to achieve better trade-off compared to existing methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses part.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FjGXVu8oql", "reviewer": "Reviewer_E9NR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1) The problem is well-motivated and the paper is mostly well-written.\n2) Based on the empirical results, the trade-off between memory and accuracy does exist. In addition, their method achieves a better trade-off when the model hits the compression ratio.\n\nWeakness:\n\n1) The algorithm is not very clear, for example, how long do I need to hit the ratio? How many iterations do I need to run when I convert one part of the model? \n\n2) The memory usage reduction claim is vague. Is that the average usage during training or is it the one when the ratio is fixed? Also, even the ratio is fixed, it seems that during the training, the memory usage is dynamic due the promotion and re-demotion operations.\n\n3) Starting with all high precision seems strange to me especially the main benefit of the method is memory usage reduction. With a high-precision model, the peak usage of memory is very high and limits the usage on edge devices. Then what's the point of reducing the memory?\n\n4) It seems the proposed method works better on small models and has similar performance on larger models compared to previous methods. \n\n5) Can we do more experiments on other tasks or larger models such as transformers?\n\n6) The figure for ImageNet is not informative, can we zoom in a little bit at the end of the training?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new algorithm to learn how to quantize the neural network during training with mixed floating precisions to reduce memory usage. They first theoretically analyze the memory-accuracy trade-off problem, and show that the problem is NP-Hard. To solve this problem, they propose a heuristic algorithm. Starting with a full precision mode, they gradually reduce the precision of a group of tensors (for both forward and backward) based on the parameter size and adjust them when too many overflows happen.  They empirically present that their method can reduce memory usage more than 2x over a baseline precision assignment while preserving training accuracy. Experiments are conducted on various datasets and models for image classification problems.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please view the points of weakness for details.\n\nThe two main points are listed as follows:\n\n1) Can we make the algorithm more clear? Especially for how many rounds I need to run to reach the target quantization ratio. \n\n2) Can we show the model can be trained in a mixed-precision way that starts with low precision? ", "broader_impact_concerns": "I don't have any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "95t8CjuZWY", "reviewer": "Reviewer_LPEm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n> + Well-motivated: using low-precision tensor in DNN training is a common technique to reduce the memory requirement and training time. However, using one precision setting for all the tensors may not be the optimal solution. It is important to design a mapping function to map different tensors to different precision.\n> Idea simple and effective: assigning tensors to low precision in size order is simple and can be supported by most devices and even pre-computed offline and the experiments validate that such a simple scheme can beat existing precision assignment methods in both commonly used models and compact models.\n> + Comprehensive ablation studies: the ablation studies cover both the studies on each single technique and hyperparameters.\n\n# Weakness\n> + Limited tasks: all the tasks are done in image classification tasks. Since the proposed method is a heuristic and greedy approach and the improvement over the previous precision assignment is not so large, it would be better to show this method is generally applicable to different tasks.\n> + Unclear hardware benefit: the authors claim that the selection of precision options (e.g., FP8 or FP32) is to support the incoming hardware. However, memory saving is measured by a self-defined low-precision ratio. Although I fully understand the hardware measurement may be difficult now because of the limited access to the new hardware, but a low-precision ratio may be too simple to evaluate the memory cost. For example, because of the proposed Precision Promotion, every tensor has the opportunity to be high-precision. Then the hardware has to design sufficient memory for such cases. Thus, memory utilization can be low and the real memory saving may be smaller than the low-precision ratio because of the cases that low-precision tensor can be high-precision and extra cost to the data type casting.\n> +  More straightforward illustration of the proposed algorithm: it would be better to add a diagram or an algorithm table to show the proposed algorithm to avoid the cases that readers lose in the formulas in Sec. 4.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work targets the problem of mapping from all tensors (arising in training) to precision levels (high or low) in DNN training. Specifically, the authors start with the largest tensor in the model, assign tensors to low precision in size order (from largest to smallest) until the model aggregate falls below the given upper bound, and keep all remaining tensors high precision. The experiments are conducted on CIFAR-10/100 and ImageNet with models as SqueezeNet, ShuffleNet-v2, MobileNet-v2, and ResNet-18.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see #Weakness above. ", "broader_impact_concerns": "No concern about the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SyX5SpyTov", "reviewer": "Reviewer_eNE7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n- Careful study of how the variables related to the mixed-precision assignment algorithm affect performance\n- Careful comparison with baselines\n\n### Weaknesses\n- The predictive performance of the method is relatively poor. This is because the authors perform intermediate operations between GEMM operations in FP8 if his group has been demoted. A better practice would be to perform just GEMM operations in low precision because 95% of parameters and computation are done in these layers; other layers have little benefit from FP8 computation. \n- No experimental results on transformers. The authors claim that it is difficult to implement their method for transformers and thus cannot be studied easily. Asides from the softmax operation (which is usually done in FP32 anyways), there is no operation in transformers that is inherently different from convolutional networks (except convolution, but this can be seen as a matrix multiplication too). \n- No experimental results at scale. Recent literature highlights that many successful low-precision methods fail at scale. Larger models introduce new problems and it is unclear if the authors algorithm can work at scale.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\n\nThis work studies mixed-precision training in 8/16-bit with the goal of performing as many operations as possible in 8-bit Floats without losing accuracy. This is achieved by using a system of promotion and demotion to and from high-precision as well as a heuristic to initialize the tensors so that a given ratio uses 8-bit tensors.\n\n**Contributions**:\n- Develop an algorithm for promotion/demotion and compare it to uniform precision assignments\n- Developing an algorithm how for assigning the precision to initial tensors\n- Studying and analyzing the trade-offs between ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think in the current form, your claims are too general. \n\nI think the following adjustments to your claims or additional experiments to support your claims would be suitable:\n- Your method is currently only shown for convolutional networks. Either highlight that your method works just for convolutional networks, or add transformer experiments. Additional experiments that would convince me that your method is more general would be to repeat your experimental method but to use functions that your software currently does not support in 16/32-bit (such as softmax). Then use this setup and train either a CLIP model (OpenCLIP) or text-only transformer on WikiText-103. \n- I think your claim that you outperform existing precision assignments is not warranted. I would see \"all GEMM in low-precision, and everything else in high-precision\" also as a heuristic not unlike your, but currently, you do not compare against it. This heuristic has been very successful (Micikevicius et al., 2022, FP8 Formats for Deep Learning), but it is known to fail in some large-scale scenarios. Otherwise, you could reduce your claims and say you outperform methods where at least some non-GEMM operations are done in FP8.\n\nI deems these claim-reductions or additional experiments critical for acceptance since the more general claims are not supported by the evidence in the paper.", "broader_impact_concerns": "Currently, the work has no Broader Impact Statement section. I think that is okay. One could make an argument that better FP8 training makes it easier to train more powerful networks with fewer resources. This can have an impact for people with the least resources to make more powerful models available, but it can also help large corporations to train even more powerful networks for the same costs. Both of these downstream effects would affect society. As such, a Broader Impact Statement could be warranted.  ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Some of the reviewers still have (minor) comments relating to presentation.  The authors are requested to address these before publication.
Decision: Accept (poster)