Towards Cheaper Inference in Deep Networks with Lower Bit-Width Accumulators
Authors: Yaniv Blumenfeld
Itay Hubara
Daniel Soudry
Venue: ICLR 2024 poster (2024)
Keywords: Deep Neural Networks
Quantized Neural Networks
Network Quantization
Accumulators
Accelerators
Inference
Computer Vision
Language Models
Primary area: infrastructure, software libraries, hardware, etc.
TL;DR: We show that model can be fine tuned for inference with 12 bit accumulators, and develop methods for training with even smaller accumulators.

Abstract: The majority of the research on the quantization of Deep Neural Networks (DNNs) is focused on reducing the precision of tensors visible by high-level frameworks (e.g., weights, activations, and gradients). However, current hardware still relies on high-accuracy core operations. Most significant is the operation of accumulating products. This high-precision accumulation operation is gradually becoming the main computational bottleneck. This is because, so far, the usage of low-precision accumulators led to a significant degradation in performance. In this work, we present a simple method to train and fine-tune DNNs, to allow, for the first time, utilization of cheaper, $12$-bits accumulators, with no significant degradation in accuracy. Lastly, we show that as we decrease the accumulation precision further, using fine-grained gradient approximations can improve the DNN accuracy.

Published as a conference paper at ICLR 2024
TOWARDS CHEAPER INFERENCE IN DEEP NETWORKS WITH LOWER BIT- WIT-W ACCUMULATORS
Yaniv Blumenfeld Technion, Israel yani vb1m66 com
Itay Hubara
Daniel Soudry Technion, Israel dan Ganiel_ ssuudryggaa com
LabanaLaba,,,,s, Israel i ttayubaa racooarraaae ra@gma 1 com
ABSTRACT
The majority of the research on the quantization of Deep Neural Networks (DNNs) is focused on reducing the precision tensors visible by high-level frameworks (e.g., weights, activations, and gradients). However, current hardware still relies on high-accuracy core operations. Most significant is the operation of accumulating products. This high-precision accumulation operation is gradually becoming the main computational bottleneck. This is because, so far, the usage of low-precision accumulators led to significant degradation in performance. In this work, we present simple method to train and fine-tune high-end DNNs, to allow, for the first time, utilization of cheaper, 12-bits accumulators, with no significant degradation in accuracy. Lastly, we show that as we decrease the accumulation precision further, using fine-gres -ra-grii gradient approximations can improve the DNN accuracy.
INTRODUCTION
Deep Neural Networks (DNNs) quantization (Hubara et al. 2017 Sun et al. 2020 Banner et al. 2018
Nagel et al. 2022] Chmiel et al. 202T) have been generally successful at improving the efficiency of neural networks' computation without harming the accuracy of the network Liang et al. (2021). The suggested methods aim to reduce the cost of the Huutiply-An-Acccumlle both training and inference. To this end, they quantize the weights, activations, and gradients. For applications utilizing such quantization methods, the cost of multiplications, commonly considered to be the computational bottleneck, can be substantially reduced. However, the accumulation of computed products is still performed with high-precision data types. Consequently, the cost of the accumulation, component of MAC operations, becomes increasingly dominant in performance breakdowns (Sakr et al. 2019 Ni et al. 2020; Chmiel et al. (022))
(MAC) operations for
For example, when the weights and activations are in the common FP8 format, van Baalen al. (2023) showed the accumulation becomes a computational bottleneck. For example, they conducted experiments estimate the raw gate count for various FP8 implementations (a first-order approx- imation for power and area) and observed a 2x reduction in gate count when employing FP16 accumulators instead of FP32. Similarly, NN et al. (2020) reported analogous findings for INT8, demonstrating that an 8-bitx8-bit multiplier consumes a comparable amount of power and silicon area to a 32-bit accumulator.
In this study, we focus on reducing the numerical precision of the accumulation operation in DNNs. Building our solution on top of the emerging FP8 format, which has gained prominence for both training and inference on the most prevalent hardware Andersch et al. (2022), we aim to optimize such DNNs, to enable inference on hardware with Low Bit-width Accumulators (LBAs).
Our main contributions are:
We propose a simple scheme for fine-tuning models with 12-bit accumulators for a variety of tasks, and show this method can already achieve strong performance. For example, we show for the first time that 12-bits accumulators can be used in ResNets on ImageNet, with no significant degradation in accuracy.
We examine more fine- mee-grained approaches, in which, for the first time, we backpropagate through the entire accumulation-compuattio graph. Though much more expensive during training, such fine-grained backpropagation can be used to significantly improve the accuracy of DNNs with LBAs at lower bit-ws.ths.

---

Published as a conference paper at ICLR 2024
2
PRELIMINARIES: QUANTIZED NEURAL NETWORKS
2.1 QUANTIZED WEIGHTS AND ACTIVATIONS
The quantization of neural networks is, by now, a standard practice for achieving efficient neural networks. Unlike traditional scientific computation, that often (Bailey 2005) requires high-precision floating point arithmetic (e.g., FP64) to achieve accurate results, it was observed (Gupta et al.) 2015) that deep neural networks maintain high accuracy when the weights and activations in the network are represented in low bit representation. As a result, training deep neural networks (DNNs) using half-precision (FP16) arithmetic became the default setup for modern Deep Learning applications (Brown et al. 2020). Lower precision representation (INT8, FP8, INT4, FP4, and Binary) (Sun et al. 2019; 2020 Courbariaux et al.) 2016) also used for a variety of deep learning applications, for either training or inference, albeit using them is more experimental and may result in lower model performance, depending on the specific application.
Quantization of Weights and Activations (W/A) has two main benefits.
Lower memory footprint: By reducing the number of bits used for representation of each numerical value, W/A quantization can significantly reduce the memory required for storing and using a neural network. Consequently. W/A quantization enables storing larger models (with more parameters and activations) on DL accelerators with finite storage and improves the computation efficiency of smaller models by mitigating memory bottlenecks.
Reduced complexity of multiplication operation: Neural networks commonly compute mul- tiplications of weight and activation pairs. When both weight and activation are represented at a lower precision, it is possible perform the multiplication operation with cheaper hardware (smaller area, less energy). This allows us to do more multiplication operations per second, provided that the hardware was designed to support these lower- peeer-rrecision operations.
Numerical values are typically represented using either fixed, or floating point format. Methods for quantization of DNNs can divided accordingly.
2.2 FIXED POINT QUANTIZATION
Given full-precision value x, a fixed number of bits B, and an integer b (exponent-bias), we define the fixed-point quantization of x as:
Rmin
x S Rmmi
Rmin = -28-b-1
Rmax = 2-b (2B-1-1)
FIXED (x) =
R Riax
x 2 Rmax
2 Round (x 2
else
As see from Eq. (T), the process of fixed-point quantization involves two explicit changes to the value of x. First, we round x 2 to an integer value. The rounding operation can be done using a variety of operations (such Floor, Ceil, Nearest-Neighhour et al. 2018)), but will result in a loss of information either way, with rounding eтror that decreases as we increase the parameter b: A 2-b If the value of x is sufficiently small |x] < 2- the quantization noise will exceed the represented value (A > (x|) and we have no way to accurately represent the value of z. We will refer this event as underflow. Second, we have a limited range for representation, that increases with the number of bits B and decreases with b. We refer to the event when æ is outside the range (Rmin; Rmax) as overflow, noting that the quantization error this case is unbounded.
Stochastic Rounding (Wang
Integer quantization a specific case of fixed-point quantization, where the exponent bias b set to While we defined the exponent-bias 6 to be an integer, it is important to note that non-integer values could have worked mathematically just as well to define valid quantization operations. The main benefit of choosing to be integer is the efficiency of computing power-of-two multiplications in hardware.
The main advantage of fixed point quantization comes from its relative simplicity. Integer multiplica- tion (and addition) are generally considered to be cheaper on hardware when compared with floating point operations.
2

---

Published as a conference paper at ICLR 2024
2.3 FLOATING POINT QUANTIZATION
Given full-precision scalar value æ, number of mantissa bits M, number of exponent bits E, and an integer b (exponent-bias), we define the floating point (Dekker 1971 quantization M/E:
[og2([=()) m 2-- Round (2 (|x|2- - - 1))
1 (1 sign(z))
RoF
|x| 2 RoF
2
Q (a) = (-1)² M.E.B
0
|a| Rur
Ror 22" 6-1 (2 2-M), Rur 2-b
2 (m 1)
else
(2)
Note that S |x|2- < 2, due to the definition of e, which helps make sense of the quantization
operation in Eq. 2 The total number of bits used for this representation is B M E + sign bit (s), M mantissa bits (m) and E exponent bits (e). As we can see, floating point representation can cover a larger range of values when compared with fixed point representation that uses the same amount of bits and exponent bias, (ROFUU depends on 22t while Rmax, Rmin depends on 2B), reducing the occurrence of overflow and underflow events.
Unlike fixed point representation, which had a fixed bound for quantization error (A ~ 2-"") within the represented range, the quantization error for floating point representation varies, depending the magnitude of x: ~ 2e-M M. As a direct result, floating point's arithmetic also adds additional complexity, in the form of swamping Higham (1993). When performing addition over two floating points values = 21 tiPp) 22 = FLOAT
F, [2) + 22), it is possible that the precision of 2 will not be sufficient for full-representation of its summands, causing the least significant bits to be swamped out resulting in 'noisy" addition operation. In the extreme case, denoted Full-Swamping, |2|| > | 22 is swamped out entirely, so = 21 despite 22 being non-zero. In contrast, fixed-point addition will always be exact, as long as the sum remains within the representation range (no overflow).
2.4 Low BIT- WIDTH ACCUMULATORS
When performing general matrix multiplication (GEMM) operation, (e.g. matrix mutrix-multiplication or convolution), each individual scalar computed during the operation can be expressed as the sum of product pairs
N
y
J.W,y
Here, y is scalar component of the output tensor of the GEMM operation, N is the accumulations size (i.e., the number of summands used per scalar output), while (r.) and (w,) N-1 are two series of scalar inputs used for the calculation of y. The values in both series originate from the input tensors, but the exact mapping, from tensors to series, will depend Appendix A for more details). Due to the common structure of the muutiply-accumulle hardware implementations of GEMM operation often rely on the fundamental Fused Multiply-Ade (FMA) operation, defined as FMA(r: (r, w, 8) = x w + 8, with x, w, 8 being scalars. Our goal in this work will be decrease the cost of the FMA component.
N-1
the performed operation (see
operation,
Previous discussed methods, such as W/A quantization, have been helpful in reducing the cost of the multiplication of FMA. In contrast, the accumulation component of FMA has been studied to a much lesser extent. In (Wang et al. 2018), the authors show that training a neural network with FP16 accumulators can result noisy training, with a modest loss of accuracy. To mitigate this, the paper recommends chunk-based based accumulation and floating-pott -point stochastic rounding. Chunk-based accumulation changes the order of accumulation, while stochastic rounding small, random noise is added to the result of high-precision summation, before the result is cast to low-precision representation. While successful at closing the gap (e.g., for ResNet18 on ImageNet), both methods may prove difficult to implement modern hardware. Specifically, the order of accumulation on DL accelerators will usually depend on their block architecture and is not easily configured. Moreover, stochastic rounding requires implicit addition operation, which is projected to increase the cost of hardware addition, negating the benefit of using LBAs.
method where
Sakr al. (2019) examined the effect of low precision accumulators training through the accu- mulation variance statistic, which they theoretically derive, given several statistical assumptions on the distribution of the summands. In Ni et al. (2020) the authors propose WrapNet, where the
3

---

Published as a conference paper at ICLR 2024
additions performed with 8 and 12 integer accumulators with wrap-arouna WrapNet is shown to perform complex inference tasks (e.g. ImageNet classification) with extreme quantization (e.g.. 7 bits activations, 2 bit weights, and 12 bits accumulators), but does suffer noticeable accuracy degradation this setup, for tasks such as ImageNet classification.
Although mostly experimental, FP16 accumulation integrated in the design of several commercial products (Agrawal et al. 2021). including the tensor cores in the Hopper architecture (NVIDIA)
Andersch al. (2022).
FINE-TUNING NEURAL. NETWORKS WITH Low-Br BIT ACCUMULATORS
One key difference between W/A quantization and quantization of the accumulators is that accumula- tion an internal FMA operation, which is not generally visible to the software user. To simulate the effect of quantized FMA component, implement the GEMM operations (convolution/ matrix multiply) in CUDA, where the FMA operation is replaced with our custom FMAq operation:
FMAq( w, Quee (Qprod (x: (t-w)) w) + s),
as illustrated Fig. [ all experiments, we use constant chunk size 16, based on the sizes exposed the user of NVIDIA's tensor cores. It is important to highlight that the product and
W,
%
We
X1
Xn-1 We-t
FMA
FMA
FMA
Ow
A
Saet Wor1
Xan- Win-1
Add
FMA
FMA
FMA
Muttiply
prod
Mtinet WNin+1
Add
S
Add
Qcc
S
FMA
FMA
FMA
FMA
Oues
Figure 1: Left: an illustration of quantized FMA component, as simulated in our work. Unlike the W/A quantization operations (Qw (Q)(ww), xx that can be efficiently performed in software, Qpeod and Que are explicitly internal hardware operations, intended to simulate the logic of cheaper hardware component. Right: Illustration chunk- based accumulation, with chunk base of n. Chunk- based accumulation is useful for reducing error caused by swamping, but the chunk size is not easily configured and will usually depend on the architecture design of the systolic array.
accumulator quantization functions (Qpred and Qace) intended to simulate the hardware, rather than suggest an implementation for it. Breaking down the FMA to components in hardware would, in practice, undermine its efficiency as it will no longer be 'fused'. Taking this into account, 2pod and Quee must remain simple and computationally efficient. For example, 'round to nearest' and stochastic rounding methods, which are taken for granted for W/A quantization, will be available to us during inference, as their hardware implementation would still perform addition internally with higher number of bits. Our quantization will instead rely on the simple 'floor' operation, implemented in software via bit-mask. For Hardware analysis of our implementation, see Appendix B
As discussed section 2.3 floating point quantization can be broken down into 3-distinct events: underflow, overflow and swamping. Eventually, low-precision model will have to handle all three events. We will, however, start by examining their individual properties, as displayed in Tab. [
Our main insight from Tab. I is that underflow events expected to have the least significant effect over the network output (They have the lowest absolute error, since the default value for the evaluate
exponent bias b, as used by the common FP32/FPI6 definitions, is 2 In Fig. the correctness this claim, and show that the wide-scope landslaee of an LBA ResNet is barely affected when ignore UF events, And yet, the large relative error induced during underflow
4

---

Published as a conference paper at ICLR 2024
Key Parameters
Absolute Error (bound):
Event
Condition
Relative Error: [
A = =Q(x)) x
Overflow (OF)
[r]
2 2
E,
0
(0%, 00)
Underflow (UF)
[a] 2
E. b
2-b
100%
Swamping
No OF/UF
M
(M)))
,2
Table 1: Properties of each type of floating-point quantization event.
(small elements are effectively replaced with zero), will cause significant optimization errors for gradient- methods: During fine-tuning, we can expect the magnitude of the weight updates to be proportional the magnitude of the corresponding weights, causing the underflow region to be particular hard region to 'escape' from. Values that are stuck at underflow are effectively excluded from the training since the forced value of prevents them from leamnngg meaningful correlations. (see Appendix for more details.)
4
(a) Baseline LBA
(b) Excluding UF
(c) Excluding Swamping
Figure 2: Wide scope loss landscapes et al. 2018) of an LBA resmet50, using pre-trained ResNet50 weights (CIFAR10, FP32). Here, compare the qualitative effect of different components in floating points quantization over the network output: In (a), we use a complete implementation of FP quantization during convolution accumulation, with 7 Mantissa and Exponent bits. In (b), repeat the previous experiment but ignore underflow events during quantization. For comparison, in (c), we repeat the original experiment, but add 16 additional bits to the mantissa, greatly diminishing the effect of swamping, without affecting the role of underflow. All landscapes appear similar, but while the effect of excluding swamping events (c) is visible, the loss landscapes of networks with (a) and without (b) underflow are hardly distinguishable.
Therefore, propose the following method: Starting off with the weights of pre-trained net- work (trained in full- full-prccision) we will design a network that utilizes quantized FMA for forward- propagation, excluding underflow events, and perform a standard gradient-based optimization (i.e. Stochastic gradient decent, while keeping the backward implementation each operation as it was with full-precision FMAs). Once we converge some accuracy value, will enable the underflow implementation and proceed with further fine-tuning.
As seen in Tab. the exponent bias (b) can be configured control underflow and overflow events, with a clear trade-off between the former and the latter. Previous works Kuzmin et al. (2022) have made the insight, that the default value our purposes, we note that the different quantization functions Qpeui and Qaee as seen in Fig. [ are likely to require different ranges for representation: Assuming the product terms u, = 0;20; are i.i.d the accumulator's value will follow the central limit theorem, and is therefore more likely to reach overflow, resulting unbounded quantization noise. To try and avoid this scenario, our setup will give smaller exponent bias to the accumulator. In our experiments, the chunk-size, so that dace = bprod logz (Chunk-Size) Size). Following the same reasoning, one may suggest that the exponent bias should depend on the sequence number in which the FMA is applied within every GEMM operation. Nevertheless, for the context of this work, we will treat all FMA units homogeneous, with the same exponent bias.
2 is not always suitable for neural networks, For
use a relative factor based on
5

---

The results of Published as a conference paper at ICLR 2024
3.1 EXPERIMENTS: IMAGE CLASSIFICATION
For our first set of experiments, we aim to check the effect low-bit accumulators have residual neural networks He al. (2016). For each experiment, we use the standard ResNet architecture and replace each GEMM operation used during tion) with our custom implementation, as described in section 3 these changes the network accuracy (zero-shot), can be severe, as we show in Appendix B
(convolution and matrix multiplica- With no adjustments, the effect of
For Qprdd Qaces we used the same amount of mantissa and exponent bits, M = 7, E = 4, setup we will denote M7E4. For overflow, we used the exponent biases: dace 10, Upred 12, but disabled underflow events for the first part of the experiment. After loading the networks with pre-trained weights, we proceed train the network for 5 epochs, using Adam optimizer with learning rate of 7/o = 10-6, and a cosine scheduler, so that 7/s = 10-5). Then, we enable underflow events and run fine-tuning again for a single epoch, using a reduced learning rate of MUF 10-7. To evaluate the benefit of the two-staged fine-tuning, we also ran the same experiment with a single stage, where underflow is enabled for 10 epochs. The baseline numbers were obtained by repeating the fine-tuning process in a non-LBA setup, which resulted in an improvement of up to 0. 0.65% over the zero-shot accuracy. Our full setup and implementation are detailed in Appendix [ this experiment are presented in Tab. 2
Model
Baseline
1-stage
no UF*
no UF -> with UF
ResNet18
70.23%
69.94%
70.01%
70.06%
ResNet34
73.87%
73.64%
73.61%
73.45%
ResNet50
76.80%
74.70%
76.60%
76.40%
Table 2: Top-1 Accuracy results: Fine-tuning ResNets with low-bit accumulators for ImageNet classification.
*Intermediate Stage:
Both training and evaluation are done without underflow.
For LBA ResNets with full-precision W/A, our results indicate that the models we suggest can train surprisingly well even without a dedicated fine-tuning regime. The dual-stage approach (Training without UF first and enabling it later) only shows clear benefit, so far, in the case of the larger, ResNe150 model. That being said, scaling the method for larger models is important, and tasks will only become more difficult from now on.
In order for model with low-bit accumulators to be commercially viable, it is vital to show that quantized accumulation still works when the weights and activations are quantized. Therefore, our next set of experiments will test the feasibility of LBA ResNets in this setting. For weights and activations, we will use 8-bit floating point representation (Wang et al. 2018). Following the results presented in Kuzmin et al. (022). use M4E3 representation with flex-bias for both weights and activations, implemented using the qtorch library Zhang et al. (2119). For our flex-bias implementation, we evaluate the maximal exponent for each tensor during forward propagation, and use the maximal integer exponent bias that is sufficient to prevent overflows (single value per tensor). The results of fine-tuning LBA ResNets this setup can be seen in Tab. as well as a comparison of our results with previous works that also used lower-bit accumulators.
We note that a direct comparison between the methods based on final accuracy alone will not be valid: the method presented in Wangg et al. (2018) is intended for quantized training, and includes several more quantized components, well as several methods that are projected reduce hardware efficiency. Meanwhile, NN et al.] (2020) proposes the cheapest implementation (Fewer bits for Weights and activations, Integer quantization), sacrificing model accuracy for hardware efficiency. Nevertheless, when aiming for cheaper inference, our LBA models were the only models to achieve accuracy on par with non-LBA models, while providing a cheaper alternative compared to models with standard accumulation.
3.2 EXPERIMENTS: LANGUAGE MODELS
To assess the capability of LBA language models, our next set of experiments will focus on the common Bert (Devlin et al. 2018) architecture, and the SQUAD (Question-Answering task. In

---

Published as a conference paper at ICLR 2024
Model
Data Type
Weights
Activations
Accumulator
Top-1 Accuracy
R8sNet18
Baseline
FP
32
32
70.23%
Baseline (FP8)
FP
8
32
69.90%
Wang et al. 20188
FP
8
16
66.95%
Ni et al. (0200)
INT
7
12
63.84%
Ours (1 Stage)
FP
8
12
69.54%
Ours (dual-stage)
FP
8
12
69.70%
ResNet34
Baseline
FP
32
32
32
73.87%
Baseline (FP8)
FP
8
8
32
73.49%
Ours (1-stage)
FP
12
73.18%
Ours (dual-stage)
FP
8
8
12
73.42%
Re
Baseline
FP
32
32
32
76.80%
Baseline (FP8)
FP
8
32
76.25%
Wang et al. (2018
FP
8
16
71.72%
Ours (1 (I-stage)
FP
8
12
74.15%
Ours (dual-stage)
FP
8
12
76.22%
Table 3: Top-1 Accuracy results: Fine-tuning ResNets with low-bit accumulators and FP8 weights and activations for ImageNet classification. Results are compared with similar models utilizing LBAAs in the literature.
this case, fine-tuning a pre-trained model is already the standard. In contrast to our experience with residual networks, breaking down the fine-tuning process into separate phases was not, in general, beneficial for the accuracy of the larger models. The exponent biases we used for the different LBA models also had be changed, avoid overflow events. In tablef we compare the results of fine-tuning LBA Bert models with the results of fine-tuning non-LBA models, as described in C.2 While LBA Bert-small has a small (A,I 0.37%) performance degradation compared with the non-LBA model, the gap is closed completely for the Bert (A,1 -0.09% and Bert-Large (A = 0.26%).
LBA (M7E4)
LBA (M7E4)
Baseline
bace prod=
Unce Dhrod 10
Model
Exact (%)
fl (%)
Exact (%)
fl (%)
Exact (%)
fl (%)
Bert-Small
71.32
80.96
70.88
80.24
71.35
80.59
Base
79.84
87.53
79.60
87.62
79.80
87.52
Bert- Large
83.22
90.40
82.97
89.97
83.25
90.66
Table 4: SQUAD v1 fine-tuning for LBA-Bert models.
Inspired by our LBA-Bert model results (which were favorable toward larger models), we tested our LBA-aware fine-tune fine-tuning method on the LLama-v2-77 v2BB model (Touvron 2023) We used the same settings and scripts QLoRA paper (Dettmers et al.) 2023), which uses frozen 4-bit weights with an additional trainable low-rank matrix in BF16. To measure performance on range of language understanding tasks, we used the MMLU (Massively Multitask Language Understanding) benchmark Hendrycks al, (2020), multiple-choice benchmark covering 57 tasks. The fine-tuning was done over the Open Assistant (OASSAI) dataset Köpf et al. (2023) using official training scripts found in the QLoRA code (i.c., llama2 guanaco__aaa _bb). We report 5-shot test accuracy tabel
Model
Baseline
M10E5
M6E5
M7E4
LLamma v2 (OASSAL)
45.3
45.4
44.3
45.1
Table 5: MMLU 5-shot test accuracy with and without LBA, for QLORA+ LL Lmamma v2 (7B parameters). For runs with exponent bits, used dynamic (per-layer) exponent-bias.
7

---

Published as a conference paper at ICLR 2024
4 BELOW 12 BITS: FINE-GRAINED GRADIENT FOR Low BIT ACCUMULATORS
Thus far, we have shown that 12 bits are sufficient for inference a variety of deep neural networks. However, the simple methods described in section 3 are not sufficient for training neural networks with lower amounts of accumulation bits.
For example, shallow cally-connected DNN trained over MNIST, will fail when using M4E3 accumulator, even when excluding underflow events. The cause of the failure is known as it is similar to quantization failures in other areas of deep neural network: Quantization changes the output of the network during forward pass, and when the change is significant enough, it is no longer feasible to rely on the gradients of non-quantized operations for optimization. Of course, we cannot the "reall" gradients with respect to quantized operation, since they are zero almost everywhere.
The common solution to this problem, with relation to the quantization of weights and activations, Through-Estimatoe (STE). In
is to replace the derivative of the quantized function with a our case, we would like to use the STE with respect to the derivatives of the quantizers Qace and Qoood inside the FMAq operation from Eq. (4. So far in this work, we used the naive "identity STE" (Bengio et al. 2013) which makes the replacement d Q(z)" = I (we will use the quotation marks to denote an STE replacement of a derivative). However, the more common STE for quantization zeros out gradients outside of the representation range (Hubara et al. 2017). For the quantizers in Eq. ① and Eq. we get:
dz
d FIXED (x)" = 1(Rmin < < < Rmmxx) QB,0 dx
d QELOAT dx (x))" = 1(xx| < RoF),
(5)
where we defined 1() as the indicator function which is equal if its input is 'true' and zero otherwise. Many alternative forms of STEs exist and have been studied in the context of W/A quantization. The implementation of STES for IBA networks, on the other hand, has several additional difficulties.
The first, most immediate problem, is that the values of the inputs of the quantization functions within the FMAq (Quee and Qreaa) are not exposed to the software or stored in memory during forward propagation. Saving these internal values is generally not feasible, since the quantization operation occurs in each FMAq, and the number of FMAqs in DNNs typically exceeds the size of weights and activations by many orders of magnitude. However, if the hardware operation is deterministic and well-known, we found we are still able to use software for re-computation of the GEMM operation, to retrieve the required values during backpropagation (1 bit per operation). Such a re-computation operation is expensive (training time is doubled, at the very least), and so far feasible only in fully connected and attention layers (not convolutional layers). To the best our knowledge, this is the first time backpropagation is used on the full computational graph of the summation operation.
Another possible problem for using standard STES for the accumulation process stems from the recursive nature of the summation operation. The STE in equation Eq. (5) sets the corresponding gradient of any overflowing value to zero. As explained in Appendix [ if this STE is used for the accumulator's quantization function, each overflow event will eliminate the gradients of all previously accumulated product pairs. Lastly, another possible problem is that, for floating point summation, other events besides overflow can potentially be important when estimating the gradient.
Motivated by the last two potential problems, in appendix D we propose, describe, and justify the practicality of several alternative methods for estimating the gradients of FMAq(1, w, 8). The different methods use different types of STE: OF passes zero on overflow of Q (using Eq. ( while DIFF passes zero on overflow, underflow, and swamping events of the FMAq. We also distinguish between a method where we apply identity STE with respect to the partial sum s, and the non-identity STE over the product-pair (I,, w) (a.k.a Immediate), to the standard method, where the STE is applied with respect to all inputs (x., w) (a.k.a Recursive). For example, defining z FMAq(a, w, 8) = Quee (Qprod (xr w) + s) and E1, 62 as some small constants, we get:
de
1 dz d.dz,,
de
(1(
|2 sl
Immediate/ / DIFF:
ds
;
W dx x du
-1
(Iww] +
E2
(6)
dz 1 dz 1 dz
Recursive OF:
1(1Q (rw) + s| < RoF))
w dr dw In Tab. 6 compare the accuracy achieved using the proposed STE variants over the MNIST dataset. We see that such fine- ain-ggrined gradient methods can indeed enable high accuracy in models with only 8-bit accumulators.
ds
8

---

Published as a conference paper at ICLR 2024
Accuracy (Top-1, %)
Accuracy
STE
Underflow
STE
Underflow
(Top-1, %)
Baseline
98.65
Immediate OF
Yes
98.47
Identity
Yes
18.28
Immediate / DIFF
Yes
11.35
Identity
No
18.28
Immediate / DIFF
No
97.67
+Identity*
Yes
42.28
Recursive OF
Yes
98.47
Table 6: Training fully-connected NN with 8-bit (M4E3) accumulators for MNIST classification. The reported accuracy matches the final accuracy of the experiment. The model's loss does not converge when using naive (Identity) STE for accumulation. Full details in Appendix C.3 *The mantissa for the accumulator was extended by 2 additional bits in this run.
As saw in the of residual neural networks (Tab, 2 and with 1-stage training, successful implementation of LBA is not guaranteed to scale to larger models. To evaluate the quality of our estimated gradients, that end, we train a small LBA transformer from scratch for masked language modeling, over modest-sized dataset (200K rows), for 15 epochs. In Tab. compare different STE variants for variety of very-low precision accumulators.
would like compare the optimization of the different approaches. To
Recursive OF (%)
Immediate OF (%)
Immediate DIFF (%)
Accumulator
Identity (%)
FP32
51.31
M3E3
20.86
19.20
14.80
24. .00
M4E3
13.88
39.57
37.23
41.94
M5E3
9.47
45.28
44.76
50.12
M6E3
14.71
46.17
46.13
50.03
M3E4
15.2
15.15
15.43
25.53
M4E4
42.93
42.81
42.81
41.50
M5E4
47.87
48.76
48.76
47.93
Table 7: Accuracy of LBA transformer for the task of Masked Language Modelling (200 rows), when using different STEs for the accumulator operation. Full details of the experiments available in Appendix C.4
Based on our results for training masked language models, using fine-graines STEs becomes crucial when the number of accumulation bits is decreased below M 4 or E = (hence, this includes all possible FP8 formats). While successful at improving the optimization, none of the STEs we have tried were successful at closing the gap with the baseline completely. when extreme accumulator quantization was applied. Out of the three proposed STEs, we recommend Immediate/ DIFF STE, which generally achieved hetter accuracy in the areas where naive, identity STE was insufficient, despite its higher cost. The Immediate/ DIFF STE may also prove more suitable in cases where the exact behavior of the FMAq is unknown (i.e., 'black-box') since its definition is agnostic to the FMAq internals.
DISCUSSION
The quantization of the accumulator in deep neural networks is a hard but necessary task in the effort to improve neural networks' efficiency, reduce cost, and cut down carbon footprint. Despite the many difficulties involving the training, the implementation, and the theoretical analysis of networks with low-bit-accumulator. our results show that LBA networks are surprisingly easy to fine-tune. By applying simple optimization methods over pre-trained networks, we show is possible to adjust the models for inference with cheaper hardware, that utilizes 12 bits accumulators. When the accumulators bit width is further reduced we alleviate the accuracy degradation by using fine- ine-grained approaches for estimating the gradient.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The research of DS was Funded by the European Union (ERC, A-B-C-Deep, 101039436). Views and opinions expressed are however those of the author only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency (ERCEA), Neither the European Union nor the granting authority can be held responsible for them. DS also acknowledges support of the Schmidt Career Advancement Chair Al.
REFERENCES
Ankur Agrawal, Sae Kyu Lee, Joel Silberman, Matthew Ziegler, Mingu Kang, Swagath Venkatara- mani, Nianzheng Cao, Bruce Fleischer, Michael Guillorn, Matthew Cohen, et al. 9.1 a 7nm 4-core ai chip with 25.6 tflops hybrid fp8 training, 102.4 tops int4 inference and workload-aware throttling. In 2021 IEEE International Solid-State Circuits Conference (ISSCC), volume 64, pp. 144-146. IEEE, 2021. (Cited on 4
Michael Andersch, Greg Palmer, Ronny Krashinsky, Nick Stam, Vishal Mehta, Gonzalo Brito, and Sridhar Ramaswamy. Nvidia hopper architecture in-depth, Apr 2022. URL https innrre--i--ae in-depth/
7/deve oper lop.r.vvdia wwt.ddi..coo./// http:///lo/.iiidaaaaaae-e-e-eeee
(Cited on 44
David Bailey. High-precision floating-point arithmetic in scientific computation. Computing in
science & engineering, 7(3):54-61 2005. (Cited on 2
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In Advances in Neural Information Processing Systems, pp. 115--1,3, 2018. (Cited on
17
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation, 2013. (Cited ()
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. (Cited
o
Brian Chmiel, Ron Banner, Elad Hoffer, Hilla Ben Yaacov, and Daniel Soudry. Logarithmic unbiased quantization: Practical 4-bit training in deep learning. arXiv preprint arXiv:2112.10765 2021.
(Cited on T
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran EI- Yaniv,, and Yoshua Bengio. Binarized neural networks. Advances in Neural Information Processing Systems, 2016. (Cited o
Theodorus Jozef Dekker. A floating-point technique for extending the available precision. Numerische Mathematik, 18(3):224-242, 1971. (Cited on
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lms. arXiv preprins arXiv:2305. 14314, 2023. (Cited on 7
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep 201 8.
bidirectional transformers for language understanding. arXiv preprint (Cited on 6)
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning, pp. 1737-1746, 2015. (Cited o
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. (Cited 66
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.033 033000 2020. (Cited on
10

---

Published as a conference paper at ICLR 2024
Nicholas J Higham. The accuracy of floating point summation. SIAM Journal Scientific Computing. 14(4):783-77) 799, 1993. (Cited on 3)
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1):6869-6888, 2017. (Cited on 8
Andreas Köpf. Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, al. Openassistant pnversaaiiossdemocratiii large language model alignment. arXiv preprint arXiv:2304.07337, 2023. (Cited on 7
Andrey Kuzmin, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, and Tijmen Blankevoort. Fp8 quantization: The power of the exponent. Advances in Neural Information Processing Systems, 35: 15:14611-11665 2022. (Cited on 5
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Neural Information Processing Systems, 2018. (Cited on
Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. Pruning and quantization for deep neural network acceleration: A survey, Neurocomputing. 461:370-400 2021. (Cited on
Markus Nagel, Marios Fournarakis, Yelysei Bondarenko, and Tijmen Blankevoort. Overcoming oscillations in quantization-aware training. arXiv preprint arXiv:2203. 1086,
/008,, 2022. (Cited on
Renkun Ni, Hong-min Chu, Oscar Castañeda, Ping-yeh Chiang. Christoph Studer, and Tom Gold- stein. Wrapnet: Neural net inference with ultro-low-resolution arithmetic. arXiv preprint w.Xiv:2007.13242. 2020. (Cited on
Charbel Sakr, Naigang Wang, Chia-Yu Chen, Jungwook Choi, Ankur Agrawal, Naresh Shanbhag, and Kailash Gopalakrishnan. Accumulation bit-width scaling for ultra-low precision training of deep networks. arXiv preprint arXiv:1901.06588 2019. (Cited on 3
Xiao Sun, Jungwook Choi, Chia-Yo Yu Chen, Naigang Wang, Swagath Venkataramani, Xiaodong Cui, Wei Zhang, Kailash Gopalakrishnan, et al. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. 2019. (Cited on
Xiao Sun, Naigang Wang, Chia-Ya Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi Viji Srinivasan, and Kailash Gopalakrishnan. Ultra-low precision 4-bit training of deep neural networks. Advances in Neural Information
Processing Systems, 53:1796-1807, 2020. (Cited on 22
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023. (Cited on 7
Mart van Baalen, Andrey Kuzmin, Suparna S Nair, Yuwei Ren, Eric Mahurin, Chirag Patel, Sundar Subramanian, Sanghyuk Lee, Markus Nagel, Joseph Soriaga, et al. Fp8 versus int8 for efficient deep learning inference. arXiv preprint arXiv:2303.
2023. (Cited on 16)
Naigang Wang, Jungwook Choi, Daniel Brand, Chia- Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. In Advances in neural information processing systems, pp. 7675-7684, 2018. (Cited 2
3
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the-art natural language processing, arXiv preprint 19100:111. 2019. (Cited on
B
14
Tianyi Zhang. Zhiqiu Lin, Guandao Yang, and Christopher De Sa. Qpytorch: A low-precision arithmetic simulation framework, 2019. (Cited on 6 13)
11

---

Published as a conference paper at ICLR 2024
A GENERAL MATRIX MULTIPLICATION: EXAMPLE
In section 2.4 we defined the FMA operation, and presented Eq. 3 as a general formula for all GEMM operation. It worth taking a moment to illustrate the connection between the known tensor operations and the formula.
For example, let us look at the simple case of matrix-muliilicationn (Y XWT X E Rddxdd W Rdaxd:). ). Here, if we wish to calculate the scalar = Yud, we can use the mapping: 2; = X Wy = Wi. In this case, all values of X and W were used exactly once in the calculation of Y. This is not always the case, however. In batch matrix multiplication, values of W will be used multiple times, paired with values of X of different batch dimension.
In convolution, the same values of W will be used to calculate every scalar in the same output channel, and the neuron in the input channel may be used to calculate a multiple values multiple output channel. This level of repetition will be, in part, what prevents us from using fine-grained STE methods on convolutional neural networks, in Sec.
B EFFECT OF QUANTIZED FMA ON ZERO-SHOT ACCURACY
To give the reader a sense the effect of low bit accumulators on deep neural networks, we include Tab. 8 where measure the zero-shot accuracy of different ResNet architectures, pretrained with full prec-sec,s,on, after replacing all FMA components with FMAq (as described in in
Mantissa Effect
Model
Baseline
MIOE5
M9E5
MSE5
M7E5
M6E5
ResNet18
69.75
69.50
68.95
66.70
57.09
20.49
ResNet34
73.31
73.17
72.68
70.46
60.07
17.19
ResNet50
76.12
75.95
75.57
73.70
64.94
19.48
Exponent Bias Effect (M7E4)
Model
b= 8
b= 9
b= 10
= 11
b= 12
Dacer prod
10,
12
R8
55.68
60.64
60.00
58.84
56.96
60.
ResNet34
50.80
63.30
63.88
62.46
59.90
63.65
ResNet50
26.41
64.25
68.69
67.57
66.12
68.49
Table 8: Zeroshot Accuracies for LBA-ResNets, with weights of pre-trained, full precision ResNets [%]
The accuracies presented in Tab. iillustrates well why M7E4 quantization was chosen: Increasing the mantissa below A bits would result a much lower zero-shot accuracy, too far for proper fine-tuning. Likewise, reducing the number of bits to E 4 already resulted lower accuracy due to overflow and underflow events, as indicated by the effect of the exponent bias. For example, the default exponent bias for E 4 is b 8, and using it for the accumulator in Resnet50 results in significant degradation in accuracy. A small increase to b = 9, increases both underflow and overflow thresholds by factor of and is sufficient for increasing the accuracy by almost 40%.
C EXPERIMENTS IMPLEMENTATION DETAILS
C. IMAGENET
Each of the ImageNet experiments were performed on a single server, containing 8 NVIDIA GPUs (RTX 2080 Ti, RTX A6000). We used total mini-batch size of 256, equally divided across the workers. For the training datasets, we used the standard RandomResizedCCopp and RandomHorizon. talFlip augmentations only. With no quantization, our model architecture was identical to the standard torchvision architecture, for all ImageNet models, while our custom GEMM kernels were used to override all forward GEMM operations (convolutions and matrix multiplications). For optimization, used the Adam optimizer, with the hyperparameters 3 (0.9. 0.999), e 10-8, Dropout was not used. As mentioned in the main text, we used cosine scheduling. the parameters of which depend on the phase in which it was used. We used 10 epochs in the 1-stage compared
10
12

---

Published as a conference paper at ICLR 2024
with 5 epochs for the dual-stage to support our claims that the gaps between the methods (where they exist) are not simply a result of better hyperparameters. The epoch count was initially chosen due to time-constraints, and was kept since the benefit of running more epochs was small.
For W/A quantization, we used the qtorch (Zhang et al., 2019) library, which provides reliable quantization functions. Weights quantization was applied during every optimization step, while the activations were quantized using dedicated modules, preceding all convolutions, except the first one, and the downsample convolutions. The input of the final fully-connected layer was not quantized as well, in accordance with prior works. The quantization function we applied used stochastic-rounding (which is not considered expensive in this case, as we are not implementing the FMAq internals and the number of quantized components is significantly lower). No other components (e.g. Gradients or Momentum) were quantized since our solution only aimed at inference.
In addition to hyperparameters used in this experiment, we have also ran set of experiments using fixed-leaming rates (no cosine annealing). In the other set, we tested a few initial learning rate values (IE-7, 3E-8, 1E-8) for few epochs, and used enough epochs to reach convergence (for training accuracy/loss). The results in this regime were slightly better than the results published in the paper: For 8bit quantized ResNets with 4ME3, we achieved 69.6% for Resnet18, 73.48% for ResNet34 and 76.35% for ResNet50. However, this required more epochs and finer-tuned hyperparameters (different models used different learning rates). In the paper, used the regime with cosine annealing since it was more robust to hyperparameter changes.
C.2 SQUAD
For the SQUAD fine-tuning experiment, we use NVIDIA GPUs (RTX 2080 Ti, RTX A6000). We used the SQUAD training script of the transformers library (Wolf et al. 2019), while using our custom, LBA-model. In our LBA model, all fully connected layers and matrix multiplication operations were modified to use LBA GEMM operations during forward propagation, with the exception of the final fully connected layer (qa-ourputs). For pre-trained models, we used either port-bas--ncaasee for Bert or mrujivall/bert-snng
for Bert-small. For optimization, the Adam optimizer, with 1000 warmup steps to a learning rate of 3- 10-5, from which we applied a cosine annealing scheduler. The batch size was configured to be 8. Our run was set for 20 epochs, but we applied early stopping once the model performance reached its peak (usually after 3 5 epochs).
C.3 MNIST
For each experiment with the MNIST setting, we used a single RTX 2080 Ti GPU with a mini- batch size of 16. Our neural network consisted of 4 fully connected layers (with LBA), and ReLU activations, with all hidden layers being 1024 neurons wide. Outside of the accumulator, all data types were with full precision. Dropout wasn't 't used (although it was shown to benefit the results slightly). and no data augmentation wasn't 't used during training. For optimization, we used Adam optimizer, with an initial learning rate of 10-3 3, with the hyper--paran paramearattt:s::: / = (0.9, 0.999), c 10-5 8 x = 0.0, and StepLR scheduler (y = 0.95). We used 100 epochs per experiment, which was usually much more than needed for convergence or divergence.
To test the STE, we replaced the default linear operations with our custom implementation, this time also implementing a custom backward operation. During backpropagation, we used new, cuda kernel that imitated the (deterministic) operation of the original GEMM operation (using the available weights and activations), but outputted a binary tensor, that indicated the value of all STEs involved in the operation (the type of STE was configurable). For recursive implementation, we modified the tensor ad-hoc to account for the recursive nature of the STE (although less efficient than the optimal implementation). After running the kernel, we used the output to adjust the computation of the weights/ neural gradients as described in section D In this experiment, we used a fixed exponent bias of 5, which was shown to perform the best among all values in its vicinity.
C.4 MASKED LANGUAGE MODELLING
Each of the Masked Language Modelling (MLM) experiments was performed single server, containing 8 NVIDIA GPUs (RTX 2080 Ti, RTX A6000, or A100). Our tests were run over the oscar: unshullled-originalagf dataset, with single tokenizer trained over the same dataset (vocabulary
13

---

Published as a conference paper at ICLR 2024
size of 1000). The dataset was chosen due to its moderate size (200K' rows), being difficult enough to show gaps in convergence while allowing us to perform meaningful optimizations with simulation kernels in moderate time. For the transformer, of 512, hidden layers, 4 attention heads, and maximum position embedding of 1024 (All other parameters were according transformers library defaults). We used the available Huggingface infrastructure (Wolf et al., (2019) to train/ evaluate the model, with Adam optimizer, an initial learning rate for 10- drop-on-platear scheduler (evaluating every 250 step. 7 = 0.1), and global mini- batch size of 64. In practice, the drop-on- on-plateau scheduling was only applied to 'failed' runs, to give them another shot for optimization, with no success (They did not converge, even well passed the 15 specified epochs). When the number of exponent bits was set to E 3, we used fixed exponent bias of for the product and accumulator.
used the Bert architecture, with the hidden size
D GRADIENT ESTIMATION FOR LBAS
Following the general equation for GEMM operation (Eq. 33). the operation can be expressed, using the recursive expression:
So
S = FMA (za, wi, St):
Sx
(8)
In this example, we add the values of the product to the intermediate accumulator sum, (S), in sequential manner. Different orderings of FMA operation are possible and can have an effect on the output (i.e., floating point addition is not commutative as 'ideal' addition, due to swamping).
Let us write the recursive expression in Eq. 8 explicitly
Sy = FMAq (z,-1, W;-1, FMAq (I:-2, u2 FMAq (...FMAq( wp. 0)))); 3 S N (9)
Our goal in this section is to find a good estimate for the derivative for 0y" De, ow,
and
0y
D.1 RECURSIVE STES
The first, and most obvious method to estimate the derivative is by using the common STE (Eq, (5)). Per our definition, FMAq contains two quantization functions. Our main concern, however, for the accumuuuullloor quantization, Quee, and our first attempt will be to quantize it directly using general STE function (STE:. R IR). By doing so,
we get the gradients:
dy dSy
d. wy
dy
dy
STE (I,, Ws, S?)
(10)
dx;
24 dw, dS
which, when expanded upon, will give us:
dy dsp
STE (FL, wk, SQ) k=+t1
II
(11)
Eq. reveals an additional, possible issue for using standard STES for the accumulation process. Usually, when applied over an overflowed activation neuron, the STE in equation Eq. (5) will set the corresponding neural gradient to zero. If the same STE is used for the accumulator's quantization function, per Eq. (II). each overflow event will eliminate the neural gradients of all previously accumulated product-pairs. Still, this approach may help calculate more accurate gradient, provided that conditions in which the STE returns ( are not commonly met. We will denote the approach in Eq. (T) as Recursive.
To perform the recursive correction to the gradient, all we really need to know is the last index (if any), in which the accumulator was overflown. While this may be more complex in cases where the values are added in non-sequential ordering, this is still feasible calculation to perform, with modest-sizee sized output. Computationally, calculating the underflow indexes is no more difficult as a task than computing the original GEMM operation, and this calculation (as well the calculation that will be presented the following section), can be done during backpropagation, to avoid using essential memory for long term.
14

---

Published as a conference paper at ICLR 2024
D.2
IMMEDIATE STES
To avoid setting too many gradients to zero, we suggest an alternative approach. First, we will re-write Eq. 99
as:
S Cgzgg000 + 012101 W2222yW2 + 14-1F1--1W-11
(12)
where
0j FMAq (Ty, wi, S S?
(13)
T2W4
If x; or w; = O, we will define a;; for simplicity. Recall S here is the value of the "quantized" accumulator in step i (Eq. ( From its definition, Og is the correction product z; W/, to account for the FMA quantization error. Our choice to express the correction this way is based on the assumption that for most steps, |S > |x; wy/- This is true, because S is, approximately, the accumulated sum of many such products. During a floating point addition, the bits of the lesser value will be the first to be swamped out, and thus entangle the quantization error with this component.
make to the
Moving on to the gradients, we are interested in the gradients of the operation inputs, dy and dy dx;
dwy""
We can use the chain rule to get:
dy"
day dx;
w; +
(14)
dr;
kmi+1
The exact expression we got for the gradient remains complex, since dos + 0, for k: >> i. Nevertheless, dz.
moving forward, we will take the approximation that vk. dos = 0. i.e., we neglect the cumulative effect that any individual scalar value has on the quantization correction we make in the following steps of the GEMM operation. We then get:
dx;
dy
dy dw;
= W;O
I,0G
(15)
dx:
Eq. [5) suggests a correction we can make to the standard backpropagation operation, which we will denote an Immediate STE. However, make any use of this correction, must first have the values (is. terms of computation, calculating a quite similar to performing the original GEMM operation. In terms of memory, however, a, scales with the number of overall FMA operations. This feasible in the case of fully connected operations, but not for GEMM operations that include a large amount of shared weights. To make sure the evaluation of a; by itself does not overburden the memory, is possible to quantize the values of o; By doing so, we get the equation:
1 dy 1 dyd Wi dx; I; dw;
FMAq (Iz W/, ST) S
Q
(16)
TTW; E1
where is a small constant, added with flexible sign to prevent invalid denominator: In our experi- ments, we have observed that the quantization of a; does not harm the quality of the optimization process, and proceeded to binarize the value. The result is that we ended up suggesting an alternative STE to the one presented in Eq. 5 which is designed address overflow only. We denote the new STE as DIFF:
{ {0
S7) C2 kew.ttes
DIFF (x;, Wi, W;,
Else
(17)
2ped (I,w;) + S < RoF
OF (Ii, Wi, ST)
Else
The DIFF STE is similar to the common Overflow STE, but is tumed to detect cases of full-swamping and cases of product underflow in addition cases of overflow. In experiments, we tested the immediate approach with both STEs. One unique advantage the DIFF STE, is that is agnostic to the specific implementation of the FMAq component. Therefore, the DIFF STE remains relevant in
15

---

Published as a conference paper at ICLR 2024
the general cases where the FMAq operation has unspecified, black-box behavior, common for hardware modules.
Our derivation in this section done in respect to the sequential ordering of FMAq operations, which is not commonplace hardware accelerators that try to achieve large degree parallelism. A more typical case, presumably common for Psstoliearrray-ike accelerators, is the chunk-based accumulation, where the accumulation is performed in two hierarchies, as seen in Fig. [ In our experiments, all simulated GEMM operations and gradient estimation used a chunk size of 16, which
means that an operation with an accumulation width of N is initiated with T6 parallel operations (i.e.. the first hierarchy). before aggregating the results (i.e., the second hierarchy). For example, in the case of recursive STE, every detection of OF or DIFF during re-computation will result in "O' in all preceding operations, just as we saw for sequential accumulation. The only difference for parallel accumulation is that the hierarchy tree can expand in two directions (Like (right), with all the arrows reversed).
N
E HARDWARE ANALYSIS
In this section, we try to give estimate for the effect of incorporation of LBA models on hardware cost (area/ power), by estimating the number of gates needed to implement qLBA with different levels of quantization.
Following an existing design of FMAq component (van Baalen et al. (2023) 2.23).figurt figure 2b), we adjusted the design for the case of FMAq with m/e quantization of weights and activations and M/E quantization of intermediate values (product, accumulator). and suggested the following gate counts, as seen in table The gate counts are all based on the gate count assumptions listed in van Baalen et al. (2023) appendix BJ, and common block designs.
FMA Components breakdown
Gate Count
Exponent Adder
(c 1) CFA + CHA
Exponent Differ
(min E. 1) 1). CFA + CHA (1 + l 1 El)
Exponent Max
E. CMUX
Mantissa MUL
(m + +3)² 2 CAND + +(m + +=) + (m + 2) -HA
Sort Exponent
(M 1M-UX
1st Shift (M + 1 >> + F)
(F 1) loogg(mmxx) CMUX
Mantissa Adder (F, + F)
(M) OFA + CHA
Leading Zero Detector
F(CAND + CORR (kma '*OOR
2nd Shift (F>>>)) k M + 1)
(M + 1) log, (kimax) ) CMUX kmax (OFA CAND)
Exponent Rebase
(E 1). UFA CHa
Final Incrementor
(M + +1CHA
Table FMA components gate-count breakdown. For the gate count, we used CAND CoR 1 for for half and full adder.
the standard gates AND2/OR2, CMUX for MUX2, and CHA = 3, CFA
We do not include Flip-Flops in gate count. For the value of F (Canvas bits, after shifting to fixed-point representation), we used 2M + 1, the maximum bit width in which two 2's complementary values with M + 1 bits can interact during addition. For kimax (the maximum shift distance), we used min (log2 (F), E), as the magnitude of the shift is bounded by both the number of exponent bits and the size of the canvas F.
Weights/ Acititts/ctivatiions bits
FMAq Bits
Canvas
Gates
M
E
F
loga (kimax)
Count
Ratio [%]
8
47
6
2208
100
10
5
21
5
1082
49
4
3
4
15
4
808
37
Table 10: Gate estimation for Quantized FMA
We summarize our numerical results in table I0 Our results show that for 8bit activations and weights (at FP format M4E3), used in the paper, any half-precision FMAq that follows our quantization
16

---

Published as a conference paper at ICLR 2024
scheme is expected to reduce the number of gates by about 50% from the gate count of full-precision accumulators. Reducing the accumulator to M7E4, as was done in the paper, will cut the number of gates by additional 25%, compared to half precision.
We conclude our 12bit accumulators will reduce the gate count by 63% compared to 32 bit FP accumulation. We note that the 16-bit accumulation gate count in analysis is not directly applicable to previous works that used 16-bits accumulators- This is because in InSunn et al.) (2019), only the output of the accumulator was quantized with no explicit quantization of the internals. Presumably, the majority of the gain there was achieved by the reduction of communication bandwidth between FMAq components, which does not affect the gate count in this analysis.
F WHY IS IT HARD TO TRAIN WITH UNDERFLOW?
Our claim that underflow cause unique problems during SGD optimization based, first and foremost, on empirical observations (see: Tab. 2 Tab. 3). In addition, we suggest several explanations to why this problem may oceur when underflow introduced during training, despite having a small effect on the loss landscape.
Consider a neural network, where specific scalar weight w is connected its scalar activation as input, and their product is = zw. Suppose w is small enough so that 2 consistently results in product underflow, i.e. proa (z) = In this case, during forward propagation, the value of x has little to no direct effect on the output neuron to which is connected. Therefore, it is reasonable to (where L is the loss) will be uncorrelated with I. Consequently, the gradient update of the weight w will be Aw &ggr, with the expected value E[Aw] x E [x]] = E g [z)) Based on previous quantization literature
assume that the computed neural gradient g 2z
dC
literatureBBaneer [Banner et al. (2018), we have approximately E[g] O, and so E[Aw] = O. Therefore, any sufficiently small weight wo will become "stuck", " so that its z cannot escape underflow for long time.
The issue is excavated by the ratio between updates magnitude, and the magnitude weight has be updated to surpass the underflow threshold. In a fully-trained model, the gradients are expected to be de 0. When transitioning to an LBA-model, we make sure to avoid significant changes to the loss aw
landscape (as indicated by the zero-shot accuracy). As a result, we can expect the relative change in gradient to remain small, A in dur |w). (Otherwise, the loss landscape would change rapidly during SGD, and we can no longer consider the process as fine-tuning).
When dealing with quantized values, it always possible that gradient step will be too small to change the value. (This is the main motivation behind stochastic rounding, which is not suitable for our case). For example, for floating point quantization without underflow/overrlow, the gradient step must be approximately A.. = |n de
dz 2 2-M |w| for the quantized value of w to 'jump' quantization level. In this case, اسسل ~ |w| means that the ability of all weights to change during fine-tuning only depends M, and the learning rate.
In the case of underflow, however, values must surpass an absolute threshold (2 (2-b), for the gradient step have any effect. Consequently, under previous assumptions, any small enough value subjected to floating point quantization is expected to receive updates which are too small to result a state-change. This is what referred to when mentioning values being 'stuck' and 'escaping'.
In LBA networks, the quantization is performed over intermediate values (products and partial accumulation values). These value do not get the explicit updates, but they will still get implicit updates by passing their respective neural gradient backwards.
17


Reviews:
[{"review_id": "JixAcFwgGm", "reviewer": "Reviewer_L15P", "summary": "The paper introduces an innovative approach to fine-tune the process of quantized accumulation by disregarding underflow effects, thereby simplifying the rounding process to a simple floor operation during low-precision accumulation. Employing this technique, deep learning inference with 12-bit floating-point accumulation maintains the same as FP32 accumulation on the ImageNet dataset. Additionally, the paper suggests a methodology for quantized backpropagation across the entire accumulation, demonstrating promising results with 8-bit floating-point accumulation on smaller datasets, such as MNIST.", "questions": "Can the distribution of the accumulations (which might follow a normal or other distribution) affect the performance of the quantization approach?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1- The paper is well-written and well-organized\n\n2- The efficacy of the methods presented in the paper is substantiated through experimental results using BERT models on the SQuAD benchmark and ResNet models on ImageNet. \n\n3- The backpropagation through the entire quantized accumulation is unique and has not been studied before.", "weaknesses": "1- It is recommended that the paper include a comparison of the computational complexity of MAC operations between the proposed method and the previous works [1,2,3]. Additionally, the author suggested discussing the quantization overhead associated with the new approach in comparison to [1,2,3].\n\n[1] Sun, Xiao, et al. \"Hybrid 8-bit floating point (HFP8) training and inference for deep neural networks.\" Advances in neural information processing systems 32 (2019).\n\n[2] Sun, Xiao, et al. \"Ultra-low precision 4-bit training of deep neural networks.\" Advances in Neural Information Processing Systems 33 (2020): 1796-1807.\n\n[3] Chmiel, Brian, et al. \"Logarithmic unbiased quantization: Simple 4-bit training in deep learning.\" arXiv preprint arXiv:2112.10769 (2021).\n\n2- The impact of chunk size requires further exploration to determine if reducing the chunk size also diminishes the bit size.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1PDKKaBXyy", "reviewer": "Reviewer_AH1V", "summary": "In addition to low-precision inputs to a matmul, this paper proposes to use low-bitwidth accumulators (LBA) to compute the dot product. Experiments show that 12-bit LBA in the forward pass is promising for both ResNet and BERT, requiring only several epochs of finetuing. LBA for training is more difficult.", "questions": "Questions are included in the weakness section.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Reducing the accumulator bitwidth is a practical way of further reducing the cost of a low-precision matmul.\n\n2. The experimental design in the main experiments in Section 3.1 is straightforward and seems to be easy to reproduce (if the CUDA kernel is open sourced).", "weaknesses": "1. The paper needs to provide more context on related works. For example, what is the key difference between the proposed method and the prior work Wrapnet? Is it training vs. non-training?\n\n2. There is a lack of evaluation or estimation on the hardware benefits of the proposed method. What will be the gate count/computational energy/latency improvement if using 12-bit accumulators compared to FP16/BF16 accumulators?\n\n3. Low-bitwidth accumulator under the context of integer quantization is not explored in the main experiments. Integer quantization is mentioned in Section 2.2 when introducing fixed-point quantization, but it seems to be disconnected from the rest of the evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0dOyeKLTDZ", "reviewer": "Reviewer_YwrL", "summary": "This paper explores how to reduce computing resource by quantize and reducing the bit width of neural network computing accumulators. Unlike most other works that mainly focus on the multiplier part of low bit width, this paper believes that the overhead of the accumulator cannot be ignored. Therefore, through optimization on floating-point representation, the ResNet series models can achieve fine-tuning and inference for the first time with a lower 12 bit width accumulator, without significant degradation in accuracy. This paper also explores the training methods for lower bit width scenarios, including adjusting the design selection of the backpropagation straight through estimator.", "questions": "1. In section 3.1, a two-staged fine-tuning is proposed, what are the references for selecting hyperparameters here(e.g, 10 epochs, learning rate)?\n2. In section 4, one method is using STE as recursive way on FMA. This seems to require all FMA operations to be unfolded in sequence. As far as I know, the FMA gradient of deep learning training seems to be treated equally, otherwise it will seriously affect efficiency and be unreality. This is because GEMM is a highly optimized parallel operation, and additional branches are not suitable for use here.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Quantization optimization of accumulators is an interesting viewpoint, as it has not been given sufficient attention.\n\n2. The method proposed in this paper has certain reference significance for the design of floating-point accumulators in deep learning accelerators.\n\n3. The discuss of STE used in section 4 seems to be a new and original study, can achieve better results in backpropagation of accumulated errors.", "weaknesses": "1. The method proposed in this paper is difficult to be applied to accelerate real-world low bit width neural networks. Although this paper claims that an accumulator with a low bit width of 12 bits FP can be implemented with cheaper hardware, standard hardware typically only provides floating-point bit widths of 8 bits, 16 bits, and 32 bits. Implementing such acceleration requires specialized hardware design, which requires more collaborative design and additional costs. The optimization of low bit floating-point accumulators under the same cost, as well as the accuracy that can be achieved by fixed-point quantization models with the same hardware cost, remains to be discussed.\n\n2. This paper lacks accurate evaluation and theoretical analysis of the error and accuracy requirements of floating-point models. Although Table 1 in Paper categorizes the errors in several cases of floating-point quantization, and Figure 2 shows several cases of errors, it is still not possible to quantitatively evaluate the impact of low bit width floating-point accumulation on model inference errors. It is difficult to make people believe with certainty the scalability and reliability of this scheme. Also, in the last paragraph before section 3.1, the author chooses b_acc=b_prod-1/2 log_2(chunk size) as an offset does not seem to guarantee that overflow will not occur under any condition. Therefore, I think that more discussion and theoretical analysis are needed regarding parameter selection and error evaluation.\n\n3. The content of this paper only includes an evaluation of the model accuracy and does not discuss or analyze the actual cost. For example, regarding the estimation of multiplier and accumulators’ power and silicon area, how much performance improvement or resource savings can be achieved through the optimization proposed in this paper. It is interesting to discuss the benefits of these optimizations in the context of the additional accuracy loss and design complexity required by the methods presented in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UN3uHOOa1g", "reviewer": "Reviewer_AQt3", "summary": "This paper studies quantization errors occurring in the accumulation phase of dot products. While many works have studied representation quantization in the past, the problem of accumulation rounding has largely been overlooked. Yet, others (Higham'83 & Sakr'19) who have studied this problem, have often focused on the problem of swamping, which causes an underflow in one of the two summands. In contrast, this work looks at the problem of overflow that occurs during large summations, when employing relatively low precision accumulators, e.g., FP16. Modeling accumulation as a sum of iid random variables, and invoking the central limit theorem, this paper formulates a prediction method for when such overflows can occur, and as a remedy, applies a scaling factor the the chunked summation to try and suppress the damage of said overflows. Extensive empirical results on several benchmarks are provided.", "questions": "Please address the above questions. Furthermore, I am also curious if we can analyze if overflows can be allowed in a controlled manner such as to lower complexity further but maintain accuracy. Indeed, overflowing is essentially related to magnitude clipping. Recent works [1] have shown that clipping, if done properly, can significant improve the quality of quantization. Is this something we can investigate for this work?\n\n[1] Sakr, Charbel, et al. \"Optimal clipping and magnitude-aware differentiation for improved quantization-aware training.\" International Conference on Machine Learning. PMLR, 2022.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-- Good problem to tackle, rounding effects in accumulators are largely overlooked in a community that puts a lot of emphasis on using quantization to lower the cost of implementation.\n\n-- Good presentation. Following concepts related to the accumulation occurring in a GEMM is often hard, due to the complexities of tensor cores and similar hardware used for DL inference and training. But the others do a good job pinpointing the problem.\n\n-- Solid empirical results on a diverse set of benchmarks such as vision and language models.", "weaknesses": "-- The proposed solution described in Section 3 is too qualitative. For instance, the underflow region is described as a \"hard\" region to \"escape\" from. Can the \"escape\" and \"hardness\" be presented in a mathematical manner? Similar for later description of parameters being \"stuck\" and so on.\n\n-- The method only applies to feedforward, what about accumulations occurring in the GEMMs of back-propagation?\n\n-- Benchmarks employed are diverse, but relatively simple (ResNets and Berts). can the empirical results be augmented with Transformer models, such as GPTs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)