Schrödinger's FP: Training Neural Networks with Dynamic Floating-Point Containers
Authors: Milos Nikolic
Enrique Torres
Jiahui Wang
Ali Hadi Zadeh
Mostafa Mahmoud
Ameer Abdelhadi
Andreas Moshovos
Venue: Submitted to ICLR 2023 (2023)
Keywords: Quantization
Hardware Acceleration
Deep Learning

Abstract: We introduce a software-hardware co-design approach to reduce memory traffic and footprint during training with BFloat16 or FP32, in order to boost energy efficiency and execution time performance. Our methods dynamically adjust the size and format of the floating-point containers used to store activations and weights during training. The different value distributions lead us to different approaches for exponents and mantissas. Gecko exploits the favourable exponent distribution with a lossless delta encoding approach to reduce the total exponent footprint by up to 58% in comparison to the FP32 baseline. To contend with the noisy mantissa distributions, we present two lossy methods to eliminate as many as possible least significant bits without affecting accuracy. Quantum Mantissa is a machine learning mantissa compression method that taps onto the gradient descent algorithm to learn the minimal mantissa bitlengths on a per-layer granularity, and obtain up to 92% reduction in total mantissa footprint. Alternatively, BitChop observes changes in the loss function during training to adjust mantissa bitlength network-wide, yielding a reduction of 81% in footprint. Schrödinger's FP implements hardware encoders/decoders that, guided by Gecko/Quantum Mantissa or Gecko/BitChop, transparently encode/decode values when transferring to/from off-chip memory, boosting energy efficiency and reducing execution time.

Under review as a conference paper at ICLR 2023
SCHRÖDINGER'S FP: TRAINING NEURAL NETWORKS WITH DYNAMIC FLOATING-POINT CONTAINERS
A Anonymous authors Paper under doublc-blind review
ABSTRACT
We introduce a software-hardware co-design approach to reduce memory traffic and footprint during training with 16 or FP32, in order to boost energy efficiency and execution time performance. Our methods dynamically adjust the size and format of the floating-por point containers used to store activations and weights during training. The different value distributions lead to different approaches for exponents and mantissas. Gecko exploits the favourable exponent distribution with lossless delta encoding approach reduce the total exponent footprint by up to 58% in comparison to the FP32 baseline. To contend with the noisy mantissa distributions, we present two lossy methods to eliminate as many possible least significant bits without affecting accuracy. Quantum Mantissa is a machine learning mantissa compression method that taps onto the gradient descent algorithm to learn the minimal mantissa bitlengths on per-layer granularity, and obtain up to 92% reduction in total mantissa footprint. Alternatively, BirChop observes changes in the loss function during training to adjust mantissa bitlength network- wide. yielding a reduction of 81% in footprint. Schrödinger's FP implements hardware encoders/decooerrs that, guided by GeckolQuantun Mantissa or Gecko/BitChopp transparently encode/decode values when transferring to/from off-chip memory, boosting energy efficiency and reducing execution time.
INTRODUCTION
Training most state-of-the-ar neural networks has become an exascale class task /Venkataramani al. 2017 Amodei et al., 2018) requiring many graphics processors Vidia (017) or specialized
accelerators, e.g., (Jouppi et al. 2017 Hab 2019 Liao et al. 2019; Cer 2019). While training is
both computationally and data demanding, is the memory transfers to off-chip DRAM for stashing (i.e., saving and much later recovering) activation and weight tensors that dominate overall execution time and energy (Jain et al. 2018) (see Fig. IT. The per batch data volume easily surpasses on-chip memory capacities, necessitating off-chip DRAM accesses which are up to orders of magnitude slower and more energy expensive. It's no wonder that reducing this overhead has been receiving attention throughout the software/hardware stack.
Chen et al. (016) and /Zheng et al. (2020) recompute rather than stash activations, whereas micro- batching strives to keep activations on chip (Huang et al., 22018). Encoding methods target specific value patterns such as zeros (Rhu et al.) 2018)) or redundant spatial information (Evans et al. 22020), or exploit underlying properties of training for certain tensors, e.g., the outputs of ReLU or Pooling JJann et al. 2018). These lossless and lossy encodings use fewer bits for stashed tensor content to reduce tensor volume. This also boosts the effective capacity of each node's main memory, which further reduces traffic during distributed training. All aforementioned methods either shift significant costs to compute or target only some values and offer only limited relief.
The most direct way to reduce tensor volume is to use more compact datatype. Initially, with the goal to demonstrate that neural networks can tackle challenging problems, training relied on single precision 32b floating-point (FP32), which still remains the datatype of choice when achieving the best accuracy is the priority. Recently, we have seen some success in training with more compact datatypes such as half- balf-precision FP16, BF (Kalamkar al. 2019), dynamic floating -point (Das) let al. 2018), and flexpoint (Köster et al. 2017) and even with using combinations with other datatypes such fixed-point (Das et al. 2018 Micikevicius al. 2018 NVIDIA Drumond et al. 2018). IBM

---

Under review as a conference paper at ICLR 2023
PIZI
Activations in Off Chip Memory
Backward Pass
Figure 1: Training process and its memory transfers. Blue Activations that are typically saved to off-chip memory during forward pass and retrieved during backward pass, Red Weights that typically stored and loaded once from off-chip memory, Gray Updates and Gradients through mini-batching during the backward pass they can often fit on-chip
managed to push the datatype to 8b (Wang et al., 2018b) and 4b (Sun et al. 2020) extremes for some cases. As Moore's 's law and Dennard scaling for semiconductors have come to an end, using more efficient datatypes during training is getting wider attention even major hardware manufacturers are investigating how use floating point with different mantissa/exponent ratios according to perceived needs of tensors (Micikevicius al. 2022). These methods require careful trial- trial-andeerror investigation where, when, and which narrow datatypes to use. This is especially true because different tensors, tasks, architectures, or layers require different datatypes. Consequently, there is no guarantee of success. The methods require trial-and-erro full training runs as whether the choice of datatypes is viable only be evaluated post mortem. Moreover, since the datatypes statically chosen they offer no opportunity to amend the choice if accuracy suffers (e.g.. significant drop with deeper networks identified by IBM (Sun al. 2020))
Obviously, knowing in advance which compact datatypes to use during training would be the best. However, given that this goal still cludes us, our work asks whether we can harness the training process itself to automatically learn them. Ideally, such a method would automatically tailor datatypes meet the demands of each tensor, layer, and network. Furthermore, it could continuously adjust datatype selection training progresses, adapting to the changing needs. In addition to accelerating training, methods such as ours can further inform efforts for selecting more efficient datatypes for inference such as those by (Micikevicius et al. 2022) or (Sun et al., 2020).
similar idea has successfully targeted fixed-point inference by using reinforcement learning (Wang et al. 2018a), clever differentiable datatype definitions (Nikolić et al. 2020). architecture search (Wu ct al. 2018), and profiling (Nikolić et al. 2018), etc. However, all of these are too expensive for training and their overheads would overshadow the benefits of a more compact training datatype.
Given that floating point remains the datatype of choice, we focus on floating-point datatype selection. We explore the possibility to dynamically and continuously adjust the mantissa bitlength (fractional bits) and the container (overall bits) for floating-poo values (activations and/or weights) for stashed tensors, and to do transparently FP, a family of methods that dynamically adjust the floating-poing encoding and complement the aforementioned training acceleration methods. Our approach is end-to-end fully-automated, requiring no input, guessing, or advanced knowledge from the operator. Schrödinger's FP can be used to reduce memory overheads and boost computation throughput. In this work, we limit our attention to boosting energy efficiency and performance by using Schrödinger FP to transparently encode values they are being stashed off- memory, and decode them to their original format as they are being read back. This application can be used plug-in over any hardware without changing the existing on-chip memory hierarchy and compute units. Similarly, Schrödinger's 's will generally work in conjunction with methods that can improve accuracy for a preselected datatype, partition, distribute, or reschedule the training work to improve energy efficiency and performance.
additional burden to the Our solution Schrödinger's
Schrödinger's 's FP uses tailored approaches for the mantissa and exponent. It dynamically adjusts mantissa bitlengths in order to store and read fewer bits per number in off-chip memory. This work explores two such methods. The first, Quantum Mantissa, harnesses the training algorithm itself to learn on- on-the-fi fly the mantissa bitlengths that are needed per tensor/layer and continuously adapts those bitlengths per batch. Quantum Mantissa introduces single learning parameter per tensor and loss
2

---

Under review as a conference paper at ICLR 2023
function that includes the effects of the mantissa bitlength. Learning the bitlength incurs negligible overhead compared to saving from the resulting reduction in off-chip traffic. The Quantum Mantissa experiments show that: 1) it reduces the mantissa bitlengths considerably, and 2) the reductions are achieved fairly soon in the training process and remain stable till the end. However, the bitlengths vary per tensor and fluctuate throughout, capturing benefits that wouldn't be possible with a static network-wide choice of datatype.
Motivated by the success of Quantum Mantissa, we explore the second mantissa adjustment method, BitChop, which requires no additional loss function and parameters. BitChop interface only needs to be notified of the per-batch updates to the loss. Using an exponential moving average of these changes, BitChop adjusts mantissa bitlength for the whole network. As long as the network seems to be improving, BirChop will attempt to use a shorter mantissa; otherwise, it will increase The method proves effective, albeit with lower reductions compared to Quantum Mantissa. This expected since 1) Quantum Mantissa harnesses the training process to continuously learn the optimal bitlengths, and 2) Quantum Mantissa adjusts bitlengths per layer whereas BirChop uses wetwork-wide one.
Most of the exponents during training exhibit a heavily biased distribution (Awad et al. 2021). Accordingly, Schrödinger's FP uses value- based approach that stores exponents using only as many bits necessary to represent their magnitude and sign. Metadata encodes the number of bits used. To reduce the metadata overhead, Schrödinger's y FP encodes exponents in groups
To maximize benefits, we present hardware-assisted implementation of Schrödinger's 's FP. The inclusion of specialized hardware units is now commonplace among all hardware vendors as the method of choice for further improving compute performance. Appendix A presents efficient hardware (de)compressors that operate on groups of otherwise unmodified floating-point values, be it FP32 or B6 The compressors accept an external mantissa length signal and pack the group of values using the aforementioned compression methods for the mantissas and exponents. The decompressors expand such compressed blocks back into the original floating point format. We demonstrate that Schrödinger's FP greatly improves energy efficiency and execution time. Our compression methods also serve as motivation for pursuing, as future work, software-only implementation that would require low-level changes in the closed-source tensor operation kernels.
We highlight the following experimental findings:
Schrödinger's FP compression techniques find the necessary mantissa and exponent bitlengths to reduce overall memory footprint without noticcable loss of accuracy: our Quan- tum Mantissa- based method reduces the tested models down to 20.8% on average (range: 14.7% 24.9%) and our BirChop- based one to 24.6% on average (range: 19.4% 28.9%)
Schrödinger's 's FP compressoo/deeompresss exploit the reduced footprint to obtain 2.94x and 2.64 performance improvement for SFPQM and SFPBc (Schrödinger's 's FP with Quantum Mantissa or BitChop, see Section 2.1.4 for details), respectively. In fact, we hit a hard performance boundary, since all layers are shifted from memory bound to compute bound, which completely flips the workload paradigm.
* Crucially, Schrödinger's FP excels at squeezing out energy savings with on average, 3.38 and 2.96x better energy efficiency for SFP qM and SFP
ADJUSTING VALUE CONTAINERS DURING TRAINING
Generally, maintaining accuracy on most real-world tasks requires training with a floating -oint approach. Floating paiin formats comprise three segments: a mantissa, an exponent, and a sign bit. Mantissas and exponents are differently distributed, necessitating different approaches. The greatest challenge is compressing mantissas since they uniformly distributed across the domain, whereas compression exploits -on-uniformity. We will present two methods to compress mantissas, a machine learning approach (Section 2.1 and a hardware- design-inspired approach (Section 2.2) In contrast, exponents can be compressed with fairly simple hardware techniques (Section 2.3
We study Schrödinger's FP with ResNet18 and ResNet50 (He et al. 2015). and MobileNet
V3 (Howard et al. 2019) trained on ImageNet (Russakovsky et al. 20144) DLRM (Naumov et al.
2019) trained on Kaggle Criteo, BERT (Devlin al. 2018) finetuned on MRPC DDolan & Brockett
3

---

Under review as a conference paper at ICLR 2023
2005) and GTP-2 2 (Radford al. 2019) finetuned on Wikitext 2 Meeity al. 2016). For clarity, report detailed results with ResNet18 with BF throughout the paper, concluding with overall performance and energy efficiency measurements for all models.
2.1 MANTISSA: Quantum Manfissa
Quantum Mantissa involves procedures for both the forward and backward pass of training. We begin by defining conventional quantization scheme for integer mantíssa bitlengths the forward pass, and then expand it to the non-integer domain, and describe how this interpretation allows bitlengths to be learned using gradient descent. Subsequently, we introduce a parameterizable loss function, which enables Quantum Mantissa penalize larger bitlengths. We then briefly touch the compute overhead of our method and the plan for final selection mantissa bitlengths. Ultimately, demonstrate the benefits of Quantum Mantissa memory footprint during ImageNet training.
2.1.1 QUANTIZATION
The greatest challenge for learning bitlengths is that they represent discrete values with obvious differentiation. To overcome this, we define a quantization method on non-integer bitlengths. We start with integer quantization of the mantissa M with bits by zeroing out all but the top bits:
Q(M, n) = M (2" - 1) << (m n)
(I)
where Q(M, n) is the quantized mantissa with bitlength m is the maximum number of bits and A represents bitwise AND).
Throughout training, we represent the integer quantization as n). This scheme does not allow
the learning bitlengths with gradient descent due to its discontinuous and non-differentiable nature.
To expand the definition to real-valued n [n] + {n). the values used in inference during training are stochastically selected between the nearest two integers with probabilities {n} and 1 - {m}:
(Q [m]), Q(M, = 1Q(M, |n| + 1), with probability {n}
with probability (n)
where [n] and [n} are floor and fractional parts of n, respectively. The scheme can be, and in this work is, applied to activations and weights separately. Since the minimum bitlength per value is 0, n clipped at O. This presents reasonable extension of the meaning of bitlength in continuous space and allows for the loss to be differentiable with respect to bitlength.
During the forward pass, the formulae above applied both activations and weights. The quantized values are saved and used in the backward pass. During the backward pass, we use the straight-throus estimator (Bengio al. 2013) Hubara et al. 2016) to prevent propagating gradients that result from the discontinuity' 's discreteness; however, we use the quantized mantissas for all calculations. This efficient quantization during the forward pass reduces the footprint of the whole process.
2.1.2 LOSS FUNCTION
On top of finding the optimal weights, the modified loss function penalizes mantissa bitlengths by adding weighted average (with weights A, not to be confused with the model's weights) of the bits my required for mantissas of weights and activations. We define total loss L as:
L Ia + (A, x
where L1 is the original loss function, 7 is the regularization coefficient used for selecting how aggressive the quantization should be, X; is the weight corresponding the importance of the tth group of values (one per tensor), and n; is the bitlength of the activations or weights in that tensor.
This loss function parameters. Since our goal is to minimize the total footprint of a training run, we weigh each layer's tensors according to their memory footprint.
be used to target any quantifiable criteria by suitable selection of the A
4

---

Under review as a conference paper at ICLR 2023
Accrlary Proughout
Weightd everage footprin
Boseiin
Wrights Actlo38015
QM
(c)
Figure 2: Quantum Mantissa on ResNet 8/mmageNet: (a) Validation accuracy throughout training, (b) Weighted mantissa bitlengths with their spread throughout training, and (c) Mantissa bitlengths for each layer at the end of each epoch. Darker dots represent the latter epochs.
2.1.3 COMPUTATIONAL AND MEMORY OVERHEADS
Quantum Mantissa adds minimal computational and memory overhead to the forward and backward passes. In the forward pass, random numbers need to be created at chosen granularity determine the quantized values. Our experiments show that per tensor/layer is sufficient and is a negligible cost.
To update the bitlength parameters in the backward pass, need compute their gradients. These are a function of the weight values and gradients, which are calculated during the regular backward pass. The extra calculations for each bitlength are proportional to the number of values quantized to that bitlength. This overhead is negligible in comparison to the total number of computations. For our experiments, the overhead is less than 2% and 0.5% for MobileNet V3 and ResNet18, respectively.
On the memory side, the only extra parameters that need to be saved are the bitlengths, two floats per layer (bitlength for weights and activations), again negligible in comparison with the total footprint. All other values are consumed as they are produced without need for off-chip stashing.
2.1.4 BITLENGTII SELECTION
Quantum Mantissa will produce non-integer bitlengths and requires non-deterministic inference. We prefer the deployed network not to have this requirement. For this reason, we round up the bitlengths and fix them for some training time to fine-tune the network to this state. While our experiments show that bitlengths converge quickly and final ones can be determined within couple of epochs, avoiding the small overhead for most of the training, we delay this action so that bitlengths have the ability to increase if needed during training. Our experiments show that this is unnecessary for the models studied; however, the overhead so small that we leave on as a safety mechanism. We round up the bitlengths for the last 10 epochs let the network regain any accuracy that might have been lost due to Quantum Mantissa. Quantum Mantissa still reduces traffic during these epochs.
Evaluation: BitLengths and Accuracy We report measurements for per-layer weights and activa- tions quantized separately using a loss function weighted to minimize total memory footprint. We train 18 the ImageNet dataset over 90 epochs, with regularizer strength of 0. learning rate of 0.1, 0.01 and 0.001 respectively at epochs 0, 30, and 60 and weight decay of 0.0001.
Quantum Mantissa excels minimizing the memory footprint whilst not introducing accuracy loss. Figure shows that throughout training, Quantum Mantissa introduces minimal changes in validation accuracy. In the end, we converge solution within 0.4% of the FP32 baseline.
Figure 2b
shows how Quantum Manrissa quickly (within couple of epochs) reduces the required mantissas for activations and weights down
2 bits on average. Throughout training, the total cumulative memory footprint is reduced to 7.8% and 25.5% of the FP32 and BFloat 16 mantissa footprint, respectively. The figure further shows that there large spread across different layers, in- dicating that granular, per-layer, approach is the right choice maximize benefits. Via the weighted loss function, Quantum Mantissa generally targets the activation bitlengths more aggressively than the weights because the activations responsible for the majority of the memory footprint.
The spread of mantissa bitlengths across the network and time is shown in Figure 2c While most layers quickly settle at 1 or bits, there exceptions that require more (up to 4b). Consequently, network- scale datatype would have to use the largest one and leave a lot of the potential untapped. For ResNet18 8, the maximum bitlength is over 2x larger than the weighted average.
5

---

Under review as a conference paper at ICLR 2023
Aacuooy Proughhot
Awslage morksso bllengh
Bonelins 6eChop
resvett BPoottt -ecutt0- F232
Dhslieddor of mand kikggp in spodt soroft
(a)
(b)
(c)
Figure 3: BirChop on ResNet 8/ImageNet: (a) Validation accuracy throughout 16 training, (b) Average mantissa bitlengths per epoch throughout training, on BFloat16 and FP32, (c) Distribution of BitChop's 's mantissa bitlengths throughout the 5005 batches of epoch 45 over BFloat16.
2.2 MANTISSA: BitChop
While Quantum Mantissa leverages training itself to greatly reduce mantissa lengths, having method that does not require introducing an additional loss function and parameters is appealing. BirChop is a run-time, heuristic method to reduce the number of mantissa bits for the forward and backward passes. At a high-level, BirChop monitors how training progresses adjusting the mantissa length accordingly: as long as the training seems to be improving the network, BirChop will attempt to use a shorter mantissa; otherwise, will try to increase its bitlength. BitChop conceptually splits the training process into periods, where period is defined as processing N batches. BirChop adjusts the mantissa at the end of each period using information on network progress.
The ideal scenario for BirChop is one where past observation periods are good indicators of forth- coming behavior. Macroscopically, accuracy improves during training which appears to be a good fit. Microscopically. however, training is noisy process. Fortunately, training is a relatively long process based on "trial-and -nd---rrrr"" which may be forgiving for momentary lapses in judgement.
There are three major design decisions that impact how successful BitChop can be: 1) what infor- mation to use as a proxy for network progress, 2) how long the period should be, and 3) at what granularity to adjust mantissa lengths. The resulting method should strike balance between capturing as much as possible the opportunity to reduce bitlength, and avoiding over-clipping as this can hurt learning progress and ultimately the final accuracy that would be achieved.
We have experimented with several options and arrived at the following choices: 1) Using an exponential moving average of the loss as a proxy for network progress, and 2) using short period where N = 1, that is a single batch. Additionally, rather than attempting to adjust mantissas at the tensor/layer level, BirChop uses the same mantissa for the whole model. Specifically, to monitor network progress, BirChop uses the loss which is calculated per batch as part of the regular training process. While the loss improves over time, when observed over short periods of time, exhibits non-monotonic behavior which is sometimes erratic. To compensate for this volatility, BitChop uses an exponential moving average Mang which updates at the end each period:
Mavg = Mavg + a * (L, - Mang)
44)
where L; is the loss during the last period and a is an exponential decay factor which can be adjusted to assign more or less significance older loss values. This smooths the loss over time while giving importance to the most recent periods. At the end of each period i, BirChop adjusts the mantissa bitlength (unchanged, lower, or higher) by comparing L, with Mang within a dynamically updated threshold T.
Bitlengths and Accuracy We report BitChop's effect on activation footprint and accuracy during full training sessions of ResNet18 as before. Figure Ba shows that the network achieves the same validation accuracy as with the baseline training. For clarity, the figure shows results for BFloat 6 only (results with FP32 were similar and accuracy was unaffected). Throughout the training process, validation accuracy under BirChop exhibits more pronounced swings compared to the baseline and to Quantum Mantissa. However, in absolute terms, these swings are small.
Evaluation:
Figure 3b]shows that BirChop reduces mantissa bitlengths to 4 bits on average when used over BFloat 16 and to 12 bits on average when used over FP32. However, mantissa bitlengths vary per batch depending on the loss as illustrated in the histogram (Figure 3c) of the bitlengths used throughout sample epoch (epoch 45) for the BF run. This shows that the training process sometimes requires the entire range of Bíloat16 whereas other times it only requires 2 bits. All across the training process, Chop reduces the total mantissa footprint of the BF baseline 64.3%. Over

---

Under review as a conference paper at ICLR 2023
Fratior of oxperent Recretts
Weg
(a)
(b)
Figure 4: Gecko on ResNet R 8/ImageNet: (a) Cumulative distribution of exponent values. (b) Post- encoding cumulative distribution of exponent bitlength
FP32 Bho reduces mantissa footprint to 52.3%. While BirChop might miss potential bitlength reductions, it is non-intrusive and has virtually overhead.
2.3 EXPONENT: Gecko
The exponents of BFloat16 and FP32 are 8b biased integers. Except for few early batches, we find that during training, the exponent values exhibit heavily biased distribution centered around 128 which represents 0. This is illustrated in Figure SouureBajae which reports the exponent distribution throughout training of ResNet18 after epoch 10. We omit gradients which are even more biased as those can be kept on-chip. Taking advantage of the relatively small magnitude of most exponents, we adopt a variable length lossless exponent encoding. The encoding uses only as many bits as necessary to represent the specific exponent magnitude rather than using 8b irrespective of the value. Due to our variable-sized exponents, 3b metadata field specifies the number of bits used. Having a dedicated bitlength per value would negate any benefits or worse, become an overhead. To amortize this cost, multiple exponents share a common bitlength that is long enough to accommodate the largest one within the group. We further observe that, especially for weights, the values exhibit spatial correlation (values that are close by have similar magnitude). Encoding differences in value skews the distribution closer zero, benefiting
approach.
The specific encoding scheme Gecko used is as follows: Given a tensor, Gecko first groups the values in groups of 64 (padding needed) which it treats conceptually of 8 exponents is group which shares a common base exponent. The base exponent per column is the exponent that appears in the first row of incoming data. The base exponent stored in 8b. The remaining exponents are stored as deltas from the base exponent. The deltas are stored as [magnitude, sign] format and using a bitlength accommodate the highest magnitude among those per row. A leading 1 detector determines how many bits are needed. The bitlength is stored using 3b and the remaining exponents are stored using the bitlength chosen.
8x8 matrix. Every column
Evaluation: BitLength We measure how many bits are needed to encode the exponents using Gecko for the duration of training of R as described previously. As representative measurements, Figure reports the cumulative distributions of exponent bitlength for one batch across 1) all layers, and 2) for a single layer, separately for weights and activations. After delta encoding, almost 90% of the exponents are lower than 16, and 20% of the weight exponents and 40% of the activation exponents need only bit. Across the whole training process, the overall compression ratio for the weight exponents is 0.56 and 0.52 for the activation exponents. The ratio is calculated as (M CO/O where M the bits used by the per group bitlength fields, C' the bits used to encode the exponent magnitudes after compression, and O the bits used encode exponents in the original format.
3 EVALUATION - PUTTING ALL TOGETHER
We study the following two Schrüdinger's FP variants: Gecko with Quantum Mantissa (SFPQM) and Gecko with BitChop (SFPBc) which are combinations our exponent and mantissa compression methods and the interaction with GIST++ (Jain et al., 2018). GIST++ is slightly modified version of Gist that uses sparsity encoding only for those tensors where doing so reduces the footprint, avoiding the increase in traffic that would occur otherwise. For instance, this is useful for MobileNet V3, BERT and GPT-2 which do not use ReLU, and as a result, exhibit very little sparsity.
We perform full training for ResNet18, ResNet50 and MobileNet V3 Small on ImageNet, DLRM on Kaggle Criteo as well as finetuning BERT on MRCP and GPT-2 on Wikitext 2, using an RTX3090/24GB with PyTorch v0. We implement Quantum Mantissa by modifying the loss
7

---

Under review as a conference paper at ICLR 2023
Table 1: SFPBc, SFPQM. BF16: Val. Accuracy/Perplexity and total memory reduction vs. FP32.
FP32
BF16
SPPqu
SEPBC Foetprint
Network
Task
Metrie
Score
Footprint
Footprint
Score
Foutprint
Score
ResNet18
Classification
Accuracy
69.94
100%
50%
69.54
14.7%
69.95
23.7%
ResNet50
Classification
Accuracy
76.06
100%
50%
75.58
20.6%
75.72
21.7%
MobileNet V38
Classification
Accuracy
65.60
100%
50%
65.26
24.9%
65.21
27.2%
BERT
Text classdication
Accuracy
84.56
100%
50%
84,31
17.9%
84.42
19.4%
GPT-2
Language Modeling
Perplexity
100%
23.5%
20.95
28.9%
DLRM
Recommendation
Accuracy
79.42
100%
50%
79.45
23. 1%
79.44
26.9%
function and adding the gradient calculations for the per tensor/layer parameters. We simulate BitChop in software. For both methods, we faithfully emulate mantissa bitlength arithmetic effects by truncating the mantissa bits at the boundary of each layer using PyTorch hooks and custom layers. We also measure Gecko's effects in software via Py PyTorch hooks. The above enhancements allow us to measure the effects our methods have on traffic and model accuracy.
3.1 MEMORY FOOTPRINT REDUCTION
First we report activation and weight footprint reduction ResNet18. Table [Sooss the cumulative total memory reduction and validation accuracies in comparison with FP32 and BFloat baselines. Combined, our compression techniques excel at reducing footprint, with little affect on accuracy.
Figure 5a shows the relative footprint of each part of the datatype with SFPQw in comparison with the FP32 and 6 baseline. Even though our methods are very effective at reducing the weight footprint (91% for mantissas and 54% for exponents), this effect is negligible in the grand scheme of things due the fact that weights are very small part of all three footprints. For the same reason, the reductions in activation footprint (92% for mantissas, 63% for exponents and 98% for sign) have far greater effect. Because of the effectiveness of Quantum Mantissa, the mantissas are reduced from the top contributor in FP32 (70%), to a minor contributor (38%). While exponents are significantly reduced too, they start to dominate with 59% of the footprint in comparison with FP32 at 24%. Similar conclusions are reached when comparing with B except for the fact that mantissas and exponents start with similar footprint.
SFPQM:
SFPBc: Figure also shows the relative footprint of the datatype components under SFPBC when compared to the FP32 and Bfloat baselines. While BirChop does reduce mantissa bitlength for the network's weights, this does not have great effect in the total memory footprint reduction due to the small size of weights when compared to activations. Although mantissa weight footprint is not reduced, weight exponent footprint is by 56%. This is why the focus the activations' mantissa bitlengths yields significant total memory footprint reduction when compared to FP32 (mantissa footprint is reduced by 81%. exponent footprint by 63% and sign by 98% in activations), and smaller but still significant reduction when compared with 6 (36% for mantissa and 63% for exponents). The reductions are not as great as with Quantuan Mantissa due to the network- wise limitation of the method and activation mantissas stay as the major contributor of footprint.
3.2 RELATIVE COMPRESSION AND COMPARISON WITH OTHER METHODS
Finally, we compare Schrödinger's FP compression against Bfloat GIST+ and JS, a simple sparse B6 zero-compression method in Figure (b] JS uses an extra bit per value to avoid storing zeros. We limit attention to activations since weights represent small fraction of the overall footprint and traffic. All methods benefit from using 16b. On ResNet18, JS and GIST++ benefit from the 30% reduction due to high sparsity induced by ReLu. GIST++ benefits even further because of its efficient compression of maximum pooling. BFPBC does even better just by finding a smaller datatype outperforming all of them, whereas S SFPq proves even better by adjusting the datatype per layer. However, S SFPBC BC and SFPQM only target the reduced datatype and there is an opportunity to build on it with the same ideas that power JS and GIST++. When combined, this further improves compression ratios to 10x and 8x for modified SFPQM and SFPBc.
MobileNet V3 Small poses a bigger challenge since it sparsely uses ReLu and uses max pooling. Accordingly, there is little potential for JS and GIST+ to exploit. SFPQM and SFPBc still get another 2x compression over Bfloat 6. JS, and GIST++. Application of ideas from JS and GIST++ to Schrödinger's 's FP compression offers only marginal gains. BERT and GPT-2 will exhibit similar results due not using ReLu.
8

---

Under review as a conference paper at ICLR 2023
1.50
+5p
Actraloo expoert
+5F 16
Actuddo
Wiigtt Eoower:
0.76
KAkr Sportty
+067 ++
+SFF-QP
0.50
1SFP .BC
<SP CM Sponity
5SP RCC S0a-pprrhy
0.25
xSP -o 665T
<SF 80 G6T
7912
0.00
PooMett
Mo
(a)
(b)
Figure 5: Schrödinger's FP: a) Relative training footprint ResNetl f 8/mageNet w/ FP32, BF16, SFPBC and SFP and b) Cumulative activation footprint w/ BFI6, sparsity only and GIST++.
Table 2: Performance and Energy Efficiency gains in comparison w/ FP32
Performance
Energy Efficiency
Network
Bfloat
SFPQN
SFPRC
Bfloat 16
SFPQM
SFPBC
ResNet18
1.53x
2.30x
2.09
2.00x
6.12x
4.22x
ResNet50
1.64x
2.53x
2.45x
1.98x
4.56x
4.48x
MobileNet V3S
1.72x
2.37x
2.14x
2.00x
3.95x
3.60x
BERT
1.88x
3.77x
3.54x
1.36x
1.78x
1.73x
GPT-2
2.13x
4.48%
3.42x
1.34x
1.70x
1.58x
DLRM
1.60x
2.19x
2.22
1.58x
2.14x
2.17x
3.3 PERFORMANCE AND ENERGY EFFICIENCY
We evaluate the execution time and energy efficiency by incorporating Gecko hardware units into hardware accelerator representative of faateoffthh--n designs. We model an accelerator with 8K units (each capable of performing 4 MACs per cycle), and 500MHz clock for a peak computer bandwidth of 16TFLOPS. We consider two baseline configurations using respectively FP32 and BFloat16, Both have 8 channels of LPDDR4-3200 DRAM memory and 32MB of on-chip buffers. Appendix B details the evaluation methodology. Area overhead of the compressor and decompressor is 0.677% % of accelerator area, excluding on-chip memory, which is negligible.
Table reports execution time improvements of Bfloat16, SFPQM, and SFPBc over the FP32 baseline. On average, SFPqM and SFPBC are 2.9x and 2.6x faster respectively, compared to 1.8x with Bfloat16. Both SFP and SFPBC significantly outperform both the FP32 baseline and Bfloat16. However, performance does not scale linearly even though SFP Qu and SFP BC reduce the memory footprint to 20.8% and 24.6% respectively: some layers that were previously memory bound during the training process become compute bound because of the reduction in memory footprint. This is the reason why even though Bfloat16 reduces the datatype to half, it does not achieve 2x speedup. This transition of most layers from memory bound to compute bound also affects the improvements in performance that SFPQM can offer, as even though it consistently achieves a lower footprint than SFPBC, this only offers an advantage for performance in the few layers that remain memory bound. SFP qM may offer bigger performance benefits if coupled with higher computational performance hardware. Regardless, while reduction in traffic may not yield a direct improvement in performance, it does improve energy efficiency.
Table 2 also shows energy efficiency improvement with 16foat16, SFPQM and SFPBc over the FP32 baseline. SFPQu and SFPBC excel at improving energy efficiency by significantly reducing DRAM traffic. Since the energy consumption of DRAM accesses greatly exceeds that of computation, and some layers are or become compute bound, SFPQM and SFPBC improve energy efficiency more than performance, achieving an average of 3.4x and 3.0x energy efficiency respectively. The dominance of DRAM access energy consumption over computation can also be seen in Bfloat 16, where the reduction to half the footprint, the use of 16-bit compute units, and the compute layers being no longer limiting factor gives 6 1.7x energy efficiency.
CONCLUSION
We explored methods that dynamically adapt the bitlengths and containers used for floating point values during training. The different distributions of the exponents and mantissas led us to tailored approaches for each. We target the largest contributors to off-chip traffic during training for both activations and weights. There are several directions for improvements and further exploration including expanding the methods to also target the gradients and refining the underlying policies they to adapt mantissa lengths. Regardless, this work has demonstrated that the methods effective. The key advantages of our methods are: 1) they are dynamic and adaptive, 2) they do not modify the training algorithm, and 3) they take advantage of value content for the exponents.
9

---

Under review as a conference paper at ICLR 2023
5 REPRODUCIBILITY STATEMENT
We will release our full code with all the necessary instructions on how to re-run our experiments by the camera-ready deadline the latest.
REFERENCES
Cerebras CSI. In https//www..erbbrrsr////oddddo Cerebras, 2019.
Gaudi training platform white paper. In Souud--rrining-Plajje - Habana, 2019.
Dario Amodei, Danny Hernadez, Girish Sastry, Jack Clark, Greg Brockman, and llya Sutskever. Open Al Blog, May 2018. URL https://oppeaa
Omar Mohamed Awad, Mostafa Mahmoud, Isak Edo, Ali Hadi Zadeh, Ciaran Bannon, Anand Jayara- jan, Gennady Pekhimenko, and Andreas Moshovos, Fpraker: A processing element for accelerating neural network training. In MICRO '21: 54th Annual IEEE/ACM International Symposium on Microarchittettree Virtual Event, Greece, October 18-22, 2021, pp. 857-869, ACM, 2021. doi: 10.1 10.1144534667522448811 URL http:: org/ 10. 1111114444666 3480106
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 2013.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. CoRR, 0bss/66040661 74, 2016.
Dipankar Das, Naveen Mellempudi, Dheevatsa Mudigere, Dhiraj D. Kalamkar, Sasikanth Avancha, Kunal Banerjee, Srinivas Sridharan, Karthik Vaidyanathan, Bharat Kaul, Evangelos Georganas, Alexander Heinecke, Pradeep Dubey, Jesús Corbal, Nikita Shustrov. Roman Dubtsov, Evarist Fomenko, and Vadim O. Pirogov. Mixed precision training of convolutional neural networks using integer operations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 2018, Conference Track Proceedings, 2018. URL
ftr//fouure forum ? i H1 35uzZ0
Jacob Devlin, Ming- Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04800 2018. URL
httt::/arxivv org hbtp:/bb///110 04805
William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third Internationnal Workshop on Paraphrasing (IWP2005), 2005. URL
Mario Drumond, Tao Lin, Martin Jaggi. and Babak Falsafi. Training DNNs with hybrid block floating point. In Proceedings of the 32Nd International Conference on Neural Information Processing Systems, NIPS'18, 18, pp. 451-461, USA, 2018. Curran Associates Inc. http
3326688
R. David Evans, Lu Fei Liu, and Tor Aamodt. JPEG-ACT: A frequency-domain lossy DMA engine for training convolutional neural networks. In Proceedings of the 47th Annual International Symposium on Computer Architecture, ISCA '20. ACM, 2020.
Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image 03385
recognition. CoRR, abs/1512.03385, 2015. hww.:httpp//e
HewlettPackard. CACTL https///ggaaaaccc/////eeettttaaa
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoe V. Le, and Hartwig Adam. Searching for mobilenetv3. CoRR, abs/1905.00225 2019. URL
httt:httt///txtti/ http:::://axxx..oog/bbbb//////// 905. 02244
10

---

Under review as a conference paper at ICLR 2023
Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. CoRR, abs//811 1...666,,,666 2018 http::/: org. abs 1811.11111.1 0111.66965.551
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran Yaniv,, and Yoshua Bengio. Binarized neural networks. In Advances in neural information processing systems, pp. 4107-4115, 2016.
Animesh Jain, Amar Phanishayee, Jason Mars, Lingjia Tang, and Gennady Pekhimenko. Gist: Efficient data encoding for deep neural network training. In Proceedings of the 45th Annual International Symposium
Computer Architecture, ISCA '18, pp. 776-789, Piscataway, NJ, USA, 2018. IEEE Press. ISBN 978-1-5386-5984-7 doi: 10.1109/SCA. 200SCCAA.0118 2018,00070 URL https: /oii.i....o wwo.or///// w1w.0..199/// ISCA 2018. 00070
Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, ISCA '17, pp. 1-12, New York, URL
NY, USA, 2017. ACM. ISBN 978-1 978-1-45003-4892-8 doi: 10.1145/3079856.2000 http://de:: acm. org/ /tgg/11.0 111111113379998 3080246
Patrick Judd, Jorge Albericio, Tayler Hetherington, Tor M. Aamodt, Natalie Enright Jerger, and Andreas Moshovos. Proteus: Exploiting numerical precision variability in deep neural networks. In Proceedings of the 2016 International Conference on Supercomputing. ICS '16, pp. 23:1-23:12, New York, NY, USA, 2016. ACM. ISBN 978 4503- doi: 10. 10.1145/2992 212114//92542622222222 10. 11w.11122222422.2222 2ttp///22222222..2222999 2926294
URL http:: acm. wwg/10.
Dhiraj D. Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. A study of BFLOATI6 for deep learning training. CoRR, abs/1995.1220 12322, 2019, URL http::tttaaaatee org/ haapg//b//1/90 /1905 12322
Urs Köster, Tristan J. Webb, Xin Wang, Marcel Nassar, Arjun K. Bansal, William H. Constable, Oguz H. Elibol, Scott Gray, Stewart Hall, Luke Hornof, Amir Khosrowshahi, Carey Kloss, Ruby J. Pai, and Naveen Rao. Flexpoint: An adaptive numerical format for efficient training of deep neural networks. In Proceedings of the 3/st International Conference on Neural Information Processing Systems, NIPS' pp. 1740- 1750, USA, 2017. Curran Associates Inc ISBN URL http: acm. org/ pttpeiitttiiioooo wttp::ecccciiicmmmeeeeee 32949377779777
Shang Li, Zhiyuan Yang, Dhiraj Reddy, Ankur Srivastava, and Bruce Jacob. Dramsim3; A cycle- accurate, thermal-capable dram simulator. IEEE Computer Architecture Letters, 1962)100-119, 2020. doi:
H. Liao, J. Tu, J. Xia, and X. Zhou. DaVinei: A scalable architecture for neural network computing. In 2019 IEEE Hot Chips 31 Symposium (HCS), pp. 1 1-44, 2019.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. CoRR, abs/1609.07843 2016. URL
07843
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed
11

---

Under review as a conference paper at ICLR 2023
precision training. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL
net/ wanee//foummttt forum?id=rlossupe.
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenth- waite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, et al. Fp8 formats for deep learning. arXiv preprint arXiv:2209.0555 05433, 2022.
Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sun- daraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii, Yinghai Lu, Raghuraman Krish- namoorthi, Ansha Yu, Volodymyr Kondratenko, Stephanie Pereira, Xianjie Chen, Wenlin Chen, Vijay Rao, Bill Jia, Liang Xiong, and Misha Smelyanskiy. Deep learning recommendation model for personalization and recommendation systems. CoRR, abss/190.00009,, 2019. URL fitts://aarae
org/ abs 906. 00091
Miloš Nikolić, Ghouthi Boukli Hacene, Ciaran Bannon, Alberto Delmas Lascorz, Matthicu Cour- bariaux, Yoshua Bengio, Vincent Gripon, and Andreas Moshovos. Bitpruning: Learning bitlengths for aggressive and accurate quantization, 2020.
Miloš Nikolić, Mostafa Mahmoud, and Andreas Moshovos. Characterizing sources of ineffectual computations deep learning networks. In 2018 IEEE International Symposium on Workload Charucterization (IISWC), pp. 86-87, 2018. doi: 10.1 10.110//SSSC22208885555500
NVIDIA. Training with mixed precision. indpx.iidxxxee//mm html
sdk Bdk/mii seed-peeciiinntraalle
NViiaa. NVIDIA Tesla V100 GPU Achitecture. In http:///mmgessmiiddddcccc/oooooo http:/cccccactttoooooooeeeeeeeee
NVIDIA, 2017.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Minsoo Rhu, Mike O' O'Connor, Niladrish Chatterjee, Jeff Pool, Youngeun Kwon, and Stephen W. Keckler. Compressing dma engine: Leveraging activation sparsity for training deep neural networks. In 2018 IEEE International Symposiun High Performance Computer Architecture (HPCA), pp. 78-91, 2018.
doi: 10.1 10.1109/HPCA.2018 2018.0011.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. 1rXiv:1409.0575 [cs], September 2014. arXiv: 1409.0575.
Xiao Sun, Naigang Wang, Chia-Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar Maghraoui, Vijayalakshmi (Viji) Srinivasan, and Kailash Gopalakrishnan. Ultra-low precision 4-bit training of deep neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1796-1807. Curran Asso- ciates, Inc., 2020. URL htattps://prroseedigg. neurips is. cc pcc/papprr/222/111 13b9 94382598 4c55be880445777777777
Swagath Venkataramani, Ashish Ranjan, Subarno Banerjee, Dipankar Das, Sasikanth Avancha, Ashok Jagannathan, Ajaya Durg, Dheemanth Nagaraj, Bharat Kaul, Pradeep Dubey, and Anand Raghunathan. ScaleDeep: A scalable compute architecture for learning and evaluating deep networks. In Proceedings of the 44th Annual International Symposium on Computer Architecture, ISCA '17, pp. 13-26, New York, NY, USA, 2017. ACM. ISBN 178---448-888888 doi: 10. 145/ 00779856.380044.. URL http: doi acm org 10. 1145 11............ 1tt.:/aa.......1111111111003335. 3080244]
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. HAQ: hardware- a automated quantization. CoRR, abs/181 .8.88.886.88 2018a. wwtp:/ttpp./aaxi..orrg/////
08886
12

---

Under review as a conference paper at ICLR 2023
Naigang Wang, Jungwook Choi, Daniel Brand, Chia- Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. Proceedings of the 32nd International Conference Neural Information Processing Systems, NIPS' 18, pp. 7686-7695, Red Hook, NY, USA, 2018b. Curran Associates Inc.
Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed precision quantization of convnets via differentiable neural architecture search. CoRR. abs/1812.00099 2018. URL BKLLttt://aaxxv.orggg wtg/ abggg/abs/111./// 1at.b///102000090000
Bojian Zheng, Nandita Vijaykumar, and Gennady Pekhimenko. Echo: Compiler-based GPU memory footprint reduction for LSTM RNN training. In Proceedings of the 47th Annual International Symposium Computer Architecture, ISCA '20. ACM, 2020.
13

---

Under review as a conference paper at ICLR 2023
A OUR HARDWARE APPROACH
This section presents the Schrödinger's FP hardware encoder/decoder units that efficiently exploit the potential created by our quantization schemes. Without the loss of generality we describe compresors/decopressos that process groups of 64 FP32 values.
Figure 6a shows that the compressor contains 8 packer units (Figure 6c). The compressor accepts one row (8 numbers) per cycle, for total of cycles to consume the whole group. Each column is treated as subgroup whose exponents are to be encoded using the first element's s exponent as the base and the rest as deltas. Accordingly, the exponents of the first row are stored as-is via the packers. For the every subsequent row, the compressor first calculates deltas prior to passing them to the packers.
Compressor:
The length of the mantissa is the same for all values and is provided by the mantissa quantizer method be Quantum Mantissa or BitChop. Each row uses a container whose bitlength the sum of the mantissa bitlength (provided externally) plus the bitlength needed to store the highest exponent magnitude cross the row. To avoid wide crossbars when packing/unpacking the confines of their original format bit positions as per the method proposed in Proteus (Judd et al., 2016). In contrast to Proteus, however, here every row uses a different bitlength, the values are floating-point, the bitlengths vary during runtime and per row, and we target training. The exponent lengths need to be stored as metadata per row. These are stored separately necessitating two write streams per tensor both however are sequential thus DRAM-friendly, The mantissa lengths either tensor/layer- or aetwork-wide and are stored along with the other metadata for the model.
values remain within
Each packer (Figure 6c) takes single FP32 number in [exponent, sign, mantissa] format, masks out unused exponent and mantissa bits, and rotates the remain bits to position to fill in the output row. The mask is created based on the exp_width and man man-width inputs. The rotation counter register provides the rotation count which is updated to (exp_ width+man width+1) every cycle. The (L,R) register pair, is used to tightly pack the encoded values into successive rows. There are needed since a value may now be split across two memory rows. When either register, its 32b (or 16b for BFloat16) are drained to memory. This arrangement effectively packs the values belonging to this column tightly within column of 32b in memory. Since each rows the same total bitlength, the 8 packers operate in tandem filling their respective outputs at exactly the same rate. As a result, the compressor produces 8x32b at a time. The rate at which the outputs are produced depends the compression rate achieved, the higher the compression, the lower the rate.
Decompressor: As Figure 6b shows, the decompressor mirrors the compressor. It takes 8 3-bit exponent widths and a mantissa length from the system, and 8x32 bits of data per cycle. Every column of 32b is fed into dedicated unpacker per column. The unpacker (Figure (6d] reads the exponent length for this row and the global mantissa length, takes the correct number of bits, and extends the data to [exponent, sign, mantissa] format.
Each unpacker handles one column of 32b from the incoming compressed stream. The combine-and- shift will combine the input data and previous data in register then shift to the left. The number of shifted bits is determined by the exponent and mantissa lengths of this row. The 32-bit data on the left of the register are taken out and shifted to the right (zero extending the exponent). Finally, the unpacker reinserts the mantissa bits that were trimmed during compression. Since each row of data uses the same total bitlength, the unpackers operate in tandem consuming data at the same rate. The net effect is that external memory see wide accesses on both sides.
B HARDWARE EVALUATION METHODOLOGY
Best practices for the evaluation of custom hardware architectures necessitates exploration and validation first via analytical modelling or via cycle-accurate simulation. Since training these networks takes several days on actual hardware, cycle-accurate simulation of the full process is impractical. To estimate performance and energy, we use the best practice approach by analytically modelling the time and energy used per layer per pass of a baseline accelerator. To do so, we use traffic and compute counts collected during the aforementioned full training runs. We record these counts each time a layer is invoked using Py Torch hooks. We model time and energy for memory accesses via DRAMSIM3 (Li) et al. 2020). For modeling on-chip structures use CACTI (HewlettPackard) for the buffers and
14

---

Under review as a conference paper at ICLR 2023
230
MUX
man Jen
Word7
Words
Words
Words
Word3
Word2
Word!
WordD
mud Ped wal 10
oom out
metn data
(a) Compressor
data
unpasker
snowcker
urpacker
unpocker
unpacker
unpacker
unpacker
unpacker
Word7
Word6
Word5
Word4
Word3
Word2
Wordt
WordO
decoder
200
MUX
Dula_out
(b) Decompressor
M0. will
withh
Date
Rocation
and
mask-and-sshtt
12
Ragister(54 biss)
Register
Regitter R
mask and extend
meta, dota
MUX
Word
cata_cot
(c) Packer
(d) Unpacker
Figure 6: Schrödinger's FP Compresoon/Decoopreessr
15

---

Under review as a conference paper at ICLR 2023
module
area per unit (um")
unit number
total area (mm"
compressor
40682.88
16
0.651
decompressor
46481.40
16
0.744
accelerator
38533.68
8000
308.27
Table Hardware Area Overhead
layout measurements for the compute units and the Gecko compressons/deomppessor.. We use a commercial 65nm process to model the processing units and Gecko hardware. We implement the units in Verilog and perform synthesis via the Synopsys Design Compiler and layout using Cadence Innovus with a target frequency of 500MHz. Synthesis uses Synopsys' commercial Building Block IP library for the target tech node. We estimate power via Innovus using traces over a representative input sample to model properly signal activity. We used nominal operating conditions to model power and latency. There are two Gecko compressor/decoopprsso units per channel.
Due to the complexity and time cost of cycle-accurate hardware simulation, we have opted for an estimated time and energy consumption analytical model based on the proposed hardware description and the network and retrieve its structure (layer input and output sizes, kernel sizes for convolutional layers, stride, bias and padding). We then calculate the compute operations that will happen for the general batch size (N) in both the forward and backward pass, well as the number of parameters that must be stored in memory for activations, weights and gradients.
architecture. To compute the analytical model, we first analyze the
To take advantage of data reuse where possible we perform the forward pass in layer-first order per batch. This allows us to read the weights per layer only once per batch. For the backward pass, we utilize the on-chip buffers for mini-batching with a layer-first order over mini-batch of samples. Mini-batching reduces overall traffic by processing as many samples as possible in a layer-first order avoiding either having to spill gradients or reading and writing weights per sample per layer. The number of samples that can fit in mini-batch depends on the layer dimensions and the size of the on-chip buffer.
Both SFPqM qM and SFPBc sample bitlengths per batch to log file for both mantissas and exponents. These bitlengths are used to compute the number of mini-batches that can fit every training step per layer on chip. Based on the number of sampled mini-batches (K) we compute the memory footprint generated on the forward pass for each method. After this, we calculate the footprint that stays on-chip and can be loaded from on-chip for the backward pass, and the footprint that goes to off-chip and has to be loaded to on-chip again for it. Based on these memory accesses, we use DRAMsim to simulate the number of compute-cycles that take the memory accesses finish and maximum cycles between compute and memory as the time constraint to calculate total computation time in the proposed hardware.
the
To calculate energy consumption and efficiency, we use the information gathered in terms of on-chip memory access cycles, off-chip memory access cycles and compute cycles. We estimate energy consumption for all components including the compressors and decompressors. We use the following equations to estimate energy consumption for our methods (all symbols are defined in table 5):
forward = E. computt food + E Poffchip act mem E out act mem + E onchip in act mem + + onchiip out act mem + F. Fread ops mem +
E offichip wigt mem
(5)
E. onchip wet mem E Edecompact + FE decompuwg wgtt E compacte ant
E backward E compute bok + E offechip in act mem E offchip wgt mem + E onchisp in mem + E onchip wet mem E read ops mem + E decomp act + decomp wgt
(6)
where,
16

---

Under review as a conference paper at ICLR 2023
Compression ratio
Compressor power (mW)
Decompressor power (mW)
0.143 0.263
10.87
13.84
0.264 0.388
12.18
14.72
0.389 0.513
12.65
15.97
0.514 0.638
13.44
15.76
0,639 0.763
14.98
15,42
Table 4: P() terms: Power consumption as a function compression ratio.
MemCh x PDRAM x Cycles offchiip in act
E offichsp in act mem
Freqcompute
MemCh x PDRAM
E offehhsp wet mem
x (Cycles offehhpp wet + Cycles offehip weg grad)
(8)
Freqcompute
MemCh PDRAMM
E offchip out act
Cycles offehip out act
(9)
Preqcompute
onnchip
= Cycles enchip oct write X P Ponchip write
(10)
E onchip nett mem. Cycles onchip vigt read P Ponchiprent read
(11)
E onchip = Cycles onchip out act read X P Ponchip read
(12)
Cycles onchip out act write X P onchii write
Cycles comp to decomp
E decomp decomp (comp ratio)
(13)
Freqcompute
Cycles decomp to Fregoompute
Fr comp comp comprattio)
(14)
E decompaac act
E decommp aetl comp ratio)
(15)
E decompp wgt
E decomp ugg(comp ratio)
(16)
compact (cmpaac(compraaio)
(17)
17

---

Under review as a conference paper at ICLR 2023
Symbol
Definition
Energy consumption of the compute module for the entirety of the computations in the forward pass Energy consumption of the compute module for the entircty of the computations the backward pass Energy consumption of the offchip memory transfers for the network input activations Energy consumnption of the offchip memory transfers for the network weights
E compute fied
E compute bek
E offtchip in mem
Eofichip wgg mem
Energy consumption of the offchip memory transfers for the network output activations
E offfchip out act mem.
Energy consumption of the onchip memory transfers for the network input activations Energy consumption of the onchip memory transfers
E Ronchipin act
E onchipp wet
for the network weights
Energy consumption of the onchip memory transfers for the network output activations
E oonchsp out not mem
Energy consumplion of loading
E Eread mem
operations from memory
Energy consumption of decompressing
E deccompact act
activations in the decompressor
Energy consumption of decompressing
E. decompp ugt
weights in the decompressor
Energy consumption of compressing
E act
activations in the compressor
Power consumption by the decompressor when loading data from offchip memory at a specific compression ratio (see Table 4
P Pdccomp (comp
Power consumption by the compressor when writing data to offchip memory at a specific compression ratio (see Table 4
P comp (compratio)
MemCh
Number of available memory channels
PDRAM
Power consumption of offchip DRAM
Frequompute
Clock frequency of the hardware accelerator
Cyeles offehss in act
Compute cycles taken to read input activations from offchip memory
Cycles offchip wyt
Compute cycles taken to read weights from offchip memory
Cycles offehip weg grad
Compute cycles taken to read weight gradients from offchip memory
Cycles offehip out oct
Compute cycles taken to read output activations from offchip memory
Cycles onchip in act write
Compute cycles taken to read input activations from onchip memory
Cyeles onchip wgtt read
Compute cycles taken to read weights from onchip memory
Cycles onchisp out act read
Compute cycles taken to read output activations from onchip memory
Cycles 8 onchip nct write.
Compute cycles taken to write oulput activations to onchip memory
P Panchip write
Power consumption of word write to onchip memory
Ponchip read
Power consumption of word read from onchip memory Compute cycles taken to decompress compressed data
Cycles comp to decomp
Cycles decomp to comp
Compute cycles taken to compress data
Table 5: Symbols definition table
C QUANTUM MANTISSA - MORE DATA
In this section, we expand the discussion of the effects of Quantum Mantissa on the training process. We first analyze the effects in detail on ImageNet, and then follow up with other tasks.
C.I IMAGENET
We show the effects of changing the regularization parameter, multiple training runs with the same proposed regularization parameter (y = 0.1), stopping early, well as the choice of whether to simply chop off the removed bits or round them to the least significant remaining bit. These effects summarized in Table
18

---

Under review as a conference paper at ICLR 2023
C.1.1 EFFECT OF THE y HYPERPARAMETER
In order to demonstrate the effect of the newly introduced hyperparameter y, we run three full training runs with varying Y {0.01, 0.1, 1.0]. All other hyperparameters are the same. Figure 7 shows that all runs are able to follow the baseline accuracy and provide significant memory footprint reductions. The cumulative training memory footprint and final validation accuracies are reported in Table o The most aggressive version (7 1.0) achieves 40x memory footprint reduction and reduces most layers activations to 0 bits (only exponent bits remain). However, there a noticeable accuracy degradation of 1.7%. The less aggressive version (?- 0.01) greatly reduces memory albeit, as expected, to lesser degree compared to 7 = 1.0. Our selected value (7 = 0.1) avoids both of these pitfalls, matches the baseline aceuracy and provides 9 compression ratio.
The y parameter is another hyperparameter that needs tuning. However, in our experience a good one easy to guess since 0.1 seems to work really well across different models and tasks. Its broad
applicability and usefulness is confirmed by the discussion above, other models in Section 3. and Section C22
In fact, all results presented in the main-body of the paper (Section 3.1 use Y 0.1.
Validation Accuracy throughout training- ResNec18
Weighted average the mantissa footprint- ResNet10
Suading
0.71
6.0
Figure 7: Quanttum Mantissa y effects on R 8/ImageNet: (a) TOP-1 validation accuracy during
training. (b) Weighted average mantissa length during training.
C.1.2 CONSISTENCY ACROSS TRAINING RUNS
To demonstrate the consistency of results across different runs we trained ResNet18 3 times with the same hyperparameters, specifically y = 0.1. Figure Sshows that all runs consistently follow the baseline accuracy and exhibit consistent memory footprint reduction. Ultimately, the final accuracy is consistent (standard deviation of O. 0.13%) and exactly matches the baseline with the mean of 69.94% across all the Quantum Mantissa runs (Table 6).
Validation Accuracy shroughout training ResNe:18
Weighted the mantissa footprint ResNet1B
Aserae Hourlera Langth
(b)
Figure 8: Quantum Mantissa consistency on ResNet 8/mmageeet with 7 = 0.1: (a) TOP-1 validation
accuracy during training. (b) Weighted average mantissa length during training.
1.3.3 CHOICE OF THE BIT REMOVAL METHOD
In addition, we discuss the way of removing the bits selected for removal. Accuracy and memory footprint can be improved by rounding. The rounding version, denoted by R, rounds the least significant remaining bit instead of just ignoring the removed bits. This modification allows Quantum Mantissa to be even more aggressive by reducing the footprint by an additional 35% to achieve a footprint reduction of 93%, without noticeable loss of accuracy (T Since the rounded version provides the best accuracy and footprint trade-off, we present it as the Quantum Mantissa approach in the main body of the paper.
19

---

Under review as a conference paper at ICLR 2023
Table 6: Accuracy and relative mantissa footprint of different Quantum Mantissa runs with R on Imagenet. Runs labeled with R use rounding instead of chopping. Runs labeled with RES use rounding instead of chopping and stop learning, round up and fix the mantissa length at epoch 15. Experiments with multiple runs show mean and standard deviation of obtained results.
TOP-1 Validation Accuracy
TOP-5 Validation
Relative Mantissa
Version
Accuracy
Footprint vs FP32
Baseline
69.94
89.34
100%
7 0.01
69. 76
89.21
13.9%
7 = 0.10
69.94 + 0.13
88,30 + 0.07
11.1% + 0.1%
7 1.00
68.25
88.40
2.5%
7 = 0.10 R
69.73 73 + 0.26
89.17 + 0.1
7.0% = 0.7%
7 0.10 RES
69.50
89.05
7.9%
C.1.4 EARLY STOPPING OF Quantum Mantissa
Finally, we present the rounding early stop version, denoted by RES. This version stops the mantíssa length learning at epoch 15, rounds them all up. and keeps them constant throughout the last 75 epochs. As result, the overhead (in our case negligible) constrained only to the first 15 epochs, with small drop in accuracy and memory footprint reduction 6). This experiment shows that ending Quantum Mantissa early viable strategy in cases where overhead an issue.
C.2 OTHER TASKS
In this section, we further analyze the effects of Quantum Mantissa on tasks such as recommendation systems and natural language processing as well as different network architectures such as transform- ers. All models we discuss this section are extremely weight heavy, as opposed to activation heavy ImageNet CNNs. As result, the weights transfer on and off chip (sometimes repeatedly transferred back and forth in chunks) will be the costliest operation. more important. Quantum Mantissa excels here due to its ability to zero in the costliest tensor during training through its additional loss that targets minimum footprint. This is clearly shown in the following section and tables 7 and g
this case, reducing weights is much
C.2.1 RECOMMENDATION
We present DLRM the Kaggle Criteo Dataset in Table 7 Since DLRM is trained only on one epoch, we tried Quantum Mantissa with stopping the mantissa length learning, rounding and fixing about 10% and 40% iterations. Both work well. We try 7 of 0.1 and 1.0. All versions match the baseline and excel at reducing mantissa footprint. Mantissa compression rates are about 10x, Quantum Mantissa practically removes all mantissa bits from many layers, only leaving the sign and the exponent.
Since DLRM is trained only for one epoch, it does not allow for enough iterations show the full potential of Quantum Mantissa. Consequently, we report the last iteration footprint as well. All iterations beyond the first epoch will exhibit this smallest footprint.
Table 7: Accuracy and relative mantissa footprint of Quantum Mantissa runs with DLRM on the Kaggle Criteo Dataset. Quantum Mantissa is disabled and mantissa lengths are fixed at the indicated iteration
TOP-1 Validation Accuracy
Relative Mantissa
Relative Mantissa Footprint
Version
Footprint ys FP32
vs FP32 - last iteration
Baseline
79.42
100%
100%
0.10. Stop at iteration 30k/300k
79.50
12.2%
11.3%
1.00, Stop at iteration 30k/300k
79.46
10.8%
10.7%
U.10. Stop iteration 120k/300k
79.45
11.8%
11.1%
1.00, Stop at iteration 120k/300k
79.43
10.7%
9.6%
C.2.2 NATURAL LANGUAGE PROCESSING
Finally, to show applicability of Quantum Mantissa natural language processing we present finetuning of BERT on the MRCP dataset and GPT-2 on the Wikitext-2 dataset Table g We finetuned BERT and GPT-2 for 5 and 3 epochs, respectively. Both models are trained with Quantum
20

---

Under review as a conference paper at ICLR 2023
Mantissa enabled for the first epoch and disabling it, rounding up and fixing the mantissa lengths for the rest. Again, we run both with 7 = 0.1 and y = 1.0, and get similar results. We repeated these 0.1 works great all round, matching the baseline accuracy and achieving 8 - 11 x mantissa compression, while y 1.0 achieves even better compression ratios small cost to performance. y 1.0 Quantum Mantissa manages to find many layers that do not need any mantissa bits.
experiments three times. The general
Similarly to DLRM training, finetuning does allow for enough iterations to show the full potential of Quantum Mantissa. Consequently, we report the last iteration footprint as well. All iterations beyond the predetermined epoch length will exhibit this smallest footprint. This last iteration is a very good estimate of mantissa reduction during a full training run.
Table 8: Accuracy/Perplexity and relative mantissa footprint of different Quantum Mantissa runs with BERT and GPT2 finetuning. Values are reported as mean standard deviation across multiple runs. For Perplexity, lower is better.
Relative Mantissa
Relative Mantissa Footprint
Version
Metric
Score
Footprint vs FP32
vs FP32 - last iteration
BERT Baseline
Accuracy
84.56
100%
100%
BERT 7 = 0.1
Accuracy
84.31
0.00
9.34% + 0.01%
7.82% + 0.01%
BERT 1.0
Accuracy
86.68 = 0.28
3.43% + 0.01%
1.25% + 0.01%
GPT-2 Baseline
Perplexity
20.95
100%
100%
GPT-2 = 0.1
Perplexity
20.96
0.00
12.3% + 0.00%
11.2% + 0.00%
GPT-2 1.0
Perplexity
21.42 = 0.00
2.68% + 0.00%
2.0% + 0.00%
D BITCHOP MORE DATA
This section analyzes in additional detail the effects that BitChop has the training process. We study: a) the effect of the exponential decay factor (a) used in the moving average, b) the variation in accuracy across different runs, and c) the effect of the threshold for the change in the moving average.
Validation acturacy throughout ResNet18
Average mantissa bitlnnghs ResNet18
60
Baadtin
Apha
Daseine
Alpha 0.8
Apha
20
Alpha 0.8
Alpha 0.9
86
30 45
10
Eposh
Epoch
Figure 9: BitChop a effects on BesNet18/Imageaee (a) T validation accuracy during training. (b) Average mantissa bitlengths during training.
Table 9: Accuracy and relative mantissa footprint of different BitChop runs with ResNet18 on Imagenet. Experiments with multiple runs show mean and standard deviation of obtained results.
TOP-1 Validation
TOP-5 Validation Accuracy
Relative Mantissa
Version
Accuracy
Footprint FP32
Baseline
69.94
89.34
100%
a 0.4
66.27 (inconsistent)
86.95 (inconsistent)
11.2%
a 0.8
69.93 + 0.15
88.64 + 0.09
23.7% + 2%
Q 0.9
70.07
89.37
82.6%
Exponential decay factor at We run three full training runs with varying E {0.4, 0.8, 0.9). All other hyperparameters remain the same. The lower the a the more influence past changes to the loss have on BitChop's 's decisions and the more resistant it becomes in changing course. Conversely, the higher the alpha the more influence recent changes to the loss are, and the more re-active BitChop becomes. This reflected in the changes in the validation accuracy and the mantissa bitlengths as
21

---

Under review as a conference paper at ICLR 2023
seen in Figure 9 quickly tries to recover from any apparent increase in loss. However, too high an a value coupled with the natural variations in the loss across different batches prevents BitChop from trimming mantissas significantly as can be seen for a = 0.9. Lower a values such as 0.4 produce erratic behaviour in validation accuracy. In this case, the heuristic is looking too far back into previous loss values and becomes slow to change course in adjusting bitlengths leading to trimming them too aggressively. mid-high a value of 0.8 balances reactiveness changes in the loss and bitlength reduction aggressiveness. The cumulative training memory footprint and final validation accuracies are reported in T
The results show that higher values better track baseline accuracy as BitChop
In summary, BitChop works best with an a value of 0.8, as it achieves low bitlengths while being consistent and matching the baseline validation accuracy. Values higher than 0.8 result in almost no bitlength compression. As such, a values between 0.6 and 0.8 were found to strike good balance between high memory compression and consistent and converging validation accuracy.
100
Validation accuracy througoout trainin- ResNec18
25
Average of mantlssa bilknng''s ResNet1B
Baseline
BtChop Run
BtChop Run
(itChop Run
60
Boseline
BiChop Run 1
20
BiChop
BitChop Run 3
10
30
40 45
70
75
86
15
80
25
30
35
00 45 50 Epoch
Epach
(a)
(b)
Figure 10: BirChop consistency on ResNet 8/EmageNe with a 0.8: (a) TOP-1 validation accuracy during training. (b) Average mantissa bitlengths during training
Variation Accuracy Across Runs: Figure TO demonstrates the robustness of BitChop by reporting how validation accuracy varies across multiple runs (all using the same hyperparameterrs. All runs of BitChop converged into baseline accuracy with a standard deviation of 15.
Validalion accuracy Broughhul training ReaNac18
Average mantissa billangf RusNat18
Threchold
80
201
Threshold
80
20
No Threshaid
Threshold
Epsch
1posh
(b)
Figure 11: BirChop moving average heuristic with and without threshold on BosNet18/ImageNe
with = 0.8: (a) TOP-1 validation accuracy during training. (b) Average mantissa bitlengths during training.
The Effeec of Having a Threshold: Figure shows the effect of using the threshold in BitChop'ss moving average heuristic. This shows that the threshold allows BirChop to achieve much lower bitlengths while achieving much lower swings in bitlength and accuracy throughout the process. Without the threshold, BitChop becomes overly reactive to minute changes producing an overall erratic behavior. Those swings in turn produce worse overall bitlengths and less memory compression.
Takeaways: There are several key takeaways from this ablation analysis:
Exponential decay factor (a) values lower than 0.6 take into account loss values that are no longer useful to tune the bitlength, resulting in a method that is too resistant in changing course and as such reduces bitlengths too aggressively, hurting accuracy. Values higher than 0.8 overemphasize recent loss behavior and as such the heuristic becomes too reactive to minor increases in loss and does not achieve significant memory compression.
The use of a threshold enables BirChop to be resistant to minor changes in the loss and as a result enables it to trim bitlengths more effectively than a heuristic without it, while maintaining baseline accuracy. Otherwise, BirChop exhibits unnecessary big swings in bitlength which as a result don't 't allow it to converge into the low bitlengths seen with the dynamic threshold.
22

---

Under review as a conference paper at ICLR 2023
Crucially, BirChop is robust it keeps validation accuracy convergence and bitlengths consistent over different training runs with the same hyperparameters.
E MODEL HYPERPARAMETERS AND DATASETS
In this section we summarize the hyperparamters we used for training and finetuning experiments.
E. RESNET18 AND RESNET50
Epochs: 90
Learning Rate: 0.1, 0.01 and 0.0001 at epochs 0, 30 and 60, respectively
Batch size: 256 and 48 for ResNet18 and ResNet50, respectively
Momentum: 0.9
Weight Decay: le-4
Dataset: ImageNet
E.2
MOBILENET V3 SMALL
Epochs: 150
Learning Rate: Cosine Annealing Schedule with starting Learning rate of 0.05
Batch size: 256
Momentum: 0.9
Weight Decay: le-4
Dataset: ImageNet
E.3 BERT
We the default hyperparameters provided by Hugging Face for the MRCP dataset.
Epochs: 5
Batch size: 32
Learning Rate: 2e-5
Dataset: MRCP
E.4 GPT-2
We use the default hyperparameters provided by Hugging Face for the Wikitext 2 dataset.
Epochs: 3
Batch size: 8
Dataset: Wikitext 2
E.5 DLRM
We the default hyperparameters provided by Facebook for the Kaggle Criteo dataset.
Epochs:
Learning Rate: 0.1
Mini batch size: 128
Bottom architecture: 13-512-256-64-16
Top architecture: 512-256-1
Dataset: Kaggle Criteo
23


Reviews:
[{"review_id": "vIbUZbXfRO", "reviewer": "Reviewer_JoBj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper dynamically adjusts the size and format of the floating point activations and weights using gradient-based approaches to improve the training energy efficiency and execution time of ML models. It also implements encoders and decoders that guided by these approaches reduce the off-chip memory accesses, further boosting the energy efficiency.", "summary_of_the_review": "See weaknesses above. I am giving a score of 3, but I can reconsider my score based on the authors' rebuttal and discussions with the other reviewers.", "strength_and_weaknesses": "Strengths:\n\n1. The paper is generally well-motivated and the problem chosen is timely, given the adverse environmental impact of training of large-scale ML models.\n\nWeaknesses:\n\n1. The paper is hard to read at times. For example, in the abstract, the authors pose Schrodinger's FP as one of the contributions, but it is only described in the supplementary materials. It is very unclear to me what is the encoding/decoding scheme, how does the bit-length reduction results in the memory footprint reduction, etc.\n2. The paper's contributions look incremental. For example, the stochastic quantization scheme employed for Quantum mantissa seems similar to a wide range of works [1-2]. The loss function to learn the optimal bit-length is not new either [3]. In fact, there are also works that progressively reduce the bit-width of the network during training [4].\n3. The authors did not compare their results with integer (fixed point) quantized neural networks which can even further reduce the memory footprint. I am curious what might be the accuracy drop compared to the proposed quantized FP16 approach.\n\n[1] Dong et al., \"Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization\", BMVC 2017\n\n[2] Gholami et al., \"A Survey of Quantization Methods for Efficient Neural Network Inference\", 2021\n\n[3] Yang et al., \"BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization\", ICLR 2021\n\n[4] Datta et al., \"ACE-SNN: Algorithm-Hardware Co-design of Energy-Efficient & Low-Latency Deep Spiking Neural Networks for 3D Image Recognition\", Frontiers in Neuroscience 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hh5Owj66O6", "reviewer": "Reviewer_5HSw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed reduced-precision quantization methods for efficient deep neural network training. One of the methods mainly proposed in this paper parameterized the number of mantissa bits so that the number of mantissa bits can be automatically adjusted during training (Quantum-Mantissa). The authors also proposed another (more computationally simple yet less effective) mantissa bit decision algorithm based on the difference in the loss (BitChop). Lastly, the authors proposed a lossless exponent bit encoding (Gecko). The authors claim that the proposed method can reduce memory transfer while maintaining original model accuracy after training.", "summary_of_the_review": "The authors proposed a set of bit-reduction methods for floating point format, but without sufficient theoretical analysis and empirical justification. It would be highly appreciated if the authors add more in-depth discussion on the algorithmic success of the proposed methods and if they demonstrate this benefit throughout more variety of applications.", "strength_and_weaknesses": "(Strengths)\n- The concept of customizing mantissa and exponent bits would be critical for improving the efficiency of challenging DNN training.\n\n(Weaknesses)\n- Although the proposed Quantum-Mantissa method is based on parameterization of bit-width for the mantissa, there is no in-depth analysis of how well this parameter is optimized. Also, there is no discussion on the impact of these techniques on the training convergence \n\n- The proposed methods are mainly applied to forward-pass computation, and thus the scope of computational savings is limited. Also, the authors presented the energy savings based on their hardware implementation, but it is hard to thoroughly verify the claimed advantages given the limited information provided by the discussion.\n\n- The evaluation is very limited; only covers two CNNs with typical types. So, it is hard to generalize the proposed methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "cibj2IyoAu", "reviewer": "Reviewer_hmeb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents two novel methods for performing dynamic quantization of the activations and weights used during training. That is the authors suggest two ways to dynamically calculate the number of bits used to compress the activations and weights during training. The first method, Quantum Mantissa, adds an additional term to the loss, so the model is able to learn the required quantization for each layer during training. The authors show that this approach can reduce the footprint significantly without sacrificing validation accuracy while incurring only modest overheads. The second method, BitChop, is a less intrusive method that greedily attempts to decrease the network-wide bitlength while the loss keeps decreasing and increases the bitlength when the loss increases. They show that BitChop also decreases the footprint significantly incurring even smaller overheads. The authors also introduce a lossless delta encoding scheme to encode the exponent bits by taking advantage of the non-uniformity of the exponent values. They call this approach Gecko. By combining the mantissa encoding schemes and Gecko the authors evaluate the entire model, which they call Schrodinger FP, on ResNet18 and MibileNet V3 Small on Imagenet. They show that Schrodinger FP has similar performance to the full model but with a significantly lower footprint that leads to a more performant and energy-efficient training scheme.", "summary_of_the_review": "While the paper tackles a very interesting problem and has very compelling results, I don't think the paper is ready for publication in its current form. The prose needs some work, and there need to be more experiments as well as standard error measurements before the paper is ready for publication.", "strength_and_weaknesses": "# Strengths\n- The paper addresses an interesting problem that seems to suggest a significant opportunity for making training large-scale models more efficient and performant.\n- The results are very compelling.\n- The techniques seem simple to implement.\n- The authors have done a great job providing references and situating their work within the rest of the contemporary research in the community.\n\n# Weaknesses\n- The paper was not very polished and difficult to read.\n- The description for quantization could be improved. I still don't understand how it works. Perhaps an example would help.\n- What does it mean to fix the bitlengths for some training time to fine-tune the network to this state?\n- Error bars and standard deviations are not calculated.\n- I would like to see a fair comparison of Quantum Mantissa used on a single quantization level for the entire network with BitChop.\n- I would like to see more ablations on the experiments tried for the decisions of BitChop.\n- Fig 2c suggests that each network layer should have a different quantization level, but it seems like BtChop only needs a single quantization level for the entire network. The reason for this is not clearly explained.\n- For Gecko, since the distribution is already available why not use an optimal value-based lossless encoding like a Huffman encoding scheme", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aBOUGP-wLVt", "reviewer": "Reviewer_pQea", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a combination of three hardware-accelerated techniques (Gecko, Quantum Mantissa, and BitChop) for optimizing the memory footprint of model training through low-precision floating point tensors. The approach is inspired by the observation that in some scenarios, the value distribution of training state, such as weights and activations, could be represented with fewer bits without harming the model performance. In particular, the paper proposes both dynamic and adaptive techniques for lossy and lossless quantization of the mantissa and exponents of training tensors.  The evaluation results show memory savings and negligible convergence impact for some image models. ", "summary_of_the_review": "Overall, I feel the paper is lacking in novelty and evaluation. ", "strength_and_weaknesses": "Strengths\n1. Scaling model size is the current approach for improving quality, and so the memory footprint of model training is an important problem for training efficiency and costs. \n2. Dynamic and adaptive techniques are likely most effective because of high variability of models, hardware, and training phases. \n3. Hardware-acceleration is probably the most efficient way to realize these optimization techniques. \n\nWeaknesses\n1. I think the evaluation is greatly weakened by only considering image models. In particular, it would have been useful to evaluate the effectiveness on NLP models, which have driven most of the interest in mixed-precision training. \n2. The paper does not discuss the handling of overflows/underflows during training. In my experience, these are important issues for mixed-precision training in practice. \n3. The opportunity for low-precision representation of training tensors because of their value distribution is not a novel observation and is well-studied. I feel that new insights to the community could be in more efficient exploitation approaches (hardware acceleration is certainly a good direction) and demonstrating generality over a range of model architectures (e.g., transformers) and tasks (e.g., NLP, multi-modal). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject