8-bit Optimizers via Block-wise Quantization
Authors: Tim Dettmers
Mike Lewis
Sam Shleifer
Luke Zettlemoyer
Venue: ICLR 2022 Spotlight (2022)
Keywords: language models
pretraining
finetuning
GPU memory

Abstract: Stateful optimizers maintain gradient statistics over time, e.g., the exponentially smoothed sum (SGD with momentum) or squared sum (Adam) of past gradient values. This state can be used to accelerate optimization significantly, compared to plain stochastic gradient descent, but uses memory that might otherwise be allocated to model parameters, thereby limiting the maximum size of models trained in practice. In this paper, we develop the first optimizers that use 8-bit statistics while maintaining the performance levels of using 32-bit optimizer states. To overcome the resulting computational, quantization, and stability challenges, we develop block-wise dynamic quantization. Block-wise quantization divides input tensors into smaller blocks that are independently quantized. Each block is processed in parallel across cores, yielding faster optimization and high precision quantization. To maintain stability and performance, we combine block-wise quantization with two additional changes: (1) dynamic quantization, a form of non-linear optimization that is precise for both large and small magnitude values, and (2) a stable embedding layer to reduce gradient variance that comes from the highly non-uniform distribution of input tokens in language models. As a result, our 8-bit optimizers maintain 32-bit performance with a small fraction of the memory footprint on a range of tasks, including 1.5B parameter language modeling, GLUE finetuning, ImageNet classification, WMT'14 machine translation, MoCo v2 contrastive ImageNet pretraining+finetuning, and RoBERTa pretraining, without changes to the original optimizer hyperparameters. We open-source our 8-bit optimizers as a drop-in replacement that only requires a two-line code change.

Under review as a conference paper at ICLR 2022
8 8-BIT BIT OPTIMIZERS VIA BLOCK-WISE QUANTIZATION
Anonymous authors Paper under double-bling blind review
ABSTRACT
Stateful optimizers maintain gradient statistics over time, e.g., the exponentially smoothed sum (SGD with momentum) or squared sum (Adam) of past gradi- ent values. This state can be used accelerate optimization compared to plain stochastic gradient descent but uses memory that might otherwise be allocated to model parameters, thereby limiting the maximum size of models trained in prac- tice. In this paper, we develop the first optimizers that use 8-bit statistics while maintaining the performance levels of using 32-bit optimizer states. To overcome the resulting computational, quantization, and stability challenges, we develop block-wise dynamic quantization. Block-wise quantization divides input tensors into smaller blocks that are independently quantized. Each block is processed in parallel across cores, yielding faster optimization and high precision quantization. To maintain stability and performance, we combine block-wise quantization with two additional changes: (1) dynamic quantization, a form of non-linear optimiza- tion that is precise for both large and small magnitude values, and (2) a stable em- bedding layer to reduce gradient variance that comes from the highly non-umin non-uniform distribution of input tokens in language models. As result, our 8-bit optimizers maintain 32-bit performance with a small fraction of the memory footprint on range of tasks, including 1.5B parameter language modeling, GLUE finetun- ing, ImageNet classification, WMT' 14 machine translation, MoCo v2 contrastive ImageNet pretraining-ffnetuning and RoBERTa pretraining, without changes the original optimizer hyperparameters. We open- spen-sourceour 8-bit optimizers as a drop-in replacement that only requires a two-line code change.
Increasing model size is an effective way to achieve better performance for given resources (Kaplan al., 2020) Henighan al., 2020, Raffel al.) [2019] Lewis et al. (02I). However, training such large models requires storing the model, gradient, and state of the optimizer (c.g., exponentially smoothed sum and squared sum of previous gradients for Adam), all in a fixed amount of available memory. Although significant research has focused on enabling larger model training by reducing or efficiently distributing the memory required for the model parameters (Shoeybi et al. 2019 Lepikhin
ett al. 2020 Fedus et al. 2021 Brown et al. 2020 Rajbhandari et al. 2020), reducing the memory footprint of optimizer gradient statistics is much less studied. This is significant missed opportunity since these optimizer states use 33-75% of the total memory footprint during training. For example, the Adam optimizer states for the largest GPT-2 [Radford et al.] 2019) and T5 (Raffel et al.) 2019) models are 11 GB and 41 GB in size. In this paper, develop fast, high-precision non-linear quantization method block-wise dynamic quantization - that enables stable 8-bit optimizers (e.g.. Adam, AdamW, W, and Momentum) which maintain 32-bit performance fraction of the memory footprint and without any changes to the original hyperparameters !
While most current work uses 32-bit optimizer states, recent high-profile efforts use 16-bit opti- mizers report difficultly for large models with more than IB parameters (Ramesh et al. 2021). Going from 16-bit optimizers to 8-bit optimizers reduces the range of possible values from ndd 65536 values to just 2 256. To our knowledge, this has not been attempted before.
Effectively using this very limited range is challenging for three reasons: quantization accuracy, computational efficiency, and large-scale stability. To maintain accuracy, it is critical to introduce some form of non-linear quantization to reduce errors for both common small magnitude values and rare large ones. However, to be practical, 8-bit optimizers need to be fast enough to not slow
We study 8-hit optimization with current best practice model and gradient representations (typically 16-bit mixed precision), isolate optimization challenges. Future work could explore further compressing all three.

---

Under review as a conference paper at ICLR 2022
Quantization
Dequantization
Updated optimizer states
Load Index values
Optimizer State
255
Chunk into blocks
-3.1
0.1
-0.03
1.2
Lookup values
-1.0
0.0329
-0.0242
1.0
Annd block-wise absmax
3.1
1.2
Denomalliee by absmax
-1003.1
0.0329-3.1
-0.02421122
1.011.2
Nomalize with absmax
0.032
-0.025
1.0
Dequantized optimizer
0.102
-0029
1.2
Find closest 8-bit value
0.0329
Find comesponding incex
170
80
255
Store index values
Update optimizer states
Figure 1: Schematic of 8-bit optimizers via block-wise dynamic quantization, see Section 2 for more details. After the optimizer update is performed in 32-bit, the tensor is chunked into blocks, normalized by the absolute maximum value of each block. Then dynamic quantization is performed, and the index stored. For dequantization, lookup in the index is performed, with subsequent de- normalization by multiplication with the block-wise absolute maximum value. Outliers are confined single block through block-wist quantization, and their effect on normalization limited.
down training, which is especially difficult for non-linear methods that require more complex data structures maintain the quantization buckets. Finally, to maintain stability with huge models beyond IB parameters, a quantization method needs to not only have a good mean error but excellent worse case performance since a single large quantization error can cause the entire training run to diverge.
We introduce new block-wise quantization approach that addresses all three of these challenges. Block-wise quantization splits input tensors into blocks and performs quantization on each block in- dependently. This block-wise division reduces the effect of outliers on the quantization process since they are isolated to particular blocks, thereby improving stability and performance, especially for large-scale models. Block-wise processing also allows for high optimizer throughput since each nor- malization can be computed independently in each core. This contrasts with tensor- wide normaliza- tion, which requires slow cross-core synchronization that is highly dependent on task-core schedul- ing. We combine block-wise quantization with two novel methods for stable, high rere-performane 8-bit optimizers: dynamic quantization and a stable embedding layer. Dynamic quantization an exten- sion of dynamic tree quantization for unsigned input data. The stable embedding layer is variation of a standard word embedding layer that supports more aggressive quantization by normalizing the highly non-uniform distribution of inputs avoid extreme gradient variation.
Our 8-bit optimizers maintain 32-bit performance at fraction of the original memory footprint. We show this for a broad range of tasks: 1.5B and 355M parameter language modeling, GLUE finetuning. ImageNet classification, WMT' 14+WMT' 16 machine translation, MoCo v2 contrastive image pretraining-finetuniin
and RoBERTa pretraining. We also report additional ablations and sensitivity analysis showing that all components block-wise quantization, dynamic quantization, and stable embedding layer - are crucial for these results and that 8-bit Adam can be used as simple drop-in replacement for 32-bit Adam, with no hyperparameter changes. We open-source our custom CUDA kernels and provide PyTorch implementation that enables 8-bit optimization by changing two lines of code.
BACKGROUND
1.1 STATEFUL OPTIMIZERS
An optimizer updates the parameters w neural network by using the gradient of the loss with respect to the weight g, Sw gradient with respect to each parameter over time for accelerated optimization. Two of the most commonly used stateful optimizers are Adam (Kingma and Ba, 2014), and SGD with momentum
ØL at update iteration t. Stateful optimizers compute statisties of the
2

---

Under review as a conference paper at ICLR 2022
Momentum for short. Without damping and scaling constants, the update rules of these optimizers are given by:
Qian 1999)
mn = go
Initialization
Momentum gt, Wi-1, m)-1) =
m, 3 1,m,-
State 1 update
(1)
W; W m
Weight update
To m. 0
Initialization
m. = f, + (1 81)
State update
Adam(g: Wr-1, 1t-1;
It 32re-1 + (1 - 32)g;
State 2 update
W. W vFete
Weight update,
where $1 and $2 are smoothing constants, c is a small constant, and is the learning rate. For 32-bit states, Momentum and Adam consume and bytes per parameter. That 4 GB and GB for a 1B parameter model. Our 8-bit non-linear quantization reduces these costs to 1 GB and 2 GB.
1.2 NON-LINEAR QUANTIZATION
Quantization compresses numeric representations to save space at the cost of precision. Quanti- zation is the mapping of a k-bit integer to real element in D, that is, Qmap : 10, 2 For example, the IEEE 32-bit floating point data type maps the indices 23..232 to the do- main [-3,4e38, +3.4c38]. We the following notation: Qi) (i) Q.map q. for example Qmap(23) + 131072) = 2.03125, for the IEEE 32-bit floating point data type.
1] + D.
To perform general quantization from data type into another require three steps. (1) Compute a normalization constant N that transforms the input tensor into the range of the domain D the target quantization data type Qmap, (2) for each element of T/NN find the closest corresponding value in the domain D, (3) store the index i corresponding to qi in the quantized output tensor TQ, To receive the dequantized tensor T we look up the index and denormalize: T = "uP (T?) N.
To perform this procedure for dynamic quantization we first normalize into the range [-1, 1] through division by the absolute maximum value: N max([T])
Then we find the closest values via a binary search:
2
T
T
arg min j=0
Q
mup
(3)
N
1.3 DYNAMIC TREE QUANTIZATION
Dynamic Tree quantization (Dttteess) 2016) is a method that yields low quantization error for both small and large magnitude values. Unlike data types with fixed exponent and fraction, dynamic tree quantization with a dynamic exponent and fraction that can change with each number.
Exponent: 1e
Linear
Sign
suantization:
datatype
made up of four parts, seen Figure 2 (1) The first bit of the data type is reserved for a sign. (2) The number of subsequent zero bits indicates the magnitude of the exponent. (3) The first bit that is set to one indicates that all following values are reserved for (4) linear quantization. By moving the indicator bit, num- bers can have large exponent 10-7 7 or precision as high as 1/63. Compared to linear quantization, dynamic tree quantization has better absolute and relative quantization errors for non-uniform dis- tributions. Dynamic tree quantization strictly defined to quantize numbers in the range [-1.0, 1.0], which is ensured by performing tensor-level absolute normalization.
1e-2
0.6 -66-3
Figure Dynamic tree quantization.

---

Under review as a conference paper at ICLR 2022
2 8-BIT OPTIMIZERS
Our 8-bit optimizers have three components: (1) block-wise quantization that isolates outliers and distributes the error more equally over all bits; (2) dynamic quantization, which quantizes both small and large values with high precision; and (3) stable embedding layer to improve stability during optimization for models with word embeddings.
With these components, performing an optimizer update with 8-bit states is straightforward. We dequantize the 8-bit optimizer states to 32-bit, perform the update, and then quantize the states back to 8-bit for storage. We do this 8-bit to 32-bit conversion clement-by-clement in registers, which means no slow copies to GPU memory or additional temporary memory are needed to perform quantization and dequantization. For GPUs, this makes 8-bit optimizers faster than regular 32-bit optimizers, as we show in Section
2.1 BLOCK-WISS WISE QUANTIZATION
Our block-wise quantization reduces the cost of computing normalization and improves quantization precision by isolating outliers. In order to dynamically quantize a tensor, as defined in Section 1.2 we need to normalize the tensor into the range [-1, 1]. Such normalization requires reduction over the entire tensor, which entails multiple synchronizations across GPU cores. Block-wise dynamic quantization reduces this cost by chunking an input tensor into small blocks of size B 2048 and performing normalization independently in each core across this block.
More formally, using the notation introduced in Section 1.2] in block-wise quantization, treat as one-dimensional sequence of elements that we chunk in blocks of size B. This means for an input tensor T with elements
have n/B blocks. We proceed to compute normalization constant for each block: N₂ = max(/T, (&)). where b the index the block 0..n/ B. With this block-wise normalization constant, each block can be quantized independently:
2 T arg min QQ,
Thi N₂
==
0<K<B
This approach has several advantages, both for stability and efficiency. First, each block normal- ization can be computed independently. Thus no synchronization between cores is required, and throughput is enhanced.
Secondly, it also much more robust to outliers in the input tensor. For example, to contrast block- wise and regular quantization, if we create input tensor with one million elements sampled from standard normal distribution, we expect less than 1% of elements of the tensor will be in the range [3, oo). However, since normalize the input tensor into the range [-1.1] this means the maximum values of the distribution determine the range of quantization buckets. This means if the input tensor contains outlier with magnitude 5, the quantization buckets reserved for numbers between 3 and will mostly go unused since less than 1% of numbers are in this range. With block- wise quantization, the effect of outliers is limited to single block. As such, most bits used effectively in other blocks.
Furthermore, because outliers represent the absolute maximum value in the input tensor, block- wise quantization approximates outlier values without any error. This guarantees that the largest optimizer states, arguably the most important, will always be quantized with full precision. This property makes block-wise dynamic quantization both robust and precise and is essential for good training performance practice.
2.2 DYNAMIC QUANTIZATION
In this work, we extend dynamic tree quantization (Section 1.3) for non-signed input tensors by re-purposing the sign bit. Since the second Adam state is strictly positive, the sign bit is not needed. Instead of just removing the sign bit, we opt to extend dynamic tree quantization with a fixed bit for the fraction. This extension is motivated by the observation that the second Adam state varies around 3-5 orders of magnitude during the training of a language model. In comparison, dynamic tree quantization already has range of 7 orders of magnitude. We refer this quantization as
4

---

Under review as a conference paper at ICLR 2022
dynamic quantization distinguish it from dynamic tree quantization in experiments. A study of additional quantization data types and their performance is detailed in Appendix B
2.3 STABLE EMBEDDING LAYER
Our stable embedding layer is a standard word embedding layer variation (Devlin et al. 2019) designed to ensure stable training for NLP tasks. This embedding layer supports more aggressive quantization by normalizing the highly non-uniform distribution of inputs to avoid extreme gradient variation. See Appendix for a discussion of why commonly adopted embedding layers (Ott al., 2019) are so unstable.
We initialize the Stable Embedding layer with Xavier uniform initialization (Glorot and Bengio 2010) and apply layer normalization (Ba et al.) (016) before adding position embeddings. This method maintains a variance of roughly one both at initialization and during training. Additionally, the uniform distribution initialization has less extreme values than normal distribution, reducing maximum gradient size. Like Ramesh et al. (202T), we find that the stability of training improves significantly if we use 32-bit optimizer states for the embedding layers. This is the only layer that
uses 32-bit optimizer states, We still use the standard precision for weights and gradients for the embedding layers- usually 16-bit. We show in Ablation Analysis in Section 4 that this change is a necessary detail.
3 8-BIT vs 32-BIT OPTIMIZER PERFORMANCE FOR COMMON BENCHMARKS
Experimental Setup We compare the performance of 8-bit optimizers to their 32- counterparts on a range of challenging public benchmarks. These benchmarks either use Adam (Kingma and Ba 2014) AdamW (Loshchilov and Hutter 2018), Momentum QQan 1999..
We do not change any hyperparameters or precision of weights, gradients, and activations/input gra- dients for each experimental setting compared the public baseline- the only change is to replace 32-bit optimizers with 8-bit optimizers. This means that for most experiments, we train in 16-bit mixed-precision (Micikevicius et al. 2017). We also compare with Adafactor (Shazeer and Stem 2018), with the time-independent formulation for 32 (Shazeer and Stern 2018) -which is the same formulation used in Adam. We also do not change any hyperparameters for Adafactor.
We report on benchmarks in neural machine translation (Ott et al. 2018 trained on WMT 16 (Sennrich et al. 2016) and evaluated on en-de WMT 14 (Macháček and Bojar 2014), large-scale
language modeling (Lewis et al.) [2021] Brown et al. 2020) and RoBERTa pretraining (Liu et al.)
2019 on English CC-100 + RoBERTa corpus (Nagell 2016 Gokaslan and Cohen, 2019: Zhu et al.
2015 Wenzek et al. (000)). finetuning the pretrained masked language model RoBERTa (Liu et al.
2019 on GLUE (Wang et al. 2018a), ResNet-50 v1.5 image classification (He et al.
2016 on
ImageNet- 1k (Deng al. 2009), and Moco v2 contrastive image pretraining and linear finetuning
CCen et al. 2020b on ImageNet-Ik 1k (Deng et al. 2009).
We use the stable embedding layer for all NLP tasks except for finetuning GLUE. Beyond this, we follow the exact experimental setup outlined the referenced papers and codebases. We consistently report replication results for each benchmark with public codebases and report median accuracy, perplexity, or BLEU over ten random seeds for GLUE, three random seeds for others tasks, and single random seed for large scale language modeling. While it is standard report means and standard errors on some tasks, others use median performance. We opted to report medians for all tasks for consistency.
In Table T we see that 8-bit optimizers match replicated 32-bit performance for all tasks. While Adafactor is competitive with 8-bit Adam, 8-bit Adam uses less memory and provides faster optimization. Our 8-bit optimizers save up to 8.5 GB of GPU memory for our largest 1.5B pa- rameter language model and 2.0 GB for RoBERTa. Thus, 8-bit optimizers maintain performance
Results
bttes:/ /tte:://oithua con/ wtt.///////ccceeec wwtp://wttccc////coooooooo wttpe//ttttttttteeeeeeeeeeeeeeet www.aa//////eeeeeeeeeee
/SEADVE
httpss http:://gitthbb con/ pyt orch ff rawry lob/maas /wxengl wttp://ss///////aaaaaaaaa
httpss//githut
inglisamp http://www.ratteeeeeee/////////e
ht https://githhb www.:/tthhhh....aaeeeoeeeooeeoot wwwp/////oooooeeeeooooooooo
5

---

Under review as a conference paper at ICLR 2022
Table I: Median performance on diverse NLP and computer vision tasks: GLUE, object classifi- cation with (Moco v2) and without pretraining (CLS), machine translation (MT), and large-scale language modeling (LM). While 32-bit Adafactor is competitive with 8-bit Adam, it uses almost twice as much memory and trains slower. 8-bit Optimizers match exceed replicated 32-bit per- formance on all tasks. We observe no instabilities for 8-bit optimizers. Time total GPU time on V100 GPUs, except for RoBERTa and GPT3 pretraining, which were done on A A100 GPUs.
Optimizer
Task
Data
Model
Metric
Time
Mem saved
32-bit Adam W
GLUE
Multiple
RoBERTa-Large
Reference
32-bit AdamW
GLUE
Multiple
RoBERTa-Large
88.6
17h
0.0 GB
32-hit Adafactor
GLUE
Multiple
RoBE-Ta--Large
88.7
24h
1.3 GB
8 8-bit AdamW
GLUE
Multiple
RoBERTa-Large
88.7
15h
2.0 GB
32-bit Momentum
CLS
ImageNet-Ik
ResNet--50
77.1
Reference
32-bit Momentum
CLS
ImageNet-1k
ResNet-50
77.1
118h 116 h
0.0 GB 0.1 GB
8- bit Momentum
CLS
ImageNet- 1k
ResNet-50
77.2
32-bit Adam
MT
WMT' 14+16
Transformer
29.3
Reference
32-hit Adam
MT
WMTT14+16
Transformer
29.0
126h
GB 0.3 GB
32-bit bit Adafactor
MT
WMT 14+16
Transformer
29.0
127h
8- bit Adam
MT
WMT 14+16
Transformer
29.1
115h
1.1 GB
02-bit Momentum
MoCo v2
ImageNet-Ik
ResNet--50
67.5
Reference
32- bit Momentum
MoCo v2
ImageNet- lk
ResNet-50
67.3
30 days
0.0 GB 0.1 GB
bit Momentum
MoCo v2
ImageNet-1k
ResNet-50
67.4
28 days
3i-bit Adam
LM
Multiple
Transformer-
308 days
0.0 GB
32-hit Adafactor
Multiple
Transformer-
316 days
GB
8-biz bit Adam
LM
Multiple
Transformer-1.5B
9.0
297 days
8,5 GB
32-bit Adam
LM
Multiple
GPT3-Medium
10.62
795 days
0.0 GB
32-bit Adafactor
Multiple
GPT3-Medium
10.68
816 days
1.5 GB
8 bit Adam
LM
Multiple
GPT3-Medium
10.62
761 days
1.7 GB
32-bi bit Adam
Masked-LM
Multiple
RoBERTa-Baie
3.49
101 days
0.0 GB
32-hit Adafactor
Masked-LM LM
Multiple
RoBEKTa-Bas
3.59
112 days
0.7 GB
8-hit Adam
Masked-LM
Multiple
RoBERTa-
3.48
days
GB
Metric: GLUE=Mean Accuracy/Correlation
CLS/MoCo = Accuracy, MT=BLEU, LM-Perplexity.
and improve accessibility the finetuning of large models for those that cannot afford GPUs with large memory buffers. We show models that are now accessible with smaller GPUs in Table 2 A breakdown of individual dataset results on GLUE can be found in Appendix (A).
The broad range of tasks and competitive results demonstrate that 8-bit optimizers are robust and effective replacement for 32-bit optimizers, do not require any additional changes in hyperparame- ters, and save significant amount of memory while speeding up training slightly.
Table 2: With 8-bit optimizers, larger models can be finetuned with the same GPU memory com- pared standard 32-bit optimizer training. We use batch size of one for this comparison.
Largest finetunable Model (parameters)
GPU size GB
32-bit Adam
8-bit Adam
RoBERTa-baas base (110M)
RoBERTa-large (355M)
11
MT5-small (300M)
MT5-base (580M)
24
MT5-base (580M)
MT5-large (1.2B)
24
GPT-2-medium (762M)
GPT-2-large (1.5B)
4 ANALYSIS
We analyze our method in two ways. First, we ablate all 8-bit optimizer components and show that they are necessary for good performance. Second, we look at the sensitivity hyperparameters

---

Under review as a conference paper at ICLR 2022
compared 32-bit Adam and show that 8-bit Adam with block-wise dynamic quantization is reliable replacement that does not require further hyperparameter tuning.
Experimental Setup We perform our analysis
strong 32-bit Adam baseline for language
modeling with transformers (Vaswani et al. 2017). We subsample from the RoBERTa corpus (Liu et al. 2019) which consists of the English sub-datasets: Books (Zhu al. (015), Stories (Trinh and Le 2018). Open WebText-1 (Gokaslan and Cohen 2019). Wikipedia, and CC-News (Nagel] 2016). We use 50k token BPE encoded vocabulary (Sennrich et al. 2015). We find the best 2-GPU-day transformer baseline for 32-bit Adam with multiple hyperparameter searches that take in a total of 440 GPU days. Key hyperparameters include 10 layers with model dimension of 1024, fully connected hidden dimension of 8 8192, 16 heads, and input sub-sequences with a length of 512 tokens each. The final model has 209m parameters.
Table 3: Ablation analysis of 8-bit Adam for small (2 GPU days) and large-scale (=1 GPU year) transformer language models on the RoBERTa corpus. The runs without dynamic quantization use linear quantization. The percentage of unstable runs indicates either divergence crashed training due to exploding gradients. We report median perplexity for successful runs. We can see that dynamic quantization is critical for general stability and block-wise quantization is critical for large- scale stability. The stable embedding layer useful for both 8-bit and 32-bit Adam and enhances stability to some degree.
Parameters
Optimizer 32-bit Adam
Dynamic
Block-wise
Stable Emb
Unstable (%)
Perplexity
0
16.7
32-bit Adam
0
16.3
8-bit Adam
90
253.0
8-bit Adam
50
194.4
209M
8-bit Adam
10
18.6
8-bit Adam
0
17.7
8-bit Adam
0
16.8
8-bit Adam
16.4
1.3B
b2- Adam
10.4
1.3B
8-bit Adam
100
N/A
1.3B
8-bit Adam
80
10.9
1.5B
32-bit Adam
0
9.0
1.5B
8-bit Adam
9.0
For the ablation analysis, we compare small and large-scale language model- ing perplexity and training stability against 32-bit Adam baseline. We ablate components individ- ually and include combinations of methods that highlight their interactions. The baseline method uses linear quantization, and we add dynamic quantization, block-wise quantization, and the stable embedding layer to demonstrate their effect. To test optimization stability for small-scale language modeling, we run each setting with different hyperparameters and report median performance across all successful runs. A successful run is a run that does not crash due to exploding gradients or di- verges in the loss. We use the hyperparameters e {le-8, Ic-7, lc-6]. 31 {0.90, 0.87, 0.93), 83 (0.999, 0.99, 0.98) and small changes in learning rates. We also include some partial ablations for large-scale models beyond 1B parameters. In the large-scale setting, we run several seeds with the same hyperparameters. We use single seed for 32-bit Adam, five seeds for 8-bit Adam at 1.3B parameters, and a single seed for 8-bit Adam at 1.5 parameters Results are shown Table 3
Ablation Analysis
The Ablations show that dynamic quantization, block-wise quantization, and the stable embedding layer are critical for either performance or stability. In addition, block-wise quantization critical for large-scale language model stability.
We compare the perplexity of 32-bit Adam vs 8-bit Adam Stable Embed- ding as we change the optimizer hyperparameters: learning rate, betas, and e. We change each hyper- parameter individually from the baseline hyperparameters 3,=0.9, 3,=0.995, e=le-7, and 1r=0.0163
Sensitivity Analysis
We chose not do the full ablations with such large models because each training takes one GPU year.

---

Under review as a conference paper at ICLR 2022
and rum two random seeds for both 8-bit and 32-bit Adam for each setting. If 8-bit Adam is perfectly insensitive to hyperparameters compared to 32- Adam, we would expect the same constant offset in performance for any hyperparameter combination. The results can be seen in Figure [] The sults show relatively steady gap between 8-bit and 32-bit Adam, suggesting that 8-bit Adam does not require any further hyperparameter tuning compared to 32-bit Adam.
17.2
17.2
Volid Perplexity
17.0
Valld Perplexity
17.0
Perplexity
17.0
16.8
16.8
16.8
16.6
Bits
16.4
18.2
16.2
18.2
18.0
10:09
16-07
16-06
16.0
0.0005
0,0100
0.0015
0,0020
0.0025
0,0030
16.0
0.9 0,995
0.90
0.8 0.999
0. 0.99
0.65 0.98
0.65, 0.99
0.90
0.90
0.9 0.9.0.90 0.99
0.9. 0.999
1e
Adam Epsilon
Learning Rate
0
Adam Betas
Figure 3: Sensitivity analysis of 8-bit 32-bit Adam hyperparameters. We can see that there is little variance between and 32-bit performance, which suggests that 8-bit Adam can be used as drop-in replacement for 32-bit Adam without any further hyperparameter tuning.
RELATED WORK
Compressing & Distributing Optimizer States While 16-bit Adam has been used in several publications, the stability 16-bit Adam was first explicitly studied for a text-to-image generation model DALL-E (Ramesh et al. 2021). They show that stable embedding layer, tensor-wis wise scaling constants for both Adam states, and multiple loss scaling blocks are critical to achieving stability during training. Our work reduces the memory footprint of Adam further, from 16 to 8-bit. In addition, we achieve stability by developing new training procedures and non-linear quantization, both of which complement previous developments.
Adafactor (Saazeer and Stern 2018) uses a different strategy to save memory. All optimizer states still 32-bit, but the second Adam state is factorized by row-column outer product resulting in a comparable memory footprint to 16-bit Adam. Alternatively, Adafactor can also be used without using the first moment (8, 0.0) [epikhin et al. 2020). This version as memory efficient as 8-bit Adam, but unlike 8-bit Adam, hyperparameters for this Adafactor variant need to be re-tuned achieve good performance. We compare 8-bit Adam with Adafactor 31 0.0 in our experiments.
AdaGrad (Duchi et al. 1 adapts the gradient with aggregate training statistics over the entire training run. AdaGrad that uses only the main diagonal as optimizer state and extensions of AdaGrad such as SM3 (Anil et al. (019)) and extreme tensoring (Chen et al. 2020a) can be more efficient than 8-bit Adam. We include some initial comparison with AdaGrad Appendix G
Optimizer sharding (Rajbhandari et al.) 2020) splits optimizer states across multiple accelerators such as GPUs/TPUs. While very effective, it can only used if multiple accelerators are available and data parallelism is used. Optimizer sharding also have significant communication overhead (Rajbhandari et al. 202T). Our 8-bit optimizers work with kinds parallelism. They can also complement optimizer sharding, they reduce communication overhead by 75%.
General Memory Reduction Techniques Other complementary methods for efficient training can be either distributed or local. Distributed approaches spread out the memory of model across several accelerators such as GPUs/TPUs. Such approaches are model parallelism (Krizhevsky et al.) 2009), pipeline parallelism (Krizhevsky al. 2009 Huang et al., 2018 Harlap et al.) (018)) and operator parallelism Lepikhin et 2020). These approaches are useful if one has multiple accel- erators available. Our 8-bit optimizers useful for both single and multiple devices.
Local approaches work for a single accelerator. They include gradient checkpointing (Chen et al. 2016), reversible residual connections (Gomez al. 2017). and offloading (Pudipeddi al. 2020
8

---

Under review as a conference paper at ICLR 2022
Rajbhandari al. 2021 All these methods save memory at the cost of increased computational or communication costs. Our 8-bit optimizers reduce the memory footprint of the model while maintaining 32- bit training speed.
Quantization Methods and Data Types While our work is the first apply 8-bit quantization to optimizer statistics, quantization for neural network model compression, training, and inference are well-studied problems. One of the most common formats of 8-bit quantization to use data types composed of static sign, exponent, and fraction bits. The most common combination is 5 bits for the exponent and 2 bits for the fraction [Wang et al., 2018b) [unn et al.) 2019, Cambier et al. 2020 Mellempudi et al.) 2019) with either no normalization or min-max normalization. These data types offer high precision for small magnitude values but have large errors for large magnitude values since only 2 bits are assigned to the fraction. Other methods improve quantization through soft constraints (L et al. 2021 or more general uniform affine quantizations (Pappalardo 2021).
Data types lower than 8-bit are usually used to prepare model for deployment, and the main focus is on improving network inference speed and memory footprint rather than maintaining accuracy. There methods that use 1-bit (Courbariaux and Bengio 2016
Rastegari et al. [2016] Courbariaux 2019 4-bits (Li et al. (019)), more bits variable amount of bits (Gong et al., 2019), See also Qin et al.
et al. 2015), 2-bit/3 values (Zhu et al. 2017; Choi et al.
(Courbariaux et al. 2014), (0200) for survey on binary neural networks. While these low-bit quantization techniques allow for efficient storage, they likely lead to instability when used for optimizer states.
The work most similar to our block-wise quantization is work on Hybrid Block Floating Point (HBFP) (Drumond et al., 2018) which uses a 24-bit fraction data type with a separate exponent for each tile in matrix multiplication to perform 24-bit matrix multiplication. However, unlike HBFP, bise dynamic quantization has the advantage of having both block-wise normalization and dynamic exponent for each number. This allows for much broader range of important values since optimizer state values vary by about orders of magnitude. Furthermore, unlike HBFP, block-wist quantization approximates the maximum magnitude values within each block without any quantiza- tion error, which is critical for optimization stability, particularly for large networks.
6 DISCUSSION & LIMITATIONS
Here we have shown that high precision quantization can yield 8-bit optimizers that maintain 32-bit optimizer performance without requiring any change hyperparameters. One of the main limita- tions of our work is that 8-bit optimizers for natural language tasks require stable embedding layer to be trained to 32-bit performance. On the other hand, we show that 32-bit optimizers also benefit from stable embedding layer. As such, the stable embedding layer could be seen as general replacement for other embedding layers.
We show that 8-bit optimizers reduce the memory footprint and accelerate optimization range of tasks. However, since 8-bit optimizers reduce only the memory footprint proportional to the number of parameters, models that use large amounts of activation memory and little memory for parameters, such as convolutional networks, have few benefits from using 8-bit optimizers. Thus, 8- bit optimizers are most beneficial for training or finetuning models with many parameters on highly memory-consttained GPUs.
wide
Furthermore, there remain sources of instability that, our knowledge, are well understood. For example, we observed that models with over IB parameters often have hard systemic diver- gence, where many parameters simultaneously cause exploding gradients. In other cases, a single parameter among those IB parameters assumed value too large, caused an exploding gradient, and led to a cascade of instability. It might be that this rare, soft cascading instability is related to the phenomena where instability disappears after reloading model checkpoint and rolling new random seed - a method standard for training huge models. This cascading instability might also be related to the observation that the larger model is, the more unstable it becomes. For our 8-bit optimizers, we primarily needed the stable embedding layer to avoid cascading instability. Thus the stable embedding layer could potentially be viewed as decreasing the probability of extreme outlier gradients. If such phenomena were better understood, could lead to better 8-bit optimizers and more stable training in general.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Anil, R., Gupta, V., Koren, T., and Singer. Y. (2019). Memory efficient adaptive optimization. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d' d'Alch-BBu,, F., Fox. E. B., and Garnett, R., editors, Advances in Neural Information Processing Systems 32: Anmual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 9746-9755.
Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint 167.0:160.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P. Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. arXiv preprint 2005::2005.14155.
Cambier, L., Bhiwandiwalla, A., Gong, T., Elibol, O. H., Nekuii, M., and Tang, H. (2020). Shifted and squeezed 8-bit floating point format for low low-precision training of deep neural networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Chen, E. J. and Kelton, W. D. (2001). Quantile and histogram estimation. In Proceeding of the 2001 Winter Simulation Conference (Cat. No. CH373044, volume 1, pages 451-459. IEEE.
Chen, T., Xu, B., Zhang, C., and Guestrin, C. (2016). Training deep nets with sublinear memory cost. arXiv preprint (K4i::6004066)
Chen, X., Agarwal, N., Hazan, E., Zhang, C., and Zhang, Y. (2020a). Extreme tensoring for low- memory preconditioning. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Chen, X., Fan, H., Girshick, R., and He, K. (2020b). Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422)
Choi, J., Venkataramani, S., Srinivasan, V., Gopalakrishnan, K., Wang, Z., and Chuang, P. (2019). Accurate and efficient 2-bit quantized neural networks. In Talwalkar, A., Smith, V., and Zaharia, M., editors, Proceedings of Machine Learning and Systems 2019, MLSys 2019, Stanford, CA, USA, March 31 April 2, 2019. mlsys.org.
Courbariaux, M. and Bengio, Y. (2016). Binarynet: Training deep neural networks with weights and activations constrained to +1 or 11. CoRR, 102.06002.2880.
Courbariaux, M., Bengio, Y., and David, J. (2015). Binaryconnect: Training deep neural networks with binary weights during propagations. In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., and Garnett, R., editors, Advances Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 3123-3131.
Courbariaux, M., Bengio, Y., and David, J.-P. (2014). Training deep neural networks with low precision multiplications. arXiv preprint arXiv: 1412.7024.
Deng. J., Dong, W., Socher, R., Li, L.,,,, Li, K., and Fei-Fei. (2009). Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee.
Dettmers, T. (2016). 8-bit approximations for parallelism in deep learning. International Conference on Learning Representations (ICLR).
Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019). BERT: pre-training of deep bidirectional transformers for language understanding. In Burstein, J., Doran, C., and Solorio. T., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171-4188 Association for Computational Linguistics.
10

---

Under review as a conference paper at ICLR 2022
Drumond, M., Lin, T., Jaggi, M., and Falsafi, B. (2018). Training dnns with hybrid block float- ing point. Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa- Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 451-461.
Duchi, J., Hazan, E., and Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7).
Dunning, T. and Ertl, O. (2019). Computing extremely accurate quantiles using t-digests. arXiv preprint arXivv1902.044 04023.
Fedus, W., Zoph, B., and Shazeer, N. (2021). Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, arXiv preprint arXiv:210100361..
Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256 JMLR Workshop and Conference Proceedings.
Gokaslan, and Cohen, V. (2019). Openwehtext corpus.
Gomez, A. N., Ren, M., Urtasun, R., and Grosse, R. B. (2017). The reversible residual network: Backpropagation without storing activations. arXiv preprint arXiv: NtxXiv:/77..00 7777005870555
Gong, R.,. Liu, X., Jiang, S., Li, T., Hu, P., Lin, J., Yu, F., and Yan, J. (2019). Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In 2019 IEEE/CVF Interna- tional Conference Computer Vision, ICCV 2019, Scoul, Korea (South), October 27 November 2, 2019, pages 4851-4860. IEEE.
Govindaraju, N. K., Raghuvanshi, N., and Manocha, D. (2005). Fast and approximate stream mining of quantiles and frequencies using graphics processors. Proceedings of the 2005 ACM SIGMOD international conference Management of data, pages 61 1-2222
Greenwald, M. and Khanna, S. (2001). Space-efficient online computation of quantile summaries. ACM SIGMOD Record, 30(2):58-66.
Harlap, A., Narayanan, D., Phanishayee, A., Seshadri, V., Devanur, N., Ganger, G., and Gib- bons, P. (2018). Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint 1.Xx::8806.0377.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778.
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray. S., et al. (2020). Scaling laws for autoregressive generative modeling, arXiv preprint arXiv:2010.144 14701.
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M. X., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. (2018). Gpipe: Efficient training giant neural networks using pipeline parallelism. arXiv preprint arXiv:1811.06965
Kaplan, J., McCandlish, S., Henighan, T., Brown, B., Chess, B., Child, R., Gray, S., Radford, A., Wu. J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.08361
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., and Socher, R. (2019). CTRL: A conditional transformer language model for controllable generation. CoRR, abss/1999.0589
Kingma, D. P. and Ba, J. (2014). Adam: method for stochastic optimization. arXiv preprint arXiv: 6412,660.
Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images.
11

---

Under review as a conference paper at ICLR 2022
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikum, M., Shazeer, N., and Chen, Z. (2020). Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint 7XXv:2006.16668
Lewis, M., Bhosale, S., Dettmers, T., Goyal, N., and Zettlemoyer, L. (2021). Base layers: Simplify- ing training of large, sparse models. arXiv preprint
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettle- moyer, L. (2020). BART: denoising eqquence-to-sequeee
pre-training for natural language gen- eration, translation, and comprehension. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, R., editors, Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, ACL 2020, Online, July 5-10, 2020, pages 7871 7871-7880. Association for Computational Linguisties.
Li J. B., Qu, S., Li, X., Strubell, E., and Metze, F. (2021). End-to-end quantized training via log-barrier extensions.
Li, R., Wang, Y., Liang, F., Qin, H., Yan, J., and Fan, R. (2019). Fully quantized network for object detection. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 2810-2815 Computer Vision Foundation IEEE.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 16692.
Loshchilov, I. and Hutter, F. (2018). Fixing weight decay regularization in adam.
Macháček, M. and Bojar, O. (2014). Results of the wmt14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293-301.
Mellempudi, N., Srinivasan, S., Das. D., and Kaul, B. (2019), Mixed precision training with 8-bit floating point. CoRR, abs/1905.12334
Micikevicius, P. Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Hous- ton, M., Kuchaiev, O., Venkatesh, G., et al. (2017). Mixed precision training. arXiv preprint arXiv:1710.03740.
Nagel, S. (2016). Co-news.
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng. N., Grangier, D., and Auli, M. (2019). fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint
Ott, M., Edunov, S., Grangier, D., and Auli, M. (2018). Scaling neural machine translation. arXiv preprint 18X6v
Pappalardo, A. (2021). Xilinx/brevvtas.
Pudipeddi. B., Mesmakhosroshahi, M., Xi, J., and Bharadwaj. S. (2020). Training large neural net- works with constant memory using a new execution algorithm. arXiv preprint 200::200020054455
Qian, N. (1999). On the momentum term in gradient descent learning algorithms. Neural networks the official journal of the International Neural Network Society, 1:145-151.
Qin, H., Gong, R., Liu, X., Bai, X., Song. J., and Sebe, N. (2020). Binary neural networks: A survey. CoRR, abs/2004.00333.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAl blog, 1(8)99
Raffel, C., Shazeer, N., Roberts, A., Lee. K.. Narang, S., Matena, M.. Zhou. Y., Li, W., and Liu, J. (2019), Exploring the limits transfer learning with a unified text-to-text transformer. arXiv preprint
12

---

Under review as a conference paper at ICLR 2022
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. (2020). Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEE
Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., and He, Y. (2021). Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. arXiv preprint arXiv:2104.0755
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021). Zero-shot text-to-image generation. arXiv preprint arXiv:2 12092.
Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. (2016). Xnor-net:
Imagenet classification using binary convolutional neural networks. In Leibe, B., Matas, J., Sebe, N., and Welling, M., editors, Computer Vision ECCV 2016 14th European Conference, Amstenlam, The Nether- lunds, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pages 525-542. Springer.
Sennrich, R., Haddow, B., and Birch, A. (2015). Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07530
Sennrich, R., Haddow, B., and Birch, A. (2016). Edinburgh neural machine translation systems for wmt 16. arXiv preprint arX1v.1606.02891.
Shazeer, N. and Stern, M. (2018). Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596-4604, PMLR.
Shoeybi. M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. (2019). Megatron- Im: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.0800 08055..0.3...
Sun. X., Choi, J., Chen, C.. Wang, N., Venkataramani, S., Srinivasan, V., Cui, X., Zhang, W., and Gopalakrishnan, (2019). Hybrid 8-bit floating point (HFP8) training and inference for deep neural networks. In Wallach, H. M.. Larochelle, H., Beygelzimer, A., d' d'Alché-Bie Buc, F., Fox, E. B., and Garnett, R., editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 4901-4910.
Trinh, T. H. and Le, Q. V. (2018). A simple method for commonsense reasoning. arXiv preprint arXiv:1806.0222 02847.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, I.,, Gomez, A. N., Kaiser, L.., and Polo- sukhin, I. (2017). Attention is all you need. arXiv preprint 1et6.06666..222
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. (2018a). Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
Wang, N., Choi, J., Brand, D., Chen, C., and Gopalakrishnan, K. (2018b). Training deep neural networks with 8-bit floating point numbers. In Bengio, S., Wallach, H. M., Larochelle, H., Grau- man, K., Disa-Bianchi, N., and Gameet, R., editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 7686-7695.
Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., and Grave, E. (2020). CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceed- ings of the 12th Language Resources and Evaluation Conference, pages 4003-4012, Marseille, France. European Language Resources Association.
Zhu, C., Han, S., Mao, H., and Dally, W. J. (2017). Trained terarry quantization. In 5th Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Zhu, Y., Kiros, R., Zemel, R.. Salakhutdinov, R.. Urtasun, R., Torralba, A., and Fidler, S. (2015). Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19-27.
13

---

Under review as a conference paper at ICLR 2022
A GLUE SCORE BREAKDOWN
Table contains the breakdown of individual scores on the GLUE datasets.
Table Breakdown of GLUE scores. Each column is the median of 10 random seeds. The mean the mean over medians.
Model
MNLI
QNLI
QQP
RTE
SST-2
MRPC
CoLA
STS-B
Mean
32-bit Adam
90.40
94.85
92.2
84.5
96.40
90.1
67.41
93.03
88.61
32-bit Adafactor
90.35
94.70
92.2
85.4
96.45
90.0
67.63
92.91
88.71
8-bit Adam
90.30
94.70
92.2
85.9
96.40
90.3
67.20
92.87
88.73
B STABILITY OF EMBEDDING LAYERS
Highly variable gradients lead to unpredictable optimization behavior and instability that man- ifests as divergence or exploding gradients. Low precision optimziers can amplify variance of gra- dient updates due to the noise introduced during quantization. While our 8-bit optimizers appear be stable for convolutional networks, similar to Ramesh et al. (2021), we find that word embedding layers are major source of instability.
The main instability from the word embedding layer comes from the fact that it is a sparse layer with non-uniform distribution of inputs which can produce maximum gradient magnitudes 100x larger than other layers. For dense layers, if given n samples arranged into l mini-batcher the sum gradients of all mini-batches always the same independent of how the samples are arranged into k mini batces. For embedding gradients, this depends on the arrangement of samples into mini-batches, This because most deep learning frameworks normalize number of total tokens in the mini-batch, rather than the frequency of each individual token. This approximation allows stable learning with a single learning rate rather than variable learning rates that depend on token frequency in each individual mini-batch. However a side-effect of this method that the magnitude of gradients for particular token can vary widely with batch sizes and between different mini-batches.
gradient by the
There are multiple recipes for initialization word embedding layers. One of the most common recipes used in all models trained with fairseq (Ott et al.] 2019) such as RoBERTa (Liu al. 2019 BART (Lewis et al. 2020), large NMT models (Ott et al. (018), and sparse expert models (Lewis
et al. 2021), is the following: Initialize the word embedding layer with N 1/VK) where k is the embedding size of the embedding layer and to scale the outputs by vk. This scheme has a variance of one the start of training for the output distribution to ensure good gradient flow.
We find this approach to induce some instability for 8-bit optimizers. We develop the stable embed- ding layer solve this instability problem.
While the full recipe for our stable embedding layer is new, components of it has been used before. The layer norm after the embedding has been used before work such as Devlin et al. (2019) and Radford et al. (209)) and enhanced precision for this particular layer was used in Ramesh et al. (2021). As pointed out above, these elements are not standard and the stable embedding layer combines three aspects that are all important: (1) enhanced precision, (2) layer norm, and (3) Xavier initialization.
C QUANTIZATION ERROR ANALYSIS
To gain more insights into why block-wise dynamic quantization works well and how it could be improved, we performed a quantization error analysis of Adam quantization errors during language model training. Adam quantization emrors the deviations between the quantized 8-bit Adam update and the 32-bit Adam updates: [ug - utel where uy = /s/ for & bits. See Background Section I. for details on Adam.
14

---

Under review as a conference paper at ICLR 2022
A good 8-bit quantization has the property that, for given input distribution, the inputs only rarely quantized into intervals with high quantization error and most often quantized into intervals with low enror.
In 8-bit, there are 255 x256 possible 8-bit Adam updates, 256 possible values for the first and 256 for the second Adam state. We look at the average quantization enror of each of these possible updates to see where the largest errors are and we plot histograms to see how often do these values with high error occur. Taken together, these two perspectives give a detailed view of the magnitude of deviations and how often large deviations occur.
We study these questions by looking at how often each of the 256 values for both Adam states are used during language model training. We also analyze the average eπrоr for each of the inputs quantized to each of the 256 values. With this analysis it easy to find regions of high use and high error, and visualize their overlap. An overlap of these regions is associated with large frequent errors that cause unstable training. The quantization error analysis is shown Figure 4
The plots show two things: (1) The region of high usage (histogram) shows how often each com- bination of 256x256 bit values used for the first Adam state $1 (exponentially smoothed running sum) and the second Adam state (exponentially smoothed running squared sum). (2) The error plots show for k-bit Adam updates uk = $1/(v5 + e) the mean absolute Adam error |u32 ugl and the relative Adam error Ju32 - usl/|u32 averaged over each bit combination. In conjunction these plots show which bits have the highest error per use and how often each bit is used. The x- axis/y-axis represents the quantization type range which means the largest positive/negative Adam states per block/tensor take the values 1.0/-1.0.
We see that block-wit wise dynamic quantization has the smallest overlap between regions of high use and high error. While the absolute Adam quantization error of block-wise dynamic quantization is 0.0061, which is not much lower than that of dynamic quantization with 0.0067, the plots also be interpreted as block-wise dynamic having rarer large errors that likely contribute to improved stability during optimization.
D FINE-GRAINED OPTIMIZER RUNTIME PERFORMANCE
Table (iolesssswws optimizer performance that is benchmarked in isolation without any training. We use a large sample of normal distribution and benchmark the average time to perform 100 optimizer updates per billion parameters in milliseconds.
Runtime performance of 8-bit optimizers vs commonly used 32-bit optimizers in millisce- onds per update per IB parameters for 32-bit gradients. This comparision was run V V100 GPU,
Table 5:
Milliseconds per update per 1B param
Optimizer
32-bit PyTorch 32-bit Apex
8-bit (Ours)
Adam
145
63
47
Momentum
58
46
34
LAMB
91
65
LARS
119
43
E ADDITIONAL QUANTIZATION DATA TYPES
This section describes additional quantization data types that we tried but which we found to per- form poorly in quantization performance or stability. While quantile quantization has an average quantization twice low dynamic quantization for any normal distribution it has sporadic large errors that lead to large Adam errors and poor model performance (see Figure 5) and even with -fate-of-the-ar quantile estimation algorithms (see Section F) quantile quantization is too slow to be practical. An overview of quantization performance of this additional quantization data types compared to dynamic quantization (without block-wise quantization) can be found in Table
15

---

Under review as a conference paper at ICLR 2022
Linear
Dynamic
Dynamic Block-wise
Regions High
150000
1st Adam State
Adom Sate
Usage
23008
23008
2nd Adam State
2nd Adam State
2nd Ademi State
Regions of High Relative Error
1st Adam State
122 Ad3m State
1st Adtmm
2nd Adam State
2nd Adamm State
2nd Adamm State
-1.0
-1.0
Regions of High
Absolute Error
1st Adium State
1st Adban State
0.0
1.0
0.8
100
2nd Adam State
2nd Adam State
2nd Adam Male
Figure 4: Good quantization methods do not have overlaps between regions of high use and high error. The plot shows that for linear quantization regions of high usage and high error overlap. For dynamic quantization regions with high relative eтrоr are used infrequently while only small regions have high usage and high absolute error. Block-wise dynamic quantization spreads out the usage large space and has the lowest overlap between regions of high use and errors. This means that not only the overall error of bliek-wise dynamic quantization lower, but also that large errors for individual parameter updates are rarer compared other methods, thus improving stability. See the main text for more details.
Table 6: Mean relative Adam and absolute quantization error for the first Adam state for different quantization methods. Results show standard error. We can that Dynamic Quantization has best relative error and that both Dynamic methods have the best absolute error.
Method
Relative Adam Error
Absolute Quantization Error
Linear
201% #17%
41.2e- 10-3..ee10
Quantile
11.9% + 0.3%
1.8e-10:0.9e-- 4.6e- 0.44e100
Inverse Dynamic
6.5% 0. 0.1%
Dynamic
4.8 % + 0.4%
1.51-10++
16

---

Under review as a conference paper at ICLR 2022
Quantile
Dynamic
0.05
0.04
Proportion Adam
0.03
0.01
0.00
-0.5
0.5
Relafive Value
Figure 5: Distribution of Adam error among each the 256 8-bit values of the first Adam state. We normalize the values into the range [-1,1]. With this, -1 indicates the largest negative value, the value that is closest to O, and so forth. See Figure for a visualization of this normalization. Quantile quantization has large errors for large values, while dynamic quantization has small erross for both small and large values while the bulk of the errors is concentrated in intermediate values.
E. INVERSE DYNAMIC QUANTIZATION
Inverse Dynamic Quantization is motivated by the hypothesis that large Adam updates are more important than small updates. Since Adam is composed of ratio of optimizer states m,/( +c), we expect that small values the second state r; produce large Adam updates. To get better quantization error for small values we can switch the dynamic exponent and the base exponent. For regular dynamic quantization the base exponent 10" and each zero decreases the exponent by factor of 10 for minimum value of 10-7 We invert this starting with base 10-7 and each zero bit increases the exponent by 10 for a maximum value of We denote this quantization as inverse dynamic quantization.
E.2 QUANTILE QUANTIZATION: LOSSY MINIMUM ENTROPY ENCODING
lossy minimum entropy encoding with & bits has the property that for any input data, the quantized outputs take the value of each of the 2* different bit representations equally often.
More formally, a lossy minimum entropy encoding can be described in the following way. Given an infinite stream of sampled real numbers x; where is distributed as X, an arbitrary probability distribution, lossy minimum entropy encoding is given by the k-bit quantization map Qmap R2* which maps values E 2²² to indices 0, 1, .... 2 which has the property that if any number of elements x; from the stream quantized to 2 we do not gain any information which is predictive of future 2
One way to fulfill this property for arbitrary probability distributions X, is divide the probability distribution function fx into 2h bins where each bin has equal area and the mid- mid-points these bins values of the guantization map mmp,, Empirically, this is equivalent to histogram with 2* bins where each bin contains equal number of values.
How do find the mid-points for each histogram bin? This is equivalent to finding the 2 overlapping values x for the cumulative distribution function Fx with equal probability mass. These values most easily be found by using its inverse function, the quantile function Qx Fx We can find the mid-points of each of the histogram bins by using the mid-points between 2 + equally
17

---

Under review as a conference paper at ICLR 2022
spaced quantiles over the range of probabilities [0, 1]:
Qx
+ Qx
++1 2
2
(5)
To find q empirically, we can estimate sample quantiles for tensor T with unknown distribution X by finding the 2 equally spaced sample quantiles via T's empirical cumulative distribution function. We refer to this quantization as quantile quantization.
To estimate sample quantiles efficiently, we devise a specialized approximate quantile estimation algorithm, SRAM-Quantless, which is more than 75x faster than other approximate quantile esti- mation approaches (Govindaraju et al. 20055] Dunning and Ertl 2019). SRAM-Quantiles uses a divide-and-conquer strategy perform sorting solely in fast SRAM. More details on this algorithm can be found in the Appendix Section
E.3 VISUALIZATION: DYNAMIC VS LINEAR QUANTIZATION VS QUANTILE QUANTIZATION
Figure G shows the mapping from each to the 255 values of the 8-bit data types their value normalized in the range [-1, 1]. We can see that most bits in dynamic quantization are allocated for large and small values. Quantile quantization is introduced in Appendix E.2
1.00
Data Type
linear
0.75
dynamic
quantile
0.50
Normalized Value
0.25
-0.25
-0.50
00.5
-1.00
50
100
150
200
250
Index
Figure 6: Visualization of the quantization maps for the linear, dynamic and quantile quantization. For quantile quantization we use values from the standard normal distribution and normalize them into the range [-1, 1].
F SRAM-QUANTILES: A FAST QUANTILE ESTIMATION ALGORITHM
To estimate sample quantiles of tensor one needs to determine the empirical cumulative distribution function (eCDF) of that tensor. The easiest way to find the eCDF is to sort given tensor. Once sorted, the quantiles can be found by using the value index q where i is the index into the sorted array, is the desired quantile and n is the total elements in the tensor. While simple, this process of estimating quantiles is computationally expensive and would render training with quantile quantization too slow to be useful.
Similar to other quantile estimation approaches, our GPU algorithm, SRAM--uantiess, uses a slid- ing windows over the data for fast, approximate quantile estimation with minimal resources. Green- wald and Khanna (2001)'s 's quantile estimation algorithm uses dynamic bin histograms over sliding windows to estimate quantiles. Extensions of this algorithm accelerate estimation by using more ef- ficient data structures and estimation algorithms (Dunning and Ertl, 2019) or by using GPUs (Govin-) is that only compute limit daraju al. 2005). The main difference between this work
18

---

Under review as a conference paper at ICLR 2022
set of quantiles that are known priori 256, to be exact - while previous work focuses general statistics which help to produce any quantile a posteriori. Thus we can devise a highly specialized algorithm which offers faster estimation.
The idea behind our algorithm comes from the fact that sorting is slow because it involves repeated loads and stores from main memory (DRAM) when executing divide-and-conquer sorting algo- rithms. We can significantly improve performance of quantile estimation if we restructure quantile estimation to respect memory hierarchies of the device on which the algorithm is executed.
On a GPU, programmable SRAM - known as shared memory - is 15x faster than DRAM but has limit size of around 64 kb per core. The BRAM--uantile the full eCDF we find the eCDF for subset of values of the tensor that fits into SRAM (about 4096 32-bit values). Once we found the quantiles for each subset, we average the quantiles atomically in DRAM.
algorithm is simple. Instead of finding
This algorithm works, because the arithmetic mean an unbiased estimator for the population mean and samples quantiles estimated via eCDFs are asymptotically unbiased estimators of the population quantile (Chen and Kelton 2001]. Thus the more subset quantiles we average, the better the estimate of the tensor-wide widd quantiles.
For estimating 256 quantiles on a large stream of numbers, our algorithm takes on average 0.064 ns to process one element in the stream, whereas the fastest general algorithms take 300 ns (Govin- daraju et al. 2005) and 5 ns (Dunning and Ert] 2019).
G ADAGRAD COMPARISONS
While the main aim in this work is to investigate how the most commonly used optimizers, such as Adam (Kingma and Ba 2014) and Momentum (Qian 1999), can be used as 8-bit variants without any further hyperparameter tuning, it can be of interest to consider the behavior of our 8-bit methods under different scenarios. For example, difference between Adam/Momentum and AdaGrad (Duchi et al. 2011) is that AdaGrad accumulates gradients statistics over the entire course of training while Adam/ Momentum very different 8-bit quantization behavior where there are large difference between the magnitude of different optimizer states. Such large differences could induce a large quantization error and degrade performance of 8-bit optimizers.
smoothed exponential decay over time. As such, this could lead to
To investigate this, we train small 209M parameter language models on the RoBERTa corpus (Liu et al. 2019). We use the AdaGrad hyperparameters introduced by Keskar et al. (019). Results are shown in T From the results we can see that our 8-bit methods do not work as well for Ada- Grad. One hypothesis is that this is due to the the wide range of gradient statistics of AdaGrad which comes from averaging the gradient over the entire course of training. To prevent poor quantization in such scenarios, stochastic rounding proved to be very effecttve from our initial experiments with other 8-bit optimizer. While we abandoned stochastic rounding because we did not see any benefits for Adam and Momentum, it could be effective solution for AdaGrad. We leave such improved 8-bit quantization methods for AdaGrad to future work.
Table 7: AdaGrad compared to Adam performance for a 209M parameter language model on the RoBERTa corpus. The 8-bit methods stable embedding layer. AdaGrad hyperparamters taken from (eskaar et al. 2019
Optimizer
Valid Perplexity
32-bit Adam
16.7
8-bit Adam
16.4
32-bit AdaGrad
19.4
8-bit AdaGrad
19.7
19


Reviews:
[{"review_id": "fuLheAd9jKJ", "reviewer": "Reviewer_QSRR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n- The authors conducted a wide range of experiments on diverse NLP and computer vision tasks performing consistent memory footprint saving without performance degradation. \n- The proposed method is stand-alone and potentially can be applied in parallel with other compression techniques such as weight, activation quantization, and pruning. \n\n**Cons:**\n- The proposed method is beneficial more to neural networks with a high amount of the parameters proportionally to the activations thus for convolutional neural networks the memory saving ratio is much smaller rather for transformer-based NNs.\n- There are several broken references that should be fixed", "summary_of_the_paper": "This paper addresses the very important problem of reducing the memory footprint of neural networks training. For that matter, the authors propose replacing standard optimizers with their 8-bit quantize counterparts. \nThe proposed scheme of optimizer state quantization has three components (I) block-wise quantization which osolat4ed outliers impact on the error (ii) dynamic quantization which quantize both small and large values with high precision and (iii) stable embedding layer which improves the stability of the optimizer during training. ", "summary_of_the_review": "The proposed paper proposed 8-bit quantized optimizers counterparts for saving memory footprints during training. \nEmpirical results show the proposed method's efficiency over a wide range of tasks and architectures. \nI believe that the proposed method introduces an important contribution towards the efficient training of NNs.\nHence I vote for its acceptance in a pre-rebuttal phase.\n***********\nPost-rebuttal discussion: \nI have fully satisfied with the author's feedback and revision version of the paper according to my and other reviewers' concerns. \nMy score remains \"8\" and I am convinced that this manuscript would significantly contribute to ML community, hence I vote for his acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "O2d1TdTc3d", "reviewer": "Reviewer_zhGJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper shows very impressive empirical results and has an open-sourced codebase which makes it reproducible. Memory saving and time-saving are especially impressive. These empirical results themselves are very valuable to the community given there are not many open-sourced codebases, to begin with.  \n\nThe methodology of dynamic (tree) quantization and stable embedding are reasonable but not surprising to the quantization research community. The authors modified the dynamic tree quantization, which also appears to be reasonably motivated. Appendix E seems to support the argument despite there being few places where strong implicit assumptions were made:  mean of gradient zero over time, and constant variance. These should be thoroughly discussed in the appendix as well. \n\nQuesion:\n1.  Why image tasks do not have as much saving as NLP tasks? In table 1, Image related tasks's savings are marginal versus NLP related tasks. Is there a way to make this work for Image, speech task as well?\n\nThere are a few minor points that do not affect my score:\n- Several broken reference links throughout the text with ??\n- Many attempts have been made on quantizing the optimizer states from 32-bit to 8-bit using non-linear quantization, but just not as successful maybe. E.g. <Sun et al. 2019, HFP8> <Pappalardo 2021, Brevitas><Li et al. 2020 End-to-end Quantized Training via Log-Barrier Extensions>  can do such jobs as well.\n", "summary_of_the_paper": "This paper proposes a non-linear block-wise quantization method to reduce the memory overhead of stateful optimizers, without sacrificing performance going from 32bits to 8 bit. The authors combine block-wise quantization with 2 methods to stabilize training: dynamic tree quantization and a stable embedding layer. Results on WMT, GLUE and Moco show the effectiveness of the method.\n", "summary_of_the_review": "This paper has very valuable shared resources, and it's worthwhile for the research community to notice.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "AujXwsKEuE", "reviewer": "Reviewer_6PcJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "```Updated Score```: See my comments why: https://openreview.net/forum?id=shpkpVXzo3h&noteId=PWzRo82xR07 and hopefully authors address the additional comments in the final version.\n\nPaper proposes block-wise quantization (dynamic) to reduce the states (momentum and second moment) in diagonal first order optimizers and successfully implements a 8-bit Adam implementation that works as well as its f32 variant while being memory efficient. Paper further identifies embedding layers as a source of instability and proposes layer norm to improve stability,  and leaves optimizer states in f32 for this layer.\n\nStrengths:\n+ Efficient 8bit implementation of Adam on GPUs\n+ Careful work to reduce quantization errors, and improvement to existing algorithms (for example quantile estimation) \n+ Blockwise quantization is very neat way to deal with outliers\n\nWeakness/Improvements:\n+ Lack of comparison to previously established SOTA for low memory optimization (See comments on AdaFactor)\n+ Results are at similarish (lower) batch sizes. It would have been interesting to see the method is applicable for large batch training. One could conjecture that heavy tailed nature noise (https://arxiv.org/abs/1912.03194) and its associated effects on the diagonal preconditioner might make quantization harder.\n+ Ignores AdaGrad line of work\n\n==AdaFactor comparisons== \n\nAdaFactor has the option to reduce the memory used by momentum states completely by replacing it with the cheaper adaptive gradient clipping, and has been used for large model training (see https://arxiv.org/pdf/2006.16668.pdf for hyper-parameters. (See beta1=0))\n[1] https://github.com/tensorflow/lingvo/blob/master/lingvo/core/optimizer.py#L1044 \n\nThe paper includes the f32 variant, but do not include the bfloat16 variant [1] comparison\n[2] https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/adafactor.py See parameter encoding.\n\n== AdaGrad and its variants == \nPaper currently misses citation of the entire AdaGrad-line of work, and comparision/implementation, I hope the authors can address these easily.\n\n[1] https://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf \n[2] Memory efficient adaptive optimization https://proceedings.neurips.cc/paper/2019/file/8f1fa0193ca2b5d2fa0695827d8270e9-Paper.pdf\n[3] https://openreview.net/pdf?id=SklKcRNYDH (ICLR 2020)\n\nDoes the proposed block quantization work well for AdaGrad variants as well? Reason to ask is, after reading the paper, it's not entirely clear if the proposed approach works for AdaGrad, SM3 and Extreme Tensoring as it accumulates statistics over the entire horizon and has a very different range of values (distribution of values) than Adam's second-moment statistics.\n\nIt would be valuable to the community if authors could address this in the paper by demonstrating improvements or showing negative results and challenges to this end.  For example, in this work authors leave compressing states for embedding layers (with layernorm) as future work.\n\n== Some related citations for block wise quantization ==\nFor weights of a network; similar idea of block-wise quantization (but for binarization) has been employed in \"Improving Bi-Real Net with block-wise quantization and multiple-steps binarization on activation\", Duy H. Le; Tuan V. Pham\n\n", "summary_of_the_paper": "The paper shows a working implementation of 8bit states for momentum and second momentum. This is achieved by using block-wise dynamic quantization for efficient compression and fast implementation. They show drastic improvement over f32 diagonal optimizers (mainly Adam) and improvement over a subset of previously proposed sub-linear memory optimizer (AdaFactor f32 variant).", "summary_of_the_review": "Authors propose 8-bit optimizers for training neural networks, and demonstrated its usefulness in one regime. There are several important ablations (mentioned in the reviews, and I will revise the score accordingly) that are missing which are important it ascertain its wide utility, as well understand its limitations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes Adam and Momentum optimizers, where the optimizer state variables are quantized to 8bit using block dynamics quantization. These modifications significantly improve the memory requirements of training models with many parameters (mainly, NLP models). These are useful contributions which will enable training even larger models than possible today. All reviewers were positive.
Decision: Accept (Spotlight)