GPT3.int8(): 8-bit Matrix Multiplication for Transformers at Scale
Authors: Tim Dettmers
Mike Lewis
Younes Belkada
Luke Zettlemoyer
Venue: NeurIPS 2022 Accept (2022)
Keywords: quantization
8-bit
transformers
inference

Abstract: Large language models have been widely adopted but require significant GPU memory for inference. We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for inference by half while retaining full precision performance. With our method, a 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation. This is made possible by understanding and working around properties of highly systematic emergent features in transformer language models that dominate attention and transformer predictive performance. To cope with these features, we develop a two-part quantization procedure, {\bf LLM.int8()}. We first use vector-wise quantization with separate normalization constants for each inner product in the matrix multiplication, to quantize most of the features. However, for the emergent outliers, we also include a new mixed-precision decomposition scheme, which isolates the outlier feature dimensions into a 16-bit matrix multiplication while still more than 99.9\% of values are multiplied in 8-bit. Using LLM.int8(), we show empirically it is possible to perform inference in LLMs with up to 175B parameters without any performance degradation. This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open source our software.

LLM int8 (): 8-bit Matrix Multiplication for Transformers at Scale
Tim Detimers
Mike Lewis
Younes Belkada
Luke Zettlemoyer
University of Washington
Facebook AI Research*
Hugging F
ENS Paris-Saclay*
Abstract
Large language models have been widely adopted but require significant GPU memory for inference. We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for inference by half while retaining full precision performance. With our method, 175B parameter 16/32-bit checkpoint be loaded, converted to Int8, and used immediately without performance degradation. This is made possible by understanding and working around properties of highly systematic emergent features in transformer language models that dominate attention and transformer predictive performance. To cope with these features, develop two-part quanti- zation procedure, LLM.int80. We first use vector-wise quantization with separate normalization constants for each inner product in the matrix multiplication, quantize most of the features. However, for the emergent outliers, we also include mixed-precision precision decomposition scheme, which isolates the outlier feature new mixed-
dimensions into 16- bit matrix multiplication while still more than 99.9% of values are multiplied 8-bit. Using I.MMMint8), we show empirically it possible perform inference in LLMs with up to 175B parameters without any performance degradation. This result makes such models much more accessible, for example making it possible use OPT- 175//BLOOM on single server with consumer GPUs. We open source our software.
Introduction
Large pretrained language models are widely adopted in NLP (Vaswani al., 2017: Radford et al., 2019; Brown et al., 2020; Zhang et al., 2022) but require significant memory for inference. For large transformer language models at and beyond 6.7B parameters, the feed-forward and attention projection layers and their matrix multiplication operations are responsible for 95%? of consumed parameters and 65-85% all computation (Ilharco et al., 2020). One way to reduce the size of the parameters is to quantize them to less bits and use low-bit-pre pirepciiin matrix multiplication. With this goal in mind, 8-bit quantization methods for transformers have been developed (Chen al., 2020; Lin et al., 2020; Zafrir et al., 2019; Shen et al., 2020). While these methods reduce memory use, they degrade performance, usually require tuning quantization further after training, and have only been studied for models with less than 350M parameters. Degradation-free quantization up to 350M parameters is poorly understood, and multi-billion parameter quantization remains an open challenge.
Majority of research done visiting researcher Faccbook Al Research.
*Other parameters come mostly from the embedding layer. tiny amount comes from norms and biases.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In this paper, we present the first multi-billion-seat Int8 quantization procedure for transformers that does not incur any performance degradation. Our procedure makes possible load 175B parame- ter transformer with 16 or 32-bit weights, convert the feed-forwan and attention projection layers to 8-bit, and use the resulting model immediately for inference without any performance degradation. We achieve this result by solving two key challenges: the need for higher quantization precision at scales beyond IB parameters and the need to explicitly represent the sparse but systematic large magnitude outlier features that ruin quantization precision once they emerge in all transformer layers starting at scales of 6.7B parameters. This loss of precision is reflected in C4 evaluation perplexity (Section 3) as well as zeroshot accuracy
these outlier features emerge, shown in Figure 1.
We show that with the first part of our method, vector-wise quantization,
Method
possible to retain performance at scales up to 2.7B parameters.
B-be baselne
16-bi baseline
For vector-wise quantization, matrix multiplica- tion can be seen as a sequence of independent inner products of row and column vectors. As such, we can use separate quantization nor- malization constant for each inner product to improve quantization precision. We can recover the output of the matrix multiplication by de- normalizing by the outer product of column and row normalization constants hefore we perform the next operation.
Mean zerostot accuracy
0.5
emergence of outlier features
1238
3508
1.38
2.78
0.78
318
648
1758
To scale beyond 6.7B parameters without perfor- mance degradation, it is critical to understand the emergence of extreme outliers in the feature dimensions of the hidden states during infer- ence. To this end, we provide a new descriptive analysis which shows that large features with magnitudes up to 20x larger than in other di- mensions first appear in about 25% of all trans- former layers and then gradually spread to other layers we scale transformers to 6B parame- ters. At around 6. 6.7B parameters, a phase shift occurs, and all transformer layers and 75% of all sequence dimensions are affected by extreme magnitude features. These outliers are highly systematic: at the 6.7B scale, 150,000 outliers occur per sequence. but they are concentrated in only 6 feature dimensions across the entire transformer. Setting these outlier feature dimensions to zero decreases top-1 attention softmax probability mass by more than 20% and degrades validation perplexity by 600-1000% despite them only making up about 0.1 0.1% of all input features. In contrast, removing the same amount of random features decreases the probability by a maximum of 0.3% and degrades perplexity by about 0.1%.
Poromaters
Figure 1: OPT model mean zeroshot accuracy for WinoGrande, HellaSwag, PIQA, and LAMBADA datasets. Shown is the 16-bit baseline, the most precise previous 8-bit quantization method as a baseline, and our new 8-bit quantization method, LLM.int80.. We can see once systematic outliers occur scale of 6. 6.7B parameters, regular quan- tization methods fail, while LLM.int8() maintains 16-bit accuracy.
To support effective quantization with such extreme outliers, we develop mixed-precision decomposi- tion, the second part of our method. We perform 16-bit matrix multiplication for the outlier feature dimensions and 8-bit matrix multiplication for the other 99.9% of the dimensions. We name the combination of vector-wise quantization and mixed precision decomposition. LLM.int80. We show that by using LLM.innt0). we can perform inference in LLMs with up to 175B parameters without any performance degradation. Our method not only provides new insights into the effects of these outliers on model performance but also makes it possible for the first time to use very large models, on a single server with consumer GPUs. While our work focuses
for example, GPT-175B/BLOOO on making large language models accessible without degradation, we also show in Appendix D that we maintain end-to-end inference runtime performance for large models, such as BLOOM-176B and provide modest matrix multiplication speedups for GPT-3 models of size 6.7B parameters or larger. We open-source our software" and release a Hugging Face Transformers (Wolf et al., 2019) integration making our method available to all hosted Hugging Face Models that have linear layers.
htttp::/gggbbbocc/////ttttttttt

---

LLM.int8()
8-bit Vector-wise Quantization
(1) Find vectar-wise constants: C Cx
(2) Quantae
(4) Dequantie
(1 X
Our (c,0C Out
Cw
W 1217//// W
127-127
2
(3) Int8 Metmul
X
W
716
W
x V, Out,,
7736
16-bit Decomposition
(1) Decompose outiiers
(2) FPI6 Matmul
W Out
Out
Regular values
X
Outliers
P2B
Figure 2: Schematic of LLM.int80. Given 16-bit foating-poing inputs Xy16 and weights W f16. the features and weights are decomposed into sub-matrices of large magnitude features and other values. The outlier feature matrices are multiplied in 16-bit. All other values are multiplied in 8-bit. We perform 8-bit vector-wise multiplication by scaling by row and column- absolute maximum of C2 and C and then quantizing the outputs Int8. The Int32 matrix multiplication outputs Out are dequantization by the outer product of the normalization constants C2 & Cw. Finally, both outlier and regular outputs are accumulated in 16-bit floating point outputs.
Background
In this work, push quantization techniques to their breaking point by scaling transformer models. We are interested in two questions: at which scale and why do quantization techniques fail and how does this related quantization precision? To answer these questions study high-precision asymmetric quantization (zeropoint quantization) and symmetric quantization (absolute maximum quantization). While zeropoint quantization offers high precision by using the full bit-range of the datatype, is rarely used due to practical constraints. Absolute maximum quantization is the most commonly used technique.
2.1 8-bit Data Types and Quantization
Absmax quantization scales inputs into the 8-bit range [-127, 127] by multiplying with Sæylo which 127 divided by the absolute maximum of the entire tensor. This is equivalent dividing by the infinity norm and multiplying by 127. As such, for an FP16 imput matrix X X,16 E Iaax 1nt8 absmax quantization is given by:
127 X f16
127
X
max X f16.,
[X
X,16
=
Sxyro (errixX161 X/16
where [] indicates rounding to the nearest integer.
Zeropoint quantization shifts the input distribution into the full range [-127, 127 by scaling with the normalized dynamic range ndz and then shifting by the zeropoint 2p.- With this affine transformation, any input tensors will use all bits of the data type, thus reducing the quantization error for asymmetric distributions. For example, for ReLU outputs, in absmax quantization all values in [-127, 0) go unused, whereas in zeropoint quantization the full [-127, 127] range used. Zeropoint quantization is given by the following equations:
1.7
nd
(I)
1 max
f16) mint X
f16)
2Pznc
X X min(X
( /
(2)
X
Indzy X X
(3)

---

To use zeropoint quantization in an operation we feed both the tensor X and the zeropoint 2p into special instruction" which adds 2Pr.1 to each element of X,s before performing a 16-bit integer operation. For example, to multiply two zeropoint quantized numbers Ass and Big along with their zeropoints ZPanc and 2Pb2ic we calculate:
Css2 multiply. cnn (Asparn Bep = (Ais 2Pa.io ) + 2Pb.1e
(4)
where unrolling is required if the instruction multiply,16 is not available such as GPUs or TPUs:
C,32 = Bis + Agg2pboe + 2a + zPa,e &Pb116
(5)
where Ais Bis computed with 1nt8 precision while the rest computed in Intl 1n32 precision. As such, zeropoint quantization can be slow if the mu instruction is not available. In both cases, the outputs are accumulated as a 32-bit integer C322. To dequantize C132, we divide by the scaling constants nd and ndb
Int8 Matrix Multiplication with 16-bit Float Inputs and Outputs. Given hidden states Xy1e Rx and weights W /16 hx with sequence dimension s, feature dimension h, and output dimension o we perform 8-bit matrix multiplication with 16-bit inputs and outputs as follows:
X W fiG Cf16
C232 = Sy16 C132
1 Sy16 B = Sy16 Q(A(16) Q(B/16),
(6)
Where Q(-) is either absmax zeropoint quantization and Czy.a and Cwyys are the respective tensor- wise scaling constants 3₂ and sw for absmax or ndz and rdw for zeropoint quantization.
Int8 Matrix Multiplication at Scale
The main challenge with quantization methods that single scaling constant per tensor is that a single outlier can reduce the quantization precision of all other values. As such, is desirable to have multiple scaling constants per tensor, such as block-wise constants (Dettmers al., 2022), so that the effect of that outliers is confined to each block. We improve upon one of the most common ways of blocking quantization, row-wise quantization (Khudia et al., 2021), by using vector-wise quantization, as described in more detail below.
To handle the large magnitude outlier features that occur in all transformer layers beyond the 6.7B scale, vector-wise quantization is no longer sufficient. For this purpose, we develop mixed- precision decomposition, where the small number of large magnitude feature dimensions (==0. 1 are represented in 16-bit precision while the other 99.9% of values are multiplied in 8-bit. Since most entries are still represented in low-precision, retain about 50% memory reduction compared to 16-bit. For example, for BLOOM-I76B, we reduce the memory footprint of the model by 1.96x.
Vector- wise quantization and mixed-precisioe decomposition are shown in Figure 2. The LLM.int80) method is the combination of absmax vector-wise quantization and mixed precision decomposition.
3.1 Vector-wise Quantization
One way to increase the number of scaling constants for matrix multiplication is to view matrix multiplication as a sequence of independent inner products. Given the hidden states X E bxc and weight matrix W /16 Ax0 we can assign a different scaling constant Czyle to each row of Xf16 and to each column of W f16. To dequantize, we denormalize each inner product result by 1/ (C For the whole matrix multiplication this is equivalent denormalization by the outer product Cz @ Curyg where C, E R and Cap RR. As such the full equation for matrix multiplication with row and column constants given by:
C fis
Cas2 S Cay S A = ((((116)) Q(B)1s).
Czyie
& Cwfi
which we term vector-wise quantization for matrix multiplication.

---

3.2 The Core of LLM.int8): Mixed-precision Decomposition
In our analysis, we demonstrate that a significant problem for billion-scale 8-bit transformers is that they have large magnitude features (columnss) which are important for transformer performance and require high precision quantization. However, vector-wise quantization, our best quantization technique, quantizes each row for the hidden state, which is ineffective for outlier features. Luckily, we see that these outlier features are both incredibly sparse and systematic in practice, making up only about O. I % of all feature dimensions, thus allowing to develop new decomposition technique that focuses on high precision multiplication for these particular dimensions.
We find that given input matrix X f16 R these outliers occur systematically for almost all sequence dimensions 8 but are limited to specific feature/hidden dimensions h. As such, we propose catted-precisionn decomposition for matrix multiplication where separate outlier feature dimensions into the set O = {ili E Z, 0 S S h). which contains all dimensions of h which have at least one outlier with magnitude larger than the threshold a. In our work, we find that 6.0 is sufficient to reduce transformer performance degradation close to zero. Using Einstein notation where all indices superscripts, given the weight matrix W /16 RAxo mixed-precision decomposition for matrix multiplication is defined as follows:
Σx W fic + Sy16 C
X
W 48
hEO
kgo
where Sy16 is the denormalization term for the Int8 inputs and weight matrices X,8 and Was.
This separation into 8-bit and 16-bit allows for precisreciiion multiplication of outliers while using memory-efficient matrix multiplication with 8-bit weights of more than 99.9% of values. Since the number of outlier feature dimensions is not larger than (|O) < 7) for transformers up 13B parameters, this decomposition operation only consumes about 0. % additional memory.
3.3 Experimental Setup
We measure the robustness of quantization methods scale the size of several publicly available pretrained language models up to 175B parameters. The key question is not how well a quantization method performs for particular model but the trend of how such method performs as we scale.
We two setups for our experiments. One is based on language modeling perplexity, which we find to be highly robust measure that is very sensitive to quantization degradation. We use this setup to compare different quantization baselines. Additionally, we evaluate zeroshot accuracy degradation on OPT models for range of different end tasks, where we compare our methods with a 16-bit baseline.
For the language modeling setup, et al., 2019) ranging between 125M and 13B parameters. These transformers have been pretrained on Books (Zhu al., 2015), English Wikipedia, CC-News (Nagel, 2016), Open WexText (Gokaslan and Cohen, 2019), CC-Stories (Trinh and Le, 2018), and English CC100 (Wenzek et al., 2020). For more information on how these pretrained models are trained, Artetxe et al. (2021).
dense autoregressive transformers pretrained in fairseq (Ott
To evaluate the language modeling degradation after Int8 quantization, we evaluate the perplexity of the 8-bit transformer on validation data of the C4 corpus (Raffel et al., 2019) which is subset of the Common Crawl corpus. 5 We use NVIDIA A40 GPUs for this evaluation.
To measure degradation in zeroshot performance, we use OPT models (Zhang et al., 2022), and we evaluate these models on the EleutherAl language model evaluation harness (Gao al., 2021).
3.4 Main Results
The main language modeling perplexity results on the 125M to 13B Int8 models evaluated on the C4 corpus can be seen in Table 1. We that absmax, row-wise, and zeropoint quantization fail as we scale, where models after 2.7B parameters perform worse than smaller models. Zeropoint quantization fails instead beyond 6.7B parameters. Our method, LLM.int80, is the only method that preserves perplexity, As such, LLM.int80 is the only method with a favorable scaling trend.
'nttps://commoncravu/
5

---

Table 1: C4 validation perplexities of quantization methods for different transformer sizes ranging from 125M to 13B parameters. We see that absmax, row-wise, zeropoint, and vector-wise quantization leads to significant performance degradation as we scale, particularly at the 13B mark where 8-bit 13B perplexity is worse than 8-bit 6.7B perplexity. If we use L.L.Mint8(. we recover full perplexity as we scale. Zeropoint quantization shows an advantage due to asymmetric quantization but is no longer advantageous when used with mixed-precistee decomposition.
Parameters
125M
1.3B
2.7B
6.7B
13B
32-bit Float
25.65
15.91
14.43
13.30
12.45
Int8 absmax Int8 zeropoint
87.76
16.55
15.11
14.59
19.08
56.66
16.24
14.76
13.49
13.94
Int8 absmax row-wise
30,93
17.08
15.24
14.13
16.49
Int8 absmax vector-wise
35.84
16.82
14.98
14.13
16.48
Int8 zeropoint vector-wise
25.72
15.94
14.36
13.38
13.47
Int8 absmax row-wise + decomposition
30.76
16.19
14.65
13.25
12.46
Absmax LLM.int8) (vector-wise decomp)
25.83
15.93
14.44
13.24
12.45
Zeropoint LLM.int8() (vector-wise + decomp)
25.69
15.92
14.43
13.24
12.45
When we look at the scaling trends of zeroshot performance of OPT models on the FleutherAl language model evaluation harness Figure 1, we see that intM. maintains full 16-bit perfor- mance as we scale from 125M to 175B parameters. On the other hand, the baseline, 8-bit absmax vector-wise quantization, scales poorly and degenerates into random performance.
Although our primary focus is on saving memory, we also measured the run time of LLM.int80) The quantization overhead can slow inference for models with less than 6.7B parameters, compared to FP16 baseline. However, models of 6.7B parameters or less fit most GPUs and quantization is less needed in practice. LLM.int8) run times is about two times faster for large matrix multiplications equivalent to those in 175B models. Appendix D provides more details on these experiments.
Emergent Large Magnitude Features in Transformers at Scale
As we scale transformers, outlier features with large magnitudes emerge and strongly affect all layers and their quantization. Given a hidden state X E Rexh where is the sequence/token dimension and h the hidden/feature dimension, we define a feature to be a particular dimension h;. Our analysis looks a particular feature dimension h; across all layers of a given transformer.
We find that outlier features strongly affect attention and the overall predictive performance of transformers. While up to 150k outliers exist per 2048 token sequence for 13B model, these outlier features are highly systematic and only representing at most 7 unique feature dimensions h; Insights from this analysis were critical to developing mixed-preed mixed-precision decomposition. Our analysis explains the advantages of zeropoint quantization and why they disappear with the use of mixed-precision decomposition and the quantization performance of small vs. large models.
4.1 Finding Outlier Features
The difficulty with the quantitative analysis of emergent phenomena is two-fold. We aim to select small subset of features for analysis such that the results are intelligible and not to complex while also capturing important probabilistic and structured patterns. We use empirical approach to find these constraints. We define outliers according to the following criteria: the magnitude of the feature at least 6.0, affects at least 25% of layers, and affects at least 6% of the sequence dimensions.
More formally, given a transformer with layers and hidden state X; E - , O...L where is the sequence dimension and h the feature dimension, we define a feature to be particular dimension h; any of the hidden states X1,- We track dimensions h;, < i < h, which have at least one value with magnitude of 2 6 and we only collect statistics if these outliers occur in the same feature dimension h; in at least 25% of transformer layers O. and appear in least 6% of all sequence dimensions across all hidden states X4. Since feature outliers only occur in attention projection

---

100
100
% layers affected
Percentage of layers or tokens affected
Percentage layers tokens affected
emergence of outlier features
60
20
emergence outlier features
10
12
C4 perplexity
Parameters in billions
(a)
(b)
Figure 3: Percentage of layers and all sequence dimensions affected by large magnitude outlier features across the transformer by (a) model size (b) C4 perplexity. Lines are B-spline interpolations of and 9 linear segments for (a) and (b). Once the phase shift occurs, outliers are present in all layers and in about 75% of all sequence dimensions. While (a) suggest sudden phase shift in parameter size, (b) suggests a gradual exponential phase shift as perplexity decreases. The stark shift in (a) co-occurs with the sudden degradation of performance in quantization methods.
[kcy/query/value/outttt)) and the feedforward network expansion layer (first sub-layer), we ignore the attention function and the FFN contraction layer (second sub-layer) for this analysis.
Our reasoning for these thresholds as follows. We find that using mixed- peeeed--reisionn decomposition,
perplexity degradation stops if we treat any feature with magnitude 6 or larger For the number of layers affected by outliers, we find that outlier features are systematic large models: they either occur in most layers not at all. On the other hand, they probabilistic in small models: they occur sometimes some layers for each sequence. As such, we set our threshold for how many layers need to be affected to detect outlier feature in such a way to limit detection to single outlier in our smallest model with 125M parameters. This threshold corresponds to that at least 25% of transformer layers are affected by outlier in the same feature dimension. The second most common outlier occurs in only a single layer 2% of layers), indicating that this is a reasonable threshold. We the affected by outlier features in our 125M model: outliers occur in at least 6% of sequence dimensions.
outlier feature.
procedure to find the threshold for how many sequence dimensions
We test models up to a scale of 13B parameters. To make sure that the observed phenomena not due to bugs software, we evaluate transformers that were trained three different software frameworks. We evaluate four GPT-2 models which use OpenAl software, five Meta Al models that use Fairseq (Ott et al., 2019). and one EleutherAl model GPT-J that uses Tensorflow-Me al., 2018). More details can be found in Appendix C. We also perform our analysis in two different inference software frameworks: Fairseq and Hugging Face Transformers (Wolf et al., 2019).
(Shazeer
4.2 Measuring the Effect of Outlier Features
To demonstrate that the outlier features essential for attention and predictive performance, set the outlier features to zero before feeding the hidden states X₁ into the attention projection layers and then compare the top-1 softmax probability with the regular softmax probability with outliers. We do this for all layers independently, meaning we forward the regular softmax probabilities values to avoid cascading errors and isolate the effects due the outlier features. We also report the perplexity degradation if we remove the outlier feature dimension (setting them to zero) and propagate these altered, hidden states through the transformer. As control, apply the same procedure for random non-outlier feature dimensions and note attention and perplexity degradation.
Our main quantitative results can be summarized four main points.

---

60
emergence outtler features
Median feature outlier magnitude
50
40
30
20
emergence outiler features)
30
20
15
C4 perplexity
C4 perplexity
Figure 4: The median magnitude of the largest outlier feature in (a) indicates a sudden shift in outlier size. This appears to be the prime reason why quantization methods fail after emergence. While the number of outlier feature dimensions is only roughly proportional to model size, (b) shows that the number outliers is strictly monotonic with respect perplexity across all models analyzed. Lines are B B-spline interpolations of 9 linear segments.
(1) When measured by the number of parameters, the emergence of large magnitude features across all layers of a transformer occurs suddenly between 6B and 6.7B parameters as shown in Figure 3a as the percentage of layers affected increases from 65% 100%. The number of sequence dimensions affected increases rapidly from 35% to 75%. This sudden shift co-occurs with the point where quantization begins fail.
(2) A Aternatively, when measured by perplexity, the emergence of large magnitude features across all layers of the transformer can be seen as emerging smoothly according to an exponential function of decreasing perplexity, as seen in Figure 3b. This indicates that there is nothing sudden about emergence and that we might be able detect emergent features before phase shift occurs by studying exponential trends in smaller models. This also suggests that emergence is not only about model size but about perplexity, which related to multiple additional factors such the amount of training data used, and data quality (Hoffmann et al., 2022; Henighan et al., 2020).
(3) Median outlier feature magnitude rapidly increases once outlier features of the transformer, as shown in Figure 4a. The large magnitude of outliers features and their asymmetric distribution disrupts Int8 quantization precision. This is the core reason why quantization methods fail starting at the 6.7B scale the range of the quantization distribution is too large so that most quantization bins are empty and small quantization values are quantized to zero, essentially extinguishing information. We hypothesize that besides Int8 inference, regular 16-bit floating point training becomes unstable due outliers beyond the 6.7B scale is easy to exceed the maximum 16-bit value 65535 by chance if you multiply by vectors filled with values of magnitude 60.
in all layers
(4) The number of outliers features increases strictly monotonically with respect to decreasing C4 perplexity as shown in Figure 4b, while relationship with model size is non-monotonic. This indicates that model perplexity rather than model size determines the phase shift. We hypothesize that model size is only one important covariate among many that are required to reach emergence.
These outliers features are highly systematic after the phase shift occurred. For example, for 6.7B transformer with a sequence length of 2048, we find about 150k outlier features per sequence for the entire transformer, but these features are concentrated in only 6 different hidden dimensions.
These outliers are critical for transformer performance. If the outliers are removed, the mean top-1 softmax probability is reduced from about 40% to about 20%, and validation perplexity increases by 600-1000% even though there are at most outlier feature dimensions. When we remove 7 random feature dimensions instead, the top-1 probability decreases only between 0.02-0.3%, %, and perplexity increases by 0. 0... This highlights the critical nature of these feature dimensions. Quantization tiny errors greatly impact model performance.
precision for these outlier features is paramount

---

4.3 Interpretation of Quantization Performance
Our analysis shows that outliers in particular feature dimensions are ubiquitous in large transformers. and these feature dimensions are critical for transformer performance. Since row-wise and vector- wise quantization scale each hidden state sequence dimension s (rows) and because outliers occur in the feature dimension h (columns), both methods cannot deal with these outliers effectively. This is why absmax quantization methods fail quickly after emergence.
However, almost all outliers have a strict asymmetric distribution: they are either solely positive or negative (see Appendix C). This makes zeropoint quantization particularly effective for these outliers, as zeropoint quantization is an asymmetric quantization method that scales these outliers into the full [-127, 127] range. This explains the strong performance in our quantization scaling benchmark in Table 1. However, at the 13B scale, even zeropoint quantization fails due to accumulated quantization errors and the quick growth of outlier magnitudes, as seen in Figure 4a.
If we use our full LLM.int80 method with mixed- -reeed-preiiion decomposition, the advantage of ze- ropoint quantization disappears indicating that the remaining decomposed features symmetric. However, vector wise still has an advantage over row-wise quantization, indicating that the enhanced quantization precision of the model weights is needed to retain full precision predictive performance.
5 Related work
There closely related work on quantization data types and quantization of transformers, as described below. Appendix B provides further related work on quantization of convolutional networks.
8-bit Data Types. Our work studies quantization techniques surrounding the Int8 data type, since it is currently the only 8-bit data type supported by GPUs. Other common data types are fixed point or floating point 8-bit data types (FP8). These data types usually have a sign bit and different exponent and fraction bit combinations. For example, a common variant of this data type has bits for the exponent and bits for the fraction (Wang al., 2018; Sun al., 2019; Cambier et al., 2020; Mellempudi et al., 2019) and uses either no scaling constants or zeropoint scaling. These data types have large errors for large magnitude values since they have only 2 bits for the fraction but provide high accuracy for small magnitude values. Jin et al. (2022) provide an excellent analysis of when certain fixed point exponen//fration bit widths are optimal for inputs with a particular standard deviation. We believe FP8 data types offer superior performance compared to the Int8 data type, but currently, neither GPUs nor TPUs support this data type.
Outlier Features in Language Models. Large magnitude outlier features in language models have been studied before (Timkey and van Schijndel, 2021; Bondarenko al., 2021; Wei et al., 2022; Luo et al., 2021). Previous work proved the theoretical relationship between outlier appearance in transformers and how it relates to layer normalization and the token frequency distribution (Gao et al., 2019). Similarly. Kovaleva et al. (2021) attribute the appearance of outliers in BERT model family to LayerNorm, and Puccetti et al. (2022) show empirically that outlier emergence related to the frequency of tokens in the training distribution. We extend this work further by showing how the scale of autoregressive models relates to the emergent properties of these outlier features, and showing how appropriately modeling outliers is critical to effective quantization.
Multi-billion Scale Transformer Quantization. There are two methods that were developed in parallel to ours: nuQmm (Park et al., 2022) and ZeroQuant (Yao et al., 2022). Both use the same quantization scheme: group-w2ise quantization, which has even finer quantization normalization constant granularity than vector-wise quantization. This scheme offers higher quantization precision but also requires custom CUDA kernels. Both nuQmm and ZeroQuant aim accelerate inference and reduce the memory footprint while we focus on preserving predictive performance under an 8-bit memory footprint. The largest models that nuQmm and ZeroQuant evaluate are 2.7B and 20B parameter transformers, respectively. ZeroQuant achieves -egraddaaai performance for 8- bit quantization of a 20B model. We show that our method allows for ger-deeradation quantization of models up to 176B parameters. Both nuQmm and ZeroQuant suggest that finer quantization granularity can be an effective means to quantize large models. These methods are complementary with I.L.Mint80) Another parallel work is GLM-1330 130B which uses insights from our work to achieve zero-degradation 8-bit quantization (Zeng et al., 2022). GL.M-130B performs full 16-bit precision matrix multiplication with bit weight storage.

---

6 Discussion and Limitations
We have demonstrated for the first time that multi-billion parameter transformers can be quantized to Int8 and used immediately for inference without performance degradation. We achieve this by using our insights from analyzing emergent large magnitude features at scale to develop mixed-precision decomposition to isolate outlier features in a separate 16-bit matrix multiplication. In conjunction with vector-wise quantization that yields our method, LLM.int80 which we show empirically can recover the full inference performance of models with up to 175B parameters.
The main limitation of our work is that our analysis is solely on the Int8 data type, and we do not study 8-bit Moating-point (FP8) data types. Since current GPUs and TPUs do not support this data type, we believe this is best left for future work. However, we also believe many insights from Int8 data types will directly translate FP8 data types. Another limitation is that we only study models with up to 175B parameters. While we quantize a 175B model to Int8 without performance degradation, additional emergent properties might disrupt our quantization methods at larger scales.
third limitation is that we do not use 1nt8 multiplication for the attention function. Since our focus on reducing the memory footprint and the attention function does not strictly needed. However, an initial exploration of this problem indicated that a solution required additional quantization methods beyond those we developed here, and we leave this for future work.
any parameters, it was
final limitation is that we focus on inference but do not study training or finetuning. We provide an initial analysis of Int8 finetuning and training at scale in Appendix E. 1nt8 training at scale requires complex trade-offs between quantization precision, training speed, and engineering complexity and represents a very difficult problem. We again leave this to future work.
Table 2: Different hardware setups and which methods can be run in 16-bit vs. 8-bit precision. We can see that our 8-bit method makes many models accessible that were not accessible before, in particular, OPT- 10T-1755/BLOOM.
Largest Model that can be run
Class
Hardware
GPU Memory
8-bit
16-bit
Enterprise
8x A100
80 GB
OPT-175B BLOOM
OPT-175B BLOOM
Enterprise
8x A100
40 GB
OPT-175B BLOOM
OPT-66B
Academic server
8x RTX 3090
24 GB
OPT-175B BLOOM
OРT-66B
Academic desktop
4x RTX 3090
24 GB
OPT-66B
OРT-30B
Paid Cloud
Colab Pro
GB
OPT-13B
GPT-J-6B
Free Cloud
Colab
GB
T01T5-1BB
GPT-2 1.3B
7 Broader Impacts
The main impact of our work is enabling access to large models that previously could not fit into GPU memory. This enables research and applications which were not possible before due to limited GPU memory. in particular for researchers with the least resources. See Table 3 for model/GPU combinations which now accessible without performance degradation. However, our work also enables resource-rise organizations with many GPUs to serve more models on the same number of GPUs, which might increase the disparities between resource-rich and poor organizations.
In particular, we believe that the public release of large pretrained models, for example, the recent Open Pretrained Transformers (OPT) (Zhang et al., 2022), along with our new Int8 inference for zero- and few-shot prompting, will enable new research for academic institutions that was not possible before due to resource constraints. The widespread accessibility of such large-scale models will likely have both beneficial and detrimental effects on society that are difficult to predict.
We thank Ofir Press, Gabriel Ilaarco, Daniel Jiang, Mitchell Wortsman, Ari Holtzman, Mitchell Gordon for their feedback on drafts of this work. We thank JustHeuristic (Yozh) and Titus von Köller for help with Hugging Face Transformers integration.
Acknowledgments
10

---

References
Artetxe, M., Bhosale, S., Goyal, N., Mihaylov. T., Ott, M., Shleifer, S., Lin, X. V., Du, J., Iyer, S., Pasunuru, R., al. (2021). Efficient large scale language modeling with mixtures of experts. arXiv preprint arXiv:2112.11 10684.
Bai. H., Zhang, W., Hou, L.. Shang, L., Jin, J., Jiang, X., Liu, Q. Lyu, M. R., and King, I. (2021). Binarybert: Pushing the limit of bert quantization. ArXiv, abs/2012.15701.
Bondarenko, Y., Nagel, M., and Blankevoort, TT. (2021). Understanding and overcoming the challenges of efficient transformer quantization. arXiv preprint arXiv:2109. 129.44888.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. arXiv preprint 2005.
Cambier, L., Bhiwandiwalla, A., Gong, T., Elibol, O. H., Nekuii, M., and Tang, H. (2020). Shifted and squeezed 8-bit floating point format for low- now-precision training of deep neural networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Chen, J., Gai, Y., Yao, Z., Mahoney, M. W., and Gonzalez, J. E. (2020). A statistical framework for low-bitwidth training deep neural networks. Advances Neural Information Processing Systems, 33:883-894.
Choi, J., Venkataramani, S., Srinivasan, V., Gopalakrishnan, K., Wang. Z., and Chuang, P. (2019).
Accurate and efficient 2-bit quantized neural networks. In Talwalkar, A., Smith, V., and Zaharia, M., editors, Proceedings of Machine Learning and Systems 2019, MLSys 2019, Stanford, CA, USA, March 31 April 2, 2019. mlsys.org.
Courbariaux, M. and Bengio, Y. (2016). Binarynet: Training deep neural networks with weights and activations constrained to or -1. CoRR, 0ts//602.02$35
Courbariaux, M., Bengio, Y., and David, J. (2015). Binaryconnect: Training deep neural networks with binary weights during propagations. In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., and Gareett, R., editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 3123-3131.
Courbariaux, M., Bengio, Y., and David, J.PP. (2014). Training deep neural networks with low precision multiplications. arXiv preprint arXiv:1412.7024.
Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, L.. (2022). 8-bit optimizers via bise quantization. 9th International Conference on Learning Representations, ICLR.
Devlin, J., Chang. M.-.,, Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805.
Dong, Z., Yao, Z., Gholami, A., Mahoney, M. W., and Keutzer, K. (2019). Hawq: Hessian aware quan- tization of neural networks with mixed- -reeidpreeision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 293-302.
Esser, S. K., McKinstry, J. L., Bablani, D., Appuswamy, R., and Modha, D. S. (2019), Learned step size quantization. arXiv preprint arXiv: 190200813.
Fan, A., Stock, P., Graham, B., Grave, E., Gribonval, R., Jegou, H., and Joulin, A. (2020). Training with quantization noise for extreme model compression. arXiv preprint arXiv:2004.07320.
Gao, J., He, D., Tan, X., Qin, T., Wang, L., and Liu, T.-Y. (2019). Represennaation degeneration problem in training natural language generation models. arXiv preprint
Gao, I.., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., Phang, J., Reynolds, I.., Tang, E.,, Thite, A., Wang, B., Wang, K., and Zou, A. (2021). A framework for few-shot language model evaluation.
11

---

Gholami, A., Kim, S.. Dong, Z., Yao, Z., Mahoney, M. w., and Keutzer, K. (2021). A survey of quantization methods for efficient neural network inference. arXiv preprint 17X1:
Gokaslan, A. and Cohen, V. (2019). io/OpenWebTene Corpe,
Openwebtext corpus.
wrlhutp://Skviioo007 github.
Gong, R., Liu, X., Jiang, S., Li, T., Hu, P., Lin, J., Yu, E,, and Yan, J. (2019). Differentiable soft quantization: Bridging full free-iiiiiiion and low- bit neural networks. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2. 2019, pages 4851-4860. IEEE.
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. (2020). Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 147.011...
Hoffmann, J., Borgeaud. S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., al. (2022). Training compute-optimal large language models. arXiv preprint arXiv:2203.55555
Ilharco, G., Ilharco, C., Turc, I., Dettmers, T., Ferreira, F., and Lee, (2020). High performance ural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, pages 24-27, Online, Association for Computational Linguisties.
Jin, Q., Ren, J.. Zhuang, R., Hanumante, S., Li, Z., Chen, Z., Wang, Y., Yang, K., and Tulyakov, S. (2022). F8net: Fixed-poin 8-bit only multiplication for network quantization. arXiv preprint arXiv:2202.05239.
Khudia, D., Huang, J., Basu, P., Deng, S., Liu, H., Park, J., and Smelyanskiy, M. (2021). Fbgemm: En- abling high-performance low-precision deep learning inference. arXiv preprint arXiv:2101.055:
Kovaleva, O., Kulshreshtha, S., Rogers, A., and Rumshisky, A. (2021). Bert busters: Outlier dimensions that disrupt transformers. arXiv preprint 0rXiv:2105.06990
Li, R., Wang, Y., Liang, F., Qin, H., Yan, J., and Fan, R. (2019). Fully quantized network for object detection. In JEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 28 131028199 119, Computer Vision Foundation IEEE.
Lin, Y., Li, Y., Liu, T., Xiao, T., Liu, T., and Zhu, J. (2020). Towards fully 8-bit integer inference for the transformer model. arXiv preprint arXiv:2009.08034.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:190..11692.
Luo, Z., Kulmizev, A., and Mao, X. (2021). Positional artefacts propagate through masked language model embeddings. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5312-5327, Online. Association for Computational Linguistics.
Macháček, M. and Bojar, O. (2014), Results of the wmt14 14 metrics shared task. In Proceedings of the Ninth Workshop Statistical Machine Translation, pages 293-301.
Mellempudi, N., Srinivasan, S., Das, D., and Kaul, B. (2019). Mixed precision training with 8-bit floating point. CoRR, 12334.
Nagel, S. (2016). Ce-news.
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng. N., Grangier, D., and Auli, M. (2019). fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint /r0.ivv/9440001
Ott, M., Edunov, S., Grangier, D., and Auli, M. (2018). Scaling neural machine translation. arXiv preprint ar 18061000.00117
12

---

Park, G., Park, B., Kwon, S. J., Kim, B., Lee, Y., and Lee, D. (2022). nuqmm: Quantized matmul for efficient inference of lcrge-scale generative language models. arXiv preprint arXiv:2206.0955)
Puccetti, G., Rogers, A., Drozd, A., and Orletta, F. (2022). Outliers dimensions that disrupt transformers driven by frequency. arXiv preprint arXiv:2205.11380
Qin. H., Gong, R., Liu, X., Bai, X., Song, J., and Sebe, (2020). Binary neural networks: A survey. CoRR, abs/2004.03333.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models unsupervised multitask learners. OpenAl blog, 1(8):9.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W. and Liu, P. J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683.
Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. (2016). Xnor-net: Imagenet classification using binary convolutional neural networks. In Leibe, B., Matas, J., Sebe, N., and Welling, M., editors, Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pages 525-542. Springer.
Sennrich, R., Haddow, B., and Birch, A. (2016). Edinburgh neural machine translation systems for wmnt 16. arXiv preprint
Shazeer, N., Cheng, Y., Parmar, N., Tran, D., Vaswani, A., Koanantakool, P., Hawkins, P., Lee, H., Hong, M., Young, C., et al. (2018). Mesh-tensorflow: Deep learning for supercomputers. Advances in neural information processing systems, 31.
Shen, S., Dong, Z., Ye, J., Ma, I.., Yao, Z., Gholami, A., Mahoney, M. W., and Keutzer, K. (2020). Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8815-8821.
Sun, X., Choi, J., Chen, C., Wang, N., Venkataramani, S., Srinivasan, V., Cui, X., Zhang, W., and Gopalakrishnan, K. (2019). Hybrid 8-bit floating point (HFP8) training and inference for deep neural networks. In Wallach, H. M., Larochelle, H.. Beygelzimer, A., d'Alché-Bur F., Fox, E. B., and Garnett, R., editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 1901-4910.
Timkey, W. and van Schijndel, M. (2021). All bark and no bite: Rogue dimensions in transformer language models obscure representational quality. arXiv preprint arXiv:2109.0444))
Trinh, H. and Le, Q. V. (2018). A simple method for commonsense reasoning. arXiv preprint arXiv: 1806.02847
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, (2017). Attention is all you need. arXiv preprint (xXXvv1706.0366
Wang, N., Choi, J., Brand, D., Chen, C., and Gopalakrishnan, K. (2018). Training deep neural networks with 8-bit floating point numbers. In Bengio. S., Wallach, M., Larochelle, H., Grauman, K., Cesa-Bianchi, Bianchi, N., and Garnett, R. editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 7686-7695.
Wei. X., Zhang, Y., Zhang, X., Gong, R., Zhang, S., Zhang, Q., Yu, F., and Liu, X. (2022). Out- lier suppression: Pushing the limit of low-bit transformer language models. arXiv preprint arXiv:2209.1 13325.
Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F,, Joulin, A., and Grave, (2020). CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4003-4012, Marseille, France. European Language Resources Association.
13

---

Wolf. T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., al. (2019). Huggingface's transformers: State-of-the- the-art natural language processing. arXiv preprini
Wu, H., Judd, P., Zhang, X., Isaev, M., and Micikevicius, P. (2020). Integer quantization for deep learning inference: Principles and empirical evaluation. arXiv preprint arXiv:2004.0966 00600..
Yao, Z., Aminabadi, R. Y., Zhang, M., Wu, X., Li, C., and He, Y. (2022). Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. arXiv preprint arXiv:2206.01880
Yao, Z.,, Dong. Z., Zheng. Z., Gholami, A., Yu, J., Tan, E., Wang. L.., Huang, Q. Wang, Y., Mahoney, M., et al. (2021). Hawq-v3: Dyadic neural network quantization. In International Conference on Machine Learning, pages 11875-1 1886. PMLR.
Zafrir, O., Boudoukh, G., Izsak. P., and Wasserblat, M. (2019). Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing -NerriPP Edition (EMC2-NIPS), pages 36-39, IEEE.
Zeng. A., Liu, X., Du, Z., Wang. Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., al. (2022). 6lm-130bs An open bilingual pre-trained model. arXiv preprint
Zhang, D., Yang, J., Ye, D., and Hua, G, (2018), Lq-nets: Learned quantization for highly accurate and compact deep neural networks. Proceedings of the European conference on computer vision (ECCV), pages 365-382.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022). Opt: Open pre-trained transformer language models. arXiv preprins 200.0:2005.01068.
Zhang, W., Hou, L., Yin, Y., Shang, L., Chen, X., Jiang, X., and Liu, Q. (2020). Ternarybert: Distillation- aware ultra-low bit bert. In EMNLP.
Zhao, C., Hua, T., Shen, Y.. Lou, Q., and Jin, H. (2021). Automatic mnixdd-precisoon quantization search of bert. arXiv preprint arX19:2112.14938.
Zhu, C., Han, S., Mao, H., and Dally, W. (2017). Trained ternary quantization. In 5th Interna- lional Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. (2015). Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19-27.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes]. [No], or [N/A]. . You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data proprietary.
Did you include the license to the code and datasets? [N/]]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
14

---

(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the limitation section
(c) Did you discuss any potential negative societal impacts of your work?[Yes] See the Broader Impacts section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?[Yes) Yes, we believe our work conforms to these guidelines.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We will include
our code the supplemental material.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? Yes] See the experimental setup section
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] Our experiments are deterministic for each model. Instead of running the same model multiple times, we run multiple models different scales. We are unable to compute error bars for these experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See the exper2imental setup section
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See experimental setup section
(b) Did you mention the license of the assets? [No] The license is permissible for all the assets that we use. The individual licenses can easily be looked up.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A] We only use existing datasets.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "y0H9BZhwLTj", "reviewer": "Reviewer_Booa", "summary": "This paper proposes a memory-efficient matrix multiplication method, mainly for multi-layer perceptron (MLP) and attention projection.\nThe method consists mainly of two parts.\nOne is vector-wise quantization to allow quantization with vector-wise independent scaling.\nThe other is mixed-precision matrix decomposition to adopt the outliers that often occur in larger Transformer models.\nThis method can reduce the required memory for inference by half while retaining full precision performance. \nThis paper first loads a 16/32-bit standard checkpoint and then converts it to Int8 for inference.\nThis method requires no post-quantization training.\nThis paper also reveals that existing quantization methods perform poorly at scale due to emergent outlier feature dimensions. \nExperimental results on LM performance indicate that the result of the proposed method achieved no degradation from the original-size models. \nThese results indicate more accessibility to larger models (The authors state that GPT-J and T5-11B can work on a single free cloud GPU, GPT-NeoX-20B on a single gaming-grade GPU, and OPT-30B on a single data-center-grade GPU. \n\n", "questions": "* The definition of O, that is, $O = \\\\{i|i ∈ \\\\mathbb{Z}, 0 ≤ i ≤ h\\\\}$, in Section 2.2 is somewhat strange. Please check it again, and if this is not wrong, then could you show how to compute this set by using some actual examples.\n\n* $ X^{h}\\_{f16}$ and $ X^{h}\\_{i8} $  in Eq. 8 should be in bold face.\n\n ", "limitations": "This paper has limitations and potential negative social impact sections.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The existence of outlier features is already discussed in the previous study, but this paper seems to be the first paper that clearly indicates the reason for poor performance on existing quantization methods due to emergent outlier feature dimensions. \n* The proposed method seems novel and unique.\n\n\nWeaknesses:\n* There are several missing definitions for notations in equations, and seems some errors (See Questions). \n* The Absmax quantization and Zeropoint quantization seem not novel enough as main contribution of this paper since a similar approach has already been used in the literature (e.g., see https://intellabs.github.io/distiller/algo_quantization.html for example.\n* The remaining part of the proposed method is Mixed-precision Matrix Decomposition. I feel that this technique is straightforward.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "i5_gS49F0DQ", "reviewer": "Reviewer_2A6h", "summary": "The paper proposes a method to quantize GPT style models using INT8 quantization. The paper finds that existing quantization methods perform poorly on quantizing GPT models. The authors identify that this was because of the outliers in feature maps, leading to large quantization errors. To address this issue, the authors propose to use finer-grained vector-wise quantization. The evaluation shows that the proposed method can quantize GPT style models without performance degradation to up to 13B parameters. ", "questions": "Please cite and describe the relationship between this work and prior works on quantization, like those mentioned in the weaknesses section. \n\nPlease provide inference latency measurements or describe the hardware implication of the proposed method. ", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper studies a timely problem and proposes practically useful techniques for quantizing large-scale GPT models.\n\n- The paper evaluates the effectiveness of their approach on some very recent large-scale models such as billion-scale GPT-NeoX, OPT-30B, and demonstrates that they can be deployed on a single GPU with Int8 quantization. \n\nWeaknesses:\n\n- The paper misses quite some related work on quantization for transformer models, making it difficult to judge the novelty of the proposed method. For example, the outliers in feature maps have already been previously identified such as in \"Understanding and Overcoming the Challenges of Efficient Transformer Quantization\" by Bondarenko et al., https://arxiv.org/pdf/2109.12948.pdf. Therefore, the findings of emergent outliers on GPT models are not that new. Furthermore, the paper claims that its method does not require quantize-aware training, but again, the post-training quantization method has also been proposed by https://arxiv.org/pdf/2109.12948.pdf. Again, no references or discussions on this prior work. Finally, mixed-precision quantization and finer-grained quantization such as group-wise quantization have been studied in prior work such as https://www.stat.berkeley.edu/~mmahoney/pubs/NeurIPS-2020-hawq-v2.pdf, http://proceedings.mlr.press/v139/yao21a/yao21a.pdf, https://arxiv.org/pdf/1909.05840.pdf, https://www.ijcai.org/proceedings/2021/0472.pdf. Many of these works are either not cited or not properly discussed in the related work. Also, given that many of these techniques have been proposed and studied by prior works, the main contribution of the paper is limited to applying to billion-scale GPT models. \n\n- The inference latency impact of the proposed method is not very clear. One of the main motivations for having INT8 quantization is that INT8 arithmetic computation is supported on commodity hardware such as Nvidia GPUs. Therefore, it appears to be important to either evaluate the inference latency of the proposed method or at least provide an explanation of the implication of the proposed method to the hardware in comparison to alternative quantization methods, e.g., those used together with QAT. \n\n\n----------------\nPost-rebuttal comments\nThe authors addressed my concerns. I increased my score from 5 to 6 to reflect my perspective after the rebuttal. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "aN3dEbw07Jm", "reviewer": "Reviewer_JbRh", "summary": "This paper explores how we can apply the uniform quantization method to large pre-trained language models. They utilize vector-wise quantization method and decomposition techniques for over 1B models and show the validation ppl of the 13B language model can be not degraded by uniform quantization method. For the 13B model, the decomposition method for outliers can improve the model performance regardless of quantization schemes.", "questions": "None", "limitations": "To support their assertions, I think this paper can be refined in two aspects: 1) additional experimental results on performance of generation models, and 2) additional arguments in the acceleration environment and implementation. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "<<Strengths>>\n- Since there have been a few studies on large transformer models (over 1B parameters), the results on this paper can be a good reference for many researchers. We can learn the existence of many outliers in big transformer models (Table 2) and the performance degradations in conventional uniform quantization methods (Table 1).\n- I think scaling to the 13B model is a big strength of this paper, and the consistent trend to 13B is also a good point of this paper. \n\n<<Weaknesses>\n- Although we can learn many things from Table 1, I have a concern on the experiments in the aspect of the limited evaluation. Many researchers have given attention to the powerful generation capability of Transformer decoders. But, this paper only measured the perplexities, the probabilistic results of language models. I think it seems to be not sufficient to note that there is no performance degradation on generation tasks. How about zero/few-shot evaluation results? With various kinds of tasks, the contribution of this paper would be strengthened.\n- I have also another concern on the acceleration of this method, the vector-wise quantization and decomposition method. There is a short explanation about acceleration in Appendix A and Table 3. But I think it is not sufficient to prove there is no acceleration overhead. There is no description on the environmental setup of inference measurement. I think this paper should include some arguments about this issue in detail.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "X1e6TnuMJco", "reviewer": "Reviewer_neVa", "summary": "The largest language models in practice become resource intensive for inference which interferes with accessibility for researchers or e.g. edge device use cases. This paper offers a new type of weight quantization to int8 data type which enables lower overheads in inference for accessibility with reduced GPU utilization. Although int8 quantization is not new for language models, prior work has been interested in smaller models (e.g. on order of BERT scale). \n\nOne of the findings of this paper was that when such practices were attempted on larger scaled models (they looked at up to 13B parameters), there was an emergence of increased density of extreme outliers found within the weight values which impacted performance of prior quantization schemes. Their solution was to perform what they refer to as vector quantization, which scales the int8 quantization derivation unique to each vector after decomposing between those with prevalent outliers (which are retained as FP16) and those without (which are converted to int8), and after performing matrix multiplication the decomposed vectors are recombined into a common FP16 matrix. Their benchmarks demonstrated that the practice was able to duplicate performance of FP32 at up to 13B parameters, outperforming prior quantization schemes for high parameterization scales.", "questions": "1. Did the researchers consider or attempt to identify whether the emergence of extreme outliers in layers correlated with emergence of sparsity of weights? It seems possible that prior work noting sparsity of deeper models (e.g. Gissen et al, ICLR 2020) may have neglected to inspect for presence of extreme outliers, or for this work vice versa. Such alignment of sparsity and outliers emergence may suggest that it is a shared phenomenon.\n\n2. The scaling methods of the paper appeared of most interest to negating potential for representation overflow in the quantization. Did author’s consider whether scaling based on a maximum value might result in a disproportionate degree of representation underflow? After all if scaling is based on extreme outliers, that suggests that a resulting absolute maximum scaled distribution will likely be “squished” into the lower registers of the data type. (Such distribution width is what is addressed when dividing by a standard deviation in z-score normalization, and wouldn’t be addressed by paper’s noted zero point extension.)\n\n3. I am wondering if scaling derivations might be redundant with similar scaling performed as a part of batch normalization, and if so have authors considered a variant with hard coded scaling assumptions assuming presence of batch norm?\n\n4. Could the authors please clarify what is meant by “feature dimensions h” (247) and how it is derived in comparison to outlier feature dimensions h_i? (I didn’t understand your example of “dimension i=888 for h=1024” (236))\n\n5. I would have been interested in additional detail for context of the noted parameterization thresholds. While I recognize that this is a quantization process for inference, I expect the presence of outlier values arises during training, and various phenomenon from overparameterization (like double descent) are found to arise by parameterization in context of scale of training tokens. (See e.g. Belkin et al 2019 \"Reconciling modern machine-learning practice and the classical bias–variance trade-off\"). I would be curious if future work may yet identify that the benefit of this form of quantization, as well as the emergence of “outlier feature dimensions”, were found to be impacted by variations of such conditions experienced during training like proximity to the interpolation threshold (which is a function of both parameterization and training sample counts).\n\n6. Does the method only work on low rank tensors as per the demonstration or can it be adapted to diverse configurations?", "limitations": "Ideally benchmarking could be extended to larger parameterizations exceeding 13B, the authors noted the possibility of additional emergence of feature dimensions. Since this is a method for inference as opposed to training authors may find that such experiments may not be hugely out of reach from a resource utilization standpoint.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. Prior language model quantizations required adaptions during training, this approach can be performed in inference independent of training regime.\n\n2. Prior language model quantizations experienced performance degradations a higher tiers of parameterization. This paper demonstrated through benchmarking that inference could be quantized without material impact to performance, demonstrated at scales up to 13B parameters.\n\n3. These methods could make large language models much more accessible to research and industry by way of lowering GPU overhead in inference.\n\n\nWeaknesses\n\n1. Authors note that the method wasn’t applied to attention layers. Paper should have provided better detail of why this method was insufficient, especially considering that the intended use case is for large language models.\n\n2. The paper introduced some interesting emergent phenomenon at larger parameterization scale (of outlier weight values) but did not attempt to draw context towards other phenomenon found in overparameterization. \n\n3. I found the reference to what author’s call “outlier feature dimensions” lacked clarity of definition.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept