Intriguing Properties of Quantization at Scale
Authors: Arash Ahmadian
Saurabh Dash
Hongyu Chen
Bharat Venkitesh
Zhen Stephen Gou
Phil Blunsom
Ahmet Üstün
Sara Hooker
Venue: NeurIPS 2023 poster (2023)
Keywords: Quantization
optimization
language modelling
efficiency
TL;DR: Explores the effects of training optimization choices on Post Training Quantization of LLMs

Abstract: Emergent properties have been widely adopted as a term to describe behavior not present in smaller models but observed in larger models  (Wei et al., 2022a). Recent work suggests that the trade-off incurred by quantization is also an emergent property, with sharp drops in performance in models over 6B parameters. In this work, we ask _are quantization cliffs in performance solely a factor of scale?_ Against a backdrop of increased research focus on why certain emergent properties surface at scale, this work provides a useful counter-example. We posit that it is possible to optimize for a quantization friendly training recipe that suppresses large activation magnitude outliers. Here, we find that outlier dimensions are not an inherent product of scale, but rather sensitive to the optimization conditions present during pre-training. This both opens up directions for more efficient quantization, and poses the question of whether other emergent properties are inherent or can be altered and conditioned by optimization and architecture design choices. We successfully quantize models ranging in size from 410M to 52B with minimal degradation in performance.

Intriguing Properties of Quantization at Scale
Arash Ahmadian *f Cohere For Al
Saurabh Dash
Hongyu Chen
Cohere
Cohere
arash@cohere.cm com
saurabh@cohere.com
charlie@cohere.com
Bharat Venkitesh Cohere bharat@cohere.. email
Stephen Gou
Phil Blunsom
Cohere
Cohere
stephen@cohere.com com
phil@cohere.cm com
Ahmet Üstün Cohere For AI
Sara Hooker
Cohere For AI
ahmet@cohere.. com
sarahooker@ccherr.coom com
Abstract
Emergent properties have been widely adopted term to describe behavior not present in smaller models but observed in larger models (Wei al. (022a) Recent work suggests that the trade-off incurred by quantization is also an emergent property, with sharp drops in performance in models over 6B parameters. In this work, we ask are quantization cliffs in performance solely factor of scale? Against a backdrop increased research focus on why certain emergent properties surface scale, this work provides useful counter-example. We posit that it is possible to optimize for quantization friendly training recipe that suppresses large activation magnitude outliers. Here, we find that outlier dimensions are not an inherent product of scale, but rather sensitive to the optimization conditions present during pre-training. This both opens up directions for more efficient quantization, and poses the question of whether other emergent properties are inherent or can be altered and conditioned by optimization and architecture design choices. We successfully quantize models ranging in size from 410M to 52B with minimal degradation in performance.
Introduction
The push for ever larger language models (LLMs) has been driven by strong correlation between performance and the number of parameters (Chowdhery al., 2022 Zhang al. 2022] Kaplan et al. 2020). This has led to new breakthroughs in downstream performance, but has also posed new challenges in making these models accessible. Larger models incur higher memory and latency because of the requirement to store many more model weights and the optimizer in fixed memory (Dehghani et al. 200TT] Treviso et al. 2022). Due to the massive size of state-of-art L.L.s, inference often requires hosting across multiple machines which limits the practical usability of such models.
To address this, much research has focused compression techniques such as quantization, which reduces the number of bits needed to represent each learned parameter in model (Gholami et al. Quantization techniques are widely used in smaller model regimes quantizing weights
2021).
"Equal Contribution
Also affiliated with the University of Toronto & the Vector Institute for Artificial Intelligence.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

stored as 32-bit or 16-bit poaating-pint numbers to 8-bit integers (INT8) produces large reductions in memory and latency.
However, at scale simple quantization tech- niques have been shown to lead to pronounced
degradation in performance (Xiao et al. 2022).
Mean Performance
This trade-off has been attributed by several re-
0.7
cent works (Dettmers et al., [2022] Zeng et al.
2022
Xiao al.J 2022; Bondarenko et al.
2021
to emergent outlier dimensions-Seaaae
transformer-basel architecture results in large activation magnitude outliers which are con- centrated along a few hidden dimensions. To solu-
ours
0.4
ours-fp16
optiint8
remedy this degradation, mixeddppeeistee tions have been proposed that handle the outlier activations separately (Dettmers et al., 2022). While effective at preventing degradation, these specialized techniques pose significant latency overhead which negates some of the benefits to memory (wu et al., 2020). In Appendix [C1.33 we demonstrated the efficiency benefits of us- ing simpler quantization techniques over mixed- precision solutions.
opt-/p16
0.3
1.38
138
308
1508
Parameter Count
Figure 1: Mean zero-shot accuracy on HellaSwag. PIQA, LAMBADA, and WinoGrad. In contrast to the OPT family, our models show minimal degra- dation after simple vectorwise quantization. Data points for OPT models are from (Dettmers et al.) 2022).
The difficulties of quantizing at scale prompt us to ask: are emergent properties due to nature or nurture? Recent work introduces intriguing and somewhat contradictory answers to this question: Models like OPT-175B (Zhang al. 2022) and FairSeq (Artetxe al. 2022) exhibit pronounced sensitivity to post-training quantization and require complex mixed-precision decomposition quantization methods (Dettmers et al. 2022 Wei et al. 2022b Bondarenko et al. 2021] Luo et al., [2020] Zeng et al., (022). On the contrary. BLOOM-1766 (Scao et al.. 2022) is casier to guantize with simple quantization recipe and relatively small performance drop (Frantar et al. 2022] Xiao et al. (022). Zeng et al. (2022) hypothesize that the observed difference in weight distribution characteristics may be due to the difference in optimization choices made during pre-training.
In this work, we seek to reconcile these observations. We posit that it is possible to optimize for quantization friendly training recipe that suppresses large activation magnitude outliers. This leads to distribution of activations and weights that are more amenable to simple INT8 quantization recipes and does not necessitate the need for complex and inefficient mixed-precision computations. Our results show that we introduce simple INT8 post-training quantization with negligible impact on performance due to choices we make during the pre-training stage. As shown in Figure [] across 8 zero-shot downstream tasks, models do not present any significant performance drop, having only 0.24% average degradation in a 52 billion parameter model.
In summary, our contributions are as follows:
We conduct controlled large scale study At 6B, maintain the same architecture and vary key optimization choices such as weight decay, gradient clipping, dropout and precision of training representation. We present results with optimal hyper-parameters across models varying from 410 million to 52 billion parameters, with each experiment variant trained from random initialization. While this requires compute intensive set-up, it allows to rigorously disentangle what factors actually influence sensitivity to quantization.
We show that reoccurring activation outliers are not a universal emergent property of LLMs at scale and can be avoided at scales as large 52B given the right optimization choices. Our 52B parameter model shows only 0.26% performance degradation across 8 tasks with INT8 PTTQ quantization of both activations and weights, which enables a 1.4-1.5x inference speedup and around 40% memory footprint reduction over half-precision inference.
We contribute fine-grained analysis of activations and weights, and show that several key weight and activation characteristies may explain the difference sensitivity between our robust models and models like OPT which have been shown to have pronounced sensitivity

---

to quantitization at scale. We hope these insights help guide future model design and pre-training strategies.
2
Background
Quantization refers to compressing weights and activations of neural network into lower-bit representations. Here, our focus is on one-shot post-training quantization (PTQ) (Xiao et al. 2022 Dettmers et al.
2022), which quantizes the network post-training without additional finetuning steps for calibration. Given the complexities of successfully training a large language model (Zhang et al.
2022
Rae et al. 2021). PTQ methods extremely attractive these techniques require the least modification to pre-trained parameters, compared to quantization-awan training methods (Zafrin
et al.
2019 Krishnamoorthi 2018) quantization-aware finetuning (Park et al.) 2022b; Yao et al. Zhu Frantar et [202]] Zhuo et al. [0022] Li et al. 2021] Hubara et al. 2020] Nagel et al. 20200)
2022
which both require updates model weights. We include more detail about each of these broad
groups of techniques in Appendix
To-date PTQ techniques that quantize both the activations and weights of a language model have proven extremely challenging at large scale (>6B parameters) leading to pronounced drops in performance. Instead, less aggressive techniques have been used such weight-only quantization (Grganoy 2023] Frantar et al. 2022; Zeng et al., (222) that leaves the activations in higher precision quantization with mixed- mixedd-preiiste precision decomposition (Dettmers et al. 2022) which decomposes the matrix multiplication to compute a small fraction of elements at a higher precision (FP16) while the bulk of the computations is performed low precision (INT8).
Weight-only quantization brings speedup to inference by reducing the amount of data movement. However, as large language models are scaled, progressively they become compute-bound and the improvements due to weight-only quantization stagnate. While mixed-precision decomposition approaches have theoretical latency benefits due to the bulk of the computation being performed at lower precision, in practice without specialized hardware (Dash et al. [022] Dash & Mukhopadhyay 2020 Hooker 2222)) GPU kernels, or additional kernel calls to prepare the inputs and weights for muxeddppreissie computation, the projected benefits cannot be realized (Dettmers et al. 2022). To further realize latency gains, we need to quantize both weights and activations into 8-bit integers (INT8) to utilize specialized integer INT8 GEMM kernels, which are supported by a wide range of hardware (e.g., NVIDIA GPUs, Intel CPUs, Qualcomm DSPs, etc.). In Appendix C.1.3] we showed 1.4x inference speedup using weight and activation quantization, compared to half half-precision baseline. In addition, weight and activations quantization further enables compressing key-value cache INT8. Since key-value cache takes up a significant part of the GPU memory during inference (Sheng et al., 2023), weight and activations quantization further contributes memory saving and high throughput inference.
Hence, the most challenging weights and activations setting is the focus of this work. More concretely, given a neural network that learns a function / parameterized by a set of weights (Wo. W W1...., W.) with corresponding activations (Xo. X, -.., X.), during quantization, activation and weight tensors denoted by X E Rexh and W E h where i denotes the sequence-length, h the input hidden units and the output hidden unitsf] - are replaced with lower-bit counterparts Wa Q and Xq by scaling and rounding the original tensors to INT8 range. We focus vector-wise quantization recipe (Dettmers et al.] 2022) to increase quantization granularity. In vector-wiss wiie quantization, we define the row-scaling vector Sx e IR' and column- for each row/column through uniform symmetric mapping (Nagel et al., 2021). After INTS matrix multiplication, dequantization of XQ Wq back to FP16 is through element- multiplication with 8x and Sw:
vector sw R" by calculating the scaling constants
XW Sx O XXQWo Sw
Where @ denotes broadcastable matrix multiplication. The quantization and dequantization steps for the above do not add much memory overhead compared quantizing with a single scaling constant for each weight or activation tensor, while significantly increasing the representation power of INT8.
We omit the batch dimension for simplicity.
3

---

3 Methodology and Experimental Setup
3.1 Methodology
Our goal is to understand whether sensitivity to widely used quantization techniques is inherently an emergent property at scale or due to optimization choices made during pre-training. Recent work has presented seem- ingly contradictory empirical findings some models such as OPT- 175B show pronounced sensitivity at scale to post-training quantization while other models such as BI.0OM-1761 16B are relatively robust to post-training quantization.
Experimental Axes
Choices
Weight decay
0.001, 0.01, 0.1
Gradient clipping Dropout Half-precision
None, 1
0, 0.1, 0.4, 0.8
bf16, fp16
Table 1: Optimization choices that are ex- plored for pre-training in controlled setup.
These models differ in numerous ways such architec- tural differences, pre-training data, training infrastruc- ture, and finally optimization choices, making chal-
lenging to attribute differences in quantization performance. To rigorously isolate what choices result sensitivity to quantization, we measure the impact of optimization choices within tightly controlled experimental setup- training the same large scale model architecture from random initial- ization while rigorously varying only key aspects of the optimization procedure. Each optimization choice is evaluated in two ways: we measure the resulting degradation after PTQ in zero-shot down- stream performance and then analyze the model weights and feature activations to understand how the characteristics scale impact quantization performance.
Training multiple multi-billion parameter size language models is extremely expensive single 52B language model takes roughly 20 days of training with 2048 TPU cores Therefore, we first conduct our controlled experiments on 410M and 6B models using early checkpoints and then validate the results at scale, by fully training 6B, 13B, and 52B parameter size models with our most quantization friendly training recipe. In practice, we found performance at early checkpoints predictive of fully trained model performance.
We briefly describe each of the axes of variations below:
Weight decay is widely used to impede over-fitting by penalizing large magnitude weights (Goodfellow et al. 2016). We experiment with range of weight decay values (0.001, 0 01, 0.1].
Weight decay
Gradient clipping Gradient clipping rescales the norm of the gradient vector if it exceeds the threshold (Pascanu et al. 2013). is widely used in LLMs to prevent exploding gradients and accelerate training convergence (Du et al. 2021 Zhang et al. 2022). We experiment with gradient norm threshold of 1 as well as training without gradient clipping.
Dropout Dropout is widely used regularization technique that drops neurons with probability of p during training (Srivastava et al. 20144; Hinton et al., 2012). We apply dropout to the output of the self-attention block and the feed-forward layer before the corresponding residual connection as described in Vaswani et al. (2077), but we do not use a dropout for the input embeddings. We experiment with [0. 1, 0. 4, O 0.8) dropout probabilities.
Half-precision data type: bf16 vs fp16 Training neural networks in seeedd-rreiiion is common technique to reduce the memory requirement and improves training time while often achieving comparable performance to full-precision training (Micikevicius et al.) 2017). In this technique, copy of weights is stored in full full-precision (fp32) whereas the forward and backward passes are done using half-precision either 66 (fp16) or bfloat16 (bf16) (Kalamkar et al. 2019 Dean et al. 2012 Abadi et al.) 2015).
We experiment with fp16 and 16 Furthermore, for each half-precision data type, we vary weight decay values of (o. 1, O. 01) to observe whether the effect of the half half-precision data type is exasperated with smaller weight decay value of 010
We include more details about the hardware and training requirements in Section 3.2

---

Weight Decay
Dropout
Gradient Clipping
Mean % PTQ Diff.
0.32 0.31
0.41
-0.09
-04
-0.27
-0.8
dropout-D.0
-0.57
-0.8
dropout=D.1
-12
wd=0.01
-12
dropaut-o,4
-12
9c=1.0
-16
wd=0.001
-1.36
-16
dropout-0.8
-16
gc-none
-1.36
(a)
(b)
Figure 2: Study of the PTQ performance when varying weight decay, dropout, and gradient clipping. In Figure 2 control weight decay value 0.001 used to minimize the effects weight decay when studying gradient clipping. Otherwise, we use weight decay of 0.1, dropout of O, and gradient clipping threshold of 1.0 as control variables.
3.2 Experimental Setup
Model We train autoregressive decoder-only Transformer models (Liu al. 2018) with standard
language modeling objective. Given an input sequence of S = [$1 [$1..... St), a language model with
parameters 0 trained to minimizes the following negative log likelihood:
L(S) Σ -logP(s:(scc::))
Our language models follow the traditional GPT style architecture reported in Radford et al.) Different from the Radford et al., we do not share the same weight matrix for input and output embedding layers. Instead, learn separate input and output projections enable higher model parallelism. We train models with parameter sizes ranging from 410 million to 52 billion. All models have maximum sequence length of 2048 tokens. We SentencePiece (Kudo & Richardson 2018 tokenizer with vocabulary of 51 51200 to tokenize the text.
Training details We pre-train models using mixture of datasets from Common Crawl and C4 (Raffel et al. (020) with AdamW (Loshchilov & Hutter 2019) optimizer and batch size of 256. scheduler with 1500 warm-up steps. We GeLU activations (Hendrycks) & Gimpel 2016). All the models are trained with mixed- preccdsooeiiioon i.e. forward and backward passes computed in bf 16 fp16 half-precision format, but the model parameters stored in fp32 in the distributed optimizer state (Rajbhandari et al.) (020). For half- malf-precision fp16, we experimented with variant by switching layernorm arithmetic from fp16 to fp32 for better numeric stability (Micikevicius et al. 2017). Without layernorm arithmetic being in fp32, the training was unable to converge.
cosine learning
To avoid an exorbitant computational cost given each variant requires training from scratch and evaluating very large scale models, first iterated 410M models but observed very minimal degradation. As a result, we scaled to 6B parameters which is the scale at which we present validate
results. Following the analysis, optimization choices to 13B and 52B models trained to more than 200k steps. We note that the 52B model required training time of roughly 20 days and in total we trained 17 models of sizes ranging from 410M to 52B to carefully measure the impact of each ablation. The details of each model's architecture and training hyper-paramettes are provided in Appendix 1
findings by scaling the most PTQ friendly
Infrastructure We use TPU-v4 chips (Jouppi et al. 2017) to train, and Nvidia A100 GPUs to evaluate models. All models trained using the FAX YYo et al. 2022) framework which enables efficient model and data parallelism. It takes approximately 72 hours on 128 cores to train a 6B parameter model for 75000 steps.
Evaluation We evaluate each model variant on Copa (test and dev set) (Wang al. 2019). HellaSwag (Zellers et al. (201)). PIQAValidation (Bisk et al., 2020). StoryCloze (Mostafazadeh et al.) 2016). WinoGrande (Sakauuchi et al. 2019) 9). Paralex (Fader et al., (2013), and LAMBADA (Paperno et al.
2016). All evaluations were done in zero-shot setting. In total, we benchmark on 8 tasks comprised Multiple choice (MC) completion, MC Co-referencing, Generation, and Question Answering (QA) types. Details of our evaluation suite given in Appendix A.2 For each experimental variant,

---

Half-precision data type: bf16 vs fp16
Mean % PTQ Diff.
1.1
0.77
0.32
0.3
-0.5
0:76
dtype-b/15 (wd=0.1)
-1.3
dype-brle
@ypc-b016
dtypemb11 (wd=0.01)
aypemfoze (ed-0:3)
dtypemfp15 (wd-0.1)
-21
dypesfol6 (wde0.001
-1.73
dtype-fp15 (wd=0.03)
15000
40000
75000
Figure 3: Study of PTQ performance when varying the precision used during training. On the left, the performance difference is plotted at 75000 steps whereas on the right, it is plotted over time. We observe that fp16 training consistently leads to models which are far more sensitive to post-training quantization.
report the average percent performance difference and the pre-quantization and post-quantization performances across tasks. The percent degradation is calculated by normalizing each task's 's absolute degradation by the corresponding pre- pre-quantization performance.
Results and Discussion
For each experimental axis, train the corresponding variants to maximum of 75000 steps. Note that due to the enormous computational cost of full-pretraining of these models, we only fully-train the final variants with optimal hyper-parameters (for >200k steps). Below we present breakdown of the degradation results and analysis for each experimental axis. All variants with the exception of dropout=O.. 8, had similar rueauuanizztiinn performance. This is important, in comparing optimization choices that still result models of comparable quality, but differing sensitivities to post-training quantization. Refer Appendix A.3 for the per-task breakdown of results.
are interested
Weight Decay As can be seen in Figure 2a we observe that higher level of weight decay during pre-training improves post-training quantization performance. We do not use gradient clipping in these experiments to isolate the impact of weight decay. A larger weight decay value (0. 1 vs 0. 0.001) results in better post-training performance (0.09% 1.36% degradation). Furthermore, as shown in Figure combining lower weight decay with fp16 further amplify sensitivity post-training quantization. A small weight decay value (0.01) can cause higher performance degradation in qust-quantization after training (1.73%).
Dropout and Gradient Clipping In Figure 2b we observe that higher levels of dropout correspond to sharper degradation in post- post-traaiing quantization. Note that Paropout 0.8 unsurprisingly leads to a poor absolute downstream performance, however, it helps establish clear trend given other data points. Figure 2c shows the relative quantization degradation for models with and without gradient clipping. When varying gradient clipping, control weight decay value 0.001 used minimize the impact of weight decay. As seen in the figure, gradient clipping shows positive impact on the quantization performance. improving robustness to post-training quantization. This suggests that gradient clipping to extent counteracts the effects of small weight-decay value which would otherwise lead to higher quantization degradation.
Half- palf--recision: bf16 vs fp16 Figure shows the quantization degradation and absolute performance for fp16 and bf16 for 6B parameter models. Training with fp16 leads to higher quantization degradation than of 16.. We relate the degradation results to the numerical stability in training. fp16 format uses a smaller range for the exponent than bf16. While bf16 uses 8 bits for the exponent, fp16 only uses 5. Most floating point formats also have denormalized numbers which allow for soft underflow. This can get exponentially closer to 0.0f for each additional bit in the mantissa. This makes underflow more of concern for floating point formats.

---

Positional Embeddings Since choice of positional embeddings is often differentiating factor be- tween architectures e.g. OPT models use learned embeddings (same as us) whereas the BLOOM fam- ily uses AliBi Press et al. (2022) we also trained 3 models using learned and sinusoidal positional em- beddings, and ALiBi with otherwise optimal hyper-parameter (bf16 training, weight seight-decay=0 1, dropout=O. o, gradient clipping-1. O. We did not see a notable difference in downstream PTQ degradation as all values were within 0.06% of another.
Notably, we observe that our findings provide insights into the quantization robustness of BLOOM- 176B, which to our knowledge the only open-source LLM (with more than 50B parameters) and a decoder-only block architecture. that is trained with bf 16; compared to simliar sized fp16 trained models such as OPT-175B (Xiao et al. 2022).
Using Early Checkpoints to Infer Converged Model Behavior Given the considerable compu- tational cost of our experiments, it is valuable explore whether converged model behavior can be inferred from checkpoints from early snapshots of training. In Figure [ we plot the relative post-training quantization degradation given checkpoints trained with different levels of precision at different steps during pre-training. We observe that quantization degradation increases with step count, but the relative impact of varying the bit representation emerges early in training which confirms the main trend. Interestingly, fp16 (wd=0. 01) variant exhibits high quantization degradation in the starting phase training as early as 15000 steps.
Outlier Analysis As a potential proxy measure of PTQ degradation, we measure the number of outlier dimensions it's been shown strongly effect activation quantization sensitivity of (LeMmeDttteers) et al. 2022). Activation feature dimensions are classified as outlier dimensions when the activation values (denoted a)) are greater than 6.0 more than 25% of layers and 6% of tokens. However, we find that threshold of 6.0 is too high to classify a feature dimension as an outlier for all the variants we consider. After correspondence with the authors, we also explored various adaptations of this outlier detection recipe presented in Dettmers et al. (2022) to make it generalizable. However, we did not observe a clear correlation between these measures and sensitivity to quantization. We refer to Appendix B.3 for detailed treatment of these replication efforts.
Scaling Insights to 52B scale To validate our experimental findings at scale and with fully trained models, we pre-train 410M, 6B, 13B, and 52B parameter models for more than 200k steps, using the best optimization choices with respect robustness in post-training quantization: weight decay of 0.1, no dropout, gradient clipping of 1, and bf16 as the half-precision format. Figure [igure[]]shows mean zero-shot accuracy for the non-quantized and quantized model using INT8 weights and activations. Compared with OPT models (Zhang et al. 2022), our fully-trained models are significantly more robust to post-training quantization starting from 6B parameter size. Our largest scale model with 52B parameters, even shows a 0.08% average improvement across the evaluation suite and 0 % degradation across LAMBADA, HellaSwag, and PIQA where OPT-66B which is the closest OPT model in terms of size, has an extreme drop in performance of 42% as reported in Dettmers et al. (2022).
Transferability of Results All the models that we experimented with are architecturally very similar to the OPT & BLOOM family they are both decoder-only, pre-layernom (where the layernorm is placed before the linear layers), and use vanilla attention. They differ in the number of blocks and the hidden dimensions of models at different sizes. Since we thoroughly experimented with different model sizes, we believe that our insights directly translate to the OPT& BLOOM families' architectures.
To evaluate our models' performance under a different quantization recipe in addition to INT8, we also test 4-bit integer (INT4) column- wise weight-only quantization, similar to Du et al. (202]). Our 52B parameter model exhibited only a 3.6% relative drop in mean zero-shot performance across the 8 evaluation tasks. It is worth noting that this quantization scheme does not require any fine-tuning or optimization and hence these results highlight the high robustness of our trained models. In comparison, when applying the same quantization scheme to BLOOM, BLOOM-176B and BLOOM- 7B show 29.5% and 18.7% degradation respectively on LAMBADA as reported in the GLM-130B technical report (Du al. 2021) while our 52B model only has 8.6% degradation.
7

---

attn-kqv-proj
layernorm-1
attn-kqv-proj
layernorm-1
10'
107
10
10
10
10
13"
10
-055
0.5
1.0
2,2
-0.4 0.0
0.4
0.2
2.30
0.75
1.30
1.2
fp16 (wd=0.01)
bf16 (wd Nodd=0001) 0.01)
ours 6b
opt_
6b
bloom
6b
Figure 4: Weight distribution of attn-kqv attn-kqv-proj and layernorm gain (g) parameter in an example block (Block 14) for both fp16/bf16 variants and our final 6B model in comparison with OPT and BLOOM. Weight distributions for all blocks are shown in Appendix B.1
5 Weight and Activation Analysis
Our results in Section 4 find that sensitivity to quantization at scale is not an inherent emergent property. Rather, it can be avoided at scales as large as 52B given the right optimization choices. We also found that outlier dimensions are not always consistent proxy for quantization sensitivity. Thus as an attempt to address the short-comings of outlier dimensions as proxy and to reconcile our observations degradation trends, we perform fine-grained analysis of activations and weights to understand how the trained distribution of our models differs from models like OPT that are far more sensitive to quantization. we hope that the metrics we propose and evaluate in this section, help further discussion about useful proxy metrics for guiding pre-training optimization choices to improve robustness to quantization. For all the metrics proposed below, we include the complete analysis for all layers in Appendix
Activations As a first step, we analyze input activations of the attention projection (attn- -attn-kqv-proo] as it is the earliest point for INT8 multiplication in a decoder block. Here, we measure root-mean- square error RMSB(X, (X X) where X denotes the de-quantized activations. Additionally, we report the mean standard deviation of the input activations measured per token. While RMSE(X. X, X) directly indicates the quantization error, standard deviation (STD) has been shown to be closely related the expected quantization error of a normally distributed input (Kuzmin et al. 2022). Figure 3 compares the 16 and fp16 variants. We observe that the RMSE and STD of f are far higher than the
6616 variant - the RMSE for the fp16 variant 6.9x the RMSE for the bf variant. This difference is even more pronounced if compare our model to the OPT: the RMSE and STD of the OPT 27.7x and 1.8x higher respectively relative to our model (Figure 5 Bottom row).
LayerNorm Since outer- eepareexpddde-mmmeee are both preceded by a layernorm. The layernorm gain parameter, g E R², directly influences the output activation tokens' spread, and can significantly vary in distribution shape as seen in Figure We generally observe that within our experimental axes, the standard deviation of the gain parameters for both the first and second layernorms are higher in significant number of layers for variants with higher degradation compared to others in the same axis. In Figure [S] we compare the standard deviation of g for self-attention layernorm and observe that STD(g) is 2x higher for the fp16 variant relative to bf 16.. 16. Our findings add further support to previous work which suggests that the gain parameters act as an outlier amplifier and thus further quantization degradation through scaling (Wei et al. 20226).
pre-norm architecture, the input activations to attn --v--rqvpoo proj and
In the bottom row of Figure we also compare STD(g) of our model relative to OPT and BLOOM. We observe that even the BLOOM model that is relatively robust to quantization has a far larger STD(g) than our model with multiplier of 5x. Interestingly, we find that OPT-6B layernorm gain parameters are all set to 1.0 while biases varied as expected. Hence, given the gain parameters appear to be hardcoded, the STD(g) of the OPT model is O. We were not able find any mention of such design decision either in Zhang et al. (2022) or the github repository: htttps://gitae Facebookresearrchettaass
com/
Attention Projection Weights Finally, we compare the weight distribution of attn-kqv-poa proj layers. As seen in Figure 4 the fp16 variant has significantly wider distribution compared to bf16.
8

---

RMSE(X, X)
Σ std(X))
std(g)
||
0.00
0.06
0.5
0.02
0.0
0.0
0.00
dfype=b/16 (wd=0.01)
dtype=fp16 (wd=0.01)
16-45
1.00
0.15
2-
0.75
10.0
7.5
0.30
0.05
0.25
0
0.00
0.00
0.0
ours 6b
bloom bloom_7.1b
opt 6b
Figure 5: Average input token activation STD, preceding layernorm gain STD, and spectral norm of linear weights, follow trends similar those observed in RMSE(X.X). Plots correspond to the attn-kqv-pan proj layer in an example block (Block 14). Comparisons for all other blocks are given in Appendix B.2 Top row: Comparison of fp16 and bf16 variants. Bottom row: Our 6B model trained with optimal hyper-parameters compared against similar sized OPT-6B and BLO0M-7.1B models
Additionally, inspired by Lin et al. (2019). we use spectral norm to measure the maximum degree of noise amplification for each token activation.
For given weight matrix W Rxxo, and input token activation noise x5 Rh the spectral norm || 2|2 is defined as
Wxx
l2
W 2 = sup
maa
|x1|22
x,
where maaz is the largest singular value of W.
As seen in Figure 5 we observe that the spectral norm of the fp16 variant is 4x higher than the bf16. In addition, on the bottom row of Figure [5] we observe that both BLOOM and our model have generally lower spectral norm than OPT 6B that is far more sensitive to quantization.
Related Work
Challenges of Quantization Scale Recently, there have been several studies to characterize the emergence of outliers at scale, and relate this to the difficulties in post-training quantization of both weights and activations (Dettmers al., 2022, Wei et 2022b Puccetti al.) 2022). Dettmers et al. (2022) depict phenomenon of emerging outliers by observing that large outlier dimensions systematically emerge certain scale (6.7B parameters) which hamper quantization attempts. Extreme outliers at scale was also empirically confirmed in follow-up works (Zeng et al., 2022] Xiao et al. 2022). The causes of outliers have also been the subject recent work, Puccetti et al. (2022) observe that in Masked Language Models (MLMs) the magnitude of hidden state coefficients corresponding to outlier dimensions correlates with the frequency of encoded tokens in pre-training data. Wei et al. (2022b) observe that LayerNorm scaling (g) amplifies the outliers and can be suppressed using a modified LayerNorm and token-wise clipping. |Wortsman et al. (2023) consider large-scale vision-language models and show that quantization techniques are more stable if the network is trained and initialized so that large feature magnitudes are discouraged.
Most mitigation strategies to quantize in the presence of outliers has required more complex quantiza- tion techniques. For example, [Dttmers et al. (2022) propose selective mixed-precision computation by only computing the outliers at higher precision. However, such setup proves difficult to map hardware, limiting the inference speedup. Xiao al. (2022) propose to smoothen out these outliers by migrating some of the activation variances into the model weights with appropriate scaling.

---

Although the authors demonstrate the ability of this framework to scale to large models, additional rescaling is required for activations which leads additional latency overhead without specialized kernels. Another limitation of Xiao et al. (2022) is that it relies on the assumption that outliers exist in activations, and that weights can bear additional outliers and still easy to quantize.
Our work is the first to show that outliers are not inherent to scaling large language models. Rather than an emerging property, they are result of particular training methods. Compared to previous methods using extensive quantization schemes with custom kernels (Dettmers et al., 2022). our work applies PTQ using simple, one-shot linear weight and activation quantizations which can take advantage of AVIDIA-provide CUTLASS kernels, leading to a significant decrease in latency and memory footprint (T)
Conclusion
We present rigorous study of the effect of how various optimization choices affect INT8 PTQ with the goal of reconciling the recent contradictory observations regarding emergent properties in Large Language Models. We show that regularization directly impacts PTQ performance and that higher levels of regularization through common techniques such as weight-decay. and gradient-clippee leads to lower post-training quantization degradation. We further demonstrate that the choice of half-precision training data type has a significant impact on PTQ performance emergent features significantly less pronounced when training with bf
Broader Impact Our work serves as a useful counter-example to scholarship which has advanced the notion that certain properties depend only on model scale (Wei al. (022a).. Rather, our results support the conclusion that optimization choices play large role whether emergent properties are present. We believe there is more work to be done here. We also hope that the insights gained from our work illustrate the significant impact the underlying hardware can have on PTQ. Currently, bf16 training is possible on TPUs and only very recently introduced to A100 & H100 GPUs. Finally, we belive our results present an impactful formula for training models which are inherently easier to quantize at scale, making these models more accessible for deploying in variety of deployment environments.
Limitations and Future Work We do not vary the architectural design (except for the positional embedding type) and training objective our experiments given our goal a controlled experimental set-up and the large computational cost of each variant. We leave this to the future work. In addition, worthwhile to explore the impact of finetuning with the recommended optimization choices on make pre-trained model more coantizzatio-friendll. as well the theory behind why these optimization choices correlate with better compression results.
Acknowledgements
We thank João Araújo, Milad Alizadeh and other colleagues in Cohere & Cohere For AI for help- fulfeedback and support. Additionally, we thank Jimmy Ba for helpful discussions surrounding the topic of this paper. We also thank Tim Dettmers for assisting in replicating the outlier dimension definition and results in int8. LLM()
10

---

References
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals. Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-Scale Machine Learning Heterogeneous Systems, January 2015. htllhttps:/// sittps://wwyy tensorflow.cooo orgg/ Software available from tensorflow.org.
Orevaoghene Ahia, Julia Kreutzer, and Sara Hooker. The low-resource double bind: An empirical study of pruming for low-resoure, machine translation. In Findings of the Association for Compu- Iational Linguisties: EMNLP 202/, pp. 3316-3333, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/202 findings-emmlp.222. URL https: http:////caattologyyyy. org/ 2021. 200....finiige....e findinginniiiii emn 282
Alham Fikri Aji and Kenneth Heafield. Compressing Neural Machine Translation Models with 4-bit Precision. In Proceedings of the Fourth Workshop on Neural Generation and Translation, pp. 35- 42, Online, July 2020. Association for Computational Linguisties. doi: 10. (0.18853/v1/2220.ngg 1.4.
URL htttps://watt aclweb. org/ anthology/2020..og
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasumuru, Giridharan Anantharaman, Xian Li, Shuohui Chen. Halil Akin. Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O'Horo, Jeffrey Wang. Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, and Veselin Stoyanov. Efficient large scale language modeling with mixtures of experts. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 11699-1111 11732, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://
aclanthology.orgg// 20222. emn emnlp-manle hwtpppmmi......
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference Artificial Intelligence, 2020.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. CoRR, 0is/2109.119455 2021. URL https
12948
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstent Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022. URL https://arxiv org/ ab0/
2204. 02311
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Training deep neural networks with low precision multiplications. arXiv e-prints, art. arXiv: 1412.004, Dec 2014.
Saurabh Dash and Saibal Mukhopadhyay. Hessian-driven unequal protection of dnn parameters for robust inference. Proceedings of the 39th International Conference on Computer-Aided Design, ICCAD '20, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781 9781450380263. doi: 10.1 4445/33400302.41557777 URt/tttss: :tatttss:///o// //doi .ar.aaorg//... 10. 1145/4400332
34 3415679

---

Saurabh Dash, Yandong Luo, Anni Lu, Shimeng Yu, and Saibal Mukhopadhyay. Robust processing in- memory with multibit reram using gressian-driver mixed-precisss computation. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 41 41(4):1006-109, 2022. doi: 10.1109/TCAD2201...20 .3078408.
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc' au- relio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, Quoe Le, and Andrew Ng. Large scale distributed deep networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc.. 2012. URL https: /ttpoeee//proooeiinngg neuriiinnnnnnnnn.ssoere..rr neurips. cttccc//pap wacppaprr... httppppppereeeeee_eeeeeppalpeeee www.fflllepppee wttpsefffeeeeeeeeeeeeeeee pttp///pape/////////ee// 0eca5700666611100822288888668813
pdf
Mostafa Dehghani, Anurag Amnb,, Lucas Beyer, Ashish Vaswani, and Yi Tay. The efficiency 12894
misnomer. CoRR, (bss/210.1120 12894, 2021. httptttppp://....ttrrrrtttteteee org/
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. GPT3.int8(: 8-bit matrix multiplication for transformers at scale. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL net/ wit.middddddiioxooddoooo
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Annual Meeting of the Association for Computational Linguistics, 2021. URL httttp:://arxiv.org//p/.. pttpp/////....//1//// 10360
pdf
Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. Paraphrase-driven learning for open question answering. In Annual Meeting of the Association for Computational Linguistics, 2013.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023. URL htttpp://a.xi..org////// wttp:////aa////...0///// 2ttp:///300007777
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training abbabb//
quantization for generative pre-trained transformers, 2022. URL
2210.17322 17323
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks, 2019. URL atttps:/aaxxv..org/bbb//////
Georgi Gerganov. llama.cpp.
htw.aub..coomggggggoo//laaaaaaa cpp 2023.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. survey of quantization methods for efficient neural network inference, 2021. URL https
//arxiv.
13630
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1.
MIT Press, 2016.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep Learning with Limited Numerical Precision. CoRR, abs/1552.00251, 2015. URL http://arxiv.. wrg. abs/ 1502.02553 02551
B. Hassibi, D. G. Stork, and G. J. Wolff. Optimal Brain Surgeon and general network pruning. In IEEE International Conference on Neural Networks, pp. 293-299 vol.1, March 1993a. doi: 10.1 109/ICNN.1993..99
Babak Hassibi, David G. Stork, and Stork Cre. Ricoh. Com. Second Order Derivatives for Network Pruning: Optimal Brain Surgeon. In Advances in Neural Information Processing Systems 5, pp. 164 171. Morgan Kaufmann, 1993b.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), 2016. URL httpp::/aan org/abs/160s 08411
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in Neural Network, arXiv e-prints, art. arXiv: 1003::5503.025 1,531, Mar 2015.
12

---

Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky. Ilya Sutskever, and Ruslan R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors, 2012.
Sara Hooker. The hardware lottery. Commun. ACM, 64(12):58-66 65, nov 2021. ISSN 0001-0782. doi: 10.1145/3467017. URL https://doi.oorr//// 10.
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. ArXiv e-prints, April 2017.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran EI- Yaniv, and Yoshua Bengio. Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations. CoRR, 0ass/609.07061.. 2016. http://///r wttp:/orrggaabbbb..0.000
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Improving post training neural quantization: Layer-wise calibration and integer programming. arXiv preprint arXiv:2006. 105788 2020.
F. N. landola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and 0.SMB model size. ArXiv e-prints, February 2016.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and Training of Neural Networks for Efficient Onteger-ArithmeticOOnn Recognition, Jun 201 8. doi: 10. 10.1 8.00286. htttttp: //xx. dow.ddo... org/
Inference. 2018 IEEE/CVF Conference on Computer Vision and Pattern org/10.1110 100 1109/
CVPR. 2018. 00286
Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, AI Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho. Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni. Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salck, Emad Samadiani, Chris Sevem, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang. Eric Wilcox, and Doe Hyun Yoon. In-datacenter performance analysis of a tensor processing unit. SIGARCH Comput. Archit. News, 45(2):1 jun 2017. ISSN 0163-5964. doi: (0.1145/3140599,33080) org 11111...111111000 3080246
URL httLhht/////a
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. study of 6 for deep learning training, 2019.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.082 100361, 2020. URLhhttps: arxiv. org/ abss/000 08361
Yulhwa Kim, Jaeyong Jang, Jehun Lee, Jihoon Park, Jeonghoon Kim, Byeongwook Kim, Se Jung Kwon, Dongsoo Lee, et al. Winning both the accuracy of floating point activation and the simplicity of integer arithmetic. In The Eleventh International Conference on Learning Representations.
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: whitepaper. arXiv preprint arXiv:1806.083.40 08342, 2018
13

---

Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 12.1865/1/DD18-2011 URL htttps:/aallathhoo.gg
Ashish Kumar, Saurabh Goyal, and Manik Varma. Resource-efficiet Machine Learning in 2 KB RAM for the Internet of Things. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp. 1935-1944, International Convention Centre, Sydney, Australia, 06-11 html
Aug 2017. PMLR. URL http://proceediinn mlr. wwt.:ppressssouiieeeep
Andrey Kuzmin, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, and Tijmen Blankevoort. Fp8 quantization: The power of the exponent, 2022.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal Brain Damage. In Advances in Neural Information Processing Systems, pp. 598-605, Morgan Kaufmann, 1990.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang. Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction, 2021. URL
https://arxiv.c org/ abs/2102. 05426
Ji Lin, Chuang Gan, and Song Han. Defensive quantization: When efficiency meets robustness, 2019.
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam M. Shazeer. Generating wikipedia by summarizing long sequences. ICLR, abs/1801.10198, 2018.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL httphtt:/pppeereieewwnooo www.:neenfffeemmcmmmm Bkg6RiCqV7
C. Louizos, M. Welling, and D. P. Kingma. Learning Sparse Neural Networks through L_0 Regular- ization. ArXiv e-prints, December 2017.
Ziyang Luo, Artur Kulmizev, and Xiao-Xi Mao. Positional artefacts propagate through masked language model embeddings. In Annual Meeting of the Association for Computational Linguistics, 2020.
Paulius Micikevicius, Sharan Narang. Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed Precision Training, 2017.
Nasrin Mostafazadeh,, Nathanael Chambers, Xiaodong He. Devi Parikh, Dhruv Batra, Lucy Van- derwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper under- standing of commonsense stories. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 839 849, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v 1/ 1098. URL
orw.:16-g1/1/10 1098
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pp. 7197-7206. PMLR, 2020.
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tij- men Blankevoort. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295. 2021.
Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta. Exploring Sparsity in Recurrent Neural Networks, arXiv e-prints, art. arXivv1704.05119, Apr 2017.
Nvidia. Nvidia a100. waw.chhhttcccueeeeeeeaeaaaeeeeae
URL http:::/eeeoorrs...iiiiiiiiie com en us- www.///en-ssss-emmmnsseee genomics-ep/
14

---

Kelechi Ogueji. Orevaoghene Ahia, Gbemileke Onilude, Sebastian Gehrmann, Sara Hooker, and Julia Kreutzer. Intriguing properties compression on multilingual models. 2022. doi: 10.48550/
ARXIV.2211.02755 URI.https httphttss///riii..oooo abs/ 2211 02738
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaclla Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), Pp. 1525-1534, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. (18856/v1//P1/ 1144. htwpo/loog.ooo/11-111111111 1144
URL
Gunho Park, Baeseong Park, Sungjae Lee, Minsub Kim, Byeongwook Kim, Se Jung Kwon, Youngjoo Lee, and Dongsoo Lee. nuqımm: Quantized matmul for efficient inference of large-scale generative language models, 2022a. htttss/////arxi org/ /bs/// 2206 2206609557
Minseop Park, Jaeseong You, Markus Nagel, and Simyung Chang. Quadapter: Adapter for gpt-2 quantization. arXiv preprint 11.xi:22111.6912,, 2022b.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 1310-1318, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL httpss:/ppoceeeding.... mmr
html
Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022.
Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice Dell Outliers dimensions that disrupt transformers are driven by frequency, 2022. URL http://aaiiv.oorr/b//22200
11380
Jerry Quinn and Miguel Ballesteros. Pieces of Eight: 8-bit Neural Machine Translation. In Pro-
ceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume (Industry Papers), pp. 114 120, New Orleans Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/1/N118-300 URL
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners.
Jack W. Rae, Sehastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang. Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganimi, Laurent Sifre. Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman. Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis amp; insights from training gopher, 2021.
abs/ 11446
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Tea Transformer, 2020.
15

---

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '20. IEFE Press, 2020. ISBN 9781728199986.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint
2019.
Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement Pruning: Adaptive Sparsity by Fine-Tuning, 2020.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié. Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:221 1005000, 2022.
Abigail See, Minh-Thang Luong, and Christopher D. Manning. Compression of Neural Machine Translation Models via Pruning. arXiv e-prints, art. arXiv:1606.09274 Jun 2016.
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y Fu, Zhiqiang Xie. Beidi Chen, Clark Barrett, Joseph E. Gonzalez, et al. High-throughput generative inference of large language models with single gpu. arXiv preprint arXiv:2303. 06665, 2023.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting Journal of Machine Learning Research, 15(1):1929-1958 2014.
Nikko Ström. Sparse Connection And Pruning In Large Dynamic Artificial Neural Networks, 1997.
Marcos Treviso, Tianchu Ji. Ji-Ung Lee, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Pedro H. Martins, André F. T. Martins, Peter Milder, Colin Raffel, Edwin Simpson, Noam Slonim, Niranjan Balasubramanian, Leon Derczynski, and Roy Schwartz. Efficient methods for natural language processing: A survey, 2022. URL https://arxiv..o org/ abs/ 2209 .0009
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp.
2017.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: stickier benchmark for general-purposs. language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL http:tttsss///pcceeddiggg newrips....... /ttpcc/ppppp/... files/ Lttles//pppr///199 pdf
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022a.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. arXiv preprint arXiv:2209.13383 2022b.
W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li. Learning Structured Sparsity in Deep Neural Networks. ArXiv e-prints, August 2016.
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. Stable and low-precision training for large-scale vasion-language models, 2023.
Hao Wu, Patrick Judd, Xiaojie Zhang. Mikhail Isaev, and Paulius Micikevicius. Integer quantization for deep learning inference: Principles and empirical evaluation. arXiv preprint arXiv:2004.09602 2020.
16

---

Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models, 2022. URL httph://aariii org/abs/ 2211 10438
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang. Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27 11111668111,,,, 2022.
Zhewei Yao, Cheng Li, Xiaoxia Wu, Stephen Youn, and Yuxiong He. A comprehensive study post-training quantization for large language models, 2023.
Joanna Yoo, Kuba Perlin, Siddhartha Rao Kamalakara, and João G. M. Araújo. Scalable training of language models using jax pjit and 1puv4, 2022. URL https://arxiv... wttp:/iv..rrggbb///2000/ 06514
Ofrr Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-Neurlii Edition (EMC2-NIPS) pp. 36-39. IEEE, 2019.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Anmual Meeting of the Association for Computational Linguistics, 2019.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang. Glm-130b: An open bilingual pre-trained model, 2022. URL /astps://arxiv... org/ abs 2210 02414
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Oit, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. URL httts:://axt:: org/ /bs/ 2205.01068
Shaojie Zhuo, Hongyu Chen, Ramchalam Kinattinkara Ramakrishnan, Tommy Chen, Chen Feng, Yicheng Lin, Parker Zhang, and Liang Shen. An empirical study of low precision quantization for tinyml. arXiv preprint 220.0.200,, 2022.
17


Reviews:
[{"review_id": "RvTXSZQDum", "reviewer": "Reviewer_egL3", "summary": "\nThe authors study the question of whether quantization (of weights and\nactivations in decoder-only language transformer architectures) fundamentally\nreduces performance at large parameter counts, or whether this degradation of\nperformance, observed in prior works on this topic, is simply a consequence of\ncertain design choices in the optimizer. They consider post-training\nquantization (PTQ) schemes, which simply quantize the weights and activations\nof a pretrained (half-precision) model at inference time (to int8/int8) without any extra\nquantization-aware retraining, and conduct experiments at the 6B parameter\nscale (which entail measuring the accuracy drop after PTQ on 8 datasets)\nvarying the magnitude of weight decay, dropout rate, gradient clipping, and\nhalf-precision type (fp16 vs. bf16). Their results suggest that larger weight\ndecay, gradient clipping, and bf16 data types improve PTQ performance; they\nthen train with appropriate settings at the 60B parameter level, and\ndemonstrate that they can significantly improve the average PTQ performance\nrelative to a comparable scale model (OPT-66B); Figure 1 highlights these\nresults. The authors then study the weights and activations in the OPT-66B\nmodel and their own model in order to assess how their design choices have\naffected various properties that played a role in prior narratives on why PTQ\nperformance might fundamentally be limited at large scales (specifically, the\nrole of \"outlier dimensions\", but also general quantization-type metrics such\nas reconstruction RMSE).\n\n", "questions": "In line 268, performance differences on BLOOM are claimed; are these results\navailable or reported? From the appendix Table 4, it seems that these results\nare referring to int4 quantization of weights only?\n\nThe authors do not test their design choices at the full 175B scale, and the\nreason for this is not discussed. It would be helpful to know whether this is\njust due to compute obstructions to training at this scale, or whether it is\nbecause the authors' methods are not scaling properly to this scale.\n\n", "limitations": "Limitations are clearly specified throughout and in the conclusion; there is perhaps somewhat of a missed opportunity to discuss potential negative societal impact of LLM quantization.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors' insights lead to a significant improvement over OPT-66B PTQ\naccuracy drop with a comparable parameter count (the accuracy actually improves\non average).  Their results also highlight useful general takeaways about how\nregularization can help mitigate PTQ performance drop, including the utility of\nbf16, weight decay, and gradient clipping, and they provide reasonable\nexplanations for why.\n\nThe authors have presented most methodological choices and details of their\nexperimental setup very clearly, which will aid reproducibility of their\nresults and the ability of researchers to build off the results of their study.\n\n", "weaknesses": "It might be helpful to include additional quantitative comparisons to prior\nwork; currently the comparisons to prior art are substantial but mostly\nqualitative. iT seems one of the cited works in the related work section,\nFlexGen, achieves INT4 quantization of OPT-175B on Lambada and WikiText with\nnegligible performance (acc/ppl) penalty.  It might also be nice to add brief\nexperimental studies demonstrating improved throughput/memory footprint of the\nauthors' PTQ approach, to substantiate the claims made at a few points in the\npaper that the authors' approach will have such benefits over the\nmixed-precision approach of Dettmers et al.\n\nThe description of the experimental protocol for section 4 is somewhat confusing -- lines\n145-150 and 193-198 both suggest that training was done using early checkpoints\non 410M parameter models before \"scaling\" to 6B models, but no results seem to\nbe reported for 410M parameter models. It seems that this detail could be\nomitted if no results of substance are reported to corroborate it. In addition,\nit is not made completely clear in section 3.2 whether the 6B experiments\nreported in section 4 are actually done via training-from-scratch or done from\nearly checkpoints; the text in lines 145-150 suggests that section 4 results\nare not actually trained from scratch, but the actual protocol in section 3.2\ndoes not clearly state this.\n\nThe description of the regularization study in the contributions section feels\nslightly misstated -- the \"controlled large-scale study\" alluded to does not\nactually seem to \"train from random initialization\" (uses early checkpoints),\nand the hyperparameter search is actually only conducted at 6B parameter models\n(from early checkpoints). This suggests an implicit hypothesis of this study is\nthat performance differences observed at \"medium-scale\" models are predictive\nof those at larger scales; the fact that this is borne out up to 52B parameter\nmodels seems to be an interesting consequence of the work, but then it does not\nseem completely accurate to claim a systematic study up to this scale as a\ncontribution.\n\nThe takeaways from the weight/activation studies in section 5 do not seem to\nlend themselves to straightforward conclusions or insights into why the\nauthors' design choices lead to superior PTQ robustness (see discussion at line\n316). It might help the presentation if the authors selected a small subset of\nthe activation/weight metrics presented which lend themselves to a coherent\ninterpretation of why the authors' model and OPT/BLOOM have different PTQ\nperformance. One key takeaway that is mentioned both in the introduction and\nthe conclusion, that the authors' work refutes the hypothesis that outliers are\nnot inherent to scaling large language models, is not even presented in the\nmain body (it has been relegated to appendix B.3), and lines 323-324 suggest\nthe authors could not find correlations between outlier detection recipes and\nPTQ performance. Discussing this study more in section 5 would seem necessary\ngiven the way the authors have presented it as a contribution.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank  **R egL3** for noting that our findings “lead to a significant improvement” and our results “highlight useful general takeaways about how regularization can help mitigate PTQ performance drop”. We are glad they found that our experimental setup is presented “very clearly”, aiding “the ability of researchers to build off the results”.\n\n> It seems one of the cited works in the related work section, FlexGen, achieves INT4 quantization of OPT-175B on Lambada and WikiText with negligible performance (acc/ppl) penalty\n\nThanks to **R egL3** for this opportunity to clarify the difference between our work and Flexgen. We note that FlexGen compresses only weights but not activations. We are tackling the harder problem of compressing both weights and activations. Thus, it is not comparable with our quantization performance. Conceretly, weights and activation quantization benefits the inference latency of both compute-bound (large batch size,long input sequence and short generation) and memory-bandwidth-bound (small batch size, short input sequence and long generation) use cases while weights only quantization (FlexGen) targets memory-bandwidth-bound use cases only. We will further clarify this difference in the final manuscript and the strength of our contribution, which is currently described briefly in Section 2 and Appendix C.\n\n> an implicit hypothesis of this study is that performance differences observed at \"medium-scale\" models are predictive of those at larger scales; the fact that this is borne out up to 52B parameter models seems to be an interesting consequence of the work, but then it does not seem completely accurate to claim a systematic study up to this scale as a contribution.\n\nWe conduct a large-scale controlled study on the relation between optimization choices (weight decay, gradient clipping, dropout and half-precision training) and quantization robustness with 13 different 6B parameter model variants. While this requires a compute intensive set-up, it allows us to rigorously disentangle what factors actually influence sensitivity to quantization. We then translate the conclusions inferred from the 6B experiments to train models ranging from 410M to 52B parameters from scratch, and validate the robustness of models trained by our recipe. The training of these model variants is incredibly compute intensive (training the 52B model took ~20 days itself) which is why we called this a “large scale” controlled study. We believe this term is justified given the breadth, rigor and computation cost of our experiments.\n\n> lines 145-150 and 193-198 both suggest that training was done using early checkpoints on 410M parameter models before \"scaling\" to 6B models, but no results seem to be reported for 410M parameter models.\n\nRegarding experiments on 410M, we did the same ablations with 410M models as we did with 6B variants, but omitted the results due to space. We will include them in the final manuscript. We note that we use 6B model experiments (trained from scratch for 75k steps) to infer the best optimization choices because as reported in Dettmers et al., \"emergent\" quantization sensitivity starts at 6B for the other models, and is representative of larger model sizes as opposed to 410M.\n\n> lines 323-324 suggest the authors could not find correlations between outlier detection recipes and PTQ performance. Discussing this study more in section 5 would seem necessary given the way the authors have presented it as a contribution.\n\nWe fully agree with **R egL3** that having a detailed discussion on outlier dimensions earlier in the manuscript would help make our argument more cohesive. We will move the main parts of the detailed outlier analysis from Appendix B.3 to section 5 for the final manuscript.\n\n> It might help the presentation if the authors selected a small subset of the activation/weight metrics presented which lend themselves to a coherent interpretation of why the authors' model and OPT/BLOOM have different PTQ performance\n\nWe agree with this suggestion from **R egL3**. We point out that RMSE($X,\\widehat{X}$) and input token activation STD strongly correlate with degradation trends we observe in OPT and BLOOM models. Comparing our model with the other two, we look at all the attention projection layers (attn-kqv-proj) across the models and find that RMSE($X,\\widehat{X}$) for OPT and BLOOM models is significantly higher than corresponding values in our 6B model. This is depicted in Figure 11, Appendix B.2. More concretely,  RMSE($X,\\widehat{X}$) values for OPT-6B and BLOOM-7.1B are up to **33.2x** and **13x** the corresponding values for our fully trained 6B model, respectively. A similar pattern exists for STD values which are consistently higher for OPT-6B and BLOOM-7.1B than that of our model, up to 3.6x and 2x, respectively.\n\nBased on **R egL3’s** suggestion, we will put more emphasis on these metrics and make more clear that RMSE and STD explain the OPT & BLOOM degradation patterns in the final version of the paper.\n\n> In line 268, performance differences on BLOOM are claimed; are these results available or reported? From the appendix Table 4, it seems that these results are referring to int4 quantization of weights only?\n\nThe numbers in lines 266-273 are regarding INT4 weight-only column-wise quantization. We compare against the INT4 BLOOM results reported in the GLM-130B technical report in Table 10. We will make this source clear in the final manuscript.\n\n> The authors do not test their design choices at the full 175B scale, and the reason for this is not discussed\n\nThis was purely due to the enormous cost of fully training a 175B parameter model given all our model variants are trained from random initialization. We note that the 52B model required training time of roughly 20 days and in total we trained 17 models of sizes ranging from 410M to 52B to carefully measure the impact of each ablation. We will add this explanation to the final version of the paper."}, {"review_id": "phFsBSCpUT", "reviewer": "Reviewer_vYyu", "summary": "There is a quantization tradeoff in LLMs, where quantization methods drop in particular for some larger models. This problem has been attributed to large activation magnitude \"outliers\". The authors note that activation outliers are not an emergent property of LLMs and that instead a byproduct of how the networks were trained.  The authors compare different training strategies  (w/o weight-decay, grad clipping, dropout, half precision) and show that while many configurations can lead to the same performance, after one-shot post training quantization (OPT), the performance can be very different. The authors then train multiple models and show that their results hold for models even at 100B parameters. ", "questions": "- The authors focus on network retraining for improved OPT quantization. For finetuning many approaches have been developed to improve the weight distribution (such as temperature scaling) Do the authors have an idea of how helpful that would for quantization if applied during finetuning?", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is clear and easy to follow.  It describes the motivation/problem, related work and the contributions clearly.", "weaknesses": "The paper proposes a training strategy for improved quantization. \n- As noted by the authors, other work has  also noted that quantization techniques are more stable if the network is trained initialized so that large feature magnitudes are discouraged [1]. Given that this recipe is provided given one architecture I was wondering if the authors could comment on whether they believe the hyperparam configurations will hold across quantization levels/architectures/ pre-training / tasks. Having a paragraph that summarizes this would make the paper more impactful. \n- One component of the recipe is using bf16. Fig 5 and other work [2] shows that bf16 is better than fp16, Given that OPT is trained used using fp16, I was wondering how much of the recipe is a benefit of using bf16? \n- Could the authors provide a summary of the ablations of the recipe?\n\n[1] Worstman et al. 2023\n[2] https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank **R vYyu** for their helpful feedback and their observations noting that our paper is “easy to follow” and “clear” in describing the motivation, problem, and related work. \n\n> As noted by the authors, other work has also noted that quantization techniques are more stable if the network is trained initialized so that large feature magnitudes are discouraged [1]. Given that this recipe is provided given one architecture I was wondering if the authors could comment on whether they believe the hyperparam configurations will hold across quantization levels/architectures/ pre-training / tasks. Having a paragraph that summarizes this would make the paper more impactful.\n\nThanks for the detailed suggestion, and the opportunity to speak to the transferability of our results. We have provided further details on our position regarding architectural transferability in the global response, but in short, we believe slight architectural differences such as the type of positional embedding do not impede the transferability of our results. \n\nWe also benchmarked on diverse types of tasks in our evaluation suite (lines 203-207), covering 8 tasks of 4 different types (Multiple choice (MC) completion, MC Co-referencing, Generation, and Question Answering). Hence, we believe that our results are not dependent on the nature of the downstream evaluation task. We will add a paragraph clarifying these details to summarize our position on the transferability of our results to settings different than ours to the final manuscript.\n\n>One component of the recipe is using bf16. Fig 5 and other work [2] shows that bf16 is better than fp16. Given that OPT is trained used using fp16, I was wondering how much of the recipe is a benefit of using bf16?\n\nWe believe that bf16 training has a significant impact on downstream PTQ as fp16 training leads to the largest swing in degradation amongst all experimental axes (Figure 3, Left). When coupled with weight decay of 0.01 and otherwise PTQ optimal hyperparameters, fp16 training leads to high degradation as early as 15k steps and at 75k steps, leads to the largest drop in PTQ performance out of all the early checkpoint model variants (Figure 3, Right). Simply switching to bf16 training leads to a 6.7x reduction in PTQ degradation (inferred from Figure 2a, second column) which is the largest for any hyperparameter we studied. Moreover, as we briefly mention in lines 244-247, BLOOM-176B which is trained with bf16, is significantly more robust to PTQ than OPT-175B which is trained with fp16. This claim is based on the results published in SmoothQuant (Table 4) where INT8 quantization of both activations and weights leads to a 4% degradation for BLOOM-176B where OPT-175B shows a 39.2% drop. \n\n>Could the authors provide a summary of the ablations of the recipe?\n\nTo understand the effects of dropout, gradient-clipping, weight-decay, and training computation datatype, **per experimental axis we train models from random initialization, varying each hyper-parameter while keeping others the same** to establish trends by comparing the mean PTQ degradation across 8 diverse tasks. In total, we train 13 6B parameter models from random initialization for 75k steps each and draw the following insights: we find that high weight-decay, gradient-clipping, no dropout, and most significantly, bf16 as the computation data-type, lead to the best PTQ performances in their respective axis. To validate that these insights carry to different scales and full training, we fully train (for more than 200k steps) a series of new, i.e. randomly initialized, 410M, 6B, 13B, and 52B sized models with the optimal hyper-parameters inferred from the previously mentioned 6B experiments, and observe minimal PTQ degradation in the evaluation suite. For example, across HellaSwag, PIQA, WinoGrande, and LAMBADA, our fully trained 52B model achieves negligible mean degradation while the similarly sized OPT-66B shows ~42% degradation. \n\n> The authors focus on network retraining for improved OPT quantization. For finetuning many approaches have been developed to improve the weight distribution (such as temperature scaling) Do the authors have an idea of how helpful that would for quantization if applied during finetuning?\n\nIn our work we completely focused on the impact of pretraining optimization choices on post-training quantization without any additional step such as fine-tuning. However, we agree that measuring the impacts of optimization choices during the fine-tuning step will be an interesting future work. We think that high regularization may have a degree of impact similar to the methods that improve weight distributions. We will add this discussion to the future work in the final manuscript. \n"}, {"review_id": "F16AOMU90z", "reviewer": "Reviewer_xK34", "summary": "Recent research suggests that the trade-off incurred by quantization is an emergent property, resulting in sharp performance drops in models with over 6 billion parameters. In this study, the authors investigate whether quantization cliffs in performance are solely determined by model scale. Considering the increased research focus on understanding the emergence of certain properties at scale, the authors provide a valuable counter-example.\n\nThe authors propose the possibility of optimizing a quantization-friendly training recipe that suppresses outliers in large activation magnitudes. They find that outlier dimensions are not an inherent consequence of scale but rather sensitive to the optimization conditions during pre-training. This discovery opens up avenues for more efficient quantization methods and raises questions about the inherent nature of other emergent properties and their sensitivity to optimization and architecture design choices.\n\nThe authors successfully apply their approach to quantize models of various sizes, ranging from 410 million to 52 billion parameters, with minimal performance degradation. This showcases the practical implications of their work and its potential to facilitate effective quantization across a broad range of model scales.", "questions": "None", "limitations": "See weaknesses.", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors conducted a controlled experiment, maintaining consistent architecture while varying key optimization choices. This approach allowed them to systematically investigate the factors influencing sensitivity to quantization.\n\nThe study rigorously analyzed the impact of optimization choices on sensitivity to quantization in models ranging from 410 million to 52 billion parameters. By carefully controlling these factors, the authors demonstrated that reoccurring activation outliers are not a universal emergent property of large language models at scale.\n\nDespite the massive scale of 52 billion parameters, the authors achieved minimal performance degradation (0.26%) across eight tasks when applying quantization to both activations and weights. This showcases the effectiveness of their optimization approach in mitigating the impact of quantization on model performance.\n\nThe study provides a detailed analysis of activations and weights, offering insights into the key characteristics influencing sensitivity to quantization. These findings can guide future model design and pre-training strategies to enhance robustness and performance.", "weaknesses": "Even though the authors do a thorough empirical analysis, their work doesn't begin to explain why we observer this quantization phenomenon.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank **R xK34** for the positive feedback, including their observations that 1) We conducted a series of “controlled experiments, maintaining consistent architecture while varying key optimization choices” which allowed us “systematically investigate the factors influencing sensitivity quantization sensitivity” and infer a set of  PTQ optimal hyperparameters. 2) By using these choice to fully train a 52B parameter model, despite its “massive scale”,  we achieved “minimal performance degradation (0.26%) across eight tasks when applying quantization to both activations and weights”. Which showcases the effectiveness of our “optimization approach in mitigating the impact of quantization on model performance”.\n \n>Even though the authors do a thorough empirical analysis, their work doesn't begin to explain why we observe this quantization phenomenon.\n\nWe thank **R xK34** for the opportunity to address this feedback. As they mention, we systematically demonstrate the effect of the key optimization choices on the quantization sensitivity in models ranging from 410M to 52B. We find that high weight-decay, gradient-clipping, no dropout, and most significantly, bf16 as the computation data-type, mitigate unfavorable outlier distributions and lead to the best PTQ performances. Validating these findings, our fully trained 52B model achieves minimal PTQ degradation (0.26%) across 8 tasks.\n  \nWe relate this quantization robustness to our models’ weight and activation distributions. To show that our recipe leads to a more quantization-friendly activation distribution, we compare RMSE($X,\\widehat{X}$) and input token activation STD of our model with that of OPT and BLOOM models for attention projection layers. As shown in Figure 11 (Appendix B.2). RMSE($X,\\widehat{X}$) values for OPT-6B and BLOOM-7.1B are up to **33.2x** and **13x** the corresponding values for our fully trained 6B model, respectively. Similarly, STD values are consistently higher for OPT-6B and BLOOM-7.1B than that of our model, up to 3.6x and 2x, respectively. \n\nTo explain why some optimization choices generate activation distributions better suited for PTQ, we investigate the models’ weights. Specifically, we find that as the choice with highest impact on PTQ performance of the 6B model variants, bf16 training leads to significantly different model parametrization. Notably, Figure 6 (Appendix B.1) shows that for all attention projection layers (attn-proj-kqv) the model trained with bf16 consistently has a narrower weight distribution compared to the fp16 counterpart, for all layers across the network. Furthermore, bf16 training results in up to 2x higher STD for the gain parameters of the layer norms. (Figure 10, Appendix B.2).  \n\nHowever, we would like to note that **R xK34** raises an important question, and understanding the exact mechanism of how these choices interact with one another can be a new research. We leave this to the future work.\n"}, {"review_id": "5YDjmhVgV0", "reviewer": "Reviewer_TcCw", "summary": "\n\nThe paper empirically studies the effect of optimization/regularization hyperparameters -- weight-decay, dropout, gradient-clipping -- on post-training quantization (PTQ) performance of large language models (LLMs). Specifically, it performs PTQ to int8 by simple affine scaling, adopts a GPT-style architecture and trains models with various hyperparameter settings, and finds that higher levels of regularization generally helps lower the performance degradation after quantization. The paper finds that at 52B parameters, training with the best hyperparameters results in a 0.26% average performance degradation, and shows that activation outliers are not a universal emergent property of LLMs. Finally it computes various statistics of weights and activations such as std and spectral norm, for the proposed model v.s. other models, arguing they may offer insight about robustness to quantization. \n\n", "questions": "See weaknesses above.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper is mostly clearly written and easy to follow. \n\n2. The problem setting appears well motivated; I'm not an expert in NLP but a quick read of related work strongly suggests to me the difficulty of quantizing transformer-based language models, in particular when both the network weights and activations are quantized.\n\n3. The proposed method for mitigating performance drop due to quantization is surprisingly simple -- just use a higher amount of regularization. The empirical result (0.26% average performance degradation) seems quite good compared to much larger performance drops experienced by other existing models at int8 precision; however this point remains to be clarified (see weakness). ", "weaknesses": "1. In the abstract, it is claimed \"we find that outlier dimensions are not an inherent product of scale, but rather sensitive to the optimization conditions present during pre-training.\" Where exactly is this result established? The main experiments (section 4) mostly focused on the performance on zero-shot NLP tasks, rather than outlier dimensions. Moreover, \"outliers\" are frequently mentioned but not defined until later in the paper, which makes this argument hard to follow and it's unclear what definitive role outliers play in post-training quantization performance. \n\n2. The paper acknowledges that there's many possible factors contributing to a LLM's sensitivity to post-training quantization, including model choices (architecture, number of parameters/scale) and optimization/regularization choices, and recent work show conflicting findings between different model families (line 130-132). While paper focuses on the effect of regularization choices using a GPT style architecture (line 178), can we say anything about the effect of different model architectures on quantized performance? e.g., would the proposed methodology apply to an OPT architecture? As is currently written, the paper cannot exclude the possibility that the relatively small performance drop experienced by their trained model is a result of the model architecture, rather than the choice of optimization/regularization. \n\n3. Related to the above: I'm not familiar with LLM evaluation, and it's unclear how to interpret the 0.26% average degradation of the paper's largest (52B) model (line 260). Is it good or bad? It certainly seems impressive compared to the 42% drop encountered by the most comparable OPT model (line 263), but it seems pretty normal compared to performance drops of 0.09% and 1.36% when using various optimization hyperparameters (Figure 2) within the GPT-style model considered by the paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank **R TcCw** for their valuable feedback and noting that the problem of quantizing both activations and weights is “well motivated “ and that our paper is “clearly written” and “easy to follow”. \n\n\n> While paper focuses on the effect of regularization choices using a GPT style architecture (line 178), can we say anything about the effect of different model architectures on quantized performance? e.g., would the proposed methodology apply to an OPT architecture?\n\nThank you for the feedback. We clarify that both our model and OPT are decoder-only models and have many architectural similarities. We provide more details on the architectural similarities of these models in the global response. In summary, we believe our results are transferable to the OPT architecture.  \n\n> I'm not familiar with LLM evaluation, and it's unclear how to interpret the 0.26% average degradation of the paper's largest (52B) model (line 260). Is it good or bad? It certainly seems impressive compared to the 42% drop encountered by the most comparable OPT model (line 263), but it seems pretty normal compared to performance drops of 0.09% and 1.36% when using various optimization hyperparameters (Figure 2) within the GPT-style model considered by the paper.\n\t\nWe would like to clarify that 0.26% degradation for the 52B model is 1) across the full set of 8 tasks we describe in lines 203-211 2) **after training from random initialization for more than 200k  steps, not after only 75k steps**. With this fully trained 52B model, we achieve 0.0% degradation across HellaSwag, PIQA, WinoGrande, and LAMBADA (all of which are included in the previously mentioned 8 tasks), whereas the similarly sized OPT-66B which has a very similar architecture to ours, shows an extremely sharp drop (42%). Thus, the 0.26% degradation across the full eval suite and 0.0% across the previously mentioned 4 tasks is indeed very impressive.\n\nThe PTQ degradations presented in Figure 2 correspond to 6B models which are only trained for 75k steps, and are not directly comparable with that of fully trained models. We use these results to establish the trends in PTQ performance w.r.t. optimization choices. As shown in Figure 3 (right), performance drop, which is due to some optimization choices such as fp16 training, increases with step count. OPT models were also fully trained using fp16, which supports our insight from the 75k 6B checkpoints, that fp16 training leads to notably worse PTQ performance, at scale. \n\n\n>In the abstract, it is claimed \"we find that outlier dimensions are not an inherent product of scale, but rather sensitive to the optimization conditions present during pre-training.\" Where exactly is this result established? The main experiments (section 4) mostly focused on the performance on zero-shot NLP tasks, rather than outlier dimensions. Moreover, \"outliers\" are frequently mentioned but not defined until later in the paper, which makes this argument hard to follow and it's unclear what definitive role outliers play in post-training quantization performance  \n\nWe will update the final manuscript to provide a focused discussion on outlier dimensions earlier in the paper, to accompany the downstream PTQ degradations as opposed to Appendix B.3, where it currently resides. \n \nWhen we use the term outliers, we are referring to large magnitude features in the activations as defined and investigated in LLM.int8(). The authors provide study of activation outliers and provide evidence that they are responsible for performance drops after activation quantization at scales beyond 6.7B. Concretely, since the scaling coefficient used in INT8 quantization to rescale all values to be within INT8 range (-127,127), is directly proportional to the maximum values, large magnitude outliers lead to asymmetric and wide quantization distributions where most quantization bins are empty and small values are quantized to zero, causing significant information loss (Dettmers et al.). \n\nIn Appendix B.3, We present a detailed description of our reproduction of outlier dimensions as outlined in LLM.int8(). As the axis with the largest effect on PTQ performance, fp16 training increases the number of outlier dimensions to 8 while bf16 training corresponds to only 0 outliers. Note that the definition of outlier dimensions in LLM.int8(), was hard to generalize due to hard-coded magnitude threshold and number of affected layers and tokens, which failed to adapt to networks with different order of magnitude of activation. Therefore, we explore alternative metrics such as standard deviation (STD) of input token activations that are more easily generalizable. Comparing our 6B model with OPT-6B and BLOOM-7.1B, STD values of the OPT-6B and BLOOM-7.1B are consistently higher than that of our model, up to 3.6x and 2x, respectively that affirms the generalizability of this metric. \n"}]

Author rebuttal:
We greatly appreciate the thoughtful and positive feedback from the reviewers. We are encouraged that reviewers found the results notable [xK34, egL3] “insights lead to a significant improvement over OPT-66B PTQ accuracy”. Reviewers also positively reviewed our experimental setup as rigorous, clear and comprehensive [xK34, egL3, vYyu] “The study rigorously analyzed the impact of optimization choices on sensitivity to quantization in models ranging from 410 million to 52 billion parameters.” We are also glad that the reviewers found the problem we try address to be well motivated  [TcCw, vYyu] “Related work strongly suggests [to me] the difficulty of quantizing transformer-based language models, in particular when both the network weights and activations are quantized.”, and the paper to be “clearly written” and “easy to follow” [TcCw, egL3, vYyu]. 

We are heartened by the reviewers' acknowledgment of the potential brought by our work for opening up avenues for more efficient quantization methods, raising questions about the inherent nature of other emergent properties “the authors demonstrated that reoccurring activation outliers are not a universal emergent property of large language models at scale”  and “provides a detailed analysis of activations and weights, offering insights into the key characteristics influencing sensitivity to quantization” [xK34, egL3, vYyu].

Below we take this opportunity to clarify some of the shared concerns between reviewers: 


**1. On the transferability of our results to the OPT family and models with different architectures**

[vYyu] and [TcCw] raised valuable points asking how transferable our insights are to other model families such as OPT.  For clarity, all the models that we compare to in the paper including OPT and BLOOM share the same GPT-style architecture.

All the models that we experimented with are architecturally very similar to the OPT family as they are both decoder-only, pre-layernorm (where the layernorm is placed before the linear layers), use vanilla attention, and use learned positional embeddings. They differ in the number of blocks and the hidden dimensions of models at different sizes. Since we thoroughly experimented with different model sizes, we believe that our insights directly translate to the OPT family’s architecture. 

However, we realize that the type of positional embedding is often a differentiating factor between architectures e.g. OPT models use learned embeddings (same as us) whereas the BLOOM family uses ALiBi. As a result, in the rebuttal phase, we trained 3 models to 75k steps using learned and sinusoidal positional embeddings, and ALiBi. We used otherwise optimal hyper-parameters (bf16 training, weight decay=0.1, dropout=0.0, gradient clipping=1.0). We did not see a notable difference in downstream PTQ degradation as all values were within 0.06% of another. 

Therefore, we believe that it is regularization choices rather than small architecture differences, significantly impact models’ PTQ performance. In support of this, our fully trained 52B model shows 0.0% degradation across a set of 4 tasks whereas a similarly sized OPT-66B model shows a 42% drop, confirming this impact. 


**2. Clarifications on our experimental setup**

As raised by [egL3], we would like to provide further clarification on the details of our training and experimental setup. By the term “early checkpoints”, we do not mean fine-tuning or training from pre-trained checkpoints, but we refer to training from scratch and stopping relatively early at 75k steps (instead of training fully for more than 200k steps). Therefore, we do not perform any form of fine-tuning from pre-trained checkpoints on any of the models reported in the paper. We would like to clarify that **all of our models including the 13 early checkpoint 6B models, and the fully trained 410M, 6B, 13B, and 52B models, are trained from scratch (from random initialization).**  This is much more computationally expensive, but we believe this is critical to perform a controlled ablation of each variant without confounding factors like different pre-training conditions. We will make this clarification explicitly in the manuscript. 


**3. Numerical results on improved efficiency of our recipe**

We thank [egL3] for suggesting that we include “improved throughput/memory footprint of our PTQ approach”. We take this opportunity during rebuttals to run additional benchmarking experiments to act on this feedback. Concretely, we benchmark end-to-end inference speed for our PTQ recipe on our 52B model for both compute-bound (large batch size, long input sequence and short generation) and memory-bandwidth-bound (small batch size, short input sequence and long generation) use cases. We use standard CUDA kernels that enable INT8 matrix multiplication. Across all settings, we find that our vector-wise PTQ approach provides a 1.4-1.5x latency speedup (or 1.4-1.5x throughput improvement) and around 40% memory footprint reduction over half-precision inference. We note that LLM.int8() is slightly slower than half-precision inference (Dettmers et. al.). Therefore, our PTQ approach is significantly faster than LLM.int8(). Additionally, Dettmers et. al. also show that LLM.int8() leads slower matrix multiplication compared to vector-wise PTQ. We include the inference speed benchmark table in the attached pdf for reference with further details and will include it in the final manuscript.  


We are grateful for the valuable insights provided by the reviewers, and we are dedicated to addressing the feedback and incorporating it into the final version of our paper to further enhance its quality and contribution.


Meta-review: This paper studies quantization in LLMs, specifically it investigates the drop in performance which usually accompanies quantization of large-scale models. The effect of quantization is currently unclear in the literature, with some models retaining their performance much better than other models of similar size. The authors study show that optimization parameters strongly affect the post-training-quantization (PTQ) performance, and use their insights to pick better hyperparameters for certain models — showing that these large models can in fact be quantized at scale, given the right optimization.


Reviewers agree that the paper is well-written, the topic is important, and the experiments are thorough and largely convincing. Thus, I recommend acceptance.
I encourage the authors to address the reviewers' comments in the camera-ready, which will result in a stronger paper. In particular, Reviewer egL3 notes several instances where the claims as phrased may not be exactly supported by the evidence. I encourage the authors to revise this language.
Decision: Accept (poster)