EasyQuant: An Efficient Data-free Quantization Algorithm for LLMs
Authors: Hanlin Tang
Yifu Sun
Decheng Wu
Kai Liu
Jianchen Zhu
Zhanhui Kang
Venue: EMNLP 2023 Main (2023)
Keywords: model quantization

Abstract: Large language models (LLMs) have proven to be very superior to conventional methods in various tasks.
However, their expensive computations and high memory requirements are prohibitive for deployment.
Model quantization is an effective method for reducing this overhead. The problem is that in most
previous works, the quantized model was calibrated using few samples from the training data, which
might affect the generalization of the quantized LLMs to unknown cases and tasks. Hence in this work,
we explore an important question: Can we design a data-independent quantization method for LLMs to
guarantee its generalization performance?
In this work, we propose EasyQuant, a training-free and data-independent weight-only quantization
algorithm for LLMs. Our observation indicates that two factors: outliers in the weight and quantization
ranges, are essential for reducing the quantization error. Therefore, in EasyQuant, we leave the outliers
(less than 1%) unchanged and optimize the quantization range to reduce the reconstruction error. With
these methods, we surprisingly find that EasyQuant achieves comparable performance to the original model.
Since EasyQuant does not depend on any training data, the generalization performance of quantized
LLMs is safely guaranteed. Moreover, EasyQuant can be implemented in parallel so that the quantized
model could be attained in a few minutes even for LLMs over 100B. To our best knowledge, we are the
first work that achieves almost lossless quantization performance for LLMs under a data-independent
setting and our algorithm runs over 10 times faster than the data-dependent methods.

Easy Quant: An Efficient Data-free Quantization Algorithm for LLMs
Hanlin Tang
Yifu Sun yi vifusunêtencent.cm com
Decheng Wu
ranchotang@tencent.cco com
woodchenwuêtencent.ccom
Kai Liu
Jianchen Zhu dickzhu@tencent.cm com
Zhanhui Kang
raccomnlu@tencent.com
kegokang@tencent.com
Abstract
2021; Smith al., 2022: Chowdhery al., 2022: Zeng al., 2022) on various tasks over traditional methods, and has attracted massive interest in how to improve and utilize those LLMs. However, the model size also grows dramatically along with im- proved performance. Hence the memory footprint and computational cost become the bottleneck for deploying those models. One promising solution alleviate this overhead model quantization (Fran- tar et al., 2023a; Xiao al., 2023), where we quan- tize weight only or weight and activation both order to reduce memory consumption and compu- tational cost.
Large language models (LLMs) have proven be very superior to conventional methods in various tasks. However, their expensive com- putations and high memory requirements are prohibitive for deployment. Model quantiza- tion is an effective method for reducing this overhead. The problem that in most previous works, the quantized model was calibrated us- ing few samples from the training data, which might affect the generalization of the quantized LLMs to unknown cases and tasks. Hence in this work, we explore an important question: Can design a free-free quantization method for LLMs to guarantee its generalization per- formance?
Although model quantization is a well-studied area for normal-sized models, such as BERT (De- vlin et al., 2018) and GPT-2 (Radford et al., 2019), it is still quite challenging task for LLMs. One major reason is that previous lossless model quan- tization algorithms require retraining for the quan- tized model, which too expensive for models over billions parameters. Beyond this, previous models are usually designed for specific domain tasks, which means the training data are sampled from limited task domains. However, recent LLMs are usually trained on various domains of data cor- pus, and they have shown to be quite effective for multi-domain zero-shot tasks. In this case, if we only retrain the quantized LLMs using partial do- main corpus, the generalization ability of LLMs might get worse. Therefore both efficiency and generalization guarantees are very important for designing LLMs quantization algorithms. To date, for low-bits weight-only quantization, several post- training algorithms have been proposed (Frantar et al., 2023a; Yao et al., 2022). However, those methods also require small calibration set sam- pled from training data, which still takes least several hours. Moreover, the use of those calibra- tion data also brings the risk of making the model overfit to the calibration set.
In this work, we propose FasyQuant, training- free and data-free weight- oolly quantization al- gorithm LLMss. Our observation indicates that factors: outliers in the weight and quantization ranges, are essential for reducing the quantization error. Therefore, in Easy Quant, we leave the outliers (less than 1%) unchanged and optimize the quantization range to reduce the reconstruction error. With these methods, we surprisingly find that EasyQuant achieves comparable performance to the original model. Since EasyQuant does not depend on any train- ing data, the generalization performance of quantized L.LMs are safely guaranteed. More- over, EusyQuant can be implemented in parallel so that the quantized model could be attained in a few minutes even for LLMs over 100B. To our best knowledge, are the first work that achieves comparable performance with data- dependent algorithms under data-free setting and our algorithm runs over 10 times faster than the data-dependent methods.
1 Introduction
Recent work has already proved the superior per- formance of Transformer (Vaswani et al., 2017)
based LLMs (Workshop, 2023; Zhang et al., 2022;
Touvron et al., 2023; Brown et al., 2020; Rae et al.,

---

Grange
Grange 1.5
Optimizing Irange
1
Arange
100
3
EasyQuant
100
1
Outlier isolation
Add
100
Q[1]
1
Q[1]
Q
100
Figure Pipeline of EasyQuant. We first find all the outliers in weight and keep them full precision (fp32/fp16/bf16). Afterward, optimize the quantization range (denoted grange) in order approximate the normal values more precisely. the end, the normal values are quantized into lower bits (denoted as Q[:)) with optimized quantization ranges and the outliers unchanged in weight.
Our Contribution: In this work, we propose a novel data-free model quantization algorithm, namely EasyQuant, that potentially improves the performance of low-bits quantized LLMs. The gen- eralization ability of LLMs is inherently guaran- teed since EasyQuant does not need any input data, By running EasyQuant for only a few minutes, we can quantize public-available OPT-176B, BLOOM- 176B, and LLAMA-65B into lower bits without significant loss on various benchmarks. To our best knowledge, this is the first data free LLM quan- tization algorithm for LLM quantization without notable system overhead.
tensor æ into k-bits representation according to
Q(æc]
cllmp
: Imin, mmax
(1)
Here s the quantization scale, Imin and mmax are the lower and upper bound for clipping, and [-1 is the rounding operator. Usually we set 4in 2 + 1) and Imax = 2 and set to be the maximum absolute value in æ.
There are two major directions for finding the best configuration in weight-only LLM quantization. The first is to minimize the reconstruction enror of the weight parameter (denoted as W), which is defined
Moreover, our work reveals the essential fac- tors that cause the performance degradation of the quantized LLMs. We show that the outliers in weights are more critical to the model's perfor- mance compared to the normal elements. Beyond this, we propose to use gradient-based method for optimizing the quantization range. These two strategies also be used in other scenarios, such weight- seight-activation quantization and quantization- aware training (QAT).
r(W) = [Q[W] W
Notice that in this case we only need to have access to the weight itself, therefore it data-free. Beyond this, recent studies (Frantar et al., 2023a; Yao et al., 2022) propose to use the output error, defined
e(w)
Σ [e[l]-WXX XED
Last but not least, we develop efficient CUDA kernels for outlier isolation in dequantization, and proved that hold 1% outliers in weights unquan- tized brings negligible (less than 0.1%) overhead w.r.t overall latency. We also propose im- plement EusyQuant in parallel for quantizing each weight the model, which means a 175B-sized model can be quantized into 4-bits within 10 min- utes.
where D is calibration set sampled from the orig- inal training data, for optimization. This regulation tries mimic the outputs from the original model directly hence achieving more promising result than reconstruction based methods.
Data-dependent calibration might weaken the generalization ability of LLMs However, the performance gain from using calibration data might jeopardize the generalization of the quantized model, because it brings the risk of making the model overfit to the calibration set. For example,
2 Background and Motivation
The most widely used quantization method, namely rounding to nearest-number (RTN), quantizes a

---

Reconsiruction Frior Model's Performance
Reconstrur Performance Outlier
Awoodmmtti Bror
Paop 100 22
scorttid an: oepieily
6045
Optinizing stes
Optimizing 200p
Figure 2: Smaller reconstruction error cannot guarantee better model performance. Straightforwardly shrinking the quantization ranges will clip most of the outliers to be very small, hence the perplexity increases severely since those outliers are critical for preserving the model's performance. However, when keeping those outliers unquantized, the quantized model achieves a better performance as the reconstruction error decreases continuously. This result clearly suggests that the outliers are more important than the normal values in weight, and optimizing the quantization ranges using gradient defined (2) can significantly increase accuracy of quantized models. More details about the experiment can be found in Section
both ZuroQuant and GPTQ involve changing the original weight by training or OBS in order to min- imize the output error, therefore the distribution of the weight's parameters might deviate from the original. Since the calibration data is usually sam- pled from few specific domains, the performance of the calibrated model on other tasks may not be guaranteed.
count for less than 0.1% the parameters, im- pose very important influence on the model's performance.
In EasyQuant, we use quantization range mini- mization and outlier isolation address these two challenges, and our results prove that Easy achieves significant improvement over RTN.
Data-free quantization challenging, but very important Although it's more challenging to use the reconstruction етгог as a regulation because it can only optimize the quantized model indirectly, still it is very important direction for researching because the generalization ability of the model is inherently guaranteed when using data-free quanti- zation since it uses no training data. Therefore in this paper, we aim to answer the following ques- tion:
Insight behind EasyQuar
As mentioned above, the weight's outliers and quantization ranges are essential the quantized model's performance. Below we present the sup- porting experiments in detail.
3.1 The quantization range can be efficiently optimized using gradient
Although the quantization operation itself is non- differentiable, the gradient of the reconstruction error () Q[x] x 12) w.r.t. the quantization range is differentiable in most cases. We proved that the gradient of the quantization range admits (see Section 4 for more details)
How can we efficiently recover the performance of the quantized model without using any input data?
In this work we propose EusyQuant, a data-free fast algorithm that could significantly improve the per- formance of quantized LLMs data-free setting, and more importantly, even outperforms the results from data-dependent guantization algorithms. Our experiments reveal that the performance gap of the lower bits (e.g. 4-bits) quantized LLMs origins from two factors:
a|| [[x] x1| Os
Σ (([[z)) (0:))=)))))))))
(.:1)
2
(2)
With this gradient, the reconstruction error can be quickly minimized within hundreds of steps (sec Figure for more details). This result indicates that by shrinking the quantization range, most of the parameters in weight can be approximated more precisely. However, as shown in Figure 2, the per- formance of the quantized weight gets even worse
1. Setting the quantization range as the maxi- mum absolute value of the weight induces large reconstruction error for low-bits quanti- zation.
2. The outliers in the weight matrix, which ac-

---

the reconstruction error decreases. This is very counter-intuitive result.
4 Methodology
4.1 Driving of the gradient in (2)
Through in-depth analysis, we realized that when decreasing the quantization range, more salient parameters outside the quantization range would be clipped out. Although most of the weights get approximated more precisely as indicated by the decreased reconstruction error, the salient pa- rameters are poorly represented. As the model per- formance drops severely in this case, we realized that those outliers are way more important than the normal elements for the model's performance.
Let's say the original scale gets an infinitely small variation As, which means
x7
if
X 0.5.
8
As
Therefore we get
Qs+Ar(T] -(s As)
As
3.2
Outliers in weight are very important, but not sufficient
=(s + As)
Before further diseuss the influence of those outliers, first provide (no) criterion for defin- ing the outliers weight. For any weight W, we say its (i, 1-th number W an (no) outlier if
this leads to
aQ[a]]
QutA. Q.(1r]
Ōs As
This gives us
Wis mean(W)) (W)l 2 n var(W),
Q [c] x: 1:11
where me) and tar(W) are the mean and variance W.
ds
=2 Q(a0] x, aQ(æ)
Now the question is: Can we hold those out- liers unchanged and straighfforwardly compress the normal elements into lower bits? Unfortunately, our result suggests that excluding the outliers from quantization solely is not enough. As shown in Table 1, the performance gap still exists even when we hold 1% numbers in fp16. The problem is that if we keep too many numbers in fp16, the overhead of the dequantization kernel would also increase and result in a decreased overall throughput.
Os
([(z) x; \
I4
2 Σ (00/2a] x3)) 24
4.2 Algorithm description
In EasyQuant, for each weight W, first select all (no) outliers (using (3)) and store its index (") Afterward, for the normal elements, we optimize the per-channel quantization range using an opti- mizer (in our case we use Adam for example) with gradients defined in (2). The final quantized weight from EasyQuant can be formulated as
3.3 EasyQuam potentially improve the performance
As shown in Scction 3.1 and Section 3.2, optimiz- ing the quantization ranges directly reduces the model's performance drops severely because clipped outliers. These key observations inspire us to design EasyQuant, in which we isolate the outliers from quantization first and then optimizing the quantization range for the remaining elements. As shown in the right part of Figure 2, with outliers being kept unquantized, the performance of the quantized model increases continuously under de- creased reconstruction. This clearly proves we can potentially improve the performance of quantized LLMs with this strategy.
Q EasyQuant [W]
=Mask * W + (1 Mask"(W)) Q[W,,
(4)
where Mask is a mask tensor defined as
Maski = { ! if (i,j) E I'().. if (f, & (wW)..
(5)
The detailed description of EasyQuant is in Algo- rithm I.

---

Threshold n (BLO0M-7BB
Baseline
2
4
6
PPL on WikiText2
11.37
12.153
12.495
12.518
12.536
Table 1: Isolating outliers in weight from quantization increase the model's performance. Here refers the hyper-parameter in the outlier criterion (nZ) as defined in (3) and baseline is the result from unquantized model. Notice that even with 10%(n 1) numbers being held unquantized. there is still large gap to the baseline. This means isolating the outliers is not enough to fully recover the accuracy of quantized models.
Algorithm EasyQuant
and 500 for LLAMA. We use symmetric quanti- zation since the normal values are symmetrically distributed with the outliers being excluded. For a fair comparison, we use per-channel quantization for weight in all algorithms (which means each column shares one common quantization range),
1: Initialize: outlier threshold n. hyper-parameters for opti- mizer A, original weight W.
2: Quantize:
3: According to (3), compute the index (W) of the
(no) outliers W.
4:
Optimizing the quantization range using optimizer A with gradient defined in (2),
5:
Quantize W into QWW.
Evaluation Tasks. As for the evaluation tasks, we mainly focus perplexity- tasks, as they are known to be particularly sensitive to model quantization Frantar et al. (2023b). The perplex- ity tasks we include are WikiTex12 (Merity et al., 2016), Penn Treebank (Marcus et al., 1994) and C4 (Raffel et al., 2020). The zero-shot tasks' results are also provided, such as PIQA (Tata and Patel, 2003), ARC (Boratko et al., 2018) and StoryCloze (Mostafazadeh et al., 2017).
6: Dequantize:
Q RaayQuant [W]
(a) W (W) Q[W,, where Mask"(W)
defined (5).
5 Experiment
We compare EasyQuant with several baselines in the INT4 quantization setting below:
Baselines:
RTN:
The model's weights are naively quan- tized according to (1).
Implementation. Since each weight can be quan- tized in parallel, therefore we use 8* A100 for run- ning Easy Quant, and we finish the quantization in 10 mins for all models. We store the index and value for all outliers together with the quantized normal values. Our dequantization kernel is built using CUDA.
ZeroQuant:
The algorithm proposed in Yao et al. (2022). Authors treat each layer as small neural network and use the origi- nal as the teacher model to distill the quan- tized one. This is cquivalently minimizing EEED [If(W"); - fWW 2 where æ are the input activations, W is the weight of the original model and S is the quantized model.
5.1 Experiment Analysis
We foeus our study on LLM by quantizing the entire BLOOM, and LLAMA model families b-b
GPTQ: This algorithm is proposed in Frantar et al. (2023a). Authors use the same objective function EZED [I"": - S x as in ZenoQuant. But they utilize OBS for min- imizing the loss function instead of using gradient-based optimizer.
Perplexity- tasks. We first study perplexity- based tasks. On LLaMA models, Table shows that EasyQuant outperforms GPTQ in most cases. For LLaMA-65B, GPTQ drops 4.21 points on PTB, performing worse than the 9 x smaller full-precision 7B model, while EasyQuant still performs well on this task. On the other tasks, EasyQuant losing only 0.4-0.7 points. BLOOM shows similar pattern (see Table 10 in ap- pendix): EasyQuant drops only 0.1-0.16 points on perplexity-based tasks. Notice that we observe smaller gap between our method and GPTQ on C4. is mostly because, as a data-calibrated quan- tization method, GPTQ uses C4 dataset for calibra-
Experiment Setup. For all models, we set the outlier threshold n E [2.5, 3] in order to ensure that the outliers account less than 1% of all numbers. For BLOOM and LLAMA, we n 3. When optimizing the quantization ranges, we use Adam as the optimizer and set the learning rate 1e for BLOOM and le - 4 for LLAMA. We choose the quantization ranges from step 100 for BLOOM

---

Perplexity- hased Task
Perplexity- Task
WikiTex12
PTB
C4
WikiText2
PTB
C4
fp16
5.68
8.80
7.08
f6
4.10
7.30
5.98
RTN
6.29
11.25
8.12
RTN
4.54
8.65
6.54
I.LAMA7B
LLAMA-33B
GPTQ
6.09
11.56
7.78
GPTQ
4.45
8.44
6.40
Quant
6.01
10.72
7.71
EuayQuant
4.34
8.45
6.37
f6
5.09
8.07
6.61
fp
3.53
6.91
5.62
RTN
5.53
9.77
7.23
RTN
3.99
10.67
6,45
LLAMA-13B
LLAMA-6SB
GPTQ
5.36
9.49
7.07
GPTQ
4.13
11.12
6.38
CuayQuant
5.29
9.37
6.97
EasyQuant
3,98
9.61
6.30
Table 2: Perplexity results for LLAMA model family
tions.
range optimization. As discussed in Section 3, both strategies impose very important influence the final model performance.
Zeroshot tasks. For most zero-shot tasks, EasyQuant achieves harmless performance with only 0.1 %-0.52% accuracy drops as shown in Table 10 in appendix and outperforms GPTQ on most cases. Here simply the implementa- tion of GPTQ on LLAMA from its git.' We note that Easy Quant can be further improved via finer- granularity grouping. However, we will not include this overhead in this paper.
We further conduct experiments proving whether the performance gain mainly comes from the out- lier isolation: Actually, outlier isolation is very important component of EasyQuant, but still not enough to fully recover the performance loss from quantization. Keeping even 10% of weights as fpl6 outliers still admits about 8% ppl increase while EasyQuan admits only 1% ppl increase. Below we present the result of 4-bit quantized BLLOM- 7B when we just keep 1% outliers in fp16 without quantization range optimization on various bench- marks.
outlier ratio
overhead
0.01%
0,027ms
0.10%
0.055ms
0.50%
0.093ms
1%
0.1 17ms
Benchmark
EasyQuant 1% fpl6 outlier
5%
0.186ms
(PPP
11.66
12.52
10%
0.2 12ms
PTB (PPL)
21.42
23.32
C4(PPL) 15.46
16.44
Table 3: Overhead of outlier isolation on A100
PIQA (ACC)
73.61%
72.74%
Table 4: Using outlier isolation solely is not enough to fully recover the performance loss. Easy consis- tently outperforms outlier isolation in all benchmarks.
Practical Latency. We evaluate the overhead EasyQuant by comparing the overhead of out- lier isolation, int4 dequantization, and matrix mul- tiplication with batch size 1, sequence length 1024, on single A100 GPU. The matrix size is 14336 x 53746 which is the same as the first FFN layer in 176B BLOOM. For outlier isolation, test the latency of outliers ratio (fraction of outliers within the weight) in 6 settings: (0.01%, 0. 0.10%. 0.50%, 1%, 5%, 10%). The matrix multiplication takes 83ms and dequantization takes 5ms. There- fore from Table we can see that recovering the outliers in weight brings almost no overhead to the overall latency.
Outlier influence. The outlier isolation is key component in EasyQuant, but it can only impose an indirect influence on the model accuracy. The interesting phenomenon we find that the outliers behave like a gating mechanism: without outlier isolation, the model achieves a much worse perfor- mance under a small reconstruction error; however, when keeping those outliers in fp16, the quantized LLM attains continuously decreased ppl under smaller reconstruction error:
Moreover, we have also conducted comple- mentary experiment testing the direct influence of the weight outlier: We prune 1% of the values according to its magnitude) in weights into and see the ppl results (as shown in Table 6). has
Ablation study. To understand the effect of un- structured outliers, we show the perplexity result of EasyQuant without outlier isolation or quantization

---

reconstruction error
int4 outlier
fpl6 outlier
Layer index
outlier fraction (%)
4.8E4
12.65
12.50
1
0.3187
3.5B3
14.73
11.61
5
0.8579
2.7E4
19.71
11.25
10
0.3953
2.3E4
NA
11.10
15
0.3975
1.9E4
NA
11.02
20
0.3962
25
0.4399
Table 5: ppl results on Wikitext2 BLOOM-7B with and without outlier isolation.
30
0.3954
Table 8: Outlier fraction distribution in different layer
index in BLOOO-7B under 3-sigma threshold
shown that the largest value (outliers) imposes the same influence on the model performance as the normal values (median), which means those out- liers share the same direct influence on the model accuracy with normal values. Therefore outlier isolation imposes a key influence on the model accuracy indirectly.
steps
quantization range
0
0.078
10
0.069
50
0.052
100
0,048
150
0.047
200
0.047
pruned weights
PPL
smallest (top-0% 1%)
11.66
Table The dynamic quantization range of different optimization steps. Here we take the quantization range of the Att.qkv module in layer as an example.
median (top-49% 50%)
19.16
largest (top-99% 100%)
19.17
Table ppl results after pruning 1% weight with differ- ent magnitude
Related Work
Model Quantization Traditional model quanti- zation algorithms mainly focus the where both parameters and activations of the model are quantized (Lin et al., 2015; Hubara et al., 2016; Tailor et al., 2021; et al., 2020). However, di- rectly quantizing the model will greatly decrease the accuracy of the models, and one important tech- nique to improve the performance is Quantization Aware Training (QAT) (Jacob et al., 2018 18), where it simulates the quantization procedure in training to improve the accuracy of the quantized model further. For Transformer based models, the bound- ary of the compression level has been continuously advanced. For example, 8-bits quantized transform- ers as in FullyQT (Prato et al., 2019) and Q8BERT (Zafrir et al., 2019), 4-bits quantized BERT in Wu et al. (2023) and tenary case as in TernaryBERT (Zhang et al., 2020).
Outlier distribution. We also explore the outlier distribution along different modules and layers. shows that the fraction of outliers shares different patterns in different modules and layers (as shown in Table 7 and 8). FFN.2 has a significantly higher fraction of outliers. However, it shows pattern along the layer index.
module name
outlier fraction (%)
Att.qkv
0.2993
Att.output
0.5036
1
0.288
FFN,2
0.7560
Table 7: Outlier fraction distribution in different mod- ules in BLOOM-7B under 3-sigma threshold
Quantization range. The dynamic of the quanti- zation range is shown in Table 9. Roughly speak- ing, this range decreases fast in the carly stage of training, which means smaller quantization range will make most of the parameters to be quantized more precisely. After certain steps of training, the quantization range becomes stable, this means we have already achieved the optimal range.
Model Quantization for LLMs. For quantizing
LLMs, due to their prohibitive training expense, only few training data for calibration. There are major directions: 1) weight-only quantization, where the weights are quantized into lower bits. Frantar (2022), authors optimize the output error on the calibration set using OBS and gradient descent. 2)
(2023a); Yao et al.

---

Activation and weight quantization, where both ac- tivations and weights are quantized into lower bits. this case, the major obstacle is the outliers in activations. LLM. (Dettmers et al., 2022) ad- dresses this problem by isolating those outliers in fp16/bf16. However, such implementation leads to large latency overhead and is even slower than fpl6 inference. Recent studies (Wei et al., 2023; Xiao et al., 2023) found that the outliers only exist in cer- tain channels, and use the LayerNorm weights (Wei al., 2023) and calibrated scales (Xiao al., 2023) to smooth those channels. Xiao et al. (2023) has already proved that we can achieve almost loss- less W8A8 quantized LLMs using a few calibra- tion data, without manipulating the original model weights.
References
Michael Boratko, Harshit Padigela, Divyendra Mikki- lineni, Pritish Yuvraj, Rajarshi Das, Andrew McCal- lum, Maria Chang, Achille Fokoue-Nkoutche, Pa- van Kapanipathi, Nicholas Mattei, al. 2018. A systematic classification of knowledge. reasoning. and context within the are dataset. arXiv preprint arXiv:/806.0000 00358.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 1877-191..
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311
7 Conclusion and Limitations
Tim Dettmers Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm.int((): 8-bit matrix multipli- cation for transformers scale.
In this paper, we propose data-free fast weight- only quantization algorithm, namely EasyQuant for LLMs, that potentially improves the quantized model's performance without using any training data. Our analysis reveals the intrinsic origins of the performance loss when quantizing the model weights into lower bits. We show that by isolat- ing the outliers from quantization, the accuracy of the quantized LLM increases accordingly with decreased reconstruction error. Our experiment proved that EusyQuant significantly outperforms RTN in a data-free setting, and also behaves bet- ter than data-dependent algorithms. EasyQuant can finish the quantization for a 176B-sized model within 10 minutes and the overhead of dequantiza- tion in EasyQuant is negligible.
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and
Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. Cite arxiv: 111.000055Coommee. 13 pages.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023a. Gptq: Accurate post- cost-training quantization for generative pre-trained transformers.
Elias Frantar, Saleh Ashkhoos, Torsten Hoefler, and Dan Alistarh. 2023b. Gptq: Accurate post-training quantization for generative pre-trained transformers.
Itay Hubara. Matthieu Courbariaux, Daniel Soudry, Ran EI- Ea-iy,iv, and Yoshua Bengio. 2016. Binarized neu- ral networks. In Advances in neural information processing systems, pages 4107-4115.
However, we also point out some limitations of work: The outlier recovery funetionality in EasyQuant requires extra CUDA kernels for im- plementation. Moreover, weight-only quantization can only reduce the memory footprint without any computation cost reduction, hence the latency of our model cannot be minimized. In addition, this outlier isolation will make the weight/activation quantization more challenging because the weight includes numbers under different precision. We have also noticed that EasyQuantcannot outper- form the data-dependent methods in all tasks, this motivates us to investigate more effective algo- rithms in future studies.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Meng- long Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. Quanti- zation and training of neural networks for efficient anieger-rithmetic...
inference. pages 2704-2713.
Zhouban Lin, Matthieu Courbariaux, Roland Memi- sevie, and Yoshua Bengio. 2015. Neural net- works with few multiplications. arXiv preprint arXiv: 151.00000
Mitch Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre. Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The penn tree- bank: Annotating predicate argument structure. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.

---

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod- els.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, 706.1126...
Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. 2017. Ls- dsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop Linking Models of Lexical, Sentential and Discourse- Semantics, pages 46-51.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong. Shanghang Zhang. Qi Zhang. Fengwei Yu, and Xianglong Liu. 2023. Outlier suppression: Pushing the limit of low-bit transformer language models.
BigScience Workshop. 2023.
Bloom: A 176b- parameter open-access multilingual language model.
Renkun Ni, Hong-min Chu, Oscar Castañeda, Ping- yeh Chiang, Christoph Suuder, and Tom Gold- stein. 2020. Wrapnet: Neural net inference with uutra-low-resolution arithmetic. arXiv preprint arXiv:2007.13242.
Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, and Yuxiong He. 2023. Understanding int4 quantization for transformer models: Latency speedup, composability, and failure cases.
Gabriele Prato, Ella Charlaix, and Mchdi Reza- gholizadeh. 2019. Fully quantized trans- former for improved translation. arXiv preprint arXiv:1910. 10485.
Guangxuan Xiao. Ji Lin, Mickael Seznec, Hao Wu. Julien Demouth, and Song Han. 2023. Smoothguant: Accurate and efficient post-training quantization for large language models.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei. Ilya Sutskever, al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Zhewei Yao. Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112. 11446.
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8hert: Quantized Sbit bert. arXiv preprint ar1.v:1910.06/88..
Aohan Zeng Xiao Liu, Zhengxiao Du, Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-1300 An open bilingual pre-trained model. arXiv preprint arXiv:2210.02412
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, (1(5555485-5551
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- baylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig. Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre- trained transformer language models.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, al. 2022. Using deep- speed and megatron to train megatron- tuatoonn-tuingg nlg 530b, targe-scale generative language model. arXiv preprint arXiv:2201.1 19900.
Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. 2020. Ternarybert: Distillation-awarr ultra-low bit bert. arXiv preprint arXiv:2009.128/
Shyam A Tailor; Javier Fermandez-Marques and Nicholas D Lane. 2021. Degree-quant: Quantization- aware training for graph neural networks. Interna- rional Conference on Learning Representations.
A Appendix
Sandeep Tata and Jignesh M Patel. 2003. Piqa: An algebra for querying protein data sets. In 15th In- ternational Conference on Scientific and Statistical Database Management, 2003., pages 141-150. IEEE,
Hugo Touvron, Thibaut Lavril. Gautier Izacard, Xavier Martinet. Marie-Aune Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.

---

Perplexity- based Task
Zero-shot Task
WikiText2
PTB
C4
PIQA
ARC-easy
ARC-Challenge
StoryCloze
fpl6
22.42
43.69
26.6
65.07%
41.71%
24.15%
61.94%
BLOOM
RTN
25.90
51.10
29.89
63.11%
39.40%
23.89%
60.15%
GPTQ
24.03
46.97
28
64.31%
40.24%
23.46%
61.17%
560M
EusyQuant
23.74
46.86
28.03
63.06%
40.32%
24.15%
59.64%
fp16
17.69
57.96
22.05
67.14%
45.41%
25.68%
63.27%
BLOOM
RTN
22.00
66.85
24.44
65.29%
42.51%
23.34%
60.66%
GPTQ
19.05
62.48
23.25 66.05%
44.49%
25.51%
62.32%
1.1B
EasyQuant
18.51
61.83
22.94 66.65%
43.73%
25.51%
62.06%
fp16
15.39
30.00
19.49 69.97%
48.11%
26.79 %
65.44%
BLOOM
RTN
16.97
33.58
21.26 67.74%
44.70%
26.45
62.95%
GPTQ
16.48
31.84
20.55 68.77%
44.49%
25.94%
64.48%
1.7B
EasyQuant
16.01
31.50
20.15 68.99%
46.89%
26.19%
65.37%
6
13.48
25.34
17.49
70.51%
53.24%
30.55 %
67.79%
BLOOM
RTN
14.76
27.68
18.76
69.86%
51.35%
29.52%
67.09%
GPTQ
14.2
26.49
18.1
69.42%
52.82%
28.92%
67.22%
3B
EasyQuant
14.01
26.12
17.96
69.80%
50.72%
28.58%
67.35%
fp16
11.37
20.83
15.20
73.72%
57.37%
33.45 %
71.99%
BLOOM
RTN
12.10
22.42
16.06
72.69%
56.14%
32.17 %
70.72%
GPTQ
11.73
21.67
15.6
72.96%
56.14%
32.25%
71.36%
7.1B
EasyQuant
11.66
21.47
15.52
73.23%
55.72%
32.51 %
71.10%
fpl6
8.11
14.59
11.71
79.16%
67.47%
44.97 %
76.89%
BLOOM
RTN
8.37
15.00
12.04
79.00%
66.33%
43.17 %
76.00%
GPTQ
8.21
14.75
11.81
79.00%
67.42%
44.10%
76.32%
176B
EasyQuant
8.21
14.75
11.87 79.05%
67.8%
44.45%
77.28%
Table 10: Perplexity and zershot results for BLOOM model family


Reviews:
[{"review_id": "aTnn1YEwG4", "reviewer": "Reviewer_Mg7b", "summary": "", "questions": "**Question1**: Is there more evidence to support that the performance drop in lower-bit quantization is due to weight outliers, such as experiments on various tasks and analysis of weight outlier distributions?\n\n**Question2**: Are there any comparisons between the proposed method and the two state-of-the-art baselines, LLM.int8() and SmoothQuant?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your great efforts and constructive comments, and we have tried our best to address your concerns about the novelty and design of our algorithm. Please let us know if you have any further questions.\n\n### Q1. Lack of novelty compared to LLM.int8() and SmoothQuant:\nHere we politely point out that LLM.int8() and SmoothQuant are orthogonal to our work because their method cannot work on low-bits weight-only quantization (the ppl becomes larger than 1E3 on 4-bit quantized LLAMA-7B). We emphasize that the novelty of our work comes from three parts:\n1.  Our work points out that the importance of outliers in weights (not activations) is underestimated, and those outliers impose a **gating** mechanism (see our answer to Q3 below) instead of a direct influence. This is a critical observation in low-bits LLM quantization that has not been found before.\n2. Outlier isolation is only one part of our work. Beyond this, we propose a fast method for updating the quantization range, which essentially helps EasyQuant to achieve superior performance compared to the previous SOTA method (Please refer to Table 1 in our paper).\n3. This is the first efficient data-independent LLM quantization algorithm that achieves a better performance than the data-dependent algorithms. The data-independent nature of EasyQuant ensures that the generalization ability of quantized LLMs is safely guaranteed. We argue that this is a **very important** requirement for LLM quantization and should not be underestimated.\n\n### Q2. The performance  improvements brought by the proposed method are minimal:\nAs shown in Table 2,3 in our paper, EasyQuant outperforms the data-independent baseline (RTN) on almost all benchmarks. For BLOOM-1.7B, it achieves 5%\\~10% relative ppl drop and 1%\\~3% absolute accuracy increase. For 3-bit quantization, EasyQuant significantly (more than 10%) outperforms both RTN and GPTQ (which is data-dependent) on LLAMA-7B:\n| Algorithm | RTN-3bits | GPTQ-3bits | EasyQuant-3bits|\n|--|--|--|--|\n| Wikitext2 | 25.54 | 8.07 | 7.16 |\nTable 1. Comparison of different methods under 3-bit quantization\n\nTherefore we politely point out that the performance gain is not trivial.\t\n\t\n### Q3. Evidence for the importance of the outliers in weights:\nThe outlier isolation is a key component in EasyQuant, but it can only impose an indirect influence on the model accuracy (as shown in Figure 1 of our paper). The interesting phenomenon we find is that the outliers behave like a **gating** mechanism:  without outlier isolation, the model achieves a much worse performance under a small reconstruction error; however, when keeping those outliers in fp16, the quantized LLM attains a continuously decreased ppl under smaller reconstruction error:\n| reconstruction error | 4.8E4 | 3.5E4 | 2.7E4 | 2.3E4 |1.9E4 |  \n|--|--|--|--|--|--|\n| int4 outlier | 12.65 | 14.73 |  19.71 | NA | NA | \n| fp16 outlier | 12.50 | 11.61 |  11.25 | 11.10 | 11.02 | \nTable 2. ppl results on Wikitext2 of BLOOM-7B with and without outlier isolation\n\nMoreover, we have also conducted a complementary experiment testing the direct influence of the weight outlier: We prune 1% of the values ( according to its magnitude) in weights into 0 and see the ppl results:\n| pruned weights | smallest (top-0%\\~1%) | median (top-49%\\~50%) | largest (top-99%\\~100%) | \n|--|--|--|--|\n| PPL | 11.66 | 19.16 |  19.17 | \nTable 3. ppl results after pruning 1% weight with different magnitude\n\nWe can see that the largest value (outliers) imposes the same influence on the model performance as the normal values (median), which means those outliers share the same direct influence on the model accuracy with normal values. Therefore outlier isolation imposes a key influence on the model accuracy indirectly.\n\n### Q4. More experiments proving whether the performance gain mainly comes from the outlier isolation:\nActually, outlier isolation is a very important component of EasyQuant, but still not enough to fully recover the performance loss from quantization. Keeping even **10%** of weights as fp16 outliers still admits about 8% ppl increase while EasyQuant admits only 1% ppl increase. Below we present the result of 4-bit quantized BLLOM-7B when we just keep 1% outliers in fp16 without quantization range optimization on various benchmarks.\n\n| Benchmark | WikiText2  (PPL-based $\\downarrow$) | PTB  (PPL-based $\\downarrow$) | C4   (PPL-based $\\downarrow$)  | PIQA  (ACC-based $\\uparrow$) |\n| -- | -- | -- | -- | -- |\n| EasyQuant | 11.01 | 21.42  | 15.46 | 73.61% |\n| 1% fp16 outlier | 12.52 |  23.32 | 16.44  | 72.74% |\n\nTable 4. Using outlier isolation solely is not enough to fully recover the performance loss. EasyQuant consistently outperforms outlier isolation in all benchmarks.\n\n### Q5. Distribution of the outliers in weights:\nWe find that the fraction of outliers shares different patterns in different modules and layers (as shown in Table 5 and 6). We will add more discussion about this in our revised version.\n| module name | Att.qkv | Att.output | FFN.1 | FFN.2 |\n| -- | -- | -- | -- | -- |\n| outlier fraction (%) | 0.2993 | 0.5036 | 0.288 | 0.7560 |\nTable 5. Outlier fraction distribution in different modules in BLOOM-7B under 3-sigma threshold\t\t\n\n| Layer index  | 1 | 5 | 10 | 15 | 20 | 25 | 30|\n| -- | -- | -- | -- | -- | -- | -- | -- |\n| outlier fraction (%) | 0.3187 | 0.8579 | 0.3953 | 0.3975 | 0.3962 | 0.4399 | 0.3954 |\nTable 6. Outlier fraction distribution in different layers in BLOOM-7B under 3-sigma threshold\t\t\n\n\n\n\nIn summary, we politely point out LLM.int8() and SmoothQuant are designed for activation quantization and are not fair baselines for comparison, as they work poorly in low-bits weight quantization. Our work reveals the importance of the weight outliers and proposes an efficient method (outlier isolation+range optimization) to overcome the performance drop in LLM quantization. The intrinsic data-independent nature of our method shall be appreciated because it ensures the generalization property of LLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation behind the proposed approach is reasonable.\n2. The authors have chosen to test their method on more sensitive language model tasks, and some results outperform the baselines.", "reasons_to_reject": "1. In the quantization community, the discussion on outliers mainly focuses on activation outliers, such as SmoothQuant and LLM.int8(). It would be helpful to provide more evidence to support the claim that the performance drop in lower-bit quantization is due to weight outliers, for example, by conducting experiments on various tasks and analyzing the distribution of weight outliers.\n2. The motivation is relatively straightforward, the paper is closely related to works such as LLM.int8(), which may result in lower novelty.\n3. The writing of this paper is somewhat difficult to understand. For instance, the introduction does not mention any technical details, and the core contribution is not discussed until Section 4.2.\n4. The experimental section has relatively weak baselines, as it does not compare the proposed method with LLM.int8() or SmoothQuant. Additionally, the performance improvements brought by the proposed method are minimal.", "questions_to_authors": "**Question1**: Is there more evidence to support that the performance drop in lower-bit quantization is due to weight outliers, such as experiments on various tasks and analysis of weight outlier distributions?\n\n**Question2**: Are there any comparisons between the proposed method and the two state-of-the-art baselines, LLM.int8() and SmoothQuant?", "paper_topic_and_main_contributions": "This paper presents a data-free quantization method specifically designed for addressing outliers in Large Language Models (LLMs). By separately quantizing the non-outlier portion of the weights, the method aims to mitigate the performance drop caused by outliers. The authors test their approach on two model families, LLaMAs and BLOOMs, using INT4 bit for both Language Model tasks and zero-shot QA tasks. The results demonstrate a slight gain in performance for some tasks and models compared to previous methods. Additionally, the authors evaluate the overhead of their proposed data-free quantization build.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The formatting of the tables is somewhat disorganized, particularly with Table 3 being excessively large. It should have a dedicated section for discussing the limitations instead of in the conclusion."}}, {"review_id": "oDLeaEAsgM", "reviewer": "Reviewer_1vKT", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your affirmation of our work. Yes, our method consists of two parts: outlier isolation and range optimization. EasyQuant is indeed the first data-independent LLM quantization algorithm that outperforms the previous data-dependent SOTA method. Moreover, EasyQuant will not make the model overfit to certain tasks since it uses no calibration data, which means the generalization ability of quantized LLMs can be safely guaranteed. Please let us know if you have any further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper makes an important contribution by revealing that outliers in weights and optimization of quantization ranges are essential factors for reducing quantization error. Specifically, the key conclusions are: Through experiments, the paper finds that directly shrinking the quantization range will clip most outliers, severely degrading model performance even as reconstruction error decreases. This counterintuitive finding underscores the importance of outliers. Keeping outliers (e.g. less than 1% of weights) at full precision alone is not sufficient to recover accuracy - quantization ranges must also be optimized. By leaving outliers unchanged and optimizing quantization ranges via gradient descent, EasyQuant is able to significantly improve accuracy versus baseline quantization methods that do not account for these factors. ", "reasons_to_reject": "No", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes an efficient data-free quantization algorithm called EasyQuant for model compression. The key contributions are:\n\n· EasyQuant is a training-free and data-free weight-only quantization method. It has the advantage of guaranteeing the generalization ability of the quantized model, which is important for universal models like LLMs.\n· The paper finds through experiments that the two main reasons for performance degradation are suboptimal quantization ranges and improper handling of outliers in weights. EasyQuant addresses this by excluding outliers and optimizing the quantization ranges.\n· Compared with data-dependent methods, EasyQuant achieves better or comparable accuracy while ensuring generalization and runs over 10x faster.\n· EasyQuant is implemented in a highly parallel way on GPUs, enabling quantization of models with over 100B parameters within minutes. This makes it practical for large models.\n\nThe paper provides useful insights on the importance of optimizing quantization ranges and handling outliers, informing future research.\nIn summary, EasyQuant presents an effective data-free approach for low-bit quantization of LLMs, achieving good accuracy and efficiency. It also sheds light on important principles in model quantization. The results and insights are valuable and novel contributions to the field of model compression.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "B86LcImzpP", "reviewer": "Reviewer_uuQc", "summary": "", "questions": "Answer the above questions.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your great efforts and constructive comments, and we have tried our best to address your about the outlier isolation design of our algorithm. Please let us know if you have any further questions.\n### Q1. Distribution of outliers among different modules, layers, and models:\nIn the Table below, we show that the fraction of outliers shares different pattern in different modules and layers; Moreover, we also find different models shares different fraction of outliers (BLOOM-176B admits 1% outliers while LLAMA-65B has about 0.2% outliers). We will add more discussion about this in our revised version.\n| module name | Att.qkv | Att.output | FFN.1 | FFN.2 |\n| -- | -- | -- | -- | -- |\n| outlier fraction (%) | 0.2993 | 0.5036 | 0.288 | 0.7560 |\nTable 1. Outlier fraction distribution in different modules in BLOOM-7B  under 3-sigma threshold\n\n| Layer index  | 1 | 5 | 10 | 15 | 20 | 25 | 30|\n| -- | -- | -- | -- | -- | -- | -- | -- |\n| outlier fraction (%) | 0.3187 | 0.8579 | 0.3953 | 0.3975 | 0.3962 | 0.4399 | 0.3954 |\nTable 2. Outlier fraction distribution in different layers   in BLOOM-7B under 3-sigma threshold\n\n### Q2. Whether the performance gain mainly comes  from the outlier isolation:\nActually, outlier isolation is a very important component of EasyQuant, but still not enough to fully recover the performance loss from quantization. Keeping even **10%**  of weights as fp16 outliers still admits about 8% ppl increase while EasyQuant admits only 1% ppl increase. Moreover, in Table 3 we show that outlier isolation somehow behaves like a **gating** mechanism for quantization range optimization: without outlier isolation, the model achieves a much worse performance under a small reconstruction error; however, when keeping those outliers in fp16, the quantized LLM attains a continuously decreased ppl under smaller reconstruction error. Therefore it is hard to judge which part is more important in EasyQuant.\n\n| reconstruction error | 4.8E4 | 3.5E4 | 2.7E4 | 2.3E4 |1.9E4 |  \n|--|--|--|--|--|--|\n| int4 outlier--PPL | 12.65 | 14.73 |  19.71 | NA | NA | \n| fp16 outlier--PPL | 12.50 | 11.61 |  11.25 | 11.10 | 11.02 | \nTable 3. PPL results on Wikitext2 of BLOOM-7B with and without outlier isolation\n\n### Q3. Under lower bits case, will more outliers be needed and will EasyQuant still work:\nFor 3-bit quantization, EasyQuant still significantly (more than 10%) outperforms both RTN and GPTQ (which is data-dependent) on LLAMA-7B using 3-sigma threshold (about 0.2% outliers), which is the same setting in 4-bit quantization:\n| benchmark (PPL) | RTN-3bits | GPTQ-3bits | EasyQuant-3bits|\n|--|--|--|--|\n| Wikitext2 | 25.54 | 8.07 | 7.16 |\nTable 4. Comparison of different methods under 3-bit quantization\n\nSince the outliers are the salient values in weights, we believe that for lower-bit quantization, we can still use the same fraction of outliers as it is in 4-bits. We will add more experiments for 3-bit quantization of EasyQuant on more benchmarks in our revised version.\n\n### Q4. More experiments about the dynamic of the quantization range:\nThe dynamic of the quantization range is shown in Table 4. Roughly speaking, this range decreases fast in the early stage of training, which means a smaller quantization range will make most of the parameters to be quantized more precisely. After certain steps of training, the quantization range becomes stable, this means we have already achieved the optimal range.\n\n| steps  | 0 |  10 | 30 |50 | 100 | 150 | 200|\n| -- | -- | -- | -- | -- | -- | -- | -- |\n| quantization range  | 0.078 | 0.069 | 0.056 | 0.052 |  0.048 | 0.047 | 0.047 |\n\nTable 4. The dynamic quantization range of different optimization steps. Here we take the quantization range of the Att.qkv module in layer 1 as an example.\n\n\n### Q5. How does the outlier threshold n affect the performance:\n| threshold | 5 | 4 | 3 | 2 | 1 |\n| -- | -- | -- | -- | -- | -- |\n| ppl on Wikitext2 | 6.20 | 6.10 | 5.98 | 5. 89 | 5.90 |\n| ppl on PTB | 11.02 | 10.86 | 10.68 | 10.40 | 10.50 |\nTable 5. LLAMA-7B ppl results under different outlier thresholds.\n\nAs shown in Table 5, increasing the outlier fraction is beneficial to the quantized model, choosing threshold=3 is to balance the outlier overhead and model performance. Increasing the outlier fraction (under acceptable overhead) is an efficient method to improve the performance of quantized LLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The first attempt to quantize the LLMs without accessing the input data.\n(2) The proposed method reveals the impacts of the outliers in weights and the quantization range on the generalization of low-bits quantized LLMs.\n(3) High quantization efficiency. For instance, EasyQuant can quantize public-available LLMs (e.g., OPT-176B, BLOOM-176B, and LLAMA-65B) into lower bits with a few minutes.\n", "reasons_to_reject": "(1) The authors argue that the outliers in the weights have the non-negligible impacts on the quantization process. Hence, some crucial questions deserve to be studied: how does the outliers distribute in the weights? whether does they always exist? is the number (or percent) of the outliers still the same for different layers?\n(2) EasyQuant tends to keep the outliers in full precision (fp32/fp16/bf16) while the normal values are quantized into low bits, which seems to be equivalent to the mix-precision quantization. Besides, it is unclear whether the performance advantages are from the valuable outliers that are kept in full precision.  \n(3) EasyQuant consider only the 4-bit case. When encountering lower bit width, will more outliers appear? Under such situation, can the quantization range be effectively optimized?\n(4) The experiments are insufficient.\n--- Instead of the fixed quantization range, it is expected to verify how the quantization scale s changes during the optimization, especially for different layers. \n--- Lack the validation experiments about how the outlier threshold n affects the performance.\n", "questions_to_authors": "Answer the above questions.", "paper_topic_and_main_contributions": "This paper proposes a data-free weight-only quantization method (EasyQuant) for large language models (LLMs), which aims to select the outliers in the weights by utilizing the mean and variance of weights, then isolate them from the quantization operation. Based on that, the quantization range is optimized using the gradient of the reconstruction error to quantize the remaining the normal values into lower bits. The extensive experiments validate the effectiveness of EasyQuant.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a quantization technique independent of training data. Reviewers see merit in the proposal, especially because it is the first data-independent quantization technique. Reviewers have some outstanding questions (primarily about outliers).
Decision: Accept-Main