Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling
Authors: Xiuying Wei
Yunchen Zhang
Yuhang Li
Xiangguo Zhang
Ruihao Gong
Jinyang Guo
Xianglong Liu
Venue: EMNLP 2023 Main (2023)
Keywords: quantization
large language models
outlier

Abstract: Post-training quantization (PTQ) of transformer language models faces significant challenges due to the existence of detrimental outliers in activations. We observe that these outliers are concentrated in specific channels and are asymmetric across channels. To address this issue, we propose the Outlier Suppression+ (OS+) framework, which contains the channel-wise shifting for asymmetry and channel-wise scaling for concentration. We show that these operations can be seamlessly migrated into subsequent modules while maintaining equivalence. Second, we propose a fast and stable scheme to calculate effective shifting and scaling values. The channel-wise shifting aligns the center of each channel for removal of outlier asymmetry. The channel-wise scaling quantitatively evaluates changes brought by migration and quantization for better quantization burden balance. We validate our OS+ under both standard and fine-grained quantization settings with models including BERT, OPT, BLOOM, BLOOMZ, and LLaMA. Comprehensive results across various tasks demonstrate the superiority of our approach. Especially, with standard quantization, OS+ can achieve near-floating-point performance on both small models and large language models on 8-bit and 6-bit. Besides, we establish a new state-of-the-art for 4-bit BERT with 15.5\% improvement. Our code is available at \url{https://github.com/ModelTC/Outlier_Suppression_Plus}.

Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling
Xiuying Wei Yunchen Zhang25, Yuhang Lit Xiangguo Zhang?,
Ruihao Gongl.2, Jinyang Guo Xianglong Liu!
State Key Lab of Software Development Environment, Beihang University 2 "SenseTime Research 'School of Computer and Communication Sciences, EPFL Yale University, 'UESTC
xiuying. seepiingwweeeeff..ctt yuhang. 1idygle,ddaeeeddd (jinyanggus, xiliu] @buaa. (zhangyunchen, zhangxiangguo, gongruihao/ #sensetime imo. com
Abstract
2019; Radford et al., 2018) hundreds of billions of parameters (Brown et al., 2020; Zhang et al., 2022; Smith et al., 2022). This necessitates the employment of compression techniques (Han et al., 2015; Hinton et al., 2015; Zoph and Le, 2016; Le- Cun et al., 1989) for practical deployment. Among these techniques, quantization (Jacob et al., 2018) has emerged as general and primary paradigm for reducing both memory footprint and computation overhead.
Post-training quantization (PTQ) of trans-
former language models faces significant challenges due to the existence of detrimental outliers in activations. We observe that these outliers are concentrated in specific channels and are asymmetric across chan- nels. To address this issue, propose the Outlier Suppression+ (OS+) framework. which contains the channel-wis shifting for asymmetry and channel-wise scaling for concentration. We show that these operations can be seamlessly migrated into subsequent modules while maintaining equivalence. Second, we propose fast and stable scheme to calculate effective shifting and scaling values. The channel-wise shifting aligns the center of each channel for removal outlier asymmetry. The channel-wise scaling quantitatively evaluates changes brought by migration and quantization for better quantization burden balance. We validate our OS+ under both standard and fine-grained quantization settings with models including BERT, OPT, BLOOM, BLOOMZ, and LLaMA. Comprehensive results across various tasks demonstrate the superiority of approach. Especially, with standard quantization, OS+ can achieve near-floating-point performance on both small models and large language models on 8-bit and 6-bit. Besides, we establish a new state-of-the the-art for 4-bit BERT with 15.5% improvement. Our code is available at https: //aa:://iiie //gthuu. com/ Moo//ddeTTC///// Outlier_Bee Sealierr_uppresai.i. Plus.
However, quantization, particularly post-training quantization (Choukroun et al., 2019; Banner al., 2018: Wu al., 2020) under the setting of limited data and GPU resources, has become increasingly challenging on these models (e.g., 12% accu- racy drop in BERT (Bondarenko et al., 2021) and catastrophic degradation in OPT-175B (Dettmers et al., 2022)). This is caused by the presence of detrimental outliers in activation (e.g., the range of distribution be 80 in BERT and even 140 in OPTs), which prevents discrete numbers from accurately representing continuous ones.
To combat the bottleneck, researchers make in- depth investigations and find that outliers mainly concentrate on certain channels. Some works (Bon- darenko et al., 2021; Dettmers et al., 2022) suggest fine-grained quantization schemes and offer extra bit levels for outlier channels. Others (Wei et al., 2022b; Xiao et al., 2022) take the activation scal- ing to scale outliers and migrate scaling values to subsequent weights for FP equivalence. However, the former might hurt the quantization accelera- tion effect while the latter determines scaling val- ues without the consideration of minimizing the change introduced by migration and quantization, which we find is sub-optimal. Mcanwhile, we also identify a new outlier characteristic that previous works overlooked but also responsible for the large tensor range.
1 Introduction
Transformer language models (e.g., BERT, LLMs) have garnered sigmificant attention due to their re- markable performance and scalable model size. These models have evolved from hundreds of mil- lions of parameters (Devlin et al., 2018; Liu et al.,
In this paper, we propose the Outlier Suppres- siont framework composed of channel-wise shift-
Corresponding author.

---

Channel index
-1.7
-1.5
1.5
1.5
6353
5.7
43
-19
-5
6354
-20
5
8725
-0.9
1.4
-1.2
1.2
-1.2
1.2
8726
-100
30
Range
(a) Original distribution
(b) Channel-wise shifting
(c) Channel-wise scaling
Figure I: Distribution of OPT-66B. Fig. la shows the original distribution with asymmetric outliers consistently occurs at certain channels, owning considerable range (-97, 43). Fig. Ih depicts the channel-wise shifting operation to decrease the tensor range by eliminating the asymmetry. Fig. Ic further scales down the outliers to threshold 5 and finally results in distribution ranging from -5 to 5.
ing and scaling to effectively pursue better quanti- zation performance while equivalently keeping the FP output. First, we find new feature of outliers that they stay in asymmetric shape across chan- nels (e.g., Fig. la, one problematic channel on OPT-66B occupies the negative axis from -97 to -58 while another one has positive values ranging from 5.7 to 43). This outlier asymmetric presenta- tion could cause significantly wide distribution of tensor like 140 even composed of channels with rel- atively small ranges like 39. Thus, propose the channel-wise shifting operation, which shifts the activation across channels to eliminate the impact of asymmetry. Together with channel-wise scal- ing for concentrated outliers, a unified migration pattern is introduced to seamlessly transfer the re- versed effects of these operations to later modules to maintain equivalent FP models. Second, we de- vise deliberate schemes determine effective shift- ing and scaling values. The shifting vector aligns the center of each channel, reducing the whole tensor range to its maximum channel range. The scaling values quantitatively minimize the interac- tive output change of the activation and weights induced by migration and quantization, achieving balanced quantization burden with fast and stable search procedure.
1. We find new feature of outliers that show asym- metric shapes across channels and then propose the channel-wise shifting operation, along with taking channel-wise scaling for the outlier con- centration attribute. unified migration pattern that migrates their reversed effects to later mod- ules is designed to guarantee an equivalent FP network.
2. We propose fast and stable ways to determine effective shifting and scaling values. Shifting values eliminate the asymmetry feature across channels while scaling values scale down out- lier channels towards a quantitative optimization objective.
3. We assess the efficacy of our approach under both standard and fine-grained quantization set- tings. On standard one, OS+ achieves near- floating-point performance on 8-bit and 6-bit BERT, OPTs, BLOOM, and BLOOMZ. On fine- grained one, OS+ surpass others by 9.41% on 4-bit LLaMA with per-token quantization and obtain lossless results on 4-bit OPT with per-group quantization.
2 Related work
Due to the space limit, we give the most relevant papers here and put a complete related work in the Appendix A. In the realm of PTQ, researchers have discovered that the poor performance of trans- former language models should be attributed extreme outliers in activations, which exhibit spe- cial characteristics from both channel and token aspects. Thus, we will introduce related works
Our algorithm can be carried out efficiently and enjoy affordability real hardware, producing more iiaantization-friende models in minutes and requiring no extra inference burden on LLMss. To this end, our main contributions can be summarized into three aspects:

---

from the two aspects.
Quantization. We indicate standard quantization as per-tensor activation quantization, per-channel, or per-tensor weight quantization here because such schemes will not separate the integer matrix mul- tiplication. Per-tensor means assigns quantization parameters for each tensor and per-channel for each output channel. Also, for some fine-grained ways, mainly consider per-token (Yao et al., 2022) and per-group (Yao al., 2023) here, which calculates quantization parameters in each token or group.
Channel aspect. Outliers consistently emerge in certain channels over different inputs. Bon- darenko et al. (2021) employs per-embedding- group quantization scheme that uses different quan- tization parameters for distinct channel groups, while Dettmers al. (2022) suggests utilizing FP16 representations for problematic channels holding signals over Wei et al. (2022b) introduces an out- lier suppression (OS) framework with one of com- ponents called Gamma Migration. Observing that outliers accumulate in certain channels, it adopts a scaling vector to scale outliers and migrates it to subsequent modules. Xiao et al. (2022) further proposes calculating scaling values by equalizing ranges between activations and weights and eval- uates on large language models. Guo al. (2023) discards normal values adjacent to outliers, mak- ing room for outliers with customized GPU sup- port. To consider the standard quantization, we find that Wei et al. (2022b) and Xiao et al. (2022) still waste a large portion of quantization levels the extreme outlier asymmetry across channels. Meanwhile, Wei et al. (2022b) simply views the scaling parameter in LayerNorm (LN) as the seal- ing vector for outliers, which might not always be consistent with the outlier distribution. Xiao et al. (2022) that adopts the heuristic way and obtains equalized ranges between activation and weights lacks quantitative evaluation of their output change induced by migration and quantization.
Method
We first present our equivalent shifting and scal- ing operations, then introduce ways to determine effective values for them.
4.1 Equivalent shifting and scaling
In this section, we comprehensively investigate out- lier features, naturally introducing the design of shifting and scaling operations, followed by uni- fied migration pattern.
4.1.1 Outlier shifting and scaling
Channel-wise shifting. For transformers, espe- cially LLMs, we find that outliers show asymmet- ric behavior among channels. Recall that Fig. la, the 8725-th channel displays a hard negative inter- val (-97, -58), while another channel dominates a positive one 43). Due to this asymmetry, even if the range of each channel is relatively small, such as 40 and 39 for outlier channels and minuscule values for normal channels, the range of the entire tensor swell to considerably large value (e.g., 140, ranging from -97 to 43), which negatively af- fects quantization performance.
Token aspect. Different tokens exhibit varying de- grees of outliers. Dettmers et al. (2022); Yao et al. (2022) introduce a novel scheme called per-token quantization that dynamically computes quantiza- tion parameters for each token. Wei et al. (2022b) investigates the clipping impact of outliers and rec- ommends finding appropriate clipping range in token-wise manner. In this paper, we focus on the channel aspect and might combine these techniques when necessary.
To handle this issue, we propose channel-wise shifting, which can eliminate the impact of asym- metry by taking the following operation:
= - 2,
(1)
where z serves as a row vector (z E R") and shifts the activation for each channel. In this way, with a carefully designed z which we will introduce in Sec. 4.2.1, 1, the new tensor X' can get rid of the out- lier asymmetry attribute. For example, by aligning the centers of each channel in Fig. 1b, the range can be reduced to 40 (the maximum channel range) from 140 (the large tensor range). Finally, note that this operation is not the conventional shifting operation for symmetric quantization, operates thannel-wiscl and provides better distribution for per-tensor quantization.
3 Preliminary
Basic Notations. We denote matrices as upper case letters (e.g., X) and vectors as lower case letters (c.g., œ). Operator @ and represent clement-wing multiplication and division for matrices vectors. We use WX X as matrix-matrix multiplication. Fur- thermore, X,,i refers to the element of the t-th token and the j-th channel in transformer models. (-) denotes the quantization function.

---

the feasibility of this algorithm on two common structures. Equivalent shifting and scaling
Effective values
LegerNorat
Qhard
(a)
MHA Pre-LN
Wy O
2W
0
y
W; O
W +
Qunnt
W, O
zW. b.
Residaal consecion
Norm
0 -
FEN Post-LN
Chans
(b)
(f 0)
W O
ZW +
shift: align channels
olomoet-atre nuttjitiatiion
sultpliattion
scale: scale down threshold across channelz
Figure 2: Left: We show the equivalent shifting and scaling operations by giving two representative examples: (a) for problematic output of Pre-LN (LayerNorm put inside residual connection) with Multi-Head Attention (MHA) structure; (b) for problematic output of Post-LN (LayerNorm put before residual connection) with Feed-Forward Network (FFN). Right: For effective shifting and scaling values, the shifting vector can align the center of cach channel to and the scaling vector would shrink outliers into the outlier threshold which is searched based on its left metric.
Channel-wise scaling. Apart from the asymmetry feature across channels, there also exists the out- lier concentration phenomenon (Wei al., 2022b) that outliers predominantly accumulate in specific channels over various inputs. For example, the 8725-th and the 6354-th channels in Fig. Ia hold more aggressive values than others. Therefore, af- ter shifting, we equip with the channel-wise scaling to narrow them down to further alleviate the quan- tization difficulty.
Linear Layer. First, consider prevalent sce- nario where linear (convolutional) layer imme- diately follows. Reversing the above operations (i.e., [X [ s w) + b) equals to updating the W E R"" and b E R= in the next layer, given by
(X + W +b
= (X W +2W +b
(3)
X(W s (=WT +
X = - 2)
(2)
According to Eq. (3), weight and bias can absorb s and z, respectively, and thus becomes:
the above equation, the row vector s E R" scales the shifted tensor for each channel and brings final coantizatiion-rrendde activation X. For example, in Fig. Ic, tensor with a size of 10 can be obtained if we scale down channels with signals over 5. De- tailed calculation of will be given in Sec. 4.2.2. Implementation. It easy to implement these operations. Take the output of LayerNorm Fig. as an example, we only need to replace its linear transformation parameters 3 and y with (B-z)0s and 7 @ 8 to achieve shifting and scaling effects. For others, we can update parameters in the former DeQuant function.
81
82
8n
81
82
8n
W =
W O
(4)
81 82 8n_
b = wW
+b.
For example, Fig. 2(a) depicts the typical chal- lenging activation (output LayerNorm) in the attention structure, all following weights and bi- ases can absorb the shifting and scaling signals without any extra computation burden.
Residual connection. Second, we consider the case where residual connection is applied after the LayerNorm structure (Post-LN) and fed into the quantized input. As shown in Fig. 2b, in ad- dition to linear layer transformation, the identity function will be substituted with channel-wise mul- tiplication and addition to maintain equivalence. We demonstrate that these increased calculations
4.1.2 Unified migration pattern
As mentioned in Eq. (1) and Eq. (2), subtract z and divide make the problematic activation resilient quantization. To keep an equivalent FP model, unified migration pattern is proposed that transfers both reversed shifting and scaling vectors to subsequent modules. We demonstrate

---

will only incur negligible inference burden in Sec. 5.5.
of activation and weight separately (i.e.. min, E -)) (X 2 $ and ming E [[Q : 8) W o s a task loss perspective is adopted by concerning their matrix multiplication output. We measure the output change after scaling and quantizing weight and activation to pursue effective factors, given by:
Finally, because 8 and z serve as shared parame- ters across tokens and batches of data, the unified migration pattern can be well-implemented and produce the same oulput without additional com- putation most of the time.
4.2 Effective shifting and scaling
minE Q((x - 2) (esess)p SW C +
Based on the equivalent shifting and scaling opera- tions, this section, we propose fast and stable scheme to pursue effective values.
outpu after scaling and quantization
(6)
(XWT bb
original FP cniput
4.2.1
Shifting values
where the mean squared error (MSE) used to quantify the difference.
The design of the shifting vector should eliminate the impact of asymmetry across channels. Thus, we devise align the center of each channel O so that the outlier channel will not occupy only the positive or negative side. In detail, z is defined as the average of the minimum and maximum signals in each channel, given by:
Multiple linear layers: Furthermore, we study the case for multiple linear layers like the attention structure (Fig. 2a), where three weights will be mul- tiplied by the same sealing vector and calculated with the same suppressed activation.
In this scenario, their matrix multiplication out- puts produced by scaled and quantized matrices are marked as Qq. Kg. Vg (Original outputs are denoted as Q, K, V). Applying Eq. (6) to three linear layers separately and simply summing the losses make it difficult to illustrate their differ- ent importance and usages. Therefore, we employ the attention mechanism post-process function to reasonably organize their scaling and quantiza- tion information, given by:
max X + min X
2j=
(5)
With the channel-wise shifting now, the tensor range reduces the largest channel range, getting rid of being defined by asymmetric outliers.
4.2.2 Scaling values
The design of the scaling vector should further scale down outliers while bringing marginal impact on following weight quantization. The following parts introduce how obtain it with the proposed optimization objective and procedure.
min B|/offaa Q,K, V., softmax (V
(7)
Challenges. Recall that the equivalent transfor- mation Eq. (4) also scales weights and poten- tially leads to inferior weight quantization, which requires us to calculate elaborate scaling values to reach quantization balance between activa- tion and weights. Nevertheless, we find previous works (Wei al., 2022b; Xiao et al., 2022) ei- ther ignore the affected following weight or take a heuristic way that simply equalizes ranges of ac- tivation and weights. Unlike them, we think the key point is to minimize their interactive output change resulting from migration and quantization (a detailed analysis is available in Table 6). Hence, new optimization objective is proposed.
Normalization and masking are omitted for nota- tion simplicity, and it can be seen that information from the first two linear layers has been encapsu- lated within the attention map.
Optimization procedure. Toward the above ob- jective, fast and stable procedure introduced search the scaling vector. First, we find that scaling down only channels with outliers can bring better performance. Because channels with normal acti- vations can exhibit more variation over different inputs, it can be difficult to find a decent scaling value for them. Also, considering that they are not responsible for low quantization performance, scal- ing them is not necessary. Second, we propose optimize an alternate variable called outlier thresh- old t, which would squeeze only channels with an activation range over into (-t, f) and keep others intact (Fig. 2). Essentially, here used to specify
Optimization objective. We first study the simple case that the problematic activation acts as the input of one linear layer (c.g., Fig. 2b).
Instead of minimizing quantization errors

---

which channel to scale down, the final scaled ac- tivation range, well as the scaling values in the following weights.
centile (Wu et al., 2020), OMSE (Choukroun al., 2019), and recent works on BERT quantization including PEG (Bondarenko et al., 2021), and Out- lier Suppresion (Wei et al., 2022b). For large models including OPT, BLOOM, and LLLMA, we mainly compare with recent works including Zero- Quant (Yao et al., 2022), and SmoothQuant (Xiao et al., 2022). For details, readers can refer to Ap- pendix C.
This technique simplifies the complex problem with numerous variables s to a single variable t. Then we adopt the simple grid search for t to min- imize the objective Eq. (6), Eq. (7). After getting the effective t, the scaling vector is calculated as:
8j = m1x01.0, max x 21))
(8)
Implementation. We randomly select 128 samples from the training dataset, in-domain data for the GLUE benchmark, and PILE (Gao et al., 2020) dataset for zero-shot tasks. A batch of data is first used to calculate effective shifting and scaling vec- tors. Then, calibration is conducted. More details can be found in Appendix C.
1
5 Experiments
The evaluations are designed to show: I. satisfac- tory predictions of our OS+ for both small and large language models with standard quantization; II. consistent performance of OS+ even lower-bit with fine-gres greigraiddd quantization; III. ablation study; III. analysis like computation complexity.
5.2 Standard quantization with OS+
In this section, we show how OS+ can help standard quantization achieve satisfying results from both the small models and LLMs aspects.
5.1 Set up
Quantization setting. Both the standard and fine- grained quantization are considered. For the stan- dard one, we take quantization nodes the same as in Wei et al. (2022b); NVIDIA (2022), always adopt per-tensor activation quantization, consider per- tensor (fastest speed) and per-channel (high perfor- mance) weight quantization, For the fine-grained quantization, we adopt per-token (Yao et al., 2022) and per-group (Yao et al., 2023) quantization. Notation: We use INT8, INT6, INT4 to denote the bitwidth of activation and weight. Specifically. INT8* refers to per-tensor weight quantization. And per-token and per-group quantization will be marked in the table below.
Method
CoLA
MNLI
QNLI
SST-2
STS-B
Avg.
FP32
59.6
84.9
91.8
93.4
89.5
83.8
INT8
MinMax
52.3
81.3
89.0
91.1
86.2
79.5
OMSE
54.8
82.1
89.7
91.3
87.7
81.6
PEG
59.4
81.3
91.1
92.7
87.9
82.5
OS
60.3
83.9
90.2
92,9
88.2
83.0
OS+
91.1
92.7
88.3
INT6
OMSE
35.4
73.7
84.7
86.3
85.8
73.5
Percentile
37.3
72.1
79.4
87.3
86.8
72.9
OS
54.4
81.8
89.8
91.9
88.7
81.2
OS+
56.0
84.5
90.9
92.4
89.5
82.8
INT4
Models and tasks. We conduct experiments on both small and large language models. First, BERT models (base and large versions) are evaluated on the GLUE benchmark (Wang et al., 2018a). Second, four of the largest OPTs ranging from 13B to 175B, biggest BLOOM (Scao et al., 2022) and BLOOMZ (Muennighoff et al., 2022) boast- ing 176 billion parameters, and LLaMA (Tou- vron et al., 2023) models including 7B, 13B, 30B, 65B sizes are chosen as representatives. Zero- shot tasks including language modeling, multiple choice, commonsense reasoning, etc. are selected for evaluation. The evaluation code is based on tar-haaness-evve eeweleaaiiin i
OMSE
4.7
38.5
52.2
50.3
0.2
41.1
Percentile
7.0
53.0
61.5
77.1
66.1
57.0
OS
28.5
57.9
72.5
80.4
67.8
62.7
OS+
50.0
80.2
85.4
91.4
86.5
78.2
Table 1: PTQ performance BERT-base models. MNLI and STS-B report the combined score. Avg. indicates the averaged results 8 tasks on GLUE benchmark (details Appendix B). means per-tensor quantization for weight. OS indicates Outlier Suppression for short.
BERT. Table gives prediction results of common PTQ algorithms. Most methods perform well on INT8* but fail on lower bits while our approach consistently achieves superior outcomes. Com- pared Wei et al. (2022b), our method outper- forms by 1.6% and 15.5% on 6-bit and 4-bit, re- spectively. In summary, our approach can achieve near- flartfloaiing point performance on high bits and
Baselines. For BERT, we adopt classical PTQ techniques as baselines, including MinMax, Per-
hpapee //g.t//ttat com

---

PIQA (f)
Winogrande (f)
HellaSwag (1)
LAMBADA (
Model
Method
FP16
INT8
INT6
FP16
INT8
INT6
FP16
INT8
INT6
FP16
INT8*
INT6
7enoQuant
54.1
53.0
52.1
51.1
26.5
25.8
42.9
0.0
OPT-13B
Smooothuunt
75.8
76.0
73.5
63.1
64.9
60.3
$2.5
52.2
49.2
68.6
68.3
65.2
76.4
68.3
ZeroQuant
54.2
25.7
9.7
OPT-30B
SmoothQuant
77.6
77.2
66.7
68.5
68.2
55.0
54.3
54.2
37.4
71.5
71.0
13.4
OS+
77.4
77.4
68.0
68,9
34,2
53.7
70.8
69.6
ZeroQuant SmonthQuat
53.2
51.9
50.7
48.0
26.1
25.7
0.6
0.0
OPT-66B
78.3
72.9
OS+
78.7
77.5
69.0
69.4
56.2
55.8
73.0
72.7
7enoQuint Smoothhuant
52.3
53.1
50.2
49.1
25.4
25.6
0.0
0.0
OPT-175B
79.7
79.7
52.6
72.5
71.2
49.1
59.3
58.9
26.0
74.7
74.6
0.5
OS+
79.6
80.0
72.5
58.5
74.7
74.2
ZeroQuant SmoothQuant
76.0
69,4
30.5
67.8
7.5
BLOOM-176B
78.8
77.7
76.7
70.3
68.6
67.6
55,9
54.1
52.1
67.7
69.2
60.2
OS+
78.4
78.1
69.8
70.3
55.2
54.8
68.0
69.2
ZeroQuant
79.1
54.0
70.9
49.6
56.3
28.2
67.6
1.4
BL.00M--76B
SmonthQumt
79.7
68.7
OS+
79.9
79.9
71.3
70.6
56.7
56.4
68.8
69.2
Table 2: Comparison among different techniques in terms of accuracy on four zero-shot tasks. INT8* specifically means per-tensor quantization for weights compared to INT8. More tasks are put in Appendix due to space limit.
reduce the performance gap to 5.6% on 4-bit. OРT and BLOOM. With standard quantization, we list 8-bit and 6-bit accuracy in Table 2. It can be observed that OS+ outperforms ZeroQuant by a large margin. While SmoothQuant suffers from non-negligible accuracy drops on much harder set- tings like the 6-bit 175B model with significantly severe outliers, ours still gives enjoyable results, owning 32.5% upswings on HellaSwag task, 27.4% boost on PIQA. Results of BLOOM models indi- cate that their quantization challenges are less se- vere than OPTs with smaller accuracy drops across methods. Our approach still beats the best of others by about 2% points on 6-bit. To conclude, with standard quantization, ours is indeed close to FP re- sults 8-bit and exhibits around point accuracy degradation on 6-bit.
large signals, even exceeding 600. Given such a challenge, provide experiments both with and without quantization of this layer in Table 3 and Table 10, respectively. In both tables, we high- light our lossless performance on 6-bit quantization while SmoothQuant still suffers in Table 10. Also, shows the superior performance of OS+ on 4-bit (e.g., 10.58% improvement on Winogrande, 10.04 PPL decrease on WikiText2).
Per-group Quantization. Additionally, per-group quantization (Yao et al., 2023), which tailors quan- tization parameters for each group of elements, is more fine-grained way. Recognizing the difficul- ties of 4-bit quantization for OPT's we illustrate an example by adopting per-group quantization with relatively large group sizes of 1024 and 512. Fig. 3 shows that OS+ continues to outperform other methods and can be more competitive under harder cases such as a group size of 1024.
5.3 Fine-grained quantization with OS+
Here, OS+ is combined with fine-grained quanti- zation to validate its wide application and go ex- tremely low bit setting like 4-bit quantization. Per-token Quantization. Per-token quantiza- tion (Yao et al., 2022), which customizes quan- tization parameters for individual tokens, can bring better predictions, especially for lower-bit quanti- zation and longer output like WikiText2 (Merity al., 2017). We opt for LLaMA models for valida- tion. It's worth noting that the structure of LLaMA differs from others in its design of element-wise multiplication of two activations as the input to the final layer in FFN, potentially resulting in very
MinMax
SmsetthQunst
08
FP16
PPL
INT4 (g1024) INT4 (g512)
INT4 (g1024) (NT4(1513)
WiiTea12
PIOA
Figure 3: Results of 4 hit quantization with group size set to 1024 and 512, respectively.

---

PIQA (f)
Winogrande (?)
HellaSwag (t)
WikiText2 (J)
Model
Method
FP16
INT6
INT4
FP16
INT6
INT4
FP16
INT6
INT4
FPI6
INT6
INT4
MinMax
77.26
55.98
66.54
49.64
71.78
32.28
6.00
473.97
LLaMA-1-7B
SmoothQuant
77.37
77.18
70.08
66.93
65.51
$2.96
72.99
72.10
58.13
5.68
5.85
16.87
56.67
72.32
5.76
14.17
MinMax
78.56
50.65
69.53
50.28
75.26
26.34
5.58
3410.45
LLaMA-I-13B
SmeothQuant
79.05
78.45
66,49
70.09
69,69
51.78
76.22
75.20
58.95
5.09
5.25
56.75
OS+
78.73
75.03
69.53
61.17
75.74
67.21
5.22
18.95
MinMax
78,40
50,00
72:45
50,12
77.25
27.09
5.09
2959.15
LLaMA-I-30B
SmoothQuant OS+
80.09
78.78
71.55
72.77
73.01
54.54
79.21
78.13
60.97
4.10
4.40
51.47
79,98
73.01
73.64
60.38
78.77
68.03
4.30
22.61
MinMax
77.58
50.27
69.46
49.33
78.72
24.59
5.25
14584.66
LLaMA-1-65B
SmoothQuant
80.85
78.40
65.02
77.11
74.30
51.14
80.73
78.57
59.78
3.56
3.77
19.37
61.72
79.76
9.33
Table Comparison in normalized accuracy, accuracy, normalized accuracy and perplexity (PPI.), respectively (Touvron et al., 2023). Techniques are equipped with per-token quantization (Yao et al., 2022). More results are put in Appendix
5.4 Ablation study
5.5 Analysis
Different activation scaling, Because scaling val- ues act in both the activation and weights, reducing quantization error for individual tensors not guarantee the minimum output change, which en- capsulates their information later forward pass, For example, in Table 6, Outlier Suppression with fixed scaling values has the smallest quantization error for weight. SmoothQuant with heuristic way has the smallest quantization eтror for activation. However, both of them did not bring the smallest quantization error for the output. This reveals the importance of directly optimizing according to the output, which is what our method exactly does. Thus, we can enjoy the best final performance.
Design choices of scaling values. In this section, we compare different scaling vector designs. In Table 4, the second row displays results without attention post-processing Eq. (7). Summing the losses of multiple linear layers, as shown, proves unwise, resulting in performance declines of about 2% and 10% OPTs. The third row removes the outlier threshold and instead learns scaling values directly. We find this process is unstable and re- quires suitable hyperparameters, causing failure on LLMs. As mentioned in Sec. 4.2.2, This instabil- ity may stem from suboptimal scaling values for normal channels with varying magnitudes.
Model storage and accuracy. Inspired by vari- ety of models with diverse sizes, also study the relationship between their storage and accuracy un- der quantization settings. Focusing on one kind of model with distinct quantization bit-width, Fig. 4 shows that 8-bit quantization which cuts storage by about half, can generally maintain original per- formance, and 6-bit quantization can lead to less performance drop on larger models. Moreover, con- sidering fixed storage constraints, discover that quantized big models typically outperform small FP models. These observations can relate model robustness, which implies that large models can
Effect of each operation. From Table 5, it can be observed clearly that by removing the shifting op- eration, the accuracy drops by about 1%-3% under difficult settings. This is because, without channel- wise shifting that initially smooths the quantiza- tion challenge, scaling factors struggle to suppress outliers effectively while producing the tolerable weight quantization burden. Furthermore, when excluding scaling effects, performance decreases significantly, with even crashed results LLMs.
OPT-66B (INT6)
BERT (INT4)
Method
PIQA
Winogrande
SST-2
MNLI
scaling
76.5
66.5
89,3
77.7
Method
OPT-666 (INT6)
BERT (INT4)
aaattcntion post process
74.5
57.4
89.1
77.1
PIQA
Winogrande
SST-2
MNI.I
outlier threshold
Fail
Fail
83.2
65.2
Ours
77.5
69.4
91.4
80.2
Table 4: Design choices of scaling factor. The second row
shifting
76.5
66.5
89.3
77.7
removes the attention post process in optimization objective. The third row chooses to learn the scaling vector directly rather than alternately optimize the outlier threshold.
shifting scaling
54.7
49.4
82.3
63.7
Table 5: Bffect of scaling and shifting operations.

---

uctivation
weight
Output chunge
Method
128
1.4XX
128
F1XX
1.4XX
munge
MSE
range
MSE
MSE
141x
.00x
ociginal
31.6)
(-0.13, 0.13)
1.44X
1.5x
OS
(:23.5.15.7)
[-0.0,,
1,43X
.4XX
SQ
(-3.5, 2.0)
(3.4, 3.5)
Our scaling
(-8.4, 8.4)
48.54
(1.2. 1.3)
0.02
133489
1,40x
1.53
Table 6: Detailed analysis of different techniques from the activation scaling aspect. OS indicates Outlier Suppression and SQ indicates SmoothQuant.
16
16
1.11X
BERT b-2
BFRT--arg-225
OPT-138-64
Figure Real lateney (x-axis) of our transformed 8-hit mod- els, 8-bit and FP16 original models over different batch sizes (y-axis). BFRT-large-255 refers to the BERT-large model with sequence length set to 256 while for OPT-13B- output length with input length set to 512. Bold numbers indicate quantization specdup.
benefit from compression more if special outliers are handled well.
13B 13B-64, 64 means
Computation Complexity. We explain our com- putation complexity calibration and deployment phases. For the calibration process, OS+ is efficient, and able to generate scaling and shifting values in about 20 minutes for OPT-175B offline. Moreover, due to the equivalent transformation, our method does not demand additional training and can be applied in a post-training setting. For deployment, we discuss inference efficiency with latency per- formance evaluated using (NVIDIA, 2022). As mentioned before, our channel-wiss shifting and scaling can be implemented by updating previous parameters, and be migrated to subsequent weights. For LLMs, our transformation does not introduce any extra computation burden and leads to favor- able latency improvements, as demonstrated in 1.5x speedup in Fig. 5. Only BERT models addi- tionally replace the identity function in the residual connection with channel-wise multiplication and addition. Such overhead is minimal, as shown in Fig. 5, resulting in comparable latency speedup.
in LLMs and other transformers. Our framework is simple to use, consisting of both scaling and shifting operations, which can be efficiently and effectively implemented. Experiments demonstrate the efficacy of our methods for suppressing outliers.
Limitations
While we have observed features of outliers and devised methods to deal with them, the underly- ing reasons for their emergence and attributes have not been fully understood. This may require in-depth analysis of the training pipeline, including the procedure and hyperparameters. Such investi- gations are time-consuming but can benefit both FP and quantized scenarios.
Ethics Statement
Our Outlier Suppression+ framework aims im- prove the quantization performance transformer language models. It can boost the development practical and green machine learning and does not incur extra ethical concerns.
6 Conclusion
We present the Outlier Suppression+ framework for uddressing asymmetric and consistent outliers
Acknowledgment
We sincerely thank the anonymous reviewers for their sincere reviews and valuable suggestions to make this better. We also thank Qi Zhang for the in- sightful discussion and Jing Liu for helping to build the code of LLaMA. This work was supported by the National Natural Science Foundation of China (No. 62022009), National Natural Science Foun- dation of China (No. 62306025), the State Key Laboratory of Software Development Environment SKLSSD-20022XX
Accuracy
68
64
100
200
250
Storage(Mi
Figure 4: Averaged accuracy on PIQA, Winogrande, LAM- BADA, and HellaSwag of OPTs with different storages. We draw circles, rectangles, and triangles to refer to FP16, the 8-bit and 6-bit models with quantized activation and weight.

---

References
Jacob Devlin, Ming-Wor Mee Chang, Kenton Lee, and Kristina Toutanova, 2018. Bert: Pre- training of deep bidirectional transformers for language understand- ing. arXiv preprint 1810v0181..04805.
Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. 2020. Binarybert: Pushing the limit of bert quantization. arxiv preprint arXiv:20/2.1570.
7hen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2019. Hawq: Hessian aware quantization of neural networks with mixed- precision. In Proceedings of the IEEE/CVF Interna- rional Conference on Computer Vision, pages 293- 302.
Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. 2018. Aciq: analytical clipping for integer quantization of neural networks.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. 2021. Understanding and overcoming the challenges of efficient transformer quantization. arXiv preprint arXiv: 12948.
Steven Esser, Jeffrey McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. 2019, Learned step size quantization. arXiv preprint arXiv:1902.08/55 090.00853........
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herve Jegou, and Armand Joulin. 2020. Training with quantization noise for extreme model compression. arXiv preprint arXiv:2004.073220.
Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2020. Zeroq: A novel zero shot quantization framework. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13169-13178.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.174 173223.....
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing. Travis Hoppe, Charles Foster, Jason Phang, Ho- race He, Anish Thite, Noa Naheshima, al. 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling. arXiv preprint XXiv:2101.00022
Zhaowei Cai and Nuno Vasconcelos. 2020. Rethinking differentiable search for mixed-precision neural net- works. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2349-2355
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. 2019. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In The IEEE International Conference on Computer Vision (ICCV).
Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. 2023. Diffrate: Differentiable com- pression rate for efficient vision transformers. arXiv preprint 2305. 17997.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalaksmmi Srinivasan, and Kailash Gopalakrishnan. 2018. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint 1rxiv:1805.06085
Cong Guo, Yuxian Qiu, Jingwen Leng, Xiaotian Gao, Chen Zhang. Yunxin Liu, Fan Yang, Yuhao Zhu, and Minyi Guo. 2022. Squant: On-the-fly data-free quan- tization via diagonal hessian approximation. arXiv preprint arXiv:2202.07477.
Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. 2019. Low-bit quantization of neural net- works for efficient inference. In 2019 IEEE/CVF International Conference on Computer Vision Work- shop (ICCVW), pages 3009-3018. IEEE.
Cong Guo. Jiaming Tang, Weiming Hu, Jingwen Leng, Chen Zhang, Fan Yang, Yumxin Liu, Minyi Guo, and Yuhao Zhu, 2023. Olive: Accelerating large lan- guage models via hardware-friendly outlier-victim pair quantization. Matrix, 17(4.2):7-1
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. 2015. Binaryconnect: Training deep neural networks with binary weights during propagations. Advances in neural information processing systems, 28.
Song Han, Huizi Mao, and William Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman cod- ing, arXiv preprint arXiv:1510.00144
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.022 150.00233.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm. int8 O: 8-bit matrix mul- tiplication for transformers at scale. arXiv preprint arXiv:2008.03339.
Edward J Hu, Yelong Shen, Phillip Wallis. Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models. arXiv preprint 10X.v:2100.09685.
Tim Detmmers and Luke Zettlemoyer. 2022. The case for 4-bit precision: k-bit inference scaling laws. arXiy preprint arXiv:2212.09720.

---

Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. 2021. Accurate post training quantization with small calibration sets. In Inter- national Conference on Machine Learning, pages 4466-4475. PMLR.
Jing Liu, Zizheng Pan, Haoyu He, Jianfei Cai, and Bo- han Zhuang. 2022. Ecoformer: Energy-saving at- tention with linear complexity. Advances in Neural Information Processing Systems, 35: 15:10295-10308.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprini arXiv: 190.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Meng- long Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. Quanti- zation and training of neural networks for efficient integer-arin proger--rithmetic-onne
inference. In Proceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR).
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- cls.
Qing Jin, Jian Ren, Richard Zhuang, Sumant Hanu- mante, Zhengang Li, Zhiyu Chen, Yanzhi Wang, Kaiyuan Yang, and Sergey Tulyakov. 2022. F8net: Fixed-poin 8-bit only multiplication for network quantization. arXiv preprint 2rX11:2202.05239.
Paulius Micikevieius, Dusan Stosic, Neil Burgess, Mar- ius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, et al. 2022. Fp8 formats for deep learning. arXiv preprint 2Xix:2209005433.
Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. 2021. 1-bert: Integer- only bert quantization. In International conference on machine learning, pages 5506-5518, PMLR.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong. Hailey Schoelkopf. et al. 2022. Crosslingual generaliza- tion through multitask finetuning. arXiv preprint arXiv:2211.072
Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. 2021. Bert busters: Outlier dimensions that disrupt transformers. arXiv preprint arXiv:2105.06990.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. 2020. Up down? adaptive rounding for post- training quan- tization. In International Conference on Machine Learning, pages 7197-7206. PMLR.
Andrey Kuzmin, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, and Tijmen Blankevoort. 2022. Fp8 quantization: The power of the exponent. arXiv preprint 2208 09225.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. 2019. Data- free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 1325-1334.
Yann LeCun, John Denker, and Sara Solla. 1989. Opti- mal brain damage. Advances in neural information processing systems, 2.
Yanjing Li, Sheng Xu, Baochang Zhang. Xianbin Cao, Peng Gao, and Guodong Guo. 2022. Q-vit: Accurate and fully quantized low- bit vision transformer. arXiv preprint arXiv:2210.0m 10.00677
NVIDIA. 2022.
Faster
transformer.
bttps://githen http://ggttuuu...m//..eeeee FasterTrans forme..
Yuhang Li, Xin Dong, and Wei Wang. 2019. Addi- tive powers- two quantization: An efficient non- uniform discretization for neural networks. arXiv preprint arXiv: 190919994.1...
Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice Orlettr 2022. Outliers dimensions that disrupt transformers driven by frequency. arXiv preprint arXiv:2205.11388
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. 2021. Brecq: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations.
Haotong Qin, Yifu Ding, Mingyuan Zhang, Qinghua Yan, Aishan Liu, Qingqing Dang, Ziwei Liu, and Xi- anglong Liu. 2022. Bibert: Accurate fully binarized bert. arXiv preprint arXiv:2203..6330.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. 2018. Improving language under- standing by generative pre-training.
Ji Lin. Jiaming Tang, Haotian Tang, Shang Yang. Xingyu Dang, and Song Han. 2023. Awq: Activation- aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.0097.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilie, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Mathias Gallé, al. 2022. Bloom: A 176b- parameter acen-aceess multilingual language model. arXiv preprint arX
Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang. 2023. Qllm: Accu- rate and efficient liw-ditwidth quantization for large language models. arXiv preprint arXiv:2310.00 030000....

---

Mingzhu Shen, Feng Liang. Ruihao Gong, Yuhang Li. Chuming Li, Chen Lin, Fengwei Yu, Junjie Yan, and Wanli Ouyang. 2021. Once quantization-wware train- ing: High performance extremely low-bit architecture search. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pages 5340-5349.
Hao Wu, Patrick Judd, Xiaojic Zhang, Mikhail Isaev, and Paulius Micikevicius. 2020. Integer quantization for deep learning inference: Principles and empirical evaluation. arXiv preprint arXiv:2004. 09602...
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien De- mouth, and Song Han. 2022. Smoothquant: Accurate and efficient post-training quantization for large lan- guage models. arXiv preprint 1.
Sheng Shen, Zhen Dong. Jiayu Ye, Linjian Ma, Zhewei Yao. Amir Gholami. Michael W Mahoney, and Kurt Keutzer. 2020. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 8815-8821.
Ke Xu, Lei Han, Ye Tian, Shangshang Yang. and Xingyi Zhang. 2023. Eq-net: Elastic quantization neural networks. In Proceedings of the IEEE/CVF Interna- rional Conference on Computer Vision, pages 1505- 1514.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. Using deep- speed and megatron to train megatron-turing nlg 530b, large-scale generative language model. arXiv preprint arXiv:2201.1 11990.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. arXiv preprint urXiv:2206.0786).
Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. 2023. Zeroquant-v2: Exploring post-training quantization in Ilms from comprehen- sive study to low rank compensation. arXiv preprint arXiv:2303.0830
Chaofan Tao, Lu Hou, Wei Zhang, Lifeng, Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. 2022. Compression of generative pre-trained lan- guage models via quantization. arXiv preprint arXiv:2203.10705.
Zhihang Yuan, Chenhao Xue, Yiqi Chen, Qiang Wu, and Guangyu Sun. 2021. Ptq4vit: Post-training quan- tization framework for vision transformers. arXiv preprint arXiv:2111.12293.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anm Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971
Ofrr Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop Energy Ffficient Machine Learning and Cognilive Computing-Neutt -eeuriS Edition (EMC2-NIPS), pages 36-39. IEEE.
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018a.
Glue: A multi-task benchmark and analysis platform for natural language understanding, arXiv preprint vrXiv:1804.017
Aohan Zeng. Xiao Liu, Zhengxiao Du Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash Gopalakrishnan. 2018b. Train- ing deep neural networks with 8-bit floating point numbers. Advances in neural information processing systems, 31.
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. 2018 Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European conference on com- puter vision (ECCV), pages 365-382.
Peisong Wang, Qiang Chen, Xiangyu He, and Jian Cheng. 2020. Towards accurate post-training -raiirrinnng net- work quantization via bit-split and stitching. In In- ternational Conference on Machine Learning, pages 9847-9856. PMLR.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2255.00666
Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. 2022a. Qdrop: Randomly dropping quantization extremely low-bit post-training quan- tization. In International Conference on Learning Representations.
Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. 2020. Teraryyber: Distillation-awarr ultra-low low bit bert. arXiv preprint arXiv:2009.1 12812.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. 2022b. Outlier suppression: Pushing the limit of low-bit transformer language models. arXiv preprini arXiv:2209.15 13325.
Xiangguo Zhang, Haotong Qin, Yifu Ding, Ruihao Gong, Qinghua Yan, Renshuai Tao, Yuhang Li, Feng- wei Yu, and Xianglong Liu. 2021. Diversifying sam- ple generation for accurate data-free quantization. In

---

The IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR).
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. 2019. Improving neural network quantization without retraining using outlier channel splitting. In International conference on machine learning, pages 7543-7552. PMLR.
Barret Zoph and Quee V Le. 2016. Neural architecture search with reinforcement learning. arXiv preprint 1rX1:7661.00158.

---

A Related work
from both channel and token aspects. In terms of channels, outliers consistently emerge in certain channels over different inputs. Bondarenko et al. (2021) employs a per-embedding grrppp quantiza- tion scheme that uses different quantization param- eters for distinct channel groups, while Dettmers et al. (2022) suggests utilizing FPI6 representa- tions for problematic channels holding signals over 6. Wei et al. (2022b) identifies this feature lying in LayerNorm's output and migrates the scaling parameter of LayerNorm to subsequent modules to attenuate outliers. Xiao et al. (2022) proposes calculating scaling values by equalizing ranges be- tween activations and weights and evaluates on large language models. Guo et al. (2023) discards normal values adjacent to outliers, making room for outliers with customized GPU support. Compared to them, we design the scaling factors that con- cem the interactive results of troublesome activa- tion and following weights to scale down channels with outliers offline. Also, we notice the asym- metric presentation of outliers and design a shift- ing operation. While we operate on corresponding channels between weights and activation, a later work (Liu et al., 2023) adopts the splitting and merging operations to transfer the quantization bur- den of outlier channels opposite channels of weights, which might encourage us to design a more flexible technique without the same or oppo- site channel index requirement. In terms of tokens, different tokens exhibit varying degrees of outliers. Dettmers et al. (2022); Yao et al. (2022) introduce novel scheme called per-token quantization that dynamically computes quantization parameters for each token. Wei et al. (2022b) investigates the clip- ping impact of outliers and recommends finding an appropriate clipping range a token-wise manner. Besides, some studies focus on weight quanti- zation, such as Dettmers and Zettlemoyer (2022); Frantar al. (2022); Zeng al. (2022); Lin et al. (2023) and some including Yuan et al. (2021); )); Li et al. (2022), investigate the quantization of Vision Transformer (ViT) models. Interestingly, several studies (Kovaleva et al., 2021; ;; Puccetti et al., 2022) explore the underlying reasons for emerging out- liers and trace them back to the pre-training phase.
Quantization. Compression has become more and more popular these days (Han et al., 2015; Hinton et al., 2015; Hu et al., 2021; Liu et al., 2022; Xu et al., 2023; Chen et al., 2023). One of its effective techniques called quantization (Jacob et al., 2018) employs low-bit representations for activation and weight in neural networks. Researchers catego- rize this approach into two pipelines: post-training quantization (PTQ) and quantization-awan train- ing (QAT). QAT (Courbariaux et al., 2015; Choi et al., 2018; Esser et al.. 2019; Li al., 2019; Gong et al., 2019; Shen et al., 2021; Zhang et al., 2018) trains the quantized model end-to-end, necessitat- ing significant GPU resources and the entire train- ing dataset. In contrast, PTQ (Choukroun et al., 2019; Wu al., 2020; Banner et al., 2018; Wang al., 2020; Zhao et al., 2019; Nagel et al., 2019) only requires hundreds of samples and limited re- source consumption, producing a calibrated model quickly. Recently, several works (Nagel et al., 2020; Hubara et al., 2021; Li et al., 2021; Wei et al., 2022a) proposed to adjust models slightly for improved PTQ performance. Besides, other types of quantization include zero-shot quantization with- out real calibration data (Cai et al., 2020; Zhang al., 2021; Guo al., 2022), mixed-precision with mixed bit-width (Dong al., 2019; Cai and Vas- concelos, 2020), and FP8 data type (Wang et al., 2018b; Kuzmin et al., 2022; Micikevicius et al., 2022: Jin et al., 2022).
Quantization of transformer language models. Recently there has been growing interest in the quantization of transformer language models. In the context of QAT, Zafrir et al. (2019) first ex- plores 8-bit quantization for BERT-like models. Shen et al. (2020) introduces group-wise quanti- zation and studics mixxddpreeisiio quantization based Hessian information. Bai et al. (2020); Zhang al. (2020); Qin al. (2022) combine distillation strategies with quantization. Kim et al. (2021) approximates the nonlinear function in trans- former architectures to enable integer-only infer- ence. Fan et al. (2020) incorporates quantization noise for enhancement. Additionally, Tao et al. (2022) investigates the challenges of quantizing generative models.
B Supplementary experiments
In the realm of PTQ, researchers have discov- ered that the poor performance of these models should be attributed to extreme outliers in activa- tions. These outliers exhibit special characteristics
BERT-base. We provide detailed results of BERT- base models on GLUE benchmarks in Table 7. In- terestingly, we find that models which are sensitive

---

Algorithm 1: Outlier Suppression+
Input: Problematic output X of LayerNorm with parameters 7, subsequent module M with weight W and bias b, grid search iteration K. (1. Effective shifting and scaling:)
min(X X.,) ! (.,)
Effective shifting vector.
loss" INF for to K do
t = max X 2)
D. Enumerate outlier threshold.
; = max(1.0,
Calculate loss;; based on Eq. (6). Eq. (7).
if loss" > loss; then
loss" x = lossg,
P Effective scaling factors.
12 Equivalent shifting and scaling:
8 (3 2) 0
Ø s*
D Fuse 2, 8" into former operations.
b =W b, W
W @
Update following modules.
return Transformed LayerNorm and subsequent module;
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST-2
STS-B
Method
Avg.
(Matt.)
(ace m/mm)
(fl/acc)
(ace)
(f1/ace)
(acc)
(acc)
(Pear/Spear.)
FP32
59.6
84.9/84.8
91.4/87.8
91.8
87.8/90.9
72.6
93.4
89.7/89.3
83.8
INT8*
MinMax
52.3
80.9/81.7
85.3/80.9
89.0
84.8/88.6
68.2
91.1
84.7/87.6
79.5
OMSE
54.8
81.9/82.2
89.7/86.0
89.7
86.1/89.5
72.2
91.3
87.2/88.2
81.6
PEG
59.4
81.3
88.5
91.1
89.4
69.3
92.7
87.9
82.5
OS
60.3
83.8/84.0
90.4/87.0
90.2
87.3/90.4
71.1
92.9
87.8/88.7
83.0
Ours
60.9
84.4/84.4
90.6/87.2
91.1
87.1/90.6
73.3
92.7
87.7/88.9
83.5
INT8
MinMax
57.1
82.8/83.5
89.9/85.8
90.8
87.8/90.7
69.7
92.8
86.8/88.6
82.3
OMSE
57.2
84.0/84.3
1/85/8
91.1
87.6/90.5
72.2
92.2
87.9/88.7
82.9
Percentile
57.1
83.9/84.1
90.7/86.7
91.3
87.7/90.7
71.1
93.4
87.7/88.7
82.9
OS
61.6
84.4/84.5
91.4/87.8
91.5
87.9/90.8
72.2
93.8
89.2/89.0
84.0
Ours
60.3
84.8/84.5
90.5/87.0
91.6
87.5/90.8
71.5
93.6
89.3/89.2
83.6
INT6
MinMax
17.7
32.5/32.5
0.7/31.9
65.2
40.9/69.0
48.0
82.0
59.8/60.3
47.1
OMSE
35.4
74.0/73.3
81.5/76.5
84.7
1/.1/
64.3
86.3
85.6/86.1
73.5
Percentile
37.3
72.4/71.7
85.1/79.9
79.4
72.6/80.2
61.7
87.3
86.4/87.3
72.9
OS
54.4
82.0/81.7
87.5/83.3
89.8
84.7/88.9
70.8
91.9
88.7/88.6
81.2
Ours
56.0
84.6/84.4
90.0/86.3
90.9
87.0/90.5
71.8
92.4
89.6/89.4
82.8
INT4
MinMax
-6.6
32.6/32.7
0.0/31.6
50.6
53.8/36.8
47.7
50.9
-0.5/00.5
29.5
OMSE
4.7
38.5/38.4
81.3/69.1
52.2
45.2/50.9
59.9
50.3
0.1/-0.4
41.1
Percentile
7.0
52.6/53.5
83.0/75.7
61.5
44.7/68.3
55.6
77.1
65.9/66.3
57.0
OS
28.5
57.5/58.3
83.9/75.7
72.5
45.4/70.8
56.7
80.4
67.8/67.9
62.7
Ours
50.0
80.6/79.9
87.6/83.1
85.4
85.0/77.5
65.7
91.4
86.4/86.5
78.2
Table PTQ performance of BERT-base models GLUE benehmark. per-tensor quantization for weight. OS indicates Outlier Suppression for short.

---

CoLAA
MNLI
MRPC
QNLI
QQP
RTE
SST-2
STS-B
Method
Avg.
(Matt.)
(acc m/mm)
(f1/acc)
(acc)
(f1/acc)
(acc)
(acc)
(Pear//pearr)
FP32
63.3
86.7/85.9
91.6/88.0
92.2
88.1/91.1
74.0
93.5
90.3/90.1
84.9
INT8"
MinMax
62.4
72.0/73.0
76.3/72.8
87.0
66.5/80.4
46.9
92.2
58.6/52.1
71.5
OMSE
59.9
82.7/83.5
87.8/83.8
89.0
79.2/86.2
47.3
92.0
83.9/83.3
78.1
Percentile
61.3
84,5/84.0
91.6/88.9
91.6
85.9/89.4
69.3
92.4
88.3/88.1
83.1
OS
62.3
85.1/84.5
90.1/86.0
91.1
87.0/90.3
75.1
92.4
88.7/88.4
83.9
Ours
62.2
85.9/85.2
90.9/87.0
92.2
87.8/90.8
71.8
93.3
89.3/89.3
84.1
INT6
MinMax
5.6
32.0/32.0
50.2/46.1
50.2
0.0/63.2
49.5
53.0
5.0/4.8
38.1
OMSE
14.0
59.3/58.4
86.
79.5
52.5/73.5
54.9
74.8
44.0/37.9
59.8
Percentile
16.4
63.5/63.8
82.0/77.2
87.0
44.8/70.7
49.8
81.7
65.7/67.8
62.8
OS
24.1
71.3/71.7
85.5/79.4
80.8
68.8/78.3
47.3
82.3
61.1/62.0
65.4
Ours
60.9
86.3/85.4
91.8/88.2
92.0
87.7/90.8
71.5
93.7
86.7/85.6
83.7
Table 8: PTQ performance BEKT-large models on GLUE benchmark. means per-tensor quantization for weight. OS indicates Outlier Suppression for short.
to different learning hyperparameters during the fine-tuning phase, such CoLA and RTE, also exhibit less favorable quantization outcomes. This suggests a possible relationship between quantiza- tion and robustness.
for MinMax and SmoothQuant, which makes us think of model design, training techniques, and efficient fine-tuning for quantization.
Implementation details
BERT-large. We also conduct experiments on BERT-large models in Table 8. Results across meth- ods indicate that quantizing BERT-large models is more challenging (e.g., MinMax suffers from considcrable accuracy drop (about 13%) on INT8* compared to BERT-base, and Outlier Suppression also fails on the 6-bit setting). Fortunately, with Outlier Suppression+, the results can be improved, yielding an 18.7 7% enhancement.
C.1 OS+
In this section, we provide detailed descriptions of our implementation with the core part distilled algorithm
BERT. On the GLUE benchmark, fine-tuned FP models used for quantization. We randomly lect 128 samples and set the batch size 32, First, batch of data used calculate the effective shifting and scaling signals for problematic acti- vations, especially outputs after LayerNorm here. Then shifting and scaling vectors are fused into former operations and absorbed in later modules. On fused models, we apply the calibration proce- dure. Particularly, great variance of token range as discussed in Yao al. (2022); Wei et al. (2022b), we incorporate the Token Wise Clipping proposed Outlier Suppres- sion which an orthogonal technique and weakens outliers from the token aspect.
ОPT. Here, we provide results of OPTs on more tasks. Table is the supplement Table 2, which further shows consistent performance enhancement OS+
LLaMA. Recall that we conduct experiments on LLaMA with different settings in the fine- grained quantization section. Table gives the results when quantizing the special and challenging structure (the last layer of FFN) in LLaMA models. It can be observed that ours still earns near- floating- point performance on 6-bit quantization and beats others by about 5%~ 14% in terms of averaged ac- curacy of the first four tasks, and four times PPL decrease for WikiText2. By comparing with the easier setting Table 3, we find that the special structure with large signals really leads to much lower 4-bit outcomes across methods, especially
BERT models, due to the
OPTs. For OPTs, quantize pre-trained mod- els and evaluate them zero-shot tasks. 128 samples are randomly extracted from one of the train datasets, namely the PILE dataset. As we have observed that LayerNorm produces severe asymmetric outliers on certain channels, the pro-

---

Name
Method
OPT-13B
OPT-30B
OFT-66B
OPT-175B
FP16
INTS"
INT8
INI6
FP16
INTS
INT8
INT6
FPL6
INIS
INTH
INI5
FPL6
INTS
INT8
INT6
LLM.mnt9 A
75.8
78.7
PIQA
ZeroQuan*
75.8
54.1
53.00
77.6
54.2
52.8
78.7
53.2
51.9
79.7
52.3
53.1
SmoothQuuat
76.0
72.5
77.2
66.7
78.3
52.0
79.7
52.6
Ours
76.4
75.9
75,8
77,4
77.6
77.4
78.7
78.6
77.5
79.6
79.5
N000
LLMin880
68.4
71.4
73.8
74,6
LAMBADA
ZeroQuant*
68.6
0.0
0.0
21.5
0.0
0.0
73.9
0.0
0.0
74.7
0.0
0.0
SmootthQuunt
68.5
61.2
71.0
13.4
72,9
0.0
74.6
0.5
Ours
68.3
68.4
65.7
70.8
70.8
69.6
73.0
73.4
72.7
74.7
74,5
74.2
LLM.m88
52.4
54,3
56.3
59.2
HelbaSang
ZeroQuant*
52.5
26.5
25.88
54.3
26.4
25,7
56.4
26.1
25.7
59.3
25,4
25.6
SmuuilQuantt
52.2
49.2
54.2
37.4
55.9
26.5
58.9
26.0
54.2
54.2
56.3
59.2
59.3
5XL5
LLM.inti
64.8
68.1
68.5
72.3
Winogrando
ZenQuunt
65.1
52.1
51.1
68.5
51.8
51.8
68.9
50.7
48.0
72.5
50.2
49.1
SmoothhQusa
68.2
55.0
08.3
52.1
71.2
49.1
68.0
68.5
68.9
72.5
71.7
ARC
LLMim
33.5
34.7
37.0
40.9
(Challlnge)
Smuuiliquant
ZeroQuant
32.8
19.5
20.7
34.6
33.8
19.8
26-7
20.6
37.3
20.8
36.5
23.9
20.4
40.3
21,8
40.5
21.2
20.6
33.5
32.7
34.5
34.7
34.6
37.5
37.2
40.3
39,9
41.0
ARC
ZenoQuam
LLMiniii
67.3
27.5
67.3
70.1
30.5
69.7
25.0
71.7
71.8
74,9
24.0
74.8
25.6
(Bas))
SmoothQuunt
69.7
55.8
70.5
74.1
28.8
Durs
67.3
67.0
70.1
70.0
68.9
71.3
71.8
74.8
74.7
74.3
I. LiMini
86.0
82.0
87.0
89.0
COPA
Z
86.0
82.0
55.0
55.0
W6.0
88,0
$5.0
SmouthQuunt
85.0
83.0
75.0
58.0
55.0
Durs
85.0
86.0
$5.0
83.0
82.0
$4.0
85.0
86.0
84.0
$8.0
89.0
91.0
LLM.mnB8
76.3
77.1
77.7
79.3
StoryClose
ZeroQuan A
49.6
44.3
77.0
48.5
48.0
77.5
49.2
79.5
47.7
48.2
Smuuil-Quant
76.0
73.5
76.9
61.4
77.3
48.8
79.
49.8
Ours
75.8
76.0
75.4
77.01
76.9
76.6
77.3
76.4
76.6
79.2
79.1
78.1
Avg
Ours
65.5
65.5
65.5
64.7
67.0
66.9
66.8
66.7
68.8
68.5
68.6
68.0
71.1
71.0
71.1
71.1
Table 9: Comparison among different techniques in terms of accuracy on eight zero-shot tasks. fine- fine-grained quantization, bringing extra computation overhead. INT8* specifically adopts per-tensor quantization for weights compared to INTS.
denotes dynamic and
posed method is applied here. After obtaining a more friiiization-friendlly model, the MinMax al- gorithm collects distribution statistics. Since di- verse tokens do not have outliers of varying degrees on these models, advanced clipping techniques are not involved.
unlike OPTs, for tasks with normalized accuracy metrics, we report the normalized accuracy metric instead of the accuracy one to align the original paper (Touvron et al., 2023). This point has also been indicated in each table below.
C.2 Baselines
BLOOM and BLOOMZ. The main pipeline is similar to OPTs. The only exception is using the Token-Wise Clipping as the calibration method be- cause these models hold different outliers among different tokens. The clipping ratios are searched as 0.5% and 1.5% for 8-bit and 6-bit BLOOM, and 0.0% and 0.5% on BLOOMZ.
We introduce the implementation details of base- lines here. MinMax obtains the minimum and maximum statistics of the tensor for the quanti- zation clipping range. Percentile (Wu et al., 2020) uses the activation distribution percentile as the quantization clipping range. Using the dev set, we search its 0.99999]. OMSE (Choukroun et al., 2019) min- imizes the mean squared error between quanti- zation and FP signals. PEG (Bondarenko et al., 2021) applies fine-grained quantization to problem- atic activation from channel perspective. Out- lier Suppression (OS) (Wei et al., 2022b) uses fixed scaling factors suppress outliers and fur- ther clips outliers in a token-wise manner. Zero- Quant (Yao et al., 2022) uses per-token quantiza- tion, assigning different quantization parameters to different tokens. This fine- ine-grained scheme from
LLaMA. The main pipeline is similar to OPTs with some small differences. First, we use the Wiki- Text2 dataset for calibration. Second, as LLaMA does not have biases, introducing channel-wise shifting might incur little overhead. Thus, for fair comparisons, we simply omit channel-wise shifting for LLaMA here. Third, when taking the harder setting that quantizes the last layer in FFN, the channel-wise scaling is also conducted thereby updating the quantization scale of up proj and weight parameters of down proj, which does not bring computation overhead during inference. Last,
within [0.999, 0.9999,

---

Model
Method
PIQA (1)
(TC+
ARC-(()
HellaS
Winggrande (
WilTTes22 (.)
FP16
INT6
INT4
FP16
INT6
INT4
FP16
INT6
INT4
FP16
INT6
INT4
RP16
INT6
INT4
RP16
INT6
INT4
MinMax
77.53
53.37
52.36
29.55
60.35
25.09
70.95
30.98
61.72
53.01
6.22
430.33
SQ
77.37
76.65
52.48
41.38
40.10
25.80
72.99
71.52
27.40
61.88
43,00
5.08
6.15
52.85
52.27
40.7M
71,68
65.11
54,85
40.32
57.66
13B
SQ
79.05
77.50
59.81
56.35
36.51
65.62
42.58
26.71
96.22
75.11
41.56
70.09
68.
48.70
5.09
5.50
79.35
OS+
78.24
02.62
57.83
37,67
43.43
30.46
74,96
52.21
68.59
$1.07
5.37
53.64
MinMax
75.92
19.46
56.31
26.30
13.69
29.18
76.14
25.00
89 69
15.62
5.51
1958
308
SQ
80.00
71.14
50.16
58,92
57.61
28.11
45,29
42.91
26.71
79.21
78,07
31.97
72.77
69.92
51.14
4.10
5.37
299
79.16
07.19
59.13
48.45
46.25
35.56
78.19
56.44
72.53
51.85
4,48
112.33
MinMars
55.18
25.35
5.08
54035
77.51
79.00
53.08
32.60
Table 10: Comparison on LLaMA-1 in terms of normalized accuracy (Touvron et al., 2023) for the first four tasks, accuracy for Winogrande and perplexity for WikiText2. The technique in each row is equipped with per-token quantization in ZeroQuant (Yao et al., 2022). This table would quantize the last layer in FFN compared to Table 3.
the token aspect also requires dynamic quantization. Meanwhile, for INT8* we implement per-group weight quantization according to its description. SmoothQuant (Xiao et al., 2022) migrates scal- ing factors to later modules to smooth problematic activation. Their scaling factors equal the range between activation and weights. For lower bits, we also search its hyper-parameter a according to its description for better performance.


Reviews:
[{"review_id": "TQJh3JPIKZ", "reviewer": "Reviewer_AfNB", "summary": "", "questions": "- Is there a specific reason for the absence of 4-bit quantization results for LLM, unlike BERT? I'm curious to know if applying shifting/scaling to INT4 quantization in LLM remains a challenging endeavor.\n- Do the authors believe that the proposed activation shifting is orthogonal and can be applied in conjunction with other PTQ methods, such as SmoothQuant? It seems that it might enhance the performance of other methods as well. I'm interested to know author's perspective on this.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive comments and feedback. Hope the following reply helps address the concerns.\n* **Q1**: Generalization and Evaluation: The paper appears to fall short in generalizing its methodology to other pre-trained GLM models. While the approach may prove potent for models like OPT where asymmetry in activation is pronounced, its effectiveness might be attenuated for GLMs that are more quantization-friendly in their activation. (This can be inferred from the performance discrepancy between OPT in INT6 and BLOOMZ in INT6). It would be beneficial to observe results on widely-used pre-trained GLMs like LLaMA to better assess the utility of this method. Additionally, it seems the paper omits a comparative analysis on the Perplexity (PPL) measure, which typically gauges a decoder model's generation ability.\n \n  **A1**: Thanks for the valuable suggestion. \n  * We investigate LLaMA models and find that they also hold the outlier concentration phenomenon (e.g., the 3968 and 3180 channels in one layer) and asymmetric outliers across channels (e.g., (1.0, 17.7) range on the 3968 channel and (-27.3, -19.6) on 3180). Thus, our methods can also address the similar problem for these models. The table below shows the results of LLaMA-7B.\n \n    | Method             | bit-width | PIQA (acc) | WikiText (PPL) | Winogrande (acc) |\n    | ------------------ | --------- | ---------- | -------------- | ---------------- |\n    |                    | FP16      | 78.29      | 9.45           | 66.93            |\n    | MinMax             | 8-bit     | 76.88      | 10.50          | 63.77            |\n    | shifting + scaling | 8-bit     | **77.80**  | **9.47**       | **67.01**        |\n    | MinMax             | 6-bit     | 67.46      | 31.00          | 55.94            |\n    | scaling            | 6-bit     | 75.73      | 10.88          | 62.04            |\n    | shifting + scaling | 6-bit     | **77.26**  | **10.01**      | **65.75**        |\n\n    It can be seen that ours can achieve near-floating point performance on 8-bit and 6-bit. Our scaling operation can even improve 20 PPL and the shifting operation further brings 2-3% accuracy improvement on PIQA and Winogrande. We will provide more experiments in the revision. \n  * Sorry for the omitting of the PPL metric. Due to the space limit, we only report the accuracy metric for the LAMBABA dataset in the original paper. We will include its PPL in the revision. To further address the concern, we conduct the experiment on the WikiText dataset with the PPL metric in the table below. It can be seen that our results are still close to FP16.\n    | WikiText (PPL)     | OPT-13B | OPT-30B | OPT-66B | OPT-175B |\n    |--------------------|---------|---------|---------|----------|\n    | FP16               | 14.13   | 13.09   | 12.30   | 11.00    |\n    | 8-bit              | 14.21   | 13.15   | 12.35   | 11.02    |\n    | 6-bit              | 14.41   | 13.34   | 12.49   | 11.13    |\n\n* **Q2**:  Insufficient Analysis: The paper appears to lack a comprehensive analysis of how the proposed shifting and scaling methods influence the reduction of quantization error. For instance, the introduction mentions that \"Existing work did not quantitatively analyze the effect of activation scaling on the quantization of subsequent layer's weight well.\" Yet, it's unclear if this paper provides an analysis beyond performance differences concerning the effects of scaling on subsequent layers. It would be helpful to clarify if there's an aspect I might be overlooking. A suggestion would be to delve deeper into how the scaling values determined by the proposed method correlate with module-specific distributions. A more in-depth analysis in this regard could significantly bolster the paper's strength. \n \n   **A2**: Thanks for your suggestion. We’d like to give clearer analyses in the following:\n   * The channel-wise shifting reduces the quantization error by decreasing the tensor range to its maximum channel range. For instance, on OPT-66B, where outliers exhibit a wide range (-93.9, 31.6), the mean squared quantization error is 209.8. The shifting operation aligns the center of each channel, changing the range to **(-13.9, 13.9)** followed by a smaller quantization error of **112.8**.\n\n  * The channel-wise scaling reduces the quantization error compared to other scaling methods by quantitatively minimizing interactive output change of activation and weights. We will make the mentioned sentence clearer in the revision. Because scaling values act in both the activation and weights, reducing quantization error for individual tensors can not guarantee the minimum output change, which encapsulates their information to later forward pass. \n\n    |                       |               activation  |       |        weight   |       | Output change |\n    |-----------------------|---------------------------|-------|-----------------|-------|---------------|\n    |                       | range                     | MSE   | range           | MSE   | MSE           |\n    | original              | (-93.9, 31.6)             | 209.8 |  (-0.13, 0.13)  | 0.001 |   18061.5     |\n    | Outlier Suppression   | (-23.5,15.7)              | 142.9 | **(-0.40, 0.41)** | **0.006** |    6182.52    |\n    | SmoothQuant           | **(-3.5, 2.0)**           | **3.65** | (3.4, 3.5)      | 0.43  |    3535.86    |\n    | Our scaling operation | (-8.4, 8.4)              | 48.54 |  (1.2, 1.3)     | 0.02  |    **1334.89**    |\n    \n    For example, in the table above, Outlier Suppression with fixed scaling values has the smallest quantization error for weight. SmoothQuant with a heuristic way has the smallest quantization error for activation. However, both of them did not bring the smallest quantization error for the output. This reveals the importance of directly optimizing according to the output, which is what our method exactly does. Thus, we can enjoy the best final performance.\n\n* **Q3** : Is there a specific reason for the absence of 4-bit quantization results for LLM, unlike BERT? I'm curious to know if applying shifting/scaling to INT4 quantization in LLM remains a challenging endeavor.\n   \n  **A3**: Thanks for your suggestion. We did not conduct experiments on 4-bit LLMs because compared to BERT, tensors in LLMs have much more elements that makes quantization challenging. For example, 4-bit weight-only quantization incurs an accuracy drop [1].\n\n   Nevertheless, motivated by ZeroQuant-v2 [2] which adopts per-group quantization for W4A8, we also combine per-group quantization with our method and report results on W4A4 in the table below.\n\n    | 1024 group size     | OPT-13B   | OPT-30B   | OPT-66B   | OPT-175B  |\n    | ------------------- | --------- | --------- | --------- | --------- |\n    | **WikiText (FP16)** | 14.13     | 13.09     | 12.30     | 11.00     |\n    | MinMax              | 17011.67  | 44.89     | 239.02    | 26.91     |\n    | Ours                | **15.48** | **13.89** | **12.90** | **11.24** |\n    | **PIQA (FP16)**     | 75.79     | 77.58     | 78.73     | 79.71     |\n    | MinMax              | 53.21     | 67.63     | 57.67     | 64.80     |\n    | Ours                | **74.16** | **76.88** | **77.69** | **79.82** |\n   \n    | 512 group size      | OPT-13B   | OPT-30B   | OPT-66B   | OPT-175B  |\n    | ------------------- | --------- | --------- | --------- | --------- |\n    | **WikiText (FP16)** | 14.13     | 13.09     | 12.30     | 11.00     |\n    | MinMax              | 1149.70   | 19.52     | 18.57     | 15.27     |\n    | Ours                | **14.50** | **13.43** | **12.66** | **11.18** |\n    | **PIQA (FP16)**     | 75.79     | 77.58     | 78.73     | 79.71     |\n    | MinMax              | 62.24     | 72.74     | 74.59     | 75.75     |\n    | Ours                | **74.96** | **77.20** | **77.97** | **79.71** |\n\n    It can be seen that ours can achieve near-floating point performance even with a big group size and can surpass MinMax by a large margin. Especially, it can be observed that the number of groups can be decreased with our methods to reach satisfying results. We will include more results on 4-bit quantization in the revision.\n  \n   [1]. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR2023\n   \n   [2]. ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation.\n\n * **Q4**: Do the authors believe that the proposed activation shifting is orthogonal and can be applied in conjunction with other PTQ methods, such as SmoothQuant? It seems that it might enhance the performance of other methods as well. I'm interested to know author's perspective on this.\n\n   **A4**: Yes, we agree with the reviewer’s point that the channel-wise shifting is orthogonal. First, it works from a different observation about outliers that they show asymmetric shape across channels. Second, it makes the activation more friendly for per-tensor quantization, giving better support to subsequent techniques. In the table below, we combine it with SmoothQuant on OPT-66B and Winogrande.\n\n   |              | SmoothQuant | Our scaling operation |\n   | ------------ | ----------- | --------------------- |\n   | w/o shifting | 51.14       | **66.5**              |\n   | w/ shifting  | **66.69**   | **69.4**              |\n  \n   It can be seen that our shifting operation can boost the performance of SmoothQuant from 51.14 to 66.69 with **15.54%** improvement.  Also, considering that our scaling operation is better than that of SmoothQuant (**66.5 vs 51.14**), combining both of our operations can finally achieve a near floating-point performance.\n\n* **Q5**: Typos Grammar Style And Presentation Improvements.\n\n  **A5**: Thanks for the suggestions. We will condense the main table to include more pre-trained models, and emphasize more about the channel-wise shifting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-structured overall, facilitating easy comprehension. It effectively represents the need for shifting/scaling and illustrates the process required in subsequent modules to maintain equivalence clearly.\n- The paper's identification of asymmetric distribution as a challenging aspect in per-tensor activation quantization is insightful. Proposing the shifting technique to address this concern adds to its novelty.\n- The authors convincingly showcase the efficiency of the proposed algorithm and its deployment efficiency, underscoring its practical applicability.", "reasons_to_reject": "- **Generalization and Evaluation**: The paper appears to fall short in generalizing its methodology to other pre-trained GLM models. While the approach may prove potent for models like OPT where asymmetry in activation is pronounced, its effectiveness might be attenuated for GLMs that are more quantization-friendly in their activation. (This can be inferred from the performance discrepancy between OPT in INT6 and BLOOMZ in INT6). It would be beneficial to observe results on widely-used pre-trained GLMs like LLaMA to better assess the utility of this method. Additionally, it seems the paper omits a comparative analysis on the Perplexity (PPL) measure, which typically gauges a decoder model's generation ability.\n\n- **Insufficient Analysis**: The paper appears to lack a comprehensive analysis of how the proposed shifting and scaling methods influence the reduction of quantization error. For instance, the introduction mentions that \"Existing work did not quantitatively analyze the effect of activation scaling on the quantization of subsequent layer's weight well.\" Yet, it's unclear if this paper provides an analysis beyond performance differences concerning the effects of scaling on subsequent layers. It would be helpful to clarify if there's an aspect I might be overlooking. A suggestion would be to delve deeper into how the scaling values determined by the proposed method correlate with module-specific distributions. A more in-depth analysis in this regard could significantly bolster the paper's strength.", "questions_to_authors": "- Is there a specific reason for the absence of 4-bit quantization results for LLM, unlike BERT? I'm curious to know if applying shifting/scaling to INT4 quantization in LLM remains a challenging endeavor.\n- Do the authors believe that the proposed activation shifting is orthogonal and can be applied in conjunction with other PTQ methods, such as SmoothQuant? It seems that it might enhance the performance of other methods as well. I'm interested to know author's perspective on this.", "paper_topic_and_main_contributions": "This paper presents a novel approach to addressing the challenges posed by outliers in the quantization of transformer language models. Taking into account the observed asymmetric characteristics of these outliers, the authors propose an efficient shifting and scaling mechanism to mold a quantization-friendly distribution. They further demonstrate that implementing this technique preserves the equivalence of the original model computations. By adapting the quantization target distribution to be more amenable to quantization, the method achieved superior and more robust performance than previous state-of-the-art methods, specifically on BERT with W4A4 and LLM with W6A6 configurations.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- It would be beneficial to condense the tasks presented in the main table and also include results for other GLM models for a comprehensive comparison.\n\n- One of the most innovative aspects of this paper's methodology appears to be the shifting of the asymmetric distribution for per-tensor activation quantization, distinguishing it from conventional quantization-friendly approaches. It would be beneficial to emphasize this aspect more prominently."}}, {"review_id": "ZTlwKsVZ7d", "reviewer": "Reviewer_uJMf", "summary": "", "questions": "**Question 1**: Is there any evidence to suggest that this outlier asymmetric distribution is static and can be estimated from a small number of samples? You could consider searching for z and s within the in-distribution dataset and then directly testing on the out-of-distribution datasets.\n\n**Question 2**: Have you tested the method on tasks that are more sensitive to parameter changes, such as language modeling tasks like Wikitext?\n\n**Question 3**: Have you tested more aggressive bits like 4-bit or 5-bit on OPTs or BLOOMZs?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the reviewer’s valuable suggestions and positive feedback on this paper. Below is the detailed response to each question. Hope the following reply helps address the concerns.\n\n* **Q1** : While the authors claim that z and s can be determined with only a small number of samples, it is difficult not to question whether these few samples can provide sufficient diversity and completeness for the distribution boundary.\n\n  **A1** : Thanks for the question. We find that outliers in transformer language models show some static features including concentration on fixed channels [1, 2, 3] and similar extreme positive and negative values across batches of samples, indicating that a small calibration size for getting s and z is enough. \n\n  To verify this, we provide an ablation study (which will be included in the revised draft). The calibration samples are randomly chosen from the PILE dataset (one of the train datasets). From the second column, it shows that sample sets of different sizes have similar ranges. \n\n   | Number of samples | min range /max range of one layer |\n   | ----------------- | --------------------------------- |\n   | 16                | -93.8750 / 31.5938                |\n   | 64                | -93.8750/ 31.7656                 |\n   | 96                | -94.0000 / 31.8125                |\n   |128               | -94.0000 / 31.8125                |\n\n   Besides, we give a possible explanation by attributing the observation to the inherent behaviors of per-trained models. Like [2], we also observe that the boundaries are often provided by high-frequency tokens like the separator one. Thus, going through batches of samples with an output length set to 512 can include high-frequency tokens and is enough for the boundaries. \n \n  [1]. Understanding and Overcoming the Challenges of Efficient Transformer Quantization. EMNLP2021\n\n  [2]. Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models. NeurIPS2022\n\n  [3]. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. NeurIPS2022\n\n* **Q2**: Although the results in Section 4 are quite good, QA and text classification tasks inherently have a higher tolerance for errors. It is uncertain how well the method would perform on tasks that are more sensitive to parameter changes, such as language modeling tasks.\n\n  **A2**: Thanks for your suggestion. We conduct experiments on WikiText here. It can be seen that our method still achieves near-floating point performance for these models on 8-bit and 6-bit quantization. This indicates that our scaling and shifting values found by PILE dataset are good enough.\n\n\n    | WikiText (PPL)     | OPT-13B | OPT-30B | OPT-66B | OPT-175B |\n    |--------------------|---------|---------|---------|----------|\n    | FP16               | 14.13   | 13.09   | 12.30   | 11.00    |\n    | 8-bit              | 14.21   | 13.15   | 12.35   | 11.02    |\n    | 6-bit              | 14.41   | 13.34   | 12.49   | 11.13    |\n\n* **Q3**: Is there any evidence to suggest that this outlier asymmetric distribution is static and can be estimated from a small number of samples? You could consider searching for z and s within the in-distribution dataset and then directly testing on the out-of-distribution datasets.\n\n  **A3**: Yes, in the table for Q1 here, the second column shows close boundaries across different calibration sizes. For experiments of LLMs, to evaluate zero-shot tasks, we adopted a different dataset PILE (one of the train datasets) for calculating s and z and quantization calibration. Then, experiments are conducted directly on the out-of-distribution datasets including LAMBADA, PIQA, Winogrande, etc. Good results in the paper indicate the superiority of parameters determined by the in-distribution dataset and suggest static features of outliers. \n\n* **Q4**: Have you tested the method on tasks that are more sensitive to parameter changes, such as language modeling tasks like Wikitext?\n\n  **A4**: Yes,  we have tested on the LAMBADA dataset with PPL and accuracy metrics. Due to the space limit, we did not report the PPL one in the original paper but will include them in the revision. Also, we conduct experiments on WikiText in the table for Q2 here.\n\n* **Q5**: Have you tested more aggressive bits like 4-bit or 5-bit on OPTs or BLOOMZs?\n\n  **A5**: Thanks for the insightful comment. We find that 4-bit quantization is much more challenging with limited bit levels to represent too many values. For example, 4-bit weight-only quantization can incur an accuracy drop [1]. \n\n    However, motivated by ZeroQuant-V2 which adopts per-group quantization for W4A8, we also combine the per-group quantization with our method and report results on W4A4 with group sizes 1024 and 512.\n\n\n    | 1024 group size     | OPT-13B   | OPT-30B   | OPT-66B   | OPT-175B  |\n    | ------------------- | --------- | --------- | --------- | --------- |\n    | **WikiText (FP16)** | 14.13     | 13.09     | 12.30     | 11.00     |\n    | MinMax              | 17011.67  | 44.89     | 239.02    | 26.91     |\n    | Ours                | **15.48** | **13.89** | **12.90** | **11.24** |\n    | **PIQA (FP16)**     | 75.79     | 77.58     | 78.73     | 79.71     |\n    | MinMax              | 53.21     | 67.63     | 57.67     | 64.80     |\n    | Ours                | **74.16** | **76.88** | **77.69** | **79.82** |\n\n    | 512 group size      | OPT-13B   | OPT-30B   | OPT-66B   | OPT-175B  |\n    | ------------------- | --------- | --------- | --------- | --------- |\n    | **WikiText (FP16)** | 14.13     | 13.09     | 12.30     | 11.00     |\n    | MinMax              | 1149.70   | 19.52     | 18.57     | 15.27     |\n    | Ours                | **14.50** | **13.43** | **12.66** | **11.18** |\n    | **PIQA (FP16)**     | 75.79     | 77.58     | 78.73     | 79.71     |\n    | MinMax              | 62.24     | 72.74     | 74.59     | 75.75     |\n    | Ours                | **74.96** | **77.20** | **77.97** | **79.71** |\n \n   It can be seen that ours can achieve near-floating point performance even with large  group size, and can surpass MinMax by a large margin. Especially, it can be observed that the number of groups can be decreased with our methods to reach satisfying results.  We will include more 4-bit quantization experiments in the revision.\n\n\n   [1]. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR2023.\n   \n   [2]. ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation.\n\n* **Q6**: Typos Grammar Style And Presentation Improvements.\n \n  **A6**: Thanks for pointing these out. We will fix the color and baseline parts in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation of the proposed approach is sound.\n2. The paper clearly discusses the rationale behind the outlier shifting and scaling, as well as the parameter calculation design, and provides easily understandable detailed experiments for its design choices.\n3. Without outlier shifting and scaling, Tables 2 and 5 show the performance drop of previous state-of-the-art methods at lower bit levels, such as INT6. Moreover, using OS+ enables LLMs to achieve almost no performance loss in zero-shot QA tasks.", "reasons_to_reject": "1. While the authors claim that z and s can be determined with only a small number of samples, it is difficult not to question whether these few samples can provide sufficient diversity and completeness for the distribution boundary.\n2. Although the results in Section 4 are quite good, QA and text classification tasks inherently have a higher tolerance for errors. It is uncertain how well the method would perform on tasks that are more sensitive to parameter changes, such as language modeling tasks.\n\n", "questions_to_authors": "**Question 1**: Is there any evidence to suggest that this outlier asymmetric distribution is static and can be estimated from a small number of samples? You could consider searching for z and s within the in-distribution dataset and then directly testing on the out-of-distribution datasets.\n\n**Question 2**: Have you tested the method on tasks that are more sensitive to parameter changes, such as language modeling tasks like Wikitext?\n\n**Question 3**: Have you tested more aggressive bits like 4-bit or 5-bit on OPTs or BLOOMZs?\n", "paper_topic_and_main_contributions": "This paper presents a method for normalizing outlier channels to mitigate the widespread outlier phenomenon in LLMs. The shift amplitude is determined by the minimum and maximum values in the collaboration datasets, while the scale value is obtained through module-level MSE loss search. The authors tested three model families: BERTs, OPTs, and BLOOMs, evaluating their performance in text classification tasks and zero-shot QA tasks using three different bit levels: INT8, INT6, and INT4. The experimental results show a significant performance improvement over the state-of-the-art LLMs quantitative methods at the INT6 bit level, and the performance loss compared to FP16 is relatively small.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In Figure 2, the colors of the element-wise multiplication and matrix multiplication modules are quite similar, making them difficult to distinguish when printed. \n2. The line spacing in the baselines parts(#407, #433) is too small and does not conform to the ACL style guidelines."}}, {"review_id": "huBObWD2K6", "reviewer": "Reviewer_vshF", "summary": "", "questions": "It is necessary to tell this method from SmoothQuant and AWQ in detail.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments and feedback. Hope the following reply helps address the concerns.\n\n* **Q1** : The idea is similar to that of SmoothQuant and AWQ. Of course, these ideas could have been developed independently, but it's clear that SmoothQuant precedes it.\n\n  **A1**: Our framework is inspired by the idea from Outlier Suppression [1], which was the first to introduce activation scaling to alleviate outlier quantization in language models. As we have stated the difference with it in the paper clearly, we’d like to highlight our differences compared to SmoothQuant and AWQ in the following: \n\n    * Compared to SmoothQuant: First, we identify a new feature of outliers that they stay in **asymmetric shape across channels**. To eliminate such a phenomenon, we propose the shifting operation to align channels for better per-tensor quantization with a migration pipeline for FP equivalence. The finding and technique did not appear in SmoothQuant.\n\n      Second, for the outlier concentration phenomenon, we propose a different and better solution, which can provide large improvements such as a **17.4% performance boost** on Winogrande. Considering scaling values act in both the activation and weights, we notice that reducing quantization error of individual tensors is sub-optimal and propose to optimize towards their interactive output change. Our scaling operation uses a fast and stable search pipeline to quantitatively evaluate the objective. However, SmoothQuant adopts a heuristic way which equalizes ranges between activation and weights and can not guarantee the minimum output change.\n\n   * Compared to AWQ: First, we completed our submitted paper in **April**, while AWQ was made public in **June**, precluding any opportunity for us to integrate ideas from it. Second, our work and that of AWQ address different problems and employ distinct methods. We concentrate on the challenging activation quantization, whereas they focus on weight-only quantization. Additionally, our introduction of the shifting operation to tackle asymmetric outliers across channels is unique and not present in theirs. Given our different problems, our scaling values do not take the same objective (Eq. (7)) as theirs and employ a different search method (Eq. (8)).\n\n\n   [1]. Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models. NeurIPS2022.\n\n* **Q2**: It is necessary to tell this method from SmoothQuant and AWQ in detail.\n\n  **A2**: Thanks for your suggestion. We will discuss differences between SmoothQuant, AWQ and us clearly in the revision to avoid any potential misunderstanding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The simple methods can suppress outliers in activations and successfully quantize LLMs without degrading accuracy. The operations can almost be absorbed into the existing architectures so as not to burden the computational complexity. The idea is similar to SmoothQuant, but this method outperforms it in per-channel INT6 quantization. The quantization algorithm is efficient in that the computation of shifting and scaling values takes only 20 minutes for 175B LLMs, and the real latency shows a 1.5x speed-up compared to FP16.", "reasons_to_reject": "The idea is similar to that of SmoothQuant and AWQ. Of course, these ideas could have been developed independently, but it's clear that SmoothQuant precedes it.", "questions_to_authors": "It is necessary to tell this method from SmoothQuant and AWQ in detail.", "paper_topic_and_main_contributions": "In this study, an efficient method to suppress outliers in the activation of large transformer language models for Quantization is proposed : Channel-wise shifting and scaling operations. These techniques can eliminate assymmetry in the distribution of activations in each channel and scale down the channels that have outliers. Moreover, the operations can be almost absorbed into the existing architectures by a simple correction of the parameters, so as not to burden the computational complexity. They also show the experimental results on BERT and LLMs (OPTs, BLOOM, BLOOMz with sizes ranging from 13B to 176B), the accuracy degradation of the proposed method is comparable to the recent similar approach, SmoothQuant in per-tensor INT8 quantization, and better than that in per-channel INT6 quantization.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gPDtehBpO3", "reviewer": "Reviewer_rLws", "summary": "", "questions": "A. What is the core novelty of the proposed channel-wise shifting and channel-wise scaling?\nB. In Table 2, is INT8* indicate per-tensor quantization while INT8 represents per-token quantization? Why does applying per-tensor quantization for weights (which is a coarser granularity compared to per-token) even achieve better performance with the proposed method?\nC. Can this method apply to LLaMa? ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank the reviewer for providing insightful suggestions on this paper. The detailed response is listed below. We hope our reply can address the questions.\n* **Q1** : The novelty of the proposed method (especially channel-wise shifting and channel-wise scaling) are questionable. The channel-wise shifting is to some extent similar to a standard normalization and is widely used in previous symmetry quantization methods. To balance the burden between weight and activation quantization is also a core contribution of the previous method SmoothQuant[1]. More discussion should be included in the revision.\n\n  **A1**: Thanks for the comment. We’d like to clarify our novelty in the following and will include more discussion in the revision.\n\n    * Channel-wise shifting: Our shifting operation aims to improve per-tensor activation quantization rather than symmetric quantization. We observed that outliers tend to be either extremely positive or negative within different channels. To address this, we align channel centers to reduce the entire tensor range to its maximum channel range. This approach is distinct from standard normalization, which, to our best knowledge, refers to weight standardization for regularization or distribution normalization of weights only during quantization-aware training. Additionally, our designed shifting contains a migration pattern to ensure FP equivalence in a post-training setting.\n\n    * Channel-wise scaling: Inspired by Outlier Suppression [1] which was the first to propose activation scaling in outlier quantization, we share the same goal with SmoothQuant [2] in addressing outlier concentration. However, our method distinguishes them by proposing a more rational approach to calculating scaling values, resulting in significant performance gains, such as a 17.4% improvement on Winogrande compared to [2]. Unlike [1] which reduces quantization error for individual tensors or [2] which equalizes weight and activation ranges, our method proposes to optimize their interactive outputs. We employ a quantitative method with a fast search process to minimize output changes.\n\n  [1]. Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models. NeurIPS2022\n\n  [2]. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, ICML 2023\n\n* **Q2**: The performance improvement of INT8 quantization on zero-shot tasks is limited (compared to SmoothQuant)\n\n  **A2**: For 8-bit quantization on OPT, both our method and SmoothQuant approach the FP16 baseline. However, it should be highlighted that our results on INT8 BLOOM close **60%** of the gap between FP16 baseline and SmoothQuant INT8 (e.g., SmoothQuant: 67.4, Ours: 67.9, FP16: 68.2). In our view, the results are still significant. \t\n\n  On the more challenging 6-bit quantization task, our method exhibits even more pronounced improvements, with a 28.3% enhancement on the StoryCloze task and a 27.4% boost on PIQA.\n\n\n* **Q3**: Figure 5 is not a straightforward visualization as the storage has been the x-axis, it is redundant to draw some circles/rectangles to use area to represent it.\n\n  **A3**: Thanks for your suggestion. In the revision, we will use different marks with consistent areas to represent different bit-widths.\n\n* **Q4**: A. What is the core novelty of the proposed channel-wise shifting and channel-wise scaling?\n\n  **A4**: We conclude our contribution in the novelty aspect in the following:\n\n    * The channel-wise shifting finds a new characteristic of outliers – their asymmetry across channels, which can significantly expand the tensor range, even with a small channel range. To mitigate this, we introduce the shifting operation to enhance per-tensor quantization while preserving FP equivalence through a migration pattern.\n\n    * The channel-wise scaling addresses outlier concentration with an innovative approach for scaling values. Instead of reducing quantization errors for individual tensors or using heuristic methods, it proposes to minimize interactive output changes of weights and activations. This optimization is achieved through a fast and stable search process, resulting in significant improvements, such as a 17.4% gain compared to SmoothQuant.\n \n* **Q5**: In Table 2, is INT8* indicate per-tensor quantization while INT8 represents per-token quantization? Why does applying per-tensor quantization for weights (which is a coarser granularity compared to per-token) even achieve better performance with the proposed method?\n\n  **A5** : INT8* indicates per-tensor quantization and INT8 only changes the weight quantization to the per-channel scheme. Note that activation always takes the per-tensor quantization but both per-tensor and per-channel schemes are common for weights.\n\n  For the better performance brought by coarser granularity, we guess the reviewer refers to cases like PIQA on OPT-13B, where INT8* achieves 76.4% accuracy, compared to 75.9% for INT8. We think this is because they have been close to the FP16 baseline at 75.8, and the PIQA datasets with limited size might make it challenging to distinguish these subtle differences among good enough results.\n\n  However, the last row in Table 2 shows the averaged results across tasks. It can be seen that per-tensor quantization only surpasses per-channel (weight) quantization by 0.1% on OPT-30B. On others, INT8 performs equally or better than INT8*.\n\n* **Q6**: Can this method apply to LLaMa?\n\n  **A6**: Our method is applicable to LLaMA, where we also observe outliers concentrated in specific channels (e.g., the 3968 and 3180 channels in one layer) and asymmetric shape across channels (e.g., (1.0, 17.7) range on the 3968 channel and (-27.3, -19.6) on 3180). Therefore, our methods effectively suppress these outliers.\n\n   In the table below, we conduct experiments on LLaMa-7B and we will include more complete results in the revision. For 8-bit and 6-bit quantization, our methods can bring close results to FP16. Especially, on 6-bit, the scaling operation can even improve 20 PPL and the shifting operation further brings a 2~3% boost on PIQA and Winogrande by aligning distribution across channels.  \n\n    | Method             | bit-width | PIQA (acc) | WikiText (PPL) | Winogrande (acc) |\n    | ------------------ | --------- | ---------- | -------------- | ---------------- |\n    |                    | FP16      | 78.29      | 9.45           | 66.93            |\n    | MinMax             | 8-bit     | 76.88      | 10.50          | 63.77            |\n    | shifting + scaling | 8-bit     | **77.80**  | **9.47**       | **67.01**        |\n    | MinMax             | 6-bit     | 67.46      | 31.00          | 55.94            |\n    | scaling            | 6-bit     | 75.73      | 10.88          | 62.04            |\n    | shifting + scaling | 6-bit     | **77.26**  | **10.01**      | **65.75**        |\n\n* **Q7**: Format: The submission has exceeded the page limits of most 8 pages.\n \n  **A7**: Thanks for the comment. We find that the limitation section is not included in the page limit. We will put the whole limitation section on page 9 for better presentation in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation of this work is clear and reasonable. Some insights on eliminating the outliers are given and are helpful for future research on LLMs.\n2. The experiment design is comprehensive, convering both lightweight LMs and LLMs. \n3. Real latency on hardware is provided, strengthening the results' credibility.\n4. The presentation quality of this paper is good, and the content is easy-to-follow.", "reasons_to_reject": "1. The novelty of the proposed method (especially channel-wise shifting and channel-wise scaling) are questionable. The channel-wise shifting is to some extent similar to a standard normalization and is widely used in previous symmetry quantization methods. To balance the burden between weight and activation quantization is also a core contribution of the previous method SmoothQuant[1]. More discussion should be included in the revision.\n\n2. The performance improvement of INT8 quantization on zero-shot tasks is limited (compared to SmoothQuant)\n\n3. Figure 5 is not a straightforward visualization as the storage has been the x-axis, it is redundant to draw some circles/rectangles to use area to represent it.\n\n[1] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, ICML 2023", "questions_to_authors": "A. What is the core novelty of the proposed channel-wise shifting and channel-wise scaling?\nB. In Table 2, is INT8* indicate per-tensor quantization while INT8 represents per-token quantization? Why does applying per-tensor quantization for weights (which is a coarser granularity compared to per-token) even achieve better performance with the proposed method?\nC. Can this method apply to LLaMa? ", "paper_topic_and_main_contributions": "This work proposes a quantization framework Outlier Suppression+ for LLMs. The target is to apply this framework to improve the efficiency of LLMs on various hardware systems with minimum computation overhead. The main contribution of this paper is first to analyze the outlier characteristics of LLMs. The channel-wise shifting and scaling operations are utilized to suppress the outliers. Outliers are further scaled down by balancing the burden between weight and activation quantization. The method is evaluated across different models (BERT, OPTs, BLOOM) and various tasks (GLUE, PIQA, HellaSwag, etc.) to demonstrate effectiveness and efficiency.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "Format: The submission has exceeded the page limits of most 8 pages.\n", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}, {"review_id": "ygMOh4PwI1", "reviewer": "Reviewer_NJj5", "summary": "", "questions": "- As you mentioned in section 3.2.2 and 4.5, the scaling factors learning process is unstable. The performance is poor if normal channels are scaled up.  Is it possible to modify the optimization objectives to improve the general robustness of the method?\n- Can you provide more details on the implementation of 8-bit model inference in section 4.4, which demonstrates realworld acceleration when compared to FP16 models?\n- How can the method  be used together with per-token quantization? The current method may have difficulty dealing with scale variance between tokens.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the valuable suggestions and thoughtful insight on this paper. The detailed response is listed below. Hope our reply can address the concerns.\n* **Q1**: The proposed method is like an extension of AWQ. However AWQ has not been even mentioned in the work.\n\n  **A1**: We’ll cite AWQ and discuss it in the revision. However, we’d like to first point out that we finished the submitted version of our paper in April while AWQ was made public in June. Thus, we did not incorporate any ideas from AWQ.\n\n  Furthermore, in our comparison with AWQ, there are key distinctions. First, we address different challenges: we focus on activation quantization, while they tackle weight-only quantization. Second, there are some differences between our designs. Our channel-wise shifting approach, aimed at addressing asymmetric outliers across channels, is unique and not present in AWQ. Similarly, for channel-wise scaling, our objectives (Eq.(7)) and the methods for searching scaling values (Eq.(8)) differ from theirs due to our distinct goals.\n\n* **A2**: The generalization of the method needs to be expanded. For example, the LayerNorm operations in llama-like models don't have beta parameters.\n\n  **Q2**: Thanks for your suggestion. We've observed that LLaMA models exhibit outliers with concentration and asymmetry, similar to OPTs. We've devised two ways to apply our methods to LLaMA models, as shown in the table below for LLaMA-7B.\n\n    * First, in cases where beta parameters are absent, we can utilize the scaling operation alone. As shown in the second row, this approach can bring 8-bit results close to FP16 and even improves 6-bit performance by 20 PPL.\n    \n    * Second, another way is to modify the model by introducing beta and following bias parameters to support both the scaling and shifting operations. While this will increase computation overhead, we find it to be manageable. For instance, the pre-trained language model, InternLM, incorporates bias with LLaMA and introduces only a minor 5% latency increase on the 7B model and even less on larger models. Besides,  the shifting operation can enhance performance, especially on more challenging tasks like 6-bit quantization.\n\n\n    | Method             | bit-width | PIQA (acc) | WikiText (PPL) | Winogrande (acc) |\n    | ------------------ | --------- | ---------- | -------------- | ---------------- |\n    |                    | FP16      | 78.29      | 9.45           | 66.93            |\n    | MinMax             | 8-bit     | 76.88      | 10.50          | 63.77            |\n    | scaling \t\t\t\t\t|8-bit     | **78.35**  | **9.47**       | 66.93        |\n    | shifting + scaling | 8-bit     | 77.80  | **9.47**       | **67.01**        |\n    | MinMax             | 6-bit     | 67.46      | 31.00          | 55.94            |\n    | scaling            | 6-bit     | 75.73      | 10.88          | 62.04            |\n    | shifting + scaling | 6-bit     | **77.26**  | **10.01**      | **65.75**        |\n\n* **Q3**: As you mentioned in section 3.2.2 and 4.5, the scaling factors learning process is unstable. The performance is poor if normal channels are scaled up. Is it possible to modify the optimization objectives to improve the general robustness of the method?\n  \n  **A3**: Thanks for your suggestion. We'd like to first clarify that our method does not directly learn the scaling factors but is a way to modify the optimization objective to efficiently and stably obtain scaling values. Recognizing that normal values often vary across samples, our approach introduces an outlier threshold in Eq. (8) to represent scaling values. Then, we inject Eq. (8) into the objective Eq. (6) and Eq. (7). Optimization in such a way scales down outlier channels while keeping normal values intact. Second, other ways to modify the objective are also worth exploring like adding a penalty term of scaling values in the objective.\n\n* **Q4**: Can you provide more details on the implementation of 8-bit model inference in section 4.4, which demonstrates realworld acceleration when compared to FP16 models?\n\n  **A4**: Thanks for your suggestion. We will provide the entire procedure from exporting improved FP models with our method to latency testing using FasterTransformer. Here are the key points:\n\n    * For LLMs, our methods only update the weights and biases of FP models, ensuring compatibility with common frameworks. Specifically, we employ FasterTransformer to directly measure the latency of 8-bit quantization using our more quantization-friendly FP models. Importantly, this design does not introduce any additional computational burden during inference.\n\n   * For BERT models, in pursuit of FP equivalence in Post-LN structure, we implement structural modifications within FasterTransformer, which replaces the shortcut branch with channel-wise addition and multiplication.  With our enhanced FP model, we measure its latency, as illustrated in Figure 4. Notably, these structural changes result in a negligible increase in latency.\n\n\n* **Q5**: How can the method be used together with per-token quantization? The current method may have difficulty dealing with scale variance between tokens. \n\n  **A5**: Our methods focus on addressing outliers in the channel dimension and can be used in conjunction with per-token quantization. The shifting operation reduces the tensor range to the maximum channel range, while the scaling operation scales down outlier channels for all tokens. Based on our more quantization-friendly FP model, per-token quantization can then be applied to deal with token variance for better quantization results. \n\n  We conduct experiments on a harder task: 5-bit quantization, and show results on OPT-66B. It can be seen that ours and per-token quantization can help each other to pursue favorable results.\n\n  |                  | PIQA  | Winogrande |\n  | :--------------- | ----- | ---------- |\n  | FP16             | 78.73 | 68.90      |\n  | per-token        | 51.90 | 50.20      |\n  | ours             | 76.50  | 67.56       |\n  | per-token + ours | **78.07**  | **68.43**      |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The experiments are extensive .The paper evaluates the proposed method on different network sizes, tasks, and quantization settings and demonstrate that it significantly improves performance after quantization.\n- The performance is good.  The 8-bit and 6-bit cases can achieves near-float point performance, and the 4-bit case is also better than other methods.", "reasons_to_reject": "- The proposed method is like an extension of AWQ. However AWQ has not been even mentioned in the work.\n- The generalization of the method needs to be expanded. For example, the LayerNorm operations in llama-like models don't have beta parameters.", "questions_to_authors": "- As you mentioned in section 3.2.2 and 4.5, the scaling factors learning process is unstable. The performance is poor if normal channels are scaled up.  Is it possible to modify the optimization objectives to improve the general robustness of the method?\n- Can you provide more details on the implementation of 8-bit model inference in section 4.4, which demonstrates realworld acceleration when compared to FP16 models?\n- How can the method  be used together with per-token quantization? The current method may have difficulty dealing with scale variance between tokens.", "paper_topic_and_main_contributions": "This paper proposes a new approach to build more quantization-friendly models by suppressing the outliers . With the grid-search technique to determine the optimal shifting and scaling values, the authors eliminate asymmetricity and scale of problematic channels, so that they can incur minimal quantization performance degradation, especially when bitwidth goes below 8-bit. The work have also evaluated the method on various networks and quantization settings to prove the efficacy.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers almost unanimously recommend acceptance with strong soundness and moderate excitement. The experimental setup and results are rigorous, the method well-motivated, and the results are strong. I recommend acceptance to the main conference.
Decision: Accept-Main