QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models
Authors: Jing Liu
Ruihao Gong
Xiuying Wei
Zhiwei Dong
Jianfei Cai
Bohan Zhuang
Venue: ICLR 2024 poster (2024)
Keywords: Network Quantization
Large Language Models
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose QLLM, an accurate and efficient low-bitwidth PTQ method designed for LLMs.

Abstract: Large Language Models (LLMs) have demonstrated unparalleled efficacy in natural language processing. However, their high computational demands and memory overheads hinder their broad deployment. To address this, two quantization strategies emerge, including Quantization-Aware Training (QAT) and Post-Training Quantization (PTQ). For LLMs, the billions of parameters make the QAT impractical due to the prohibitive training cost and thus PTQ becomes more prevalent. In existing studies, activation outliers in particular channels are identified as the biggest challenge to PTQ accuracy. They propose to transform the magnitudes from activations to weights, which however offers limited alleviation or suffers from unstable gradients, resulting in a severe performance drop at low-bitwidth. In this paper, we propose QLLM, an accurate and efficient low-bitwidth PTQ method designed for LLMs. QLLM introduces an adaptive channel reassembly technique that reallocates the magnitude of outliers to other channels, thereby mitigating their impact on the quantization range. This is achieved by channel disassembly and channel assembly, which first breaks down the outlier channels into several sub-channels to ensure a more balanced distribution of activation magnitudes. Then similar channels are merged to maintain the original channel number for efficiency. Additionally, an adaptive strategy is designed to autonomously determine the optimal number of sub-channels for channel disassembly. To further compensate for the performance loss caused by quantization, we propose an efficient tuning method that only learns a small number of low-rank weights while freezing the pre-trained quantized model. After training, these low-rank parameters can be fused into the frozen weights without affecting inference. Extensive experiments on LLaMA-1 and LLaMA-2 show that QLLM is able to obtain accurate quantized models efficiently. For example, QLLM quantizes the 4-bit LLaMA-2-70B within 10 hours on a single A100-80G GPU, outperforming the previous state-of-the-art method by 7.89% on the average accuracy across five zero-shot tasks. Code is available at [ZIP Lab](https://github.com/ziplab/QLLM) and [ModelTC](https://github.com/ModelTC/QLLM).

Published as a conference paper at ICLR 2024
QLLM: ACCURATE AND EFFICIENT LOW-BITWIDTH QUANTIZATION FOR LARGE LANGUAGE MODELS
Jing Liu Ruihao Gong".,, Xiuying Wei2.4, Zhiwei Dong , Jianfei Cai', Bohan Zhuang*
1 ZIIP Lab, Monash University "Sense Research Beihang University
"School of Computer and Communication Sciences, EPFL
University of Science and Technology Beijing
ABSTRACT
Large Language Models (LLMs) have demonstrated unparalleled efficacy in natu- ral language processing. However, their high computational demands and memory overheads hinder their broad deployment. To address this, two quantization strate- Training (QAT) and Post-Training
gies emerge, including Quantization-Aaan Quantization (PTQ). For LLMs, the billions of parameters make the QAT imprac- tical due the prohibitive training cost and thus PTQ becomes more prevalent. In existing studies, activation outliers in particular channels are identified as the biggest challenge to PTQ accuracy. They propose to transform the magnitudes from activations to weights, which however offers limited alleviation or suffers from unstable gradients, resulting in severe performance drop at low-bitwidth. In this paper, we propose QLLM, an accurate and efficient bow-bitwidth PTQ method designed for LLMs. QLLM introduces an adaptive channel reassembly technique that reallocates the magnitude of outliers to other channels, thereby mitigating their impact on the quantization range. This is achieved by channel disassem- bly and channel assembly, which first breaks down the outlier channels into sev- eral sub-channels to ensure a more balanced distribution of activation magnitudes. Then similar channels are merged to maintain the original channel number for effi- ciency. Additionally, an adaptive strategy is designed to autonomously determine the optimal number of sub-channels for channel disassembly. To further com- pensate for the performance loss caused by quantization, we propose an efficient tuning method that only learns small number of low-rank weights while freezing the pre-trained quantized model. After training, these low-rank parameters can be fused into the frozen weights without affecting inference. Extensive experiments on LLaMA-1 and LLaMA-2 show that QLLM is able to obtain accurate quantized models efficiently. For example, QLLM quantizes the 4-bit LLaMA-2- 2-0B- within 10 hours on a single A A100-80G GPU, outperforming the previous state-of-the- method by 7.89% on the average accuracy across five zero-shot tasks. Code is available at 7IP Lab and ModelTCC.
INTRODUCTION
Recently, Large Language Models (LLMs) such as GPT-4 (OpenAI, 2023) and LLaMA (Touvron al., 2023a;b) have achieved unprecedented advancements in natural language processing (NLP). These models excel in a range of tasks, from advanced reasoning in code and mathematics to clas- sification and question answering. However, their extraordinary performance is accompanied by substantial computational demands and vast model sizes. For example, GPT-3 (Brown et al., 2020), the precursor to GPT-4, already contains a stunning 175 billion parameters, requiring a minimum of 325 GB of memory for storage in half-precision (FP16) format. This necessitates the use of at least x 80GB NVIDIA 100 or 8x48GB NVIDIA A40 GPUs during the inference phase. As a result, deploying these models to real-world applications poses significant challenges.
Work done during an internship at SenseTime Research. 1 "ooresponning author. Email: bohan www.gzuuunnnnl..cooco

---

Published as a conference paper at ICLR 2024
Maximum
Minumum
200
20-
Value
Value
Value
-200
-50
-20
-400
-100-
-40
-600
5000
10000 15000 20000
5000
10000 15000 20000
5000
10000 15000 20000
Channel
Channel ID
Channel ID
(a) Pre-trained model
(b) SmoothQuant
(c) QLLM
Figure I: An illustration of the channel wise maximum and minimum values for the input activations of linear layer in LLaMA-6SB for (a) original pre- pre-trained model (b) after SmoothQuant (Xiao al., 2023) and (c) after our channel reassembly.
In light of the aforementioned challenges, network quantization (Zhou et al., 2016) emerges as a compelling solution, which maps weights and/or activations to lower-bit representations, resulting in a much lower memory footprint and faster inference. Existing quantization methods for LLMs training (QAT) (Liu et al., 2023) and post- can be classified into two types: quantization-awaat Xiao et al., 2023). Although with promising training quantization (PTQ) (Wei et al., 2022b; 2023; performance, QAT suffers from unbearable training costs as it needs fine-tune the whole quan- tized model with quantization parameters using large amount of data, rendering it impractical for the efficient deployment of LLMs. This practical limitation has shifted the spotlight towards PTQ which only uses little data to tune the quantized weights. However, when it comes to extremely low-bitwidth quantization for LLMs, e.g., 4-bit weight and/or activation quantization, existing PTQ methods (Xiao al., 2023; Dettmers et al., 2022) suffer from significant performance degradation.
Recent studies (Dettmers et al., 2022: Xiao et al., 2023; Wei et al., 2023) have revealed a unique pattern in LLMs' activations that is they contain specific outlier channels with significantly large magnitudes. This renders existing quantization methods less effective, as the outliers amplify the quantization range of layer activations, causing the vast majority of normal activation values to be quantized imprecisely and consequently leading to notable performance degradation. This issue will worsen with the prevalent use of layer-wise or token-wise activation quantization, common practice for maximizing hardware efficiency. To tackle this challenge, recent studies (Xiao et al., 2023: Wei al., 2022b: 2023; Shao al., 2023) have focused on smoothing activation outliers by transitioning the magnitudes from activations to weights through mathematically equivalent transformation. Such a transformation can be learned using either gradient-free methods (Xiao et al., 2023; Wei et al., 2022b; 2023) or gradient-based methods (Shao et al., 2023). However, as shown in Figure 1, for exceedingly pronounced activation outliers (those 50 x larger than others), the former offers only limited alleviation while the latter suffers from unstable gradients. As result, both methods leads to significant performance degradation in low- bitwidth quantization. To compensate for the performance drop of quantization, widely adopted PTQ strategy (Wei et al., 2023; Shao et al., 2023; Yao et al., 2022) further proposes to tune the quantized LLM directly by minimizing the block-wise reconstruction error. In LLMs, the tuned block refers to the Attention- FFN module. However. considering the huge number of parameters in an LLM, this approach still requires substantial training overheads and demands a significant amount of GPU memory.
In this paper, we propose QLLM, an accurate and efficient low-bitwidth post-training quantization method tailored for LLMs. To handle the outlier issue, we introduce a gradient-free channel re- assembly technique that redistributes the large activation magnitude of the outlier channels across the channels. Specifically, we first disassemble the outlier channels into several sub-channels. By spreading the magnitude of outliers, it ensures a more uniform activation range across channels, facilitating a balanced and precise quantization and thus improving the performance of quantized LLMs. We then introduce channel assembly, which fuses similar channels together to maintain the original channel count. Moreover, given the varying outlier patterns across different layers and the existence of extreme outliers, propose an adaptive strategy determine the optimal number of disassembled channels for each layer, which is based on minimizing the reassembly error between the original output activations and the counterpart with the reassembled input activations.
To further improve the performance of the quantized LLMs, motivated by low-rank parameter- efficient fine-tuning paradigm LoRA (Hu al., 2022; Dettmers et al., 2023a), we further propose
2

---

Published as a conference paper at ICLR 2024
an efficient gradient- based error correction strategy that freezes the pre-trained model and intro- duces a small set of learnable low-rank weights into each layer of the LLM. Then, QLLM learns the low-rank weights by minimizing block-wise quantization error sequentially. Owing to the re- duced number of trainable parameters, both the training time and GPU memory requirements are significantly reduced. Such efficiency gain enables us perform multi-block reconstruction that simultancously reconstructs a collection of consecutive Attention-FFN blocks, further mitigating the quantization error accumulation during propagation in low-bit LLMs. Notably, after training, these learnable low-rank weights can be seamlessly merged with the frozen weights followed by quantization, thereby ensuring no additional computational burden during inference.
Our contributions can be summarized as follows: 1) We introduce a simple yet effective channel reassembly method to suppress activation outliers in L.MMs, which is accomplished by initially dis- assembling the outlier channels to make activations more quantizationfriendly and subsequently merging similar channels so as to preserve the original channel count for efficiency. We also pro- pose to determine the optimal number of disassembled channels for each layer, considering the diverse outlier patterns across layers and the presence of extreme outliers. The overall process is gradient-free and enjoys high efficiency. 2) An efficient eтror correction mechanism is proposed to further enhance the gradient-free channel reassembly. It leverages the learning of low-rank pa- rameters to counteract quantization error in a structured way, leading to substantial reduction in training time and GPU memory requirements without incurring any additional inference overhead. 3) Extensive experiments show the promising performance and training efficiency of QLLM. For example, QLLM quantizes 4-bit LLaMA-2-705 within 10 hours, and outperforms previous SOTA methods by 7.89% on the average accuracy across five zero-shot tasks.
2 RELATED WORK
Network quantization (Zhou et al., 2016) which represents the weights, activations, and even gradients with low precision, is an effective method to reduce the model size and computational burden. Existing techniques fall into two primary categories: quantization- awwae training (QAT) (Esser et al., 2020; Kim et al., 2021; Li et al., 2022) and post-training quantization (PTQ) (Nagel et al., 2020; Li et al., 2021; Wei et al., 2022a). QAT incorporates the quantization process directly into the training phase and jointly learning the quantizer well model parame- ters (Zhang et al., 2018; Jung et al., 2019; Choi et al., 2019; Bhalgat et al., 2020; Esser et al., 2020; Liu al., 2022) with the help of straight-through estimator (STE) (Bengio al., 2013), which greatly mitigates the accuracy degradation caused by compression. However, the training cost of QAT can be prohibitively high, primarily because it requires fine-tuning the quantized model on the original training dataset of the pre-trained model. PTQ offers a less resource- resource-intensiv alternative, allowing models be quantized after being fully trained with only a small amount of data. To reduce the performance drop, several methods have been proposed to perform layer- wise (Nagel et al., 2019; 2020; Wu et al., 2020; Hubara et al., 2020; Li al., 2021) or even block-wise calibration (Li al., 2021). Further innovations delve into outlier mitigation, adopting strategies like clipping (Banner et al., 2019; McKinstry et al., 2019; Choukroun et al., 2019) or value splitting (Zhao et al., 2019) for weights and activations to improve the precision by allocating more bits to the intermediate values. However, for LLMs, a recent study (Liu et al., 2023) has found that MinMax quantization, which maintains the full value range, performs better than clipping-based methods, as outliers are critical to the performance. Different from these methods, our QLLM targets quantization for LLMs.
Network quantization.
Quantization LLMs. Given constraints such limited training data and intensive computa- tional demands, prevailing quantization techniques for LLMS are primarily based on PTQ. Existing LLM quantization approaches can be classified into two categories: weight-only quantization (Fran- tar et al., 2022; Park et al., 2023; Lin et al., 2023; Dettmers et al., 2023b; Chai et al., 2023; Cheng et al., 2023; Dettmers et al., 2023a; Kim et al., 2023; Chee et al., 2023: Lee al., 2023) and weight- activation quantization (Dettmers et al., 2022; Xiao et al., 2023; Wei et al., 2022b; 2023; Yao et al., 2022; 2023; Yuan et al., 2023; Liu et al., 2023; Wu et al., 2023). The former focuses on compressing the vast number of weights in LLMss to reduce the memory footprint, while the latter compresses both weights and activations into low-bit values, aiming to accelerate computation-intensii trix multiplication. To handle the different value ranges of weight matrices, recent studies have delved into more fine- mne-griined quantization, such as channel-wise quantization (Frantar et al., 2022) or group-wist quantization (Frantar et al., 2022; Lin et al., 2023). To further compensate for the performance drop for extremely low-bitwidth quantization, QLoRA (Dettmers et al., 2023a), and INT2.1 (Chai et al., 2023) introduce additional full-precision weights (Yao et al., 2023). While our
ma-
3

---

Published as a conference paper at ICLR 2024
method also presents small set of low-rank weights, it stands apart from QLoRA and INT2.1 as our learnable parameters can be reparameterized into pretrained weights followed by quantization. Recent research (Dettmers et al., 2022) has shown that activation outliers exist in some feature di- mensions across different tokens. Several works (Wei et al., 2022h; 2023; Xiao et al., 2023; Shao 2023) have been proposed to migrate the quantization difficulty from activations to weights within the same channel, based on gradient-free methods (Wei et al., 2022b; Xiao et al., 2023; Wei al., 2023) or gradient-based methods (Shao et al., 2023). However, when dealing with very pro- nounced activation outliers, the existing methods often show limited improvement or incur unstable gradients. In a notable difference, our proposed QLLMs method efficiently redistributes the large activation magnitudes of outlier channels among all channels, offering distinctive approach com- pared to these existing methods.
PRELIMINARIES
Basic notations. In this paper, matrix is marked as X and vector is denoted by x. The LLMs usually have two core parts: multi-head self-attention (MSA) layers and feed-forward network (FFN) layers, which are mainly composed of linear layers. Here, we give the formulation of linear layers the output channel k:
V&
M
x Wik: ik:
where E R refers input, W M denotes the weight, and y R stands for the output. In this way, the numbers of input and output channels are M and N, respectively.
Quantization. We adopt uniform quantization for both weights and activations because of its hardware-friendly nature (Jacob et al., 2018). For matrix with floating-poin values such as FP16 FP32, the b-bit quantization quantizes it in the following way:
X, = quant(X) = clamp (1 + 3,0, - 1),, where a max(X) (Xmin())
3
min (X)
2
2)
where the function clamp(v, Umin, Umax) clips any value into the range of [Vmin Unaxx] and |-] is rounding operator that returns the nearest integer of given value. Here, denotes the scaling factor and (3 represents the zero-point value.
Recent studies (Dettmers al., 2022; Xiao et al., 2023; Wei al., 2022b) point out that there extremely large outliers in certain channels of activations in LLMs, which makes the quantization challenging balance the accurate representation for large values and small numbers. To tackle this problem, some approaches (Bondarenko et al., 2021; Yuan et al., 2023) adopt fine-grained quan- tization scheme, which assigns different quantization parameters for different channels. However, such a way needs delicate kernel design and clearly increases computation overhead for inference. Also, some works Wei et al. (2022b); Xiao al. (2023) propose to use channel-wise scaling between activation and weights, which still remains outliers under extreme cases, shown in Figure 1.
4 PROPOSED METHOD
In this section, we propose the adaptive channel reassembly framework to redistribute input acti- vation outliers across multiple channels. The framework consists of three components: channel disassembly for decomposing the outlier channel, channel assembly for balancing the efficiency, and an adaptive strategy to find the suitable reassembly ratio for each layer. The channel reassem- bly technique is gradient-frre and efficient to implement. What's more, it can be equipped with a gradient-based and well-designed error correction module for further enhancement.
4.1
ADAPTIVE CHANNEL REASSEMBLY
4.1.1 CHANNEL DISASSEMBLY
In this part, we introduce channel disassembly to decompose the input outlier channels into several sub-channels, which can reduce the outlier magnitude and make the activations more quantization--rient friendlly without altering the layer output.
Considering that outliers tend to be concentrated in specific channels across various inputs and the desire preserve their information during quantization, we propose break down these outlier channels into several sub-channels to redistribute their large values. Without loss of generality, by assuming the M-th channel as the outlier channel, we can disassemble it into *y and replicate this
4

---

Published as a conference paper at ICLR 2024
channel T times, reducing the outlier magnitude by a factor of T. Simultaneously, is also natural to duplicate the corresponding weight channel times, enabling us to maintain the equivalent output:
Yk = Σm x, Wik +
XM WM +
XM W
Wik- Mk-
T
(3)
times
The equation above produces the same output with the original linear layer equation in Eq. (1) and introduces additional T channels for both the input and the weight.
Taking into account that the quantization range impacts accuracy, we introduce an outlier threshold, denoted as 0, to identify the outlier channels and determine the number of sub-channels together, with 7 = maxx ( 1))0 This approach ensures that channels with values smaller than 0 remain unchanged with T while the magnitude of outliers are divided by T.
Our channel disassembly method allows us to retain outlier information with an equivalent output and ease the quantization difficulty with much smaller value range. Its only drawback is the increase in the number of channels, which may lead to additional computational costs and will be addressed in the next subsection.
4.1.2 CHANNEL ASSEMBLY
Note that the input channel count increases to M substantial quantity of channels in LLMs, it is possible to omit some unimportant channels merge similar input channels puts. To achieve this, a straightforward method is to use channel pruning (Ma et al., 2023; Sun et al., 2023) that removes the unimportant channels directly. However, such method may result in substan- tial information loss, especially when T is large. Motivated by recent studies (Bolya et al., 2023; Bolya & Hoffman, 2023) that combine similar tokens, we propose a channel assembly method that delves into merging T 1 similar input channels. Given channels and in alignment with token merging techniques (Bolya et al., 2023; Bolya & Hoffman, 2023), our goal is to aggregate them by calculating the average of their input features, denoted as x,tx, and utilizing the aggregated feature in subsequent computations, which defined
- T' after channel disassembly. Given the
keep the original channel count M for efficiency while maintaining out-
x, WiR + W, X. (Wa + W
(4)
where W + W represents the merged weight. With the aim of minimizing the information loss of channel assembly in Eq. (4), we can define distance metric D(i,j) between channels and j as
D(i,)) = x; (.: Wak)
x, (
W
(5)
where ll represents the l norm. The above distance metric takes into account the difference in both input activations and weights between the two channels.
With the channel distance defined, the next step is to determine which channels to aggregate effi- ciently, with the goal of redueing the total channel count by T - 1. To address this, propose using bipartite soft matching (Bolya et al., 2023; Bolya & Hoffman, 2023) that first partitions the channels into two sets, each containing roughly equal sizes, and subsequently finds the 7' most similar pairs between these two sets (see Appendix A for details). Note that we do not assemble the channels that are disassembled from the outlier channels since they play a critical role in the performance of LLMs. After the channel reassembly, including both disassembly and assembly, we acquire the reassembled input activations that are more amenable to quantization, along with the corresponding reassembled weights for layer!.
4.1.3 ADAPTIVE REASSEMBLY
In this section, we present a method to adaptively determine the appropriate reassembly ratio for each layer. For channel disassembly, selecting high value for with small 0 substantially reduces outlier magnitudes and benefits quantization, while resulting in a larger increase in channel merging error due to a higher merging ratio. Conversely, choosing a small with a large will not increase the channel count much, making it easier for the assembly stage to keep the information while likely still retaining outliers, causing significant guantization errors. Therefore, is crucial to carefully determine the outlier threshold 0 or the reassembly channel number T.
5

---

Published as a conference paper at ICLR 2024
However, it is hard to choose 0 in practice distinct layers have different patterns of outliers, as shown in Figure D. Motivated by (Wei et al., 2023), we propose an adaptive strategy to find the optimal 9 by minimizing the reassembly eπror between the original output activations and their counterparts generated with the reassembled input activations for each layer.
Note that our channel reassembly technique can yield the reassembled activation X E RLXM with a sequence length of L, which can then be fed into MSA layer or FFN layer. For ex- ample, let us consider a case where X is fed into MSA layer. A standard MSA layer calcu- lates queries, keys and values with three learnable projection matrices Wq, Wx, Wy E RMXN as Q = xWq, K = XWK,, = XWv. where X E RLXM represents the original input activation. Let Wq. Q.. WK. Wy be the reassembled projection weights. In this way, the constructed queries, keys, and values can be formulated as Q quant (uuant(t)uuuuuuu (W).) (mant(X)quantii) W k). = quaan(X)quant(( (X)unt(()uuuuuu We then find 0 by solving the problem as arg ming [Sottaxx(QB) QK V - Softmax QKTT
(6)
where ||:||p denotes the Frobenius norm. To solve problem (6) efficiently, we use grid search fol- lowing (Choukroun et al., 2019; Wei et al., 2023) (see Algorithm Appendix for details).
4.2 EFFICIENT GRADIENT-BASEI ERROR CORRECTION
Based on the above gradient-free adaptive channel reassembly, efficient gradient-based error correction technique is further proposed for improving the performance of the quantized LLMs using a small set of calibration data.
Inspired by recent developments in parameter-efficiint fine-tuning methods (Hu et al., 2022; Dettmers et al., 2023a), the efficient error correction introduces two low-rank parameters A E &R and B E R with a rank of r into each projection layer of our QLLM. Then, we can obtain the output Y of quantized linear layer by Y quant Jquant (W) + quant AB. Instead of directly tuning the quantized weights, we learn the introduced low-rank parameters by minimiz- ing the reconstruction error between the original and the quantized outputs of the Attention-FFN block. Thanks to the reduced number of trainable parameters, both the optimization cost and GPU memory usage can be significantly reduced, Such efficiency gain allows us further suppress the accumulation of quantization error during forward propagation via structured reconstruction, i.e., performing multi-block reconstruction for QLLM, which simultaneously adjusts collection of con- secutive Attention blocks by focusing reconstructing the final block output.
After the reconstruction, we only need to store the quantized weight quant (W (W + AB), which does not introduce extra inference costs. Note that it is inevitable that the absorption process will introduce additional quantization errors. To counteract this, following (He et al., 2017; Nagel et al., 2020: Hubara et al., 2020), perform reconstruction sequentially rather than in parallel, which enables us to account for the quantization error stemming from the previous layers.
4.3 EFFICIENCY DISCUSSION
Reassembly efficiency. Our adaptive channel reassembly stands out for its efficiency, mainly at- tributed to its gradient-free nature, which excludes the need for backward propagation. The main source of computational expense of our method comes from the channel assembly, which requires the calculation of pairwise distances. Fortunately, the utilization of efficient bipartite soft match- ing eliminates the need to compute distances for every pair of channels, enhancing the efficiency. For the gradient-basse eтror correction, the reduced number of parameters significantly lowers its optimization cost, rendering more efficient than directly adjusting the quantized weights.
Inference efficiency. The inference overhead of channel disassembly and assembly is small for two reasons. 1) recent studies (Xiao et al., 2023; Wei et al., 2023) have revealed that activation outliers are often concentrated in specific channels across various inputs. This property is also reflected in similar channels for assembly as well. Therefore, we are able to pre-calculate the channel indices for disassembly and assembly using small number of calibration data, significantly reducing run- time overhead. 2) Both channel disassembly and assembly can be implemented efficiently the previous layer - 1 is linear layer. Please refer to Appendix B for more details. In cases where the preceding layer I - is a non-linenn lon-linear layer, such as a layer normalization (Ba et al., 2016), we introduce additional disassembly and assembly layers that are designed to decompose and aggregate

---

Published as a conference paper at ICLR 2024
Table 1: Performance comparisons of different methods for weights and activations quantization on LLaMA-1 model family. PPL denotes the perplexity.
PPL
Accuncy (1)
Model
Method
WikiTiou2
Avg.
PIQA
ARC-c
ARC-c
HellsSwag
Winngradd:
W16116
5.68
7.08
6.38
77.37
52.48
41.38
72.99
66.93
62.25
W646
SQ
6115
7.61
6.53
76.65
53.11
71.52
61.38
W646
0S+
5.00
76.82
$1.35
71,42
W6A65
OmniQuant
5.96
7.43
6,70
77.09
51,89
40.87
7L.61
65.03
61.30
W646
QUIM
5.8
7.34
6.62
77,26
52.02
04
71,40
65.19
61.38
LLaMA 1-78
W4A8
QLIM
596
7.49
6.73
76.17
50.84
40.02
70:75
66.22
60:00
W4A4
SQ
52.85
104.35
78.00
49,50
30.40
25.80
27.00
46.00
36.28
W4A4
LLM QAT
51.50
27, .0
90
31.10
51,90
37.26
W444
LIM-QAT
55,00
43.34
W444
OS+
40.32
62.73
39.98
30.29
44.39
52.96
46.07
W444
OmniQuant
11.26
14.51
12.89
66.15
45.20
31,14
56,44
53.43
50.47
W444
QLLM
9.65
12.29
10.97
68.77
45.20
31.14
51.84
W1
5.09
6.1
5.85
79.05
50.00 19.84
44.62
76.22
70.00
65.96
W6A6
SQ
5.50
7.03
6.27
77.80
42.58
75.11
68.11
W646
OS+
5.37
78.29
56.90
43.09
75,09
64.52
W646
OmniQuuat
5.28
6.84
6.06
78
57.26
42.91
82
68,27
64.54
LLMMA-1-138
W646
QLIM
5.28
6.05
77.91
57.70
42.92
75.02
69.14
W4A3
QLLM
5.33
6.32
78.29
57.03
74.06
68.35
64.18
WHAT
3Q
79:35
T20.24
99.99 99.50
55.55
34.31
26.71
41.36
46770
41:41
W444
OS+
53.64
63.00
50.38
51.54
47.77
W444
OmniQuaat QLLM
10.87
13.78
12:33
69.09
47.39
W4A4
8.41
10.58
9.50
71.38
47.60
$4,30
63.70
59.43
55.28
W16A16
4.10
5.98
5.04
50,09
58.92
79.21
72.77
67.28
W646
SQ
5.37
77.14
57.61
42.91
78.07
65.13
W646
OS+
4.48
80.14
58.92
45.05
77.96
71.96
66.81
W646
Quniiquant
4.38
6.22
5.30
79.81
79.79
45.22
78.95
72.21
LLAMA-1-30B
W646
QLLM
4.30
6.17
5.24
79.65
58.08
44.11
78.38
75.24
WHAS
QLIM
4:40
6.22
5.31
70.11
57.87
44.62
78.03
72:23
66.37
W444
SQ
309.65
245.87
322.76
50.16
28:11
26.71
31.97
51.14
37.62
W444
US+
112.33
67.63
46.17
34.30
52.64
WHA1
OmniQQuant
10.33
12.49
11.41
71.21
49.45
34.47
64.65
59.19
55.79
W444
QUIM
8.37
11.51
9.94
73.83
50.67
40
67.91
58.56
57.87
W16A16
3,56
5.62
4.59
30.85
58.75
46.25
50,73
77.11
68.74
W646
SQ
4.00
6.08
5.04
77.97
$4.67
44.62
77.51
72.61
65:48
W6A6
OS+
79.67
55.
45.22
78.03
73.95
66.51
W646
OmniQuant
3,75
4.79
$1.01
46.35
79.91
75.69
LLaMA-1-65B
W6A6
QLIM
3.73
5.80
4.77
80.14
$7.79
45.05
79.74
74.59
67.46
W4A8
QLLM
3.78
8.82
80.14
58.59
46.42
79.71
74.66
SQ
112.02
T18.96
T15.39
61.67
40.15
55.06
46.19
50.83
4621
W4A4
OS+
32.60
68.05
43.98
35.32
50.73
$4,30
50.48
Ommiqquat
9.17
11.28
10.23
71.81
48.02
35.92
66.81
59.51
56.41
W4A4
QLLM
6.87
8.98
7.93
73.56
52.06
39.68
70,94
62.9
59.83
channels during runtime, with the channel indexes for decomposition and aggregation calculated of- fline using calibration data. The pseudo codes of channel disassembly and assembly during runtime can be found at Section D of supplementary material. Moreover, benefiting from our efficient kernel implemented by Triton (Tillet et al., 2019) and limited reassembly ratio searched by our adaptive strategy (See Figure C), the introduced inference cost is controlled within small level.
5 EXPERIMENTS
Models and datasets. We apply QLLM to quantize the LLaMA-1 (Touvron et al., 2023a) and LLaMA-2 (Touvron et al., 2023b) families. To evaluate the performance of the quantized LLM, we report the zero-shot accuracy on various benchmarks, including PIQA (Bisk et al., 2020), ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019). and WinoGrande (Sakaguchi et al., 2021). Additionally, we evaluate the perplexity, a key indicator of a model's generative performance that correlates significantly with zero-shot outcomes, on WikiText2 (Merity et al., 2017), PTB (Marcus et al., 1993) and C4 (Raffel et al., 2020).
Quantization settings. In alignment with prior research (Dettmers et al., 2022: Shao et al., 2023), we use per-channel weight quantization and per-token activation quantization. Following (Shao et al., 2023; Liu et al., 2023), we quantize all weights and intermediate activations, with the ex- ception of the Softmax output probability, which is maintained at full precision. Following Om- niQuant (Shao et al., 2023), we focus on 4- and 6-bit weights and activations quantization. Ad- ditionally, also explore 4-bit weights and 8-bit activations quantization, aiming for hardware-
7

---

Published as a conference paper at ICLR 2024
friendly configurations while maintaining high performance. We exclude 8-bit quantization as SmoothQuant (Xiao et al., 2023) able to achieve lossless performance.
Compared methods. We compare our QLLM with several state-of-the-art (SOTA) PTQ quantiza- tion methods, such as OmniQuant (Shao et al., 2023), SmoothQuant (SQ) (Xiao et al., 2023), Outlier Suppression+ (OS+) (Wei al., 2023) and recent QAT method LLM-QAT (Liu et al., 2023). For fair comparisons, we reproduce SmoothQuant and Outlier Suppression+ with per-channel weight quantization and per-token activation quantization.
Implementation details. Following OmniQuant (Shao et al., 2023), we construct the calibration set with 128 randomly sampled sequences from WikiText2, each with sequence length of 2048. QLLM begins by applying channel reassembly prior to all linear projection layers, excluding the attention output projection layer, followed by performing error correction on the resulting model. The rank r of the introduced low-rank parameters is set to 4, and these parameters are trained for 10 epochs with a mini-batch size of We carry out the reconstruction using 4 Attention-FFi blocks. AdamW (Loshchilov & Hutter, 2019) with linear learning rate decay scheduler is used following (Yao et al., 2022). The learning rate is set to 5 x 10 most experiments; for LLaMA- 2-70B, it is set to 1 x 10-4 All training experiments are conducted on single NVIDIA A100 80G GPU. We use the Language Model Evaluation Harness toolbox (Gao et al., 2021) for evaluation.
5.1 MAIN RESULTS
We report the results LLaMA-1 and LLaMA-2 families in Table 1, and Table A in Appendix. Note that W6A6 has limited hardware support in real-world applications. However, our QLLM still demonstrates performance benefits in these settings, consistently surpassing OmniQuant in terms of lower perplexity across all models on both Wiki'Text2 and C4 and achieving comparable accu- racy on zero-shot tasks. Remarkably, with W4A8 quantization, our method incurs only a minimal performance reduction. While the absolute performance gains with 6-bit quantization might seem modest, this is partly due to the less pronounced effect of activation outliers this bitwidth. When focusing on extremely www-bitwitth quantization (i.e., 4-bit), activation outliers serve as the perfor- mance bottleneck, thereby highlighting the importance of suppressing the outliers. In this case, our QLLM achieves significantly higher zero-shot accuracy and much lower perplexity than the con- tenders. For example, QLLM quantized 4-bit LLaMA-1-65B outperforms OmniQuant counterpart by average of 3.42% in accuracy across five zero-shot tasks. Remarkably, for LLaMA-7B, QLLM even surpasses the QAT method, LLM-QAT + SQ. by 8.6% on the average accuracy, which strongly demonstrates the efficacy of our QLLM.
Table 2: Perplexity results of different components in channel re- assembly. "CD" stands for channel disassembly. "CA" represents channel assembly. "CP" indicates channel pruning. "Adaptive" refers to the adaptive strategy. "y" is the channel expansion ratio.
Table 3: Inference through- put comparisons using 2048-token segment on RTX 3090 GPUs: 1x GPU for LLLMA-1-7B and 2x GPUs for LLaMA-1-133.
CD
CA
CP Adaptive
LLaMA-1-138
WikeTeat2
PTB
C4
Avg.
0.90
$39.59
343.45
344,13
Throughput (tokeensi))
Model
0.03
8.01
13.52
10.27
10.60
0,05
7.85
13.38
10.13
10.45
PP16
3252
WSA8
5676
0.01
8.68
15.16
11.12
11.65
LLAMA-I-7B
W4A16
5708
0,83
8.72
14.99
11.03
11.58
11.29
QULM
6385
0.07
9.39
15.98
11.84
12.40
FP16
1910
0.91
8.98
16.34
11.37
13.50
12.23
LLaMA-1-138
WHA16
WXAR
3179
0.07
0.05
11.23
9.60
21.61
18.11
19.79
13.4
13.70
17.54
QULMM
W444
3873
3730
11.12
5.2 ABLATION STUDIES
Effect of different components in channel reassembly. To show the effectiveness of the diverse components involved in channel reassembly, we apply different methods with our efficient error correction to yield 4-bit LLaMA-13B and show the results in Table 2. For channel disassembly, we determine 0 by exploring different channel expansion ratios y. We observe that our method with channel disassembly significantly surpasses the counterpart that does not utilize it. With the
8

---

Published as a conference paper at ICLR 2024
increasing expansion ratio 7, the performance of the quantized model be further improved. These results strongly show that channel disassembly is able to make activations more quantization- friendly by decomposing the outlier channels.
Furthermore, by incorporating channel assembly, our method manages to preserve the original chan- nel count with little performance drop. In comparison channel pruning, our channel assembly leads to lower information loss, thereby achieving much better performance, especially at higher y Rather than determining 0 using a predefined expansion ratio, our method, equipped with an adap- tive strategy, is capable of autonomously finding optimal O, resulting in near-lossless performance compared the approach utilizing only channel disassembly. The resulting expansion ratios for different layers are shown in Figure C of the Appendix.
Table 4: Comparisons between efficient error correction (EEC) and tuning quantized weights di- rectly (TQW) for 4-bit 1LaMA-1--65B "OOM" indicates out of memory.
#Amn-FFN Black
Method
WikiTes22
PTB
C4
Avg.
Training Time (GPU Hours)
GPU Memory (GB)
TQW
6.34
17.61
9.36
11.17
12:16
31.84
BEC
8.31
13.77
10.76
10.95
7.79
19:00
TQW
6.25
11.18
8.66
12.13
52.45
BEC
7.62
11:47
9.39
9.49
7.79
28,60
TQW
00M
BEC
6.87
11.36
8.98
9.07
7.77
47.71
Effect of efficient gradient-based error correction. After channel reassembly, implement our QLLM to produce 4-bit I. LLaMA-7B models with our efficient gradient- -aase error correction (EEC) and tuning quantized weights directly (TQW) outlined in Section 4.2 further improve the perfor- mance of quantized I.LMs and show the results in Table 4. Compared with TQW which tunes all quantized weights, EEC focuses on learning small set of low-rank weights, which significantly reduces training costs and GPU memory usage while delivering comparable performance. More- over, the reduced GPU memory demand allows EEC to quantize LLaMA-1-663 on single 24GB consumer- grade GPU, such as the NVIDIA RTX 4090, a task that is not feasible with TQW. Due to the page limited, we put more results in Section L of the supplementary material.
Inference efficiency. To assess the inference efficiency of our channel reassembly technique, we measure the inference speed of QIL on NVIDIA RTX 3090 GPUs We employ W4A4 kernels from QUIK (Ashkboos et al., 2023) codebase. We also conduct a comparative analysis using weight quantization only, utilizing CUDA kernels from AutoGPTQ' As shown in Table 3, our 4-bit QLLM only incurs 4% additional cost relative to W4A4 but achieves notable 1.96x speedup over FP16. Notably, our channel reassembly strategy substantially mitigates losses attributed to quantizing out- liers (see Table E), with only slight extra computational overhead. For the detailed inference cost of channel disassembly and assembly, please refer to Section N of the supplementary material.
6 CONCLUSION AND FUTURE WORK
In this paper, we have proposed an accurate and efficient post-training quantization approach for low-bit LLMs, dubbed QLLM. The core of our QLLM lies in a novel adaptive channel reassem- bly paradigm that effectively addresses activation outliers, a pivotal factor contributing to the per- formance bottleneck quantizing LLMs. The key idea involves reallocating outlier magnitudes to other channels, accomplished through process of channel disassembly followed by assembly. We have further proposed a quantization-awarr parameter-efffiiien fine-tuning strategy that lever- ages calibration data to compensate for the information loss resulting from quantization. Extensive experiments on LLaMA model series have demonstrated the promising performance and training efficiency of QLLM. In terms of limitations, our proposed channel reassembly involves introduc- ing additional operations to decompose and aggregate channels during runtime, thereby incurring additional inference costs. A potential solution improve inference efficiency to explore kernel fusing (Wang et al., 2010), aiming to fuse disassembly, assembly and layer normalization into a sin- gle operator. Another way is to aggregate more similar or unimportant channels (Sun et al., 2023) than those disassembled to achieve higher speedup.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We sincerely thank Shenghu Jiang for his help in implementing the efficient Triton kernel.
REFERENCES
Joshua Ainslie, James Lee-Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305. 13245, 2023.
Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, and Dan Alistarh. Towards end-to-end 4-bit inference on generative large language mod- els. arXiv preprint arXiv:2310.09255 2023.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1677.000 60450065555 2016.
Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for repid-deployment. NeurlPS, 32, 2019.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 0rXiv:1308.344: 2013.
Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. In CVPR, pp. 696-697, 2020.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In AAAI, volume 34, pp. 7432-7439, 2020.
Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. In CVPR, pp. 4598-4602, 2023.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In ICLR, 2023.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort, Understanding and overcoming the challenges of efficient transformer quantization. In EMNLP, pp. 7947-7969, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 13:1877-1901, 2020.
Yuji Chai, John Gkountouras, Glenn G Ko, David Brooks, and Gu- Wei. Int2. 1: Towards fine-tunable quantized large language models with error correction through low-rank adaptation. arXiv preprint arXiv:2306.088 08662 2023.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. Quip: 2-bit quantization of large language models with guarantees. arXiv preprint urXiv:2307.I /3304, 2023.
Wenhua Cheng, Weiwei Zhang, Haihao Shen, Yiyang Cai, Xin He, and Kaokao Lv. Opti- mize weight rounding via signed gradient descent for the quantization of 1lms. arXiv preprint arXiv:2309.05516. 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%, * chatgpt quality, March 2023. URL https:
/ //Imsys.ozrs
3011:/20022000----200000000000-3 30-vicuna//
Jungwook Choi, Swagath Venkataramani, Vijayalakshmi Viji Srinivasan, Kailash Gopalakrishnan, Zhuo Wang, and Pierce Chuang. Accurate and efficient 2-bit quantized neural networks. PMLR, 1:348-359, 2019.
10

---

Published as a conference paper at ICLR 2024
Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural net- works for efficient inference. In ICCVW, pp. 3009-3011 IEEE, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprini arXiv: 1803.0557, 2018.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 O: 8-bit matrix multiplication for transformers scale. NeurlPS, 35:30318 1--3332,, 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint
2023a.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk-
boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized repre- sentation for near-lossless Iim weight compression. arXiv preprint arXiv:2306.03077 2023b.
Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra S. Modha. Learned step size quantization. In ICLR, 2020.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization for generative pre-trained transformers. In ICLR, 2022.
Lco Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot lan- guage model evaluation, September 2021. URL https: / wttp//........... 5371628.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In ECCV, pp. 544-560, 2020.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In ICCV, pp. 1389-1397, 2017.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Improving post training neural quantization: Layer-wise calibration and integer programming. arXiv preprint arXiv:2006.10516 2020.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmeticonnly inference. In CVPR, pp. 2704-2713, 2018
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning quantize deep networks by optimizing quantization intervals with task loss. In CVPR, pp. 4350-4359, 2019.
Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee. Memory-efficient fine-tuning compressed large language models via sub-4-bit integer quantization. arXiv preprint arXiv:2305. 1315, 2023.
Schoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. I-bert: Integer- only bert quantization. In ICML, pp. 1506-5518. PMLR, 2021.
Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Funhyeok Park. Owq: Lessons learned from activation outliers for weight quantization in large language models. arXiv preprint (xXiv:2306.02222 2023.
Yanjing Li. Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, and Guodong Guo. Q-vit: Accu- rate and fully quantized low-bit vision transformer. NeurIPS, 35:34451-3444 2022.

---

Published as a conference paper at ICLR 2024
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit post-training quantization by block reconstruction. ICLR, 2021.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for Il compression and acceleration. arXiv preprint arXiv:2306.009778 2023.
Zechun Liu, Kwang- Ting Cheng, Dong Huang, Eric P Xing, and Zhiqiang Shen. Nonuniform-to uniform quantization: Towards accurate quantization via generalized straight -thogug estimation. In CVPR, pp. 4942-4952, 2022.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arXiv preprini arXiv:2305./7888 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. arXiv preprint arXiv:2305. 1627, 2023.
Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational Linguistics, 19(2)3133-330 1993.
Jeffrey L McKinstry, Steven K Esser, Rathinakumar Appuswamy, Deepika Bablani, John V Arthur, Izzet B Yildiz, and Dharmendra S Modha. Discovering low-precision networks close to full- precision networks for efficient inference. In 2019 Fifth Workshop on Energy Efficient Machine -eeurrP Edition (EMC2-NIPS), pp. 6-9. IEEE, 2019.
Learning and Cognitive
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR, 2017.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In ICCV, pp. 1325-1334, 2019.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In ICML, pp. 7197-7206. PMLR, 2020.
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. URL https://api.
semant icccholar.... org/ wtt//orruu.seeooooooooooo CCrpporrrsssc ID 257552285.
Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Bycongwook Kim, Youngjoo Lee, and Dongsoo Lee. Lut-gemm: Quantized matrix mul- tiplication based on luts for efficient inference in large-scale generative language models. arXiv preprint wwXiv:2206.09557, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21(555585- 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications the ACM, 64(9)999-106, 2021.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint (xXi:23308.13377 2023.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11699 2023.
Philippe Tillet, Hsiang-Tssie Tsung Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In MAPI., pp. 10-19, 2019.
12

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307. 099888, 2023b.
Guibin Wang, YiSong Lin, and Wei Yi. Kernel fusion: An effective method for better power ef- ficiency on multithreaded gpu. In 2010 IEEE/ACM Int'l Conference on Green Computing and Communications & Int'l Conference Cyber, Physical and Social Computing, pp. 344-350, 2010.
Ying Wang. Yadong Lu, and Tijmen Blankevoort. Differentiable joint pruning and quantization for hardware efficiency. In ECCV, pp. 259-277,
Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. QDrop: Randomly drop- ping quantization for extremely low-bit post-training quantization. In ICLR, 2022a.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Feng- wei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer lan- guage models. NeurlPS, 35:17402-17414 2022b.
Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xian- glong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09145. 2023.
Di Wu, Qi Tang, Yongle Zhao, Ming Zhang, Ying Fu, and Debing Zhang. Easyquant: Post-training quantization via scale optimization. arXiv preprint arXiv:2006. 16669, 2020.
Xiaoxia Wu, Zhewei Yao, and Yuxiong He. Zeroquant-fp: laap forward in llms post-training w4a8 quantization using foating-poin formats. arXiv preprint arXiv:2307.0977 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In ICML, pp. 38087- 38099, PMLR, 2023.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. NeurIPS, 35:27168-22 271188227711--11 2022.
Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. Zeroquant-v2: Exploring post-training quantization in Ilms from comprehensive study to low rank compensation. arXiv preprint wxXiv:2303.08300 2023.
Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.01089, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? In ACL, pp. 4791-4800, 2019.
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In ECCV, pp. 365-382, 2018.
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. In ICML, pp. 7543-7552, 2019.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.088. 05685, 2023.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: (606606660) 2016.
13

---

Published as a conference paper at ICLR 2024
Appendix
A MORE DETAILS ABOUT BIPARTITE SOFT MATCHING
As mentioned in Section 4.1.2, we use bipartite soft matching (Bolya et al., 2023) to determine which channels to aggregate efficiently. Support that we want aggregate channels. The step-by-step bipartite soft matching algorithm is shown follows:
1. Divide the channels into two sets and B, each of approximately equal size.
2. For each channel in A, construct an edge its most similar counterpart in
3. Select the T most similar edges.
4. Aggregate the channels that remain connected, according to Eq. (4).
5. Concatenate the two sets to form the assembled channel set.
B MORE DETAILS ABOUT THE EFFICIENT IMPLEMENTATION FOR CHANNEL REASSEMBLY
As mentioned in Section 4.3, the channel disassembly and assembly can be implemented efficiently if the previous layer / - is linear layer. Specifically, let W E RCxM be the weights of preceding linear layer, where C' and M denotes the input and output channel number for layer / 1, respectively. For channel disassembly, we enlarge the output channels of the preceding linear layer weights by:
W if f W
(A)
otherwise,
and adjust the input channels of the current layer's weight by:
w
ii M -
w
(B)
W
otherwise.
M:
Similarly, for channel assembly, suppose that we are aggregating channel j to channel i. Then, channel assembly can be implemented by reducing the output weight channels of the preceding linear layer by:
W!
W :
(C)
2
and adjusting the input channels of the current layer's weight by:
w W w
(D)
C ALGORITHM OF ADAPTIVE CHANNEL REASSEMBLY
We summarize our proposed adaptive channel reassembly in Algorithm
D PSEUDO-CODES OF CHANNEL DISASSEMBLY AND ASSEMBLY
We show the PyTorch style pseudo-codes of channel disassembly and assembly during runtime in Figure A.
E MORE RESULTS ON LLAMA-2 FAMILY
We provide additional results for the LLaMA-2 family in Table A. The observations from these results consistent with the phenomena identified in the LLaMA-1 family. Note that the varied performance of OmniQuant on W6A6 and W4A4 for LLaMA-2-70B can be attributed to the archi- tecture of I.L.A-2-770B which employs grouped-query attention (Ainslie et al., 2023) where each
14

---

Published as a conference paper at ICLR 2024
Algorithm 1: Algorithm of Adaptive Channel Reassembly for one layer in LLM.
Input: Input activation x E RRM, linear layer weight W E RMX,, grid search iteration P.
Set _* to co.
Function ReAssemb 1y :
// Channel disassembly
Calculate the total sub-channels number by n = Di= (max( [x,)//
Perform channel disassembly using Eq. (3).
I/ Channel assembly
Find the most similar channel pairs using bipartite soft matching with the distance metric in Eq. (5).
Perform channel assembly using Eq. (4).
return ReAssembly Error L using Eq. (6);
Calculate the max value of each channel m E RM.. do
for p E 2.,,,... P)
Calculate the threshold by 0 = min mi) + f (max(() min(m)).
[=ReAssemb) y (0);
if L then
L -
Adopt the final channel reassembly using the found 0": ReAssembly (f"). return One reassembled layer of L.M..
Table A: Performance comparisons of different methods for weights and activations quantization
LLaMA-2 model family.
PPL.
Accurwy (8)
Model
Bits
Method
WikiTexm2
C4
Arg.
PIQA
ARC-e
ARC-c
HellaSwag
Winogrande
Avg.
W16A16
5.47
76,82
53.62
72.87
W6A6
SQ
71.76
W6A6
OS+
76.22
52.74
40.70
71.89
65.19
61.35
W6A6
OmniQuant
5.87
7.48
6.68
76.77
52,90
40.61
71.86
61.09
61.25
W6A6
QLLM
5,72
2.31
6.52
77.48
52.99
39.33
71.38
65.98
61.43
LLaMA-2-7B
W44A
QLLM
3.91
7.50
6.71
76.
51.73
59.35
71.27
65.59
60.81
WTTA
SQ
101.77
93.21
97.19
60.17
55.23
27.13
37.06
19357
41.87
W444
OS+
63.11
39
28
47.31
45.93
W444
OmniQuant
14.61
W444
QLLM
11.75
13.26
12.51
67.68
44.40
30.89
58.45
$6.59
$1.60
W16A16
4.8
6.47
5.68
78.84
57.91
44.28
76.63
69.85
65,50
W6A6
SQ
5.19
6.77
5.98
78.29
57.41
43.86
75.02
66.93
64.30
W6A6
OS+
78.29
59.
43.34
75.37
67.56
64,74
W6A6
OmniQuant
5.14
6.74
5.94
78.56
57.11
43.60
75.36
68.35
64.60
W6A6
QLLM
75.10
64.87
LL-MA-2-138
W1AB
QLLM
5.17
78.67
57.11
41.89
75.33
W-TA4
SQ
29.82
44.05
36.95
61.30
40.25
550.22
42.24
49.96
45.10
W444
OS+
64.47
41.46
32.17
59,30
51.38
49.76
W114
OmniQuant
12.28
14.64
13.16
69:80
47.22
33.79
59.34
55.19
53.13
W444
QLIM
9.09
11.13
10.11
70.46
48.5 48
34.39
62.80
$5.41
54.31
W16A16
3.32
5.52
4.42
81.01
59.68
47.95
80.87
76.95
69.29
W646
SQ
3,69
5.88
79.87
57.55
79.01
67.18
W6A6
OS+
W6A6
OmnaQQann
3.71
5.91
80.20
60.27
46.84
80.55
76.01
68.77
11-MA-2-70R
W6A6
QLIM
3.55
5.76
4.66
80.63
59.01
45.99
79.64
75.37
68.13
W4AA
OLLM
3.6
5.76
3,68
80.79
58.59
47.44
79.42
75.77
68.40
W4A4
SQ
26.01
34.61
30.31
64.09
41.84
52.00
54.21
51.07
48.64
W1A1
OS+
66.16
42.72
34.90
56.93
$2.96
50.73
W4A4
OruniQuant
41.10
47.72
52.99
23.89
33.88
W444
QLLM
7.00
58.62
indicates no learnable equivalent transformation (Shao et al., 2023) on queries, keys, values, or attention
output due to incompatibility with grouped-query attention (Ainslie et al., 2023) in LLaMA- 2-BBB model.
group of queries shares single key and value head. Such architecture makes the learnable equiv- alent transformation in OmniQuant incompatible with grouped-query attention. In W6A6 settings, the impact of activation outliers is relatively minor, enabling partial learnable equivalent transforma-
15

---

Published as a conference paper at ICLR 2024
def channel ditammnbblssssnnblyya num_split):
won
imput with shape of [batch, tokens, channels] nun split: the number of sub-channels for each channel with shape of [channels]
B, N, shape vB N, C)
scaling 1.0 nun split compute the scaling factor each channel scaling scale channel num.split, dim=1) # perform channel deconposition
werch.repeat.nnterleee
x. x.ehape [1]
vBvieu N, C)
def chaneel arc_idx, dst.idx):
une
input with shape of [batch, tokens, channels] src_idx: the channel index that will be merged in set with shape of [Anun nerun_e mewged..haameelel [channels] dat idx: the channel index that will marged in net with shape of [#nun merged channels]
onn
B, N, shape
ori are idx arange C, 2, devicoex.device) # index for set
ori_dst_idx torch. arange C, 2, device=x. 8 get index for set idx] divide channels into two
are, x(..., ori arc idx], x(..., erc.C shapee [-1] the channel number in set dst.C - shape [-1] get the channel number in set
channel merged channel mask = indicates devicesx.device dtype=x.dtype)
n.idx = ori.rr_.dd(ssc.idd) chamnel_mask(m.ddx]
t1, arc. shape
sub.src = aaa. (dr-ga index=arc.ids exppndxn, t1, r)) # get channela that will be merged
dat dst.acatter reducaa-1, dat idx expand r),, sub are, reducesmode) merge channels are - sie. N, sre_C, 1) dat dien,v N, dst C, 1)
concat set and set if are == dat C: merged.x torch. (Erc, dst], dium-1) view(B, N, () else:
merged_x terch. carch.ca :-1, :], dat], (am=-1).vied
B, N, src.C + dst.C
merged.x torch. corrreo, [mergedx, erc(..., :l reshape dim view(
8, N, arc.C dat.C
remove merged channels
merged nerged_x. selett,,, (channel_naa) 0) nonzero() squeeze())
return merged_x
Figure A: PyTorch style pseudo codes of channel disassembly and assembly during runtime.
tions suffice in maintaining performance. However, in the W4A4 settings, the effect of activation outliers becomes more prominent. Under these conditions, the partial learnable equivalent transfor- mation insufficient to address the outier issue, leading to notably poorer performance. Notably, our QLLM significantly outperforms the state-of-the-art post-training quantization (PTQ) methods, demonstrating a substantial margin of improvement in 4-bit quantization. For example, QLLM quantized 4-bit LLLMMA-2770B outperforms SmoothQuant counterpart by average of 7.89% on the accuracy, which shows the promising results our method.
F MORE RESULTS ON CHAT MODELS
To demonstrate the generalization ability of our QLLM on chat models, we apply QLLM to quan- tize LLaMA-2-7B-CAI -hha-ttt and LLaMA-2-13B-CH optimized for dialogue use cases. We include the concurent state-of state-of-the-art quantizaiton method, OmniQuant, for comparisons. We GPT-4 to assess the performance of the quantized models
to 4-bit. These models are instruction-tune -uuned and
16

---

Published as a conference paper at ICLR 2024
set of 80 sample questions in Vicuna benchmark (Chiang et al., 2023). To eliminate the potential position bias (Zheng et al., 2023), we conducted the comparisons in both orders (a vs. and vs.a) for each pair, amounting to total of 160 trials. From Table B, QLLM consistently achieves much better performance than OmniQuant.
Table B: Performance comparisons between QIL and OmniQuant for chat models.
Model
Case
Former Win
Tie
Former Lost
2--B Chal
QLLM vs. OuniQQuatt
137
4
LLaMA-2-13B-Char
QLLM vs. OmniQuant
116
24
20
G MORE RESULTS IN TERMS OF CHANNEL REASSEMBLY
To further show the effectiveness of our channel reassembly (CR), we compare the average block- wise reconstruction enror across the entire network before and after applying CR and show the results on a calibration set with 128 randomly selected 2048-token segments from WikiText2 in Table C. The results clearly demonstrate that using CR significantly lowers the reconstruction error, and thus improves the performance of the quantized models.
Table C: Bise reconstruction error before and after channel reassembly (CR).
Model
Method
Reconstruction Error
LLaMA-1-7B
wro CR
4.71
LLaMA-1-7B
w/ CR
2.74
LLaMA-1-BE
wfo CR
7.67
LLaMA-1-13B
1.71
H MORE RESULTS IN TERMS OF CHANNEL DISASSEMBLY ONLY
To further demonstrate the effectiveness of channel disassembly (CD). apply CD without effi- cient error correction (EEC) to obtain 4-bit LLaMA-1-13B and show the results in Table D. We observe that the absence of both CD and EEC leads to significant decline in the performance of the quantized model. Notably, using CD alone substantially reduces the performance degradation asso- ciated with quantization. Moreover, increasing the channel expansion ratio 7 further improves the model's performance, which strongly shows the benefits of using CD to decompose the outlier chan- nels. By incorporating both CD and EEC, the performance improvement is even more pronounced, underscoring the efficacy of EEC in conjunction with CD.
Table D: Perplexity results of channel disassembly (CD) with and without efficient erro correction (EEC). "y" is the channel expansion ratio. We report the perplexity of W4A4 LLMMA-1-13B on WikiText2 (Merity al., 2017), PTB (Mareus et al., 1993) and C4 (Raffel et al., 2020).
CD
EEC
WikiTex12
PTB
C4
Avg.
1702.34
1853.58
1159.41
1571.78
0.01
19.34
45.36
23.25
29.32
0.01
8.31
14.44
10.74
/
0.03
12.11
14.38
/
0.03
8.01
13.52
10.27
10.60
0.05
11.4
23.53
13.62
16.18
/
0.05
7.85
13.38
10.13
10.45
/
0.07
11.13
23.47
13.45
16.02
0.07
7.81
13.35
10.11
17

---

Published as a conference paper at ICLR 2024
I MORE COMPARISONS WITH OTHER OUTLIER HANDLING METHODS
To further show the effectiveness of channel reassembly, we compare our method with previous outlier handling methods which employ gradient-free methods to learn mathematically equivalent transformations. For fair comparisons, we do not apply efficient error correction. From Table E, all methods exhibit comparable performance at 6-bit quantization. However, for 4-bit quantization, channel reassembly significantly surpasses other methods by a large margin, particularly for larger models.
Table E: Performance comparisons of channel reassembly (CR) with previous outlier handling methods across five zero-shot tasks,
Model
#Bits
Method
PIQA
ARC-c
ARC-c
HellaSwag
Winogrande
Avg.
76.65
40.10
W6A6
OS+
76.82
51.35
41.13
71.42
65.98
61.34
W6A6
CR
76.88
52.31
40.87
71.37
64,33
61.15
LLaMA-I-7B
W4A4
SQ
49.80
30.40
25.80
27.40
48.00
36,28
W4A4
OS+
62.73
39.98
30.29
44.39
52.96
46.07
W4A4
66.92
42.55
32.34
54.31
50.04
49.23
W6A6
SQ
77.80
56.36
42.58
75.11
68.11
63.99
W6A6
OS+
78.29
56.90
43.09
75.09
69.22
64.52
1LAMA-I-13R
W6A6
78.02
42.41
70.01
W444
56.55
34.51
26.71
41.56
48.70
41.41
W444
OS+
63.00
40.32
30.38
53.61
51.54
47.77
W4A4
CR
67.57
43.77
31.48
60.78
56.04
$1.93
J MORE RESULTS IN TERMS OF THE EFFICIENT ERROR CORRECTION ONLY
Using EEC only without channel reassembly results in suboptimal performance it suffers from activation outlier issues. To demonstrate this, we applied EEC only to quantize LLaMA-1-1 1-7B to 4- bit, using the same training settings as our QLLM but with varying numbers of calibration samples. From Table F, even with an increased amount of calibration data, the performance of the EEC only significantly lags behind our QLLM. These results strongly demonstrate the effectiveness of channel reassembly in addressing activation outliers, thereby substantially improving performance.
Table F: Performance comparisons with different methods under various numbers of calibration samples. We report the perplexity of W4A4 LLaMA-1-7B WikiText2 (Merity al., 2017), PTB (Marcus et al., 1993) and C4 (Raffel et al., 2020).
Method
#Samples
WikiTexx2
PTB
C4
Avg.
16.64
38.58
27.85
EEC
256
14.94
37.70
33.62
28.75
EFC
512
12.35
29.39
31.59
24.44
QLLM
128
9.65
16.56
12.29
12,83
K MORE RESULTS IN TERMS OF TUNING QUANTIZED WEIGHTS ONLY
The effectiveness of TQW highly dependent on our channel reassembly. To demonstrate this, applied TQW only quantize LLaMA-1-77 to 4-bit using the same training settings as QLLM and show the results in Table G. The results clearly indicate that the absence of our adaptive channel reassembly results in significantly reduced performance for TQW. This underscores the vital role of channel reassembly in addressing activation outliers and thus improving model performance.
L MORE COMPARISONS BETWEEN EFFICIENT ERROR CORRECTION AND TUNING QUANTIZED WEIGHTS DIRECTLY
To further show the effectiveness of our efficient error correction (EEC), we conduct more compar- isons between EEC and tuning quantized weights (TQW) directly small model and report the
18

---

Published as a conference paper at ICLR 2024
Table G: Performance comparisons with different methods. We report the perplexity of 4-bit LLaMA-1-7B on WikiTex12 (Merity et al., 2017), PTB (Marcus et al., 1993) and C4 (Raffel et al., 2020). "CR" denotes our adaptive channel reassembly.
Method
WikiTex12
PTB
C4
Avg.
TQW we CR
13.13
42.81
32.07
29.34
TQW CR
8.90
14.75
11.63
11.76
sults in Table H. The results show that employing EEC not only maintains comparable performance but also markedly improves training speed and significantly reduces GPU memory usage over TQW. It is worth noting that there is a trade-off between GPU memory (i.e., #Attn-FFN blocks) and per- formance. Leveraging EEC even allows simultaneouslly. thereby significantly improving performance while preserving a similar training speed and a reasonable increase in GPU memory.
perform reconstruction for 16 AttentionFFFN blocks
Table H: Perplexity comparisons between efficient error correction (EEC) and tuning quantized weights directly (TQW) for 4-bit LLaMA-1-7B. "OOM" indicates out of memory.
#Atto-FFN Block
Method
WikiTex12
PTB
C4
Training Time (GPU Hours)
GPU Memory (GB)
CR
14.12
25.30
16.58
18.67
TQW
10.10
16.19
12.95
13.08
1.49
10.9
EEC
11.21
19.29
14.06
14.85
1,06
7.95
TQW
9.74
14.79
11.68
12.07
1.49
17.62
2
EEC
10.61
18.56
13.64
14.27
1.05
11.62
TQW
8.90
14:75
11.63
11.76
30.95
4
EFC
9.65
16.56
12.29
12.83
1.05
18.91
TQW
00M
8
14.98
11.63
33,50
TOW
00M
16
EEC
9.13
14.95
11.60
11.89
1.05
62.70
M EFFECT OF THE WEIGHT MERGING IN EFFICIENT ERROR CORRECTION
As explained in Section 4.2, for quant X ))uunnt(( W) + (X) the low-rank weights A and bring not only additional inference overhead due to the matrix multiplication between the full-precision AB and quant(X XX) but also extra storage burden. To address this, we perform weight merging by quant(WV ( + AB) after the reconstruction, which effectively avoids overhead but intro- duces additional quantization error. For 4-bit quantization, low-rank weights into the frozen weights using quant W + AB) does not lead to an increase in outliers. This finding is supported by the notably low MSE levels for channel- bhineel-wise Pag, P999, and maximum/minimmn values before and after the weight merging process across in Table I. More- over, our weight merging only leads to small quantization error, shown in Table J. Note that small deviations can aggregate throughout the network, leading to the performance drop. To address this, shown in Section 4.2, we further employ sequential reconstruction to mitigate errors from previous layers, resulting in only a negligible performance drop. To demonstrate this, we compare the performance QLLM with and without the weight merging. From Table K, the weight merging only leads to slight increase in perplexity.
empirically observe that merging the
Table I: The maximum MSE of channel-wise Pøa, Pau9, and maximum/minimun values before and after the merging process across all layers of 4-bit LL-MA---7B.
MSE
MSE
MSE...
MSE
5.42 x10-6
3.48 x10-6
7.11110-6
8.71 x10
Table J: The maximum MSE of channel-wise Poo. Poes, and maximum/minimum values before and after the merging process across all layers of 4-bit LLaMA-1-7B.
Model
MSE of Weight Merging
LLaMA 1-7B
4.04 x10-7
LLaMA
x10-7
19

---

Published as a conference paper at ICLR 2024
Table K: Effect of the weight merging (WM) in the efficient error correction. We report the perplex- ity on WikiText2 (Merity et al., 2017), PTB (Marcus et al., 1993) and C4 (Raffel et al., 2020).
Model
Method
WikiTex12
PTB
C4
1LMMA-I-7B
w/o
15.93
LLLMA-I-7B
w/ WM
9.65
16.56
12.29
12.83
LLaMA-1-13B
WHO WM
8.29
13.69
10:40
10.79
LLaMA-1-138
WM
8.41
14.38
10.58
N MORE RESULTS REGARDING INFERENCE EFFICIENCY
Following (Guo et al., 2020; Wang al., 2020), we use Bit-Operation (BOP) count to measure the theoretical inference complexity of our QLLM. From Table L,, our 8-bit QLLM incurs only marginal increase in BOPs when compared to the INT8 model but substantially lower than those of the FP16 counterpart, which shows the efficiency of our method.
Table L: Bit-Operation (BOP) count comparisons of different models. We report the results of LLaMA-1-7B with mini-batch size of "L" denotes the sequence length.
512
1024
2048
EP16
875.52I
1.766.40T
3,604.48T
7,493.121
INT8
231,58T
467.64T
952.561
1976.16T
231.58T+1.0M
467.64142.14M
952.56T-4422M
1976.161+8.56M
We further show the inference time of channel disassembly and assembly of our QLLM in Table M. From the results, channel disassembly results in additional inference costs due the extra channels. These additional channels often don't align with GPU-friendly multiples like 32 or 64, leading to less efficient GPU use. Using our channel assembly maintains the original channel count, ensuring better GPU utilization and mitigating the extra inference costs from disassembly. As result, the quantized models with both channel disassembly and assembly achieve higher throughput compared to the ones with disassembly only, which demonstrates the necessity of channel assembly.
Table M: Inference throughput (tokens/s) comparisons of different models. The throughput is mea- sured with a 2048-token segment on NVIDIA RTX 3090 GPUs: 1x GPU for LLaMA-1-7B and 2x GPUs for LLaMA-1-13B. "CD" stands for channel disassembly. "CA" represents channel assembly. "Adaptive" refers to the adaptive strategy. ">" is the channel expansion ratio. "OOM" indicates out of memory.
Model
Method
CD
CA
Aduptive
Inference Throughput (tokens's)
FP16
3252
W4A16
5708
W4A4
6667
W474
0.01
6322
LL-MA-1-7B
W4A4
6310
W4A4
0.01
6365
W4A4
0.05
6334
W444
6318
W4A4
FP16
1910
W8A8
3179
W4A16
2026
W444
3873
W4N4
0.01
3728
LLaMA-1-13B
W4A4
0.05
3725
W444
0.1
3678
W4AT
0.01
3737
W4A4
3728
W4A4
0.1
3681
W4A1
3730
20

---

Published as a conference paper at ICLR 2024
O MORE RESULTS REGARDING TRAINING EFFICIENCY
We assess the training efficiency of our method in comparison to OmniQuant on a single NVIDIA A100 80G GPU. The GPU training hours for both methods are presented in Table N. The results reveal that the training cost of our QLLM can be up to 1.93x faster than OmniQuant, showing the exceptional training efficiency of our QLLM.
Table N: The training time (GPU Hours) comparisons of our QLLM with OmniQuant.
Methed
OmniQuant
QLLM
1.M.M-2-77R
1.98
1.05
LLaMA-2-138
3,46
1.79
LLaMA-2-708
14.52
9.05
P EFFECT OF DIFFERENT CALIBRATION SETS
We apply QLLM to yield 4-bit LLaMA-7B using different calibration sets and report the results in Table O. From the results, we observe that the choices of calibration set have a minor effect, as the performance remains relatively consistent across different sets. For fair comparisons, following OmniQuant (Shao et al., 2023), we use WikiTeex22 as a calibration set by default.
Table O: Effect of different calibration sets. We report the perplexity & of W4A4 LLaMA-7B on WikiText2 (Merity al., 2017), PTB (Marcus et al., 1993) and C4 (Raffel et al., 2020).
Evaluation Set
WikiToxt2
PTB
C4
Averago
WikiText2
9.65
16.56
12.29
12.83
Calibration Set
C4
10.56
17.44
12.08
13.36
Q EFFECT OF DIFFERENT NUMBERS OF CALIBRATION SAMPLES
We apply QLLM to yield b-tit LLaMA-7B using different numbers of calibration samples from WikiText2 and show the results in Table P. The results reveal a positive correlation between the performance of the quantized model and the number of calibration samples, indicating that utiliz- ing more samples generally leads to better performance. This trend underscores the importance of the calibration phase, where leveraging larger sample pool can provide representation of the data distribution, enabling more accurate quantization. However, it is also im- perative to consider the computational and memory overhead associated with an increasing number of calibration samples. There is an inherent trade-off between achieving higher model performance and maintaining computational efficiency.
comprehensive
Table P: Effect of different # calibration samples. We report the perplexity 1 of W4A4 LLaMA-7B on WikiText2 (Merity et al., 2017), PTB (Marcus et al., 1993) and C4 (Raffel et al., 2020).
#Samples
WikiTexz
PTB
Average
16
10.72
18:43
13.66
14.27
32
10.12
17.35
12.84
13.44
64
10.15
16.23
12.26
12,88
16.56
256
9.60
15.75
11.79
12.38
R MORE RESULTS ABOUT THE EXPANSION RATIOS OF THE QUANTIZED LLM
In this section, illustrate the detailed expansion ratios for the input activations of different layers in the 4-bit LLaMA-1-7B and LLaMA-1-13B obtained by our adaptive strategy in Figures and C. From the results, adaptive strategy allocates higher expansion ratios to the shallower MSA
21

---

Published as a conference paper at ICLR 2024
layers and to the deeper down projection layer in the FFN, which indicates that these layers possess a greater number of outliers. To substantiate this observation, we further plot the channel-wist maximum and minimum values for the input activations across different layers in Figure D. These visual representations further underscore the effectiveness our adaptive strategy in identifying and addressing the presence of outliers in different layers.
Channel expansion ratio
MSA
0.08
FFN.up_prd
FFN.down_Pro proj
0.06
0.04
0.02
0.00
0
10
20
30
Block index
Figure An illustration of the searched expansion ratios using our adaptive strategy for 4-bit LLaMA-1-7B.
0.10
Channel expansion ratio
MSA
FFN.up_pro
0.08
FFN.down_pro proj
0.06
0.04
0.02
0.00
10
20
30
40
Block index
Figure C: An illustration the searched expansion ratios using our adaptive strategy for 4-bit LLLMA---113B
22

---

Published as a conference paper at ICLR 2024
Layer
Layer
Layer
Layer
MSA
-22
-30
Charell
Charnal
Charoet
Cheend
FFN.up_proj
2030
2006
2003
4000
Chaneen
1:00
2.5
6.0
FFN.down_pro
3000
50000
2500
1596
15000
2500
30000
2,000
2000
3000S
Figure D: An illustration the thannel-wisc maximum and minimum input activation values for the MSA, up projection and down projection layers in FFN of different blocks in LLaMA-1-13B.
23


Reviews:
[{"review_id": "rTIoug7iAi", "reviewer": "Reviewer_dSZP", "summary": "The paper is trying to address the challenge of LLM quantization associated with the well-known \"channel-specific outlier problem\", i.e. some specific channels in activations tend to have much larger range compared to the others, which will cause difficulty in choosing quantization scaling factor and degradation in accuracy. The main solution proposed is to disassemble an outlier channel into T channels, therefore, the magnitude of each channel becomes 1/T of the original channel. In order not to increase the computational cost by too much, the second part of the proposed method is to search for similar channels and merge them by averaging the activations and summation of the corresponding weights. As a result, total number of channels will be the same as the original tensors after these channel disassembly and assembly steps. Furthermore, a LORA/QLORA-like tuning is applied to greatly alleviate the cost of PTQ while enabling simultaneous tuning of multiple blocks, which is critical to improve model performance (perplexity). Finally, the inference benchmark results show that the proposed method will add ~20% overhead compared to INT8.", "questions": "1. Table 2 shows \"Channel Disassembly only\" approach with just 1% of channel expansion ratio can already achieve comparable results with the final method. In fact, the reason to use channel assembly is mainly to reduce the computational cost. Author may want to add some comments or examples regarding the overhead incurred by this 1% extra channels, in order to justify the need for Channel Assembly method.  \n2. The paragraph of \"Inference efficiency\" as well as Table 4 didn't specify whether this \"FP\" model is FP32 or FP16. If it's FP32, author may also want to include FP16 results since FP16 is used as the baseline on accuracy tables. Also it is understandable that the implementation for INT computation may not be optimized. Therefore, instead of comparing the absolute run time, it would be helpful to include the additional Ops of QLLM compared to INT8.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed channel decomposition/disassembly method is simple and effective.\n2. The method to determine disassembly threshold (theta) is reasonable and computationally affordable.\n3. Outperforms previous methods at W4A4 on Llama-1 and Llama-2.\n4. acceptable overhead compared to INT8.", "weaknesses": "1. W6A6 results are included but only referenced a few times, mostly used in a sentence like \"QLLM is comparable with other methods at W6A6.\" Given that authors in general would like to impress readers by \"being better than others\" rather than \"being comparable to others,\" W6A6 really doesn't make a strong point in the main article. Plus W6A6 has almost no real HW support... The author might want to consider adding a few more comments/discussion on W6A6 results or moving them to appendix and make the main article more concise.  \n2. A little clarification about the inference time benchmark would be helpful. For example, readers might be interested in comparing the QLLM results with other quantization implementations, like weight-only quantization W4A16. Take auto-gptq (https://github.com/PanQiWei/AutoGPTQ/tree/main#inference-speed) as a reference, the \"speed\" usually is in the unit of token/sec or msec/token. Table 4 only says \"inference time\" and the unit is ms, which is a little unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XHJ5cQRnhR", "reviewer": "Reviewer_KacJ", "summary": "This work, QLLM, mainly addresses activation outliers for quantizing LLMs. QLLM proposes to first break the outlier channels into several channels, and then merge similar channels to keep the original channel number (by bipartite soft matching). To decide how many channels an outlier channel needs to be broken into, QLLM conducts a grid search of the outlier threshold hyper-parameter $\\theta$ using layer-wise reconstruction error as the objective. Then the channel-reassembled network is more suitable for activation quantization. Finally, QLLM adopts LoRA training with block-wise reconstruction error as the objective to restore performance. Experiments are conducted with LLaMA v1 and v2 models. Evaluations are conducted on WikiText, PTB, and C4 for PPL, and 5 zero-shot benchmarks for zero-shot accuracy. The algorithm performance experiments use the W6A6 and W4A4 settings, and the inference efficiency experiments use the W8A8 setting.", "questions": "* Important points are listed in the weakness section.\n* Sec 4.1.1 uses $\\max(x_M)$ to determine the number of disassembly channels $T$ according to $\\theta$, why don't we use $\\max(|x_M|)$?\n* I wonder whether the CD results in Table 2 use the LoRA tuning? I would like to see the results of using CD without LoRA tuning as a reference.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* This paper is well-written, well-organized, and easy to follow.\n* Motivation: The activation outlier issue is important for quantizing activations in LLMs\n* Reasonable method: Channel splitting is a reasonable method to address the activation outlier issue.", "weaknesses": "My major concern in this paper is whether its current evaluation can fully demonstrate its practicability.\n\n**About the inference efficiency**\n* While the algorithm perf. experiments are conducted with W6A6 and W4A4, inference efficiency experiments are only conducted with W8A8. I think implementing W4A4 kernels on some NVIDIA GPUs and demonstrating better efficiency-algo. perf. trade-off compared with W8A8 and existing W4A4 will make this paper stronger.\n* Table 4 shows QLLM has inference overhead compared to methods without channel reassembly. Does this overhead come from the additional disassembly and assembly layers for non-linear layers? A thorough breakdown of this overhead for differently-sized models (larger models, more compressed models such as 4bit) will help illustrate whether the method is really practical.\n* As the channel-assembly process introduces additional approximations, only applying the channel-disassembly process results in the best perplexity (which is shown in Table 2). So, to further demonstrate the necessity of applying channel assembly to keep the channel number fixed, the authors can show the inference efficiency when using the expanded channel number (only apply CD).\n\n**About the algorithm performance**\n* I'm curious how much influence the reassembly process still has as the training data amount goes up. I.e., do the authors compare with using only reconstruction-based LoRA tuning without CD, CA, and CP, under exactly the same training settings but maybe with more calibration data (e.g., 256 instead of 128).\n* A suggestion instead of a weakness: To raise more interest in using this method, I recommend doing some evaluations on chat models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tVUFCMjEfF", "reviewer": "Reviewer_wUVT", "summary": "In this paper, the authors propose QLLM, a precise and efficient post-training quantization method specifically designed for Large Language Models (LLMs). They address the challenge of activation outliers by introducing a gradient-free channel reassembly technique that redistributes the magnitudes of outlier channels across all channels. This ensures a more balanced activation range, facilitating accurate quantization and improving the performance of quantized LLMs. The authors also introduce channel assembly to maintain the original channel count by merging similar channels. Additionally, they propose an adaptive strategy to determine the optimal number of disassembled channels for each layer, based on minimizing the reassembly error. The proposed QLLM method achieves accurate quantized models efficiently, as demonstrated by extensive experiments on LLaMA-1 and LLaMA-2 datasets. For example, QLLM outperforms the previous state-of-the-art method by 7.89% on average accuracy across five zero-shot tasks for the 4-bit LLaMA-2-70B model, trained within 10 hours on a single A100-80G GPU.", "questions": "- I am interested in whether the proposed CHANNEL DISASSEMBLY and CHANNEL ASSEMBLY methods can be extended to handle multiple inputs, specifically in the case of $Y=XW$ where $X\\in\\mathbb{R}^{batchsize\\times M}$. I have some concerns regarding the effectiveness of Equations (3) and (4) on $X\\in\\mathbb{R}^{batchsize\\times M}$ if the outliers differ among $X_{i,:}$, as the derivation relies on the approximation of the summation of scalars. Additionally, the authors mention that \"with the channel indexes for decomposition and aggregation calculated offline using calibration data\". Did the authors imply that the outliers often occur at the same index across different inputs? Otherwise, the effectiveness of the proposed methods may vary.\n\n- The notation of $\\beta$ in Equation (3) appears to correspond to $\\lfloor\\frac{\\min(X)}{\\alpha}\\rceil$.\n\n- It would be beneficial to include the baseline results of $\\gamma=0$ in Table 2.\n\n- According to Table 6 and the abstract, the training time of efficient error correction is reported to be around 1 hour, while the total time cost is 10 hours, suggesting that the adaptive CD/CA takes approximately 9 hours. Furthermore, the baseline result of tuning quantized weights directly (TQW) with 4 Attn-FFN Blocks seems to be quite strong, as it only takes one and a half hours with much smaller GPU memory overhead compared to the 16 Attn-FFN Blocks EEC results, and it even achieves better performance. Could the authors provide further clarification on this matter?\n\n- Did the FP results reported in Table 4 refer to FP32 or FP16 results?\n\n- I am curious about the baseline results of tuning quantized weights directly (TQW) with EFFICIENT GRADIENT-BASED ERROR CORRECTION. Did this setting outperform the proposed CD/CA + EEC?\n\n- Could the authors provide pseudo-codes to provide more detailed explanations of the \"additional disassembly and assembly layers that are designed to decompose and aggregate channels during runtime\"?\n\n- Could the authors further elaborate on why OmniQuant consistently performs well on W6A6 but performs extremely poorly on LLaMA-2-70B with W4A4?\n\n- It is worth noting that \"DIFFERENT NUMBERS OF CALIBRATION SAMPLES\" play a crucial role in performance. Did the authors follow the same setting as other baseline methods?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The concepts of CHANNEL DISASSEMBLY and CHANNEL ASSEMBLY proposed in this paper appear to be novel. The findings presented in Table 2 provide evidence for the effectiveness of CD and CA.\n- The methodology described in this paper is straightforward and comprehensible.\n- The authors have conducted thorough experiments across various settings, which is commendable.", "weaknesses": "## Major Concern:\n- While I acknowledge the efficiency of CHANNEL DISASSEMBLY and CHANNEL ASSEMBLY in the form of $y=xW^{l-1}W^{l}$, as explained in \"MORE DETAILS ABOUT THE EFFICIENT IMPLEMENTATION FOR CHANNEL REASSEMBLY\", I have reservations regarding its applicability to scenarios involving multiple inputs $X=\\\\{x_1,x_2,...,x_K\\\\}\\in\\mathbb{R}^{K\\times M}$ and a non-linear transformation $y=\\phi(xW^{l-1})W^{l}$, where $\\phi(\\cdot)$ represents a normalization layer followed by an activation function like GELU. Despite the authors' claim that \"the channel indexes for decomposition and aggregation are calculated offline using calibration data, which only introduces a minor extra overhead\", I remain unconvinced that the overhead is negligible. \n\n## Minor Comments:\n- It would be beneficial to provide a proof demonstrating that the approximation error introduced by Equation (4), in conjunction with the subsequent quantization error, is indeed smaller than the quantization error resulting from direct quantization.\n- I am also curious about the potential occurrence of outliers when employing $quant(W+AB)$ in the equation $y=quant(x)quant(W+AB)$. Additionally, I would appreciate further insights into the quantization error between $quant(X)quant(W+AB)$ and $quant(X)(quant(W)+AB)$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IMKz4ffJeD", "reviewer": "Reviewer_rXfg", "summary": "In this paper, the authors proposed QLLM for low-bit weight-and-activation quantization of LLMs. QLLM consists of three techniques: channel disassembly to reduce outlier magnitude and channel assembly to maintain channel numbers, an adaptive strategy to find the optimal expansion ratios, and a low-rank fine-tuning method to reduce quantization error. Experiments show that the proposed method outperforms existing quantization methods, especially under low-bit setting (W4A4).", "questions": "Please see the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is generally well-written and easy to follow. \n2. The proposed method outperforms existing ones under low-bit weight and activation quantization (W4A4). \n3. The ablation study is comprehensive, showing the effectiveness of each components (e.g., channel assembly, adaptive expansion rate, etc.)\n4. The efficient fine-tuning method seems like an efficient way to restore performance.", "weaknesses": "1. The proposed method has a significant advantage *only* under very low-bit settings like W4A4. However, under W4A4, all methods (including QLLM) cannot achieve a reasonable accuracy. Take Table 1 as an example, the W4A4 LLaMA-65B accuracy of QLLM is only 59.83% (average), which is even lower than the FP16 accuracy of LLaMA-7B, which is 62.23%. The huge drop in accuracy makes the setting impractical. On the other hand, the proposed method does not outperform existing work like OmniQuant under higher precisions (e.g., W6A6). Therefore, it is questionable whether the proposed method has a practical advantage. \n2. The proposed disassembly and assembly method will lead to inevitable inference overhead when there is LayerNorm or activation functions, since extra layers are inserted in the the forward graph (also confirmed in Table 4, 27% overhead compared to INT8 under context length 256, which is negligible). Furthermore, it is unclear what kind of configurations (e.g., disassembly ratios) are used for the INT8 setting in Table 4, since there is no accuracy number reported under this setting. If we use a larger disassembly ratio, will the overhead be larger?\n3. It seems from Table 6 does directly updating the quantized weights (i.e., QAT) is always better than the LoRA tuning. Does it mean if we have enough GPUs for parallelized fine-tuning (e.g., FSDP), we should use QAT for the best accuracy?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)