Title: From Comprehensive Study to Low-Rank Compensation: Exploring Post-Training Quantization in LLMs
Venue: NeurIPS 2023
Accepted: no
Keywords: Understanding Quantization

Int8

Empirical Investigation

Post-Training Quantization
Abstract: Post-training quantization (PTQ) has recently been demonstrated as a viable method to reduce memory consumption and compute cost for large language models. However, a comprehensive study on the effect of different quantization schemes, model families, and quantization bit precision has been lacking. In this work, we provide an extensive analysis of these components. We examine the impact of PTQ on weight-only, activation-only, and weight-and-activation quantization using various methods such as round-to-nearest (RTN), GPTQ, ZeroQuant, and their variants, applied to two different model families (OPT and BLOOM) with sizes ranging from 125M to 176B. We contribute by: (1) conducting a sensitivity analysis, revealing that activation quantization is generally more sensitive to weight quantization, and smaller models typically perform better than larger models in terms of activation quantization; (2) evaluating and comparing existing PTQ methods to optimize model size reduction and minimize accuracy impact, discovering that current methods can hardly achieve original model quality for quantization with either INT4-weight or INT4-weight-and-INT8-activation; (3) based on these insights, we optimize existing methods and introduce a technique called Low Rank Compensation (LoRC), which uses low-rank matrix to enhance model quality recovery with a negligible increase in model size

Abstract

Post-training quantization (PTQ) has emerged as a promising technique for mitigating memory consumption and computational costs in large language models (LLMs). However, a systematic examination of various quantization schemes, model families, and quantization bit precision has been absent from the literature. In this paper, we conduct a comprehensive analysis of these factors by investigating the effects of PTQ on weight-only, activation-only, and weight-and-activation quantization using diverse methods such as round-to-nearest (RTN), GPTQ, ZeroQuant, and their variants. We apply these methods to two distinct model families with parameters ranging from 125M to 176B. Our contributions include: (1) a sensitivity analysis revealing that activation quantization is generally more susceptible to weight quantization, with smaller models often outperforming larger models in terms of activation quantization; (2) an evaluation and comparison of existing PTQ methods to optimize model size reduction while minimizing the impact on accuracy, revealing that none of the current methods can achieve the original model quality for quantization with either INT4-weight or INT4-weight-and-INT8-activation; (3) based on these insights, we propose an optimized method called Low-Rank Compensation (LoRC), which employs low-rank matrices to enhance model quality recovery with a minimal increase in model size.

1 Introduction

Large language models (LLMs) like Codex [15] and ChatGPT [24] have demonstrated breakthrough performance across various benchmarks, such as natural language understanding and generation, and are now integrated into everyday applications. However, efficiently serving LLMs has become a pressing concern due to their significant memory consumption and computational demands. Unlike classification or diffusion models, LLMs present unique challenges, as they involve two distinct phases: prompt and generation. The prompt phase is primarily compute-bound, while the generation phase, with low batch size and KV cache, is mainly memory-bound [26].

As the progression of hardware bandwidth lags behind that of computational demand [14], the resource 27 demands of extra-large models such as MT-NLG-530B [30]-which necessitates the deployment of 28 multiple nodes for operation-escalate, adding to the complexities of cross-node communication. 29 This has emphasized the urgency to curtail both the size and computational expense of Large Language 30 Models (LLMs). An increasingly effective solution to these issues is post-training quantization (PTQ). 31 This method aids in the reduction of training prerequisites while simultaneously lowering the bit 32 precision of weights and activations to either INT4 or INT8. 33

While the effectiveness of post-training quantization (PTQ) has been underscored in a number of 34

recent studies [36, 12, 35, 7], a comprehensive, systematic investigation into several key dimensions 35

of this technique remains to be undertaken. Specifically, the extant literature falls short in providing 36

0.0

0.2

0.4

0.6

0.8

1.0

Model-size (Bit numbers in log scale)

1e12

10.0

12.5

15.0

17.5

20.0

22.5

25.0

27.5

30.0

Model-quality (Perplexity, decreasing in y-axis)

OPT (Quality and size tradeoff)

0.0

0.5

1.0

1.5

2.0

1e11

10

12

14

16

18

20

PTQ

RTN

FP16/INT8

LoRC

0.0

0.5

1.0

1.5

2.0

2.5

Model-size (Bit numbers in log scale)

1e12

10.0

12.5

15.0

17.5

20.0

22.5

25.0

27.5

30.0

Model-quality (Perplexity, decreasing in y-axis)

BLOOM (Quality and size tradeoff)

0

1

2

3

4

5

1e10

15.0

17.5

20.0

22.5

25.0

27.5

30.0

PTQ

RTN

FP16/INT8

LoRC

thorough coverage of the functionality of various PTQ methods or the sensitivity of disparate models. 37 Moreover, despite current quantization methods demonstrating promising results in the reduction of 38 model sizes, the question persists as to whether these methods are achieving their optimal potential in 39 minimizing Large Language Models (LLMs) sizes. 40

With these observations in mind, our study sets forth to address two salient questions: (1) When 41 subjected to quantization, do LLMs of varying sizes and pretraining data exhibit similar behavior? (2) 42 Are existing quantization methods truly leveraging their full potential in reducing the sizes of LLMs? 43

44

45

46

47

48

49

50

Contribution. To elucidate these queries, we undertake an exhaustive examination of the impact of PTQ on weight-only, activation-only, and combined weight-and-activation quantization. This investigation incorporates a range of PTQ methods, including round-to-nearest (RTN), GPTQ [12], ZeroQuant [36], and their respective variants. To broaden the scope of our analysis, we focus on two distinct model families, OPT [40] and BLOOM [28], spanning model sizes from 125M to a massive 176B. Our code will be made available for reproduction. In summary, we make the following contributions:

51

52

53

54

55

56

57

58

59

60

61

62

(1) We provide a thorough sensitivity analysis to demonstrate that a) Activation quantization is generally more sensitive to weight quantization; Smaller models usually have better activation quantization performance than the relative larger model. b) Different model families show different INT8 activation quantization behaviors; Particularly for large models, BLOOM-176B has small accuracy drops (about 1 perplexity or PPL) but OPT-30B and -66B experience worse performance.

(2) We carry out a detailed evaluation and comparison of current PTQ methods, utilizing optimal configurations to maximize model size reduction while minimizing accuracy impact. We found that the current existing method can barely achieve less than 0.1 PPL points degradation for quantization with either INT4-weight or INT4-weight-and-INT8-activation (W4A8). To recover the 0.1 PPL, we strive to push the boundaries of employing fine-grained quantization (FGQ) techniques. We observe FGQ is able to recovered points degradation of <0.1 PPL for large models (>13B) for INT4 weight quantization, but there are still non-negligible model quality drops.

(3) Based on the above understanding, we further optimize existing methods and introduce a technique 63 called Lo w R ank C ompensation (LoRC), which employs low-rank matrix factorization on the 64 quantization error matrix. Complementary to FGQ, LoRC plays a crucial role in enhancing the full 65 model quality recovery, while there is little increase of the model size. 66

67

In Figure 1, we provide model size and quality trade-offs for both OPT and BLOOM families.

As can be seen, using LoRC on top of PTQ methods from [36, 12] and fine-grained quantization, 68

we set a new quantization Pareto frontier for LLMs. Meanwhile, we recommend the following 69 setting for quantizing LLMs with LoRC (Note that activation quantization should be only applied if 70 necessary): (1) For larger models (>10B), fine-grained (block size 64-256) 4-bit weight quantization 71 plus 8-bit activation quantization (block size 64-256) with PTQ can be used for real deployment; (2) 72 For middle-size models (<10B and >1B), per-row INT8 quantization plus fine-grained (block size 73 64-256) INT8 activation quantization can be used with PTQ from [12, 36]; (3) For smaller models 74 (<1B), per-row W8A8 (INT8 weight and INT8 activation) RTN is enough based on [36]. 75

76

2 Related Work

Different quantization methods [29, 38, 9, 41, 1, 8, 31, 19] for transformer-based models [32] have 77 been explored for a while. However, most of those works need quantization-aware finetuning or 78 even expensive quantization-aware knowledge distillation [17]. Due to the cost of training/finetuning 79 LLMs [25, 18, 31, 34, 33], it is a challenge for practitioners/researchers to do finetuning/distillation 80 on those LLMs, particularly for models like GPT-3-175B [4] and BLOOM-176B [28]. 81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

Post-training quantization (PTQ) [37, 3] is an alternative way to quantize the model with no/minimal finetuning requirement. Along this line, several recent works focus on LLMs (beyond the millionparameter scale). [36] proposes vector-based INT8 quantization with layer-by-layer knowledge distillation to overcome the training cost and quantization error introduced by LLMs. [6] uses similar vector-based INT8 quantization weight plus mixed-precision (INT8/FP16) quantization for activation to overcome the sensitivity of activation quantization. However, the inference speed of [6] is generally even slower than FP16 baseline [2] due to the difficulty of implementing mixed-precision calculation within a single tensor. More recently, [12] extends OBQ [10, 16, 21] on LLMs for INT4 weight-only quantization and shows great efficiency on quantization and latency, and [35] shows the outliers from activations can be smoothed out by migrating the quantization difficulty from activations to its associated weights. However, [35] can only work for W8A8 quantization as lower weight precision (INT4) itself already leads to significant accuracy degradation, and the accuracy drop is larger than 0.1 PPL points, which as discussed in the later section is sub-optimal. [7] shows the scaling law of weight-only quantization with the simplest round-to-nearest baseline, but it does not consider the weight-and-activation quantization and/or the above PTQ optimization methods. As can be seen from Figure 1, by using PTQ optimization methods, the model quality can be significantly improved. Please also see Appendix E for more detailed numbers.

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

Different than existing works, our paper extensively tests the effect of (1) different quantization schemes, e.g., symmetric and asymmetric quantization, (2) different PTQ methods, e.g., [36, 12], (3) different model families, e.g., [28, 40], (4) different quantization coverage, e.g., weight-only and weight-and-activation quantization, and (5) other discussions, e.g., the effect of quantization granularity. As such, we provide a much more comprehensive understanding of post-training quantization for large language models compared to the previous works.

3 Would different model families behave similarly on quantization?

There are mainly two categories of PTQ for LLMs, i.e., weight-only quantization [12] and weightand-activation quantization [6, 36, 35]. In the latter, it is uniformly observed across all studies that activation quantization demonstrates greater sensitivity than weight quantization. However, prior research tends to concentrate on a single (family) model to emphasize the necessity of their proposed quantization technique. A comprehensive and systematic evaluation of this PTQ methodology, particularly the sensitivity of weight/activation quantization for varying model sizes and distinct model families, has yet to be undertaken. Hence, we conduct an examination on both the OPT [40] and BLOOM [28] families to elucidate the quantization sensitivity of weight and activation.

Sensitivity setting. We use the zero-shot validation perplexity (PPL) differential on three datasets, namely, Wikitext-2 [23], PTB [22], and C4 [27], before and after the quantization of these LLMs to illustrate their sensitivity, as PPL is significantly correlated to zeroshot/few-shot accuracy measurement [7]. Specifically, a higher PPL drop indicates enhanced quantization sen-

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

sitivity. For simplicity, we also categorize quantization sensitivity (or quantization loss) into three different classes as depicted in Table 1. Notably, the threshold is chosen because when the model size approximately doubles (e.g., 13B vs. 30B, and 30B vs. 66B), the PPL improvement is about 0.5 (see Table 2). The sensitivity (or loss) incrementally increases as the class number ascends. From a practical standpoint, we favor lower quantization sensitivity (accuracy loss), making Class -1 the optimal-loss post-training quantization.

We employ both symmetric and asymmetric quantization to gauge the quantization sensitivity and highlight the advantage of asymmetric quantization. Particularly, we implement per-row quantization [12] for weight quantization and per-token quantization for activation [36].

Robustness of Weight-only Quantization for Large Models. The results of weight-only quantization in OPT and BLOOM models are summarized in Table 2. INT8 weight-only quantization, either symmetric or asymmetric, results in negligible accuracy loss (less than 0.05, i.e., Class -1). Consequently, for tasks oriented towards generation, FP16 weight can simply be replaced with INT8 weight to reduce memory usage. For INT4 quantization, the asymmetric method outperforms the symmetric approach in accuracy, attributable to its superior utilization of the quantization range. Interestingly, larger models exhibit better tolerance to low-precision quantization (i.e., INT4) than smaller models, with a few exceptions such as OPT-66B. 1 Particularly, BLOOM-176B shows PPL degradation (around 0.3 points) in Class -2, which could explain why the large GLM-130B [39] can operate with INT4 weight-only quantization out of the box with acceptable accuracy impact.

Challenge Encountered in Activation Quantization for Large Models. Activation quantization has consistently proven more difficult than weight quantization [36, 6], as illustrated in Table 2. When compared to weight-only quantization, activation-only quantization indicates that asymmetric quantization can significantly improved performance over symmetric quantization. Moreover, contrary to weight-only quantization, smaller models typically exhibit better tolerance to activation quantization, as their hidden dimension is smaller and the activation dynamic range is also narrower than larger models [36]. It should be noted that for models larger than 10B, all fall into Class -3, indicating a degradation of more than 0.5 PPL points.

The last two rows of Table 2 show that different model families exhibit significantly different behaviors. BLOOM does not exhibit divergence issues even up to a model size of 176B, whereas OPT displays very poor performance from a model size of 6.7B (larger models with INT8 activation have even worse PPL). This could again be attributed to the Layer Norm issue within the OPT-family 1 .

Findings 1 on Sensitivity Analysis. (1) INT8 weight-only quantization can serve as a standard method for reducing memory costs in LLMs, with negligible degradation in accuracy. (2) INT4 weight-only quantization for small models results in substantial accuracy degradation ( Class -3), but this effect lessens as the model size increases ( Class -2). (3) Contrary to (2), INT8 activation results in minimal accuracy drops for small models ( Class -1) but larger models exhibit greater drops ( Class -3). (4) With INT8 activation, BLOOM shows no divergence issues up to a model size of 176B, whereas OPT performs poorly from ≥ 6.7B model sizes.

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

4 Are existing quantization methods optimally harnessing the potential to minimize LLMs sizes?

Numerous lightweight optimization-based methods have been proposed, which update the model weights during quantization. These methods such as [36, 12, 35], unlike quantization-aware training, only require a small portion of the training data and a limited training time. Particularly, GPTQ [12] and ZeroQuant [36], have proven to be effective and efficient in terms of GPU resources, time cost, and data usage for INT4 weight quantization. 2 In this work, we focus on the variants of GPTQ and ZeroQuant as well as the most straightforward baseline, round-to-nearest neighborhood (RTN).

RTN directly applies PTQ on the trained data and follows the procedure detailed in Section A to perform the quantization. Specifically, for symmetric quantization, we set S = max ( abs ( x )) and Z = 0 ; for asymmetric quantization, we set S = max ( x ) -min ( x ) and Z = min ( x ) .

GPTQ extends the OBQ [10]. It tries to optimize the following non-linear least square problem, min ˆ W ∥ Wx -ˆ Wx ∥ 2 2 where W is the weight, x is the activation, and ˆ W is a quantized weight. GPTQ employs second-order methods to obtain a closed-form solution. In addition, the quantization for each weight matrix is performed column-/row-wisely and the quantization errors from previous columns will be passed to those columns not yet quantized. See[10, 12] for more details.

ZQ-Global is the original method proposed in [36], where authors treat each layer as a small neural network (a.k.a., subnetwork) and use the FP16 subnetwork as the teacher model to distill the quantized one with a few hundred iterations, i.e., min ˆ θ | f θ ( x ) -f ˆ θ ( x ) | 2 2 , where θ is a set of weights, ˆ θ is the quantized version, fθ is the subnetwork with parameters θ , and x is the input. Thus, it can significantly reduce the GPU resource requirement and time cost.

ZQ-Local is an extension mode of ZQ-Global for further GPU requirement reduction and training cost reduction. Particularly, instead of using each transformer layer as the subnetwork, we treat each linear layer as the subnetwork. This method can be viewed as an iterative first-order optimization method (e.g., SGD) to solve min ˆ W ∥ Wx -ˆ Wx ∥ 2 2 .

Experimental Setup. We compare the four methods mentioned above on weight-only and weightand-activation quantization. As weight quantization is always static (i.e., it does not change during inference), there is virtually no system performance difference between symmetric and asymmetric quantization. 3 We use asymmetric quantization for better accuracy, and the conclusions would hold similarly for symmetric quantization. For parameters used for GPTQ, ZQ-Local, and ZQ-Global, please refer to Appendix B. An interesting finding for ZeroQuant is that the hyperparameters (e.g., learning rate and its scheduler) provided in the original work [36] are sub-optimal. In this work, we find the best configurations for ZQ-Local and ZQ-Global and denote them as ZQ-Local ∗ and ZQ-Global ∗ , respectively, with the best tuned results. To ensure consistent and comparable results, we set a fixed random seed for our experiments. In the context of post-training quantization, varying the random seed has minimal impact on the final results, as indicated in more detail in Table B.1.

Evaluation of Weight-only Quantization. The results from weight-only quantization using OPT and Bloom are presented in Table 3. The findings indicate that the larger models tend to be less sensitive to INT4 weight-only quantization. This observation holds true across all methods (RTN, GPTQ, ZQ-Local ∗ , and ZQ-Global ∗ ) with the exception of OPT-66B, which shows greater degradation than OPT-30B. It is noteworthy that light-weight optimization-based methods significantly outperform the RTN baseline in terms of accuracy. For instance, these methods substantially reduce the degradation in perplexity of OPT-30B/66B compared to baseline. Most quantized models with parameters greater than 6.7B fall under Class II, indicating their potential for real-world applications. For instance, the quality of INT4 OPT-30B (66B) is superior to that of INT8 OPT-13B (30B).

Among the optimization-based methods, ZQ-Global ∗ generally performs better on smaller models (those with fewer than 1B parameters), while GPTQ excels on larger models. ZQ-Local ∗ does not outperform GPTQ or ZQ-Global ∗ -- a reasonable outcome given that GPTQ employs a closed-form solution to solve the non-linear quadratic problem and ZQ-Global ∗ optimizes a larger subnetwork. The inferior performance of ZQ-Global ∗ compared to GPTQ for larger models is unexpected since ZQ-Global ∗ optimizes an entire transformer layer while GPTQ only optimizes a single linear layer.

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

A plausible explanation is that larger models are more sensitive to weight updates, necessitating more advanced fine-tuning methods.

Evaluation of Weight and Activation Quantization. The evaluation results for existing methods using W4A8 quantization are presented in Table 3. The three light-weight optimization-based methods outperform RTN significantly, underscoring their efficacy. However, all of the results fall into either Class -2 or Class -3. This suggests that for certain applications, it might be more beneficial to use smaller models with fewer parameters rather than larger, quantized models.

Among quantization-based methods, ZQ-Global ∗ and ZQ-Local ∗ generally outperform GPTQ, which is anticipated given that GPTQ was originally designed for weight-only quantization. ZQ-Global ∗ performs better than ZQ-Local ∗ in most cases except for the two largest models, OPT-66B and Bloom-176B, despite having larger trainable parameters in one step. This again signifies the need for a more suitable and advanced optimization method for large language models (LLMs).

Finding 2 on Comparisons. (1) GPTQtypically performs better for weight-only quantization, while ZeroQuant (including both ZQ-Global ∗ and ZQ-Local ∗ ) yields superior results for weight and activation quantization. (2) The tested optimization-based methods cannot achieve Class -1 quantization error for either INT4 weight-only or W4A8 quantization with the exception of GPTQ on OPT-30B with weight-only quantization.

4.1 Fine-grained Quantization and Its Evaluation

With PTQ and row-wise quantization, achieving Class -1 quantization error is challenging for both weight-only and weight-and-activation quantization. Generally, utilizing a smaller model with INT8 weight is more advantageous than employing a model that is twice as large with INT4 weight.

One potential solution to this issue is the implementation of finer-grained quantization schemes [5], where every k elements possess their own scaling factor and/or zero point. This approach can significantly reduce quantization error. In the extreme case, where every single element has its own scaling factor, the original FP16 number can be precisely recovered. Importantly, block-k quantization can be implemented on modern GPUs, one of the most prevalent deep learning architectures, since the compute unit (streaming multiprocessor) of GPUs processes tiles of data (e.g., 128 by 128 tiling size) for matrix computation.

Although fine-grained quantization can substantially narrow the gap between the quantized tensor and its floating-point counterpart, the application of RTN still results in a non-trivial accuracy gap. Consequently, we build upon fine-grained quantization by employing existing optimization-based methods to further enhance accuracy. Specifically, we utilize GPTQ and ZQ-Global for all models and settings and apply ZQ-Local to OPT-66B and Bloom-176B. For the hyperparameters used in ZQ-Global and ZQ-Local, we select the top three identified in Section 4 for all models, except for Bloom-176B, for which we only use the top-performing hyperparameter to reduce training costs.

4-bit Weight Quantization. We hereby present the W4A16 results for OPT and BLOOM, as delineated in Table 4, corresponding to an array of quantization block sizes. The performance sees a significant improvement with smaller block sizes compared to per-row quantization. The point of diminishing returns, however, varies for different model sizes. For example, smaller models (such as OPT-6.7B and BLOOM-1.7b) continue to see substantial gains until the block size reduces to 32. In contrast, for larger models (those exceeding 10B, with OPT-66B as the excep-

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

tion), the benefits derived from smaller block sizes wane rapidly around block-256/512. Most crucially, for models equal to or larger than 13B, a smaller quantization block size results in quantization error being classified under Class -1, indicating virtually negligible degradation in accuracy.

Activation Quantization (W4A8). To comprehend the benefits of fine-grained quantization on activation, we analyze the quantization between per-row and a block size of 128, with INT4 weight, as highlighted in Table 5. For models of considerable size, specifically those equal to or exceeding 1B, the application of such fine-grained activation quantization (Case-1) results in a

Table 6: BLOOM-176B with different quantization block sizes on activation. Here weight is asymmetrically quantized with block size 128. See more in Table E.22.

substantial reduction in quantization error compared to per-row activation (Case-2). By implementing fine-grained activation quantization with weight quantization (Case-3), we are able to almost restore the performance to the level of their W4A16 counterparts.

Furthermore, we detail the impacts of varying activation quantization block sizes in Table 6 on BLOOM-176B, with INT4 weight. A trend of superior accuracy is observed with smaller block sizes in contrast to larger ones. However, the enhancement in performance reaches a saturation point when the size smaller or equal to 256, which corresponds to the range of values INT8 can represent. Despite INT8's capability to signify 256 distinct values, activation quantization errors persist due to the application of uniform quantization.

Finding 3 on FGQ. (1) Larger models ( ≥ 10B) are capable of attaining Class -1 error for 4-bit quantization. These models can leverage low-precision quantization as the model size with INT4 is similar to an INT8 model that is half its size, with improved accuracy. On the other hand, smaller models ( ≤ 10B) typically reach only Class -2 or Class -3 error levels. (2) For larger models (>10B), the difference between fine-grained weight-and-activation quantization and fine-grained weight-only quantization is insignificant. (3) The advantage of fine-grained activation quantization fades for larger models when the block size reaches 256.

5 Proposed Method to Further Push the Limit of Post-training Quantization

Building on the investigation and conclusions drawn from previous sections, it has become apparent that there is still a need for an advanced methodology to further refine the existing methods, with the objective of fully realizing the original fp16 PPL quality. In this section, we introduce a simple yet effective method called LoRC (Low Rank Compensation) to optimize the current existing quantization error and further bridge the gap between the quality of the original model and its quantized counterparts.

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

LoRC is inspired by the employment of low-rank matrix factorization on the quantization error matrix E := W -ˆ W , where W represents the original weight and ˆ W is the quantized weight. LoRC approximates the error E with ˆ E = ˆ U ˆ V by using two low-rank matrices ˆ U and ˆ V . This results in a more accurate approximation of the original weight matrix W by ˆ W lorc = ˆ W + ˆ E , thereby reducing quantization errors: ∥ W -ˆ W ∥ ≥ ∥ W -ˆ W lorc ∥ . LoRC consists of two steps:

Step I: Implement Singular Value Decomposition (SVD) on the error matrix E = U Σ V , where U ∈ R d in × d in and V ∈ R d out × d out are unitary matrices, and Σ ∈ R d in × d out is a diagonal matrix with its diagonal elements ordered in a descending manner.

The objective of LoRC is to achieve a good approximation of the error matrix E using low-rank matrices, with minimal impact on the increase in model size. For instance, consider the standard transformer models [32], where each layer is comprised of a multi-headed attention (MHA) module and a multi-linear perception (MLP) module. Let h represent the hidden dimension and l the number of layers. The total number of parameters is 12 lh 2 as each layer contains 4 h 2 for MHA (for key, query, value, and projection matrices), and 8 h 2 for MLP (two matrices of sizes h × 4 h and 4 h × h ). With the addition of low-rank LoRC to the six matrices in each layer, the total number of parameters for l layers would amount to 18 hml . 4 Consequently, the ratio of parameters added to the existing model is 3 m/ 2 h . It's important to note that the low-rank dimension m can be as small as 4 or 8 (which we will discuss in detail in a later section) while the standard hidden dimension h ≥ 768 , making the number 3 m/ 2 h ≤ 0 . 016 .

Significantly, LoRC can be viewed as a supplementary feature to existing quantization methodologies such as RTN, GPTQ, and ZeroQuant-local/Global, and can be seamlessly integrated with FGQ. We have conducted experiments to evaluate the performance of LoRC on both OPT and BLOOM, applying 4-bit, 3-bit, and 2-bit weights by setting the activation to FP16. 5 Based on the discoveries in the preceding sections, we utilize the GPTQ quantization strategy. To gain a comprehensive understanding of LoRC, we include the results with and without the application of FGQ. The datasets and hyperparameters are consistent with those detailed in earlier sections.

Evaluation Results. The findings are showcased in Table 7, split into two sections: coarse-grained weight quantization (per-row) and fine-grained quantization (block-size 256). Notably, we observe that the two low-rank matrices, ˆ U and ˆ V , can be quantized to 8-bit without any performance discrepancy (Table 8). Thus, the two low-rank matrices for LoRC in Table 7 are INT8 with a low-rank dimension of m = 8 .

Table 8: Results of W4 asym A16 quantization with LoRC approximating ˆ E = ˆ U ˆ V on OPT model family. ˆ U and ˆ V can be represented with FP16 or INT8, of which the performance are represented below. There is hardly any difference between FP16 and INT8.

Several key observations can be made. Firstly, LoRC consistently boosts performance across all bit sizes and block sizes, as indicated by the lower perplexity scores when LoRC is activated. Secondly, the enhancement brought about by LoRC becomes more substantial as the bit size diminishes, especially noticeable for W2A16, which displays a markedly greater impact compared to W4A16 and W3A16 in most scenarios. Lastly, the

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

combination of fine-grained quantization with LoRC yields the most impressive results, underscoring the efficacy of LoRC when integrated with FGQ. Overall, the results emphasize the benefits of using LoRC for enhanced performance in weight quantization and its compatibility with FGQ. Notably, recovering the last 0.05-0.1 perplexity can be challenging, but with LoRC, we are able to nearly recover the original model quality for INT4 quantization.

Ablation Study on the Low Rank Dimension m . An essential aspect of the LoRC method is on the optimal low-rank dimension, denoted as m , explained in Step II . To explore this, we varied m in the range of 1, 4, 8, 16, and 32 for OPT-1.3b/6.7b/30b models, and applied W4A16 GPTQ quantization. The outcomes are depicted in Table 9, indicating that the enhancements achieved through LoRC begin to plateau as the dimension m surpasses 4. The most optimal performance for OPT-6.7b is realized when m = 8 .

This observation may seem counterintuitive initially, as one might anticipate that larger LoRC dimensions would yield more significant improvements. To gain a more comprehensive understanding, we conducted an analysis of the eigenvalues of the actual error matrix E = W -ˆ W for each matrix. By randomly selecting 20 matrices from MHA and MLP layers, we plotted the eigenvalues of E as a curve, depicted in Figure 2. The two plots reveal a rapid flattening of eigenvalues after index 8, which elucidates why increasing the LoRC dimension does not considerably enhance performance. Hence, a sensible dimension for ˆ U and ˆ V in the LoRC methodology could be 8. 6

6 Discussion

Conclusion. In this work, we provide a comprehensive study of post-training quantization (PTQ) on large language models with different PTQ methods (e.g., RTN, GPTQ, ZeroQuant), and with different quantization coverage (weight-only and weight-and-activation quantization), etc. We find that PTQ methods are critical to improving the quantized model quality, and that fine-grained quantization (FGQ) can bring acceptable accuracy and model size trade-off. Finally, we introduced an optimization technique called Low Rank Compensation (LoRC), which works synergistically with PTQ and FGQ, playing a crucial role in enhancing full model quality recovery with a minimal increase in model size.

Limitation. Despite quantizing over 10,000 experiments, our study was constrained by our computing resources. This restriction made us choose between diversifying the model sizes and varying the tasks. We strategically limited our datasets to WikiText, PTB, and C4 to concentrate on a broad range of quantization methods. Consequently, our general findings are more robust concerning the two model families and three datasets examined in this paper. However, caution should be exercised when generalizing these findings to tasks that are dissimilar to those covered in this study.

Future Opportunity. Throughout the paper, we see several unresolved problems from current quantization schemes and/or algorithms, and we find potential directions for LLM compression: (1) Although we use fine-grained quantization schemes in the paper, the real implementation is missing. How to efficiently implement odd bit precision is also challenging. [12] demonstrated that 3-bit can achieve better throughput in the generation phase by packing all 3-bit numbers in continuous memory space. However, this method is sub-optimal as the dequantization step needs to connect bits from different bytes. One possible way to implement odd bits, e.g., 5 bits, is to use two integer matrices with INT4 and INT1. During the dequantization stage, we couple the two matrices together. (2) How to combine PTQ with other lightweight compression techniques, e.g., post-training pruning [20, 11], is an interesting direction to further reduce the memory consumption and compute cost.

References 352

Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and 353 Irwin King. Binarybert: Pushing the limit of bert quantization. arXiv preprint arXiv:2012.15701 , 354 2020. 355

Big-Science. Bloom inference. https://github.com/huggingface/transformers-blo 356 om-inference/tree/main/bloom-inference-scripts , 2022. 357

Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming 358 the challenges of efficient transformer quantization. arXiv preprint arXiv:2109.12948 , 2021. 359

Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, 360 Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are 361 few-shot learners. arXiv preprint arXiv:2005.14165 , 2020. 362

Bita Darvish Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, 363 Anna Vinogradsky, Sarah Massengill, Lita Yang, Ray Bittner, et al. Pushing the limits of 364 narrow precision inferencing at cloud scale with microsoft floating point. Advances in neural 365 information processing systems , 33:10271-10281, 2020. 366

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix 367 multiplication for transformers at scale. arXiv preprint arXiv:2208.07339 , 2022. 368

Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. 369 arXiv preprint arXiv:2212.09720 , 2022. 370

Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar371 mendra S Modha. Learned step size quantization. arXiv preprint arXiv:1902.08153 , 2019. 372

Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Remi Gribonval, Herve Jegou, 373 and Armand Joulin. Training with quantization noise for extreme fixed-point compression. 374 arXiv preprint arXiv:2004.07320 , 2020. 375

Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post376 training quantization and pruning. arXiv preprint arXiv:2208.11580 , 2022. 377

Elias Frantar and Dan Alistarh. Massive language models can be accurately pruned in one-shot. 378 arXiv preprint arXiv:2301.00774 , 2023. 379

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training 380 quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 , 2022. 381

Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. 382 A survey of quantization methods for efficient neural network inference. arXiv preprint 383 arXiv:2103.13630 , 2021. 384

Amir Gholami, Zhewei Yao, Sehoon Kim, Michael W Mahoney, and Kurt Keutzer. Ai and 385 memory wall. RiseLab Medium Post , 2021. 386

GitHub. Github copilot. https://github.com/features/copilot/ , 2021. 387

Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain 388 surgeon. In Advances in neural information processing systems , pages 164-171, 1993. 389

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. 390 Workshop paper in NIPS , 2014. 391

Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and 392 Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint 393 arXiv:1909.10351 , 2019. 394

Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. I-bert: 395 Integer-only bert quantization. In International conference on machine learning , pages 5506396 5518. PMLR, 2021. 397

Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and 398 Amir Gholami. A fast post-training pruning framework for transformers. arXiv preprint 399 arXiv:2204.09656 , 2022. 400

401

402

Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems , pages 598-605, 1990.

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Using Large Corpora , page 273, 1994.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations , 2017.

OpenAI. Openai chatgpt. https://openai.com/blog/chatgpt/ , 2022.

Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization. arXiv preprint arXiv:1802.05668 , 2018.

Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. arXiv preprint arXiv:2211.05102 , 2022.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 , 2022.

Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-BERT: Hessian based ultra low precision quantization of bert. In AAAI , pages 8815-8821, 2020.

Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990 , 2022.

Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. Compression of generative pre-trained language models via quantization. arXiv preprint arXiv:2203.10705 , 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pages 5998-6008, 2017.

Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, and Yuxiong He. Understanding int4 quantization for transformer models: Latency speedup, composability, and failure cases. arXiv preprint arXiv:2301.12017 , 2023.

Xiaoxia Wu, Zhewei Yao, Minjia Zhang, Conglong Li, and Yuxiong He. Extreme compression for pre-trained transformers made simple and efficient. arXiv preprint arXiv:2206.01859 , 2022.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10438 , 2022.

Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. arXiv preprint arXiv:2206.01861 , 2022.

444

445

446

447

448

449

450

451

452

453

454

455

Ali Hadi Zadeh, Isak Edo, Omar Mohamed Awad, and Andreas Moshovos. Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) , pages 811-824. IEEE, 2020.

Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8BERT: Quantized 8bit bert. arXiv preprint arXiv:1910.06188 , 2019.

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414 , 2022.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. Ternarybert: 456 Distillation-aware ultra-low bit bert. arXiv preprint arXiv:2009.12812 , 2020. 457

Peer reviews:
comments: SUMMARY:
This paper conducts a comprehensive analysis of various quantization methods for large language models (LLMs). Some interesting takeaways were shared, for example, activation quantization is generally more susceptible to weight quantization; none of the current quantization methods can achieve the original model quality. Based on such insights, the paper also proposes an optimized method called Low-Rank Compensation (LoRC), which employs low-rank matrices to enhance model quality recovery with a minimal increase in model size.

STRENGTHS:
- The motivation of the paper is solid. With the rapid development of LLM, it is essential to study the methodology to deploy LLM on more accessible hardware, where quantization is an important category of the approach. Therefore, a comprehensive study of these methods is necessary.  

- The insights shared by this paper are helpful, e.g., it is interesting to know activation quantization is generally more susceptible to weight quantization.


- The proposed improvement based on the observation is reasonable.

WEAKNESSES:
- The proposed method is based on low-rank approximation, which can be viewed as a sparsification-based method. It is kind of out of the scope of the proposed method.

QUESTIONS:
- Is it possible to open-source the code for reproducibility?

LIMITATIONS:
Not applicable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

comments: SUMMARY:
This paper analyzes post-training quantization (PTQ) techniques in large language models, exploring various schemes, model families, and bit precision. The authors propose an optimized method called Low-Rank Compensation (LoRC) to enhance model quality recovery with minimal size increase.

STRENGTHS:
1. An evaluation and comparison of existing PTQ methods provide some insights for the community.
2. The paper is well-written.

WEAKNESSES:
Reading from paper provides a satisfying experience, but I must admit that my understanding of the LLM field is limited. Thus, my suggestions may be wrong, please directly point them out.

1. The points in Figure 1 are too dense and lack recognition.
2. Although the method is proposed for LLMs, it's also better to compare it with some traditional quantization methods on ResNet-series.

QUESTIONS:
The authors could refer to the Weaknesses.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

comments: SUMMARY:
This paper studied the post-training quantization method for 4-bit weight quantization and W4A4 quantization. The authors further proposed a Low-Rank Compensation (LoRC), to enhance model quality with low-rank matrices.

STRENGTHS:
The paper is well-written and easy to follow.

WEAKNESSES:
The novelty is not very significant. The post-training quantization with finer-grained and zero-shift is not a new idea.

The sensitivity analysis is conducted with one particular quantization method, and the conclusion should be conditioned on that quantization method, but not general enough to conclude that PTQ exhibits the same behavior. I would suggest the authors to compare different quantization functions, such as minmax/percentile to deliver a more comprehensive conclusion.

The accuracy improvement is marginal. Figure 1 didn’t show much improvement compared to the previous naïve baseline of RTN.

QUESTIONS:
Have the authors tried other different baseline quantization method, e.g., minmax/percentile quantization?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on systematic examination of various post training quantization techniques in large language models. Experimental analysis include comparison of different model sizes, different numerical precision, and quantization of only weights vs activations. In addition, the Low Rank Compensation method is proposed to enhance model quality recovery.\n\nSTRENGTHS:\nThe research on LLMs is rapidly growing, but computational and memory capabilities are still limited when deploying huge models. Therefore, quantization is a must. The analysis presented in this work is definitely needed to advance LLMs deployment in multiple use cases. The paper flows logically and is well structured. The proposed compensation method is interesting.\n\nWEAKNESSES:\n1. Quantization may be input data specific. Same model topology when trained on different data may behave differently. It’s mentioned that you “use the zero-shot validation perplexity (PPL) differential on three datasets, namely, Wikitext-2 [23], PTB [22], and C4 [27], before and after the quantization”, however results presented in the following tables doesn’t indicate what error was achieved on which dataset. Could you clairfy?\n2. Quantization may be operator specific. It would be interesting to list operators present in both model topologies and identify layers that were specifically sensitive to quantization.\n\nQUESTIONS:\n1. Please clarify the finding about the layer norm in the OPT family. Were all weights and biases 1 and 0 in all layer norm layers? Are you sure the model was initiated correctly?\n2. Does you compensation technique require any fine tuning? It wasn’t clear from the text. \n3. How does your compensation technique compare to e.g., LAPQ https://arxiv.org/pdf/1911.07190.pdf where quantization error is used for layer smoothing in a similar way.\n4. Does your compensation technique work in a layer-by-layer manner? How can you ensure that error doesn’t accumulate across layers?\n\nLIMITATIONS:\nListed limitations and future work directions are clear and make sense.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}