Title: decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Points
Venue: NeurIPS 2024
Accepted: no
Keywords: quantization; large language model; optimization
Abstract: Quantization emerges as one of the most promising compression technologies for deploying efficient large models in recent years. 
However, existing quantization schemes suffer from significant accuracy degradation at very low bits, or require some additional computational overhead when deployed, making it difficult to be applied to large-scale applications in industry.
In this paper, we propose decoupleQ, achieving a substantial increase in model accuracy, especially at very low bits.

decoupleQ abandons the traditional heuristic quantization paradigm and decouples the model parameters into integer and floating-point parts, then transforming the quantization problem into a mathematical constrained optimization problem, which is then solved alternatively by off-the-shelf solution methods. decoupleQ gets rid of any tricks for dealing with outliers, sensitive channels, etc., and focuses only on the basic optimization objective to achieve high model accuracy on extreme low bit quantization.
 Quantization via decoupleQ is linear and uniform, making it hardware-friendlier than non-uniform counterpart, and enabling the idea to be migrated to high-bit quantization to enhance its robustness. 
 
decoupleQ has achieved comparable accuracy as fp16/bf16 for 2-bit quantization of large speech models in our company.
The code (including the W2 CUDA kernels) is attached and will be made public.

1

Abstract

Quantization emerges as one of the most promising compression technologies for deploying efficient large models in recent years. However, existing quantization schemes suffer from significant accuracy degradation at very low bits, or require some additional computational overhead when deployed, making it difficult to be applied to large-scale applications in industry. In this paper, we propose decoupleQ, achieving a substantial increase in model accuracy, especially at very low bits.

decoupleQ abandons the traditional heuristic quantization paradigm and decouples the model parameters into integer and floating-point parts, then transforming the quantization problem into a mathematical constrained optimization problem, which is then solved alternatively by off-the-shelf solution methods. decoupleQ gets rid of any tricks for dealing with outliers, sensitive channels, etc., and focuses only on the basic optimization objective to achieve high model accuracy on extreme low bit quantization. Quantization via decoupleQ is linear and uniform, making it hardware-friendlier than non-uniform counterpart, and enabling the idea to be migrated to high-bit quantization to enhance its robustness.

decoupleQ has achieved comparable accuracy as fp16/bf16 for 2-bit quantization of large speech models in our company. The code (including the W2 CUDA kernels) is attached and will be made public.

1 Introduction

Serving large models (1; 2; 37; 38) in industry is budget-consuming because of the huge computational, IO and storage cost. Model compression (10; 11; 16) has therefore become a necessity to alleviate this pain. Among which, Post-Training Quantization (PTQ) (9; 26) has gained more and more popularity among researchers and engineers because it does not require heavy GPU-hours training with labeled datasets.

However, previous quantization schemes remain confined within the traditional heuristic quantization 25 paradigm, e.g., how to deal with outliers (33; 35), how to deal with sensitive channels (6), how 26 to determine the clipping range (29), and so on. These methods have achieved some success, but 27 the quantization at extreme low bit often suffers from significant accuracy degradation, thus failing 28 to meet the launching requirements of industrial practice. There are also some other options to 29 mitigate the accuracy loss. QuIP (4) pushes the accuracy limits of 2-bit quantization and can achieve 30 performance close to fp16/bf16. However, compared to traditional quantization schemes, its inference 31 imposes an additional burden due to the need to multiply two random orthogonal matrices to de32 quant the weights. N2UQ (20) fit the real-value distribution with non-uniform grids then quantize 33 them into equidistant output levels. But it need to train to get the input thresholds. SpQR (7) 34

and SqueezeLLM (14) use mixed-precision quantization or non-uniform scheme to safeguard the 35 important channels, but they need customized hardware support. 36

In order to alleviate the above pains in industry, we proposed decoupleQ, which completely abandons 37 the traditional heuristic quantization paradigm and instead decouples the model parameters into 38 integer and floating point parts, then transforming the quantization problem into a mathematical 39 constrained optimization problem, which is then solved alternatively by off-the-shelf solution methods. 40 The integer part contains the main weights of the model, and the floating-point part contains scales 41 and zero points induced via quantization. decoulpeQ starts from an abstract objective function and 42 thus does not need any tricks to deal with the minutiae of traditional quantization paradigm, such as 43 outlier, salient weights (19), and so on. Quantization via decoupleQ is linear and uniform, making it 44 hardware-friendlier than non-uniform counterpart, and enabling the idea to be migrated to high-bit 45 quantization to enhance its robustness. 46

47

48

49

decoupleQ contains two stages: 1. layer-wise minimization, defined in Eq. 1, is used to optimize the integer part and the floating-point part; 2. block-wise minimization, defined in Eq. 2, is used to further optimize the floating-point part while freezing the integer part 1 .

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

Layer-wise minimization is to minimize the ℓ 2 loss of the outputs between pre- and post-quantization for a linear layer:

where X ∈ R batch × d in is the input of this layer, W 0 ∈ R d in × d out is the pre-trained full precision weight, d in and d out are the input and output dimensions respectively. The objective is to find a matrix ˜ W with quantized-then-dequantized elements to minimize Eq. 1.

Some works (4; 8; 9; 13; 25) started from Eq. 1 and achieved some success, but they still haven't thought outside the box of traditional quantization. GPTQ series (8; 9) fake-quantize the first element of W 0 and then update the the remaining elements so as to keep Eq. 1 minimized. This process is then continued element by element until all elements are fake-quantized. However, on the one hand, they do not give any indication of how scale and zero point should be calculated, and on the other hand, the optimization problem formulated for updating the remaining elements is unconstrained (explained in detail later). decoupleQ models Eq. 1 as a constrained optimization problem, as shown in Eq. 6. It no longer needs to pay attention to some of the minutiae unique to quantization, such as outliers, clipping threshold, etc., but abstracts the essence of the problem from a higher level.

In the second stage, block-wise minimization is used to further improve the model accuracy:

where ˜ Block ( · ) is a common transformer block (32) with quantized weights. In this stage, we freeze the integer part of the weights, and train the scales, zero points and norm layers.

decoupleQ implements 2-bit uniform quantization and achieves state-of-the-art accuracy in Llama1/2 (30; 31). Like traditional uniform quantization, decoupleQ does not incur additional inference burden and only requires a linear transformation to convert the quantized weights into floating point ones.

Our main highlights are summarized as follows:

New insight: We abandoned the traditional quantization paradigm, and no longer need to focus on some of the minutiae unique to quantization, but abstracts the essence of the problem from a higher level and transforms it into a constrained optimization problem.

Extreme low-bit: decoupleQ achieves 2-bit uniform quantization with performance matching fp16/bf16 for industrial applications in the ASR model in our company, and we will also release the W2A16 CUDA kernel as one of our core contribution.

Extensibility: As a bonus, if labeled datasets are available, the idea of decoupleQ can be easily extended to supervised fune-tuning (sft) to further improve model accuracy, or the adaptation to the downstream sub-tasks.

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

2 Related Works

Quantization can be roughly divided into Quantization Aware Training (QAT) (21; 33) and PostTraining Quantization (PTQ) (4; 35). In this paper, we focus on weight-only quantization in PTQ, and we will only summarize a few works that are closely related to our work.

PTQ is commonly used for LLM quantization because it does not require a lot of GPU hours of training with labeled datasets. AdaRound (25) and BRECQ (18) start from the rounding operation and explore whether to round up or down is better. SqQR (7) and OWQ (17) use mixed-precision quantization strategy to protect sensitive parameters, while AWQ (19) opts for scaling up the weights of sensitive channels to reduce the loss of quantization of sensitive channels. OmniQuant (29) use gradient decent to optimize for the weight clipping threshold and the rescale factors. In decoupleQ, we abandon patchwork solutions and transform the quantization into a principled traditional optimization problem by decoupling the model parameters into integer and floating-point parts.

GPTQ (9) is an influential work, and it quantizes the current weights and then updates the remaining weights to minimize the ℓ 2 loss of the output of the layer between pre- and post-quantization. As we will see later, this update actually approximates much, and GPTQ does not optimize for the scale and zero point reduced by quantization.

QALora (36) also decouples model parameters at a certain level and uses labeled datasets to fine-tune the zero points. decoupleQ takes this idea a step further, optimizing the scales, zero points and norm layers with supervised fine-tuning, while freezing the integer weights.

3 Methods

3.1 Preliminaries

For a linear layer with input dimension d in and output dimension d out , quantization maps the weights with high-precision into discrete level, and the previous scheme can be described as follows:

where W 0 ∈ R d in × d out is the pre-trained full precision weights, s and z are the scale and zero point (what we call floating-point part above), ⌊·⌉ is the round-to-nearest function, ̂ W ∈ R d in × d out is the quantized integer-point matrix (what we call integer part above), ˜ W is the de-quantized floating-point matrix, α and β are the lower and upper bounds of the range of integer representations, respectively. For example, in 2-bit weight only linear quantization scheme, the value of each entry of ̂ W is limited to one of {-2 , -1 , 0 , 1 } , and α = -2 , β = 1 in this case. To get the values of ˜ W , previous methods (8; 9) show that layer-wise ℓ 2 loss between the outputs pre- and post-quantization is well related to the model accuracy, i.e., to optimize the following objective function,

where X ∈ R batch × d in is the input of this linear layer, generated by a small set of calibration dataset, and H = X T X .

In the very low-bit quantization regime, the model accuracy can be further improved via finer-grained grouping. This would impose additional overhead on inference. For example, when groupsize = 64 , it imposes an average overhead of 0.5 bit per element (FP16/BF16 for scale s and zero point z ). The extra overhead is acceptable compared to the model accuracy gain.

3.2 decoupleQ

When a model is quantized, only the integer part ̂ W and the floating-point part ( s, z ) in Eq. 4 are delivered to the downstream inference engine, and the inference process does not need to know how ̂ W and ( s, z ) are obtained at all. That is, if we can find the values of ̂ W and ( s, z ) to minimize Eq. 5 by other methods, then we don't need to use Eq. 3. So, we can decouple the model parameters into integer part ̂ W and floating point part ( s, z ) , which are then optimized alternatively via off-the-shelf

125

126

127

solution methods. decoupleQ views the process of solving for ̂ W and ( s, z ) in Eq. 4 as a constrained optimization problem independent of the previous quantization paradigm! We only need to regard Eq. 4 as an ordinary affine transformation, in which the value of s can be 0 or even negative.

In per-channel quantization, each column of the weight matrix is optimized independently of each 128 other. For simplicity of notation, we only focus on one column in ̂ W later and re-define the notations. 129 Based on Eq. 5, the optimization problem of decoupleQ in the first stage, layer-wise minimization, 130 can then be formulated as: 131

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

where the objective function is: 132

w ∈ R d in is one column of ̂ W , b ∈ R d in is the corresponding column of W 0 , s ∈ R ng is the scale and z ∈ R ng is the zero point, ng is the number of groups when grouping-quantization. The operations w.r.t ( s, z ) , i.e., ∗ s and + z , need to be broadcasted to each group. In this paradigm, we have completely abandoned the traditional framework of quantization and instead transformed quantization into a constrained optimization problem 6, which is then solved to achieve the purpose of quantization. ( s, z ) in problem 6 have lost the traditional meaning of scale and zero point, and are just two optimization variables.

Transforming the traditional quantization problem into problem 6 is the soul of decoupleQ! Problem 6 is a quadratic programming problem with an additional non-convex constraints w i ∈ Z . Quadratic programming has been studied for many years and there are now many well-established solution (24; 34). We provide one solution in the next subsection, which may not be efficient or optimal.

The core idea of decoupleQ is to decouple the model weights into the integer part w and the floating-point part ( s, z ) , with the integer part occupying most of the model's expressive power. The extensibility of the idea of decoupleQ is that we can freeze the integer part of the entire model, and use labeled data to train the ( s, z ) as well as other floating point parameters. The advantage of this is that on the one hand, it can further improve the accuracy of the model, on the other hand, it can fit specific downstream sub-tasks while maintaining the generalization ability of the model.

3.3 Optimization via Alternative Iteration

The problem 6 is not easy to solve because of the non-convex constraint w i ∈ Z . After obtaining a good initialization (explained in detail later), we solve for w and ( s, z ) alternately and iteratively. In each round of alternation, the objective function 7 w.r.t ( s, z ) is an unconstrained quadratic function, thus ( s, z ) can be readily determined analytically : by differentiating the objective function and equating the derivative to zero, followed by solving the resultant linear system of equations. While for w , the problem become problem 8:

For problem 8, one solution is to round-and-clip one element of w to be integer in [ α, β ] and then 158 update the remaining. And then this process is then performed sequentially for all elements. After the 159 j -th element has been rounded-and-clipped, the objective for the updating then becomes problem 9. 160

problem 9 is also intractable, and we can make two levels of approximation: 161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

In the first-level approximation 10, only the non-convex constraint w i ∈ Z is discarded, while in the second-level approximation 11, both the non-convex constraint w i ∈ Z and the convex constraint w i ∈ [ α, β ] are discarded. Intuitively, problem 11 is much simpler to solve than problem 10, but solving problem 10 will lead to a better convergence of the primary objective( 6) than solving problem 11. GPTQ (9) provides an efficient analytical solution for problem 11, which we will directly utilize in our experiments. ( GPTQ updates the remaining elements by considering only the second-level approximation 11 and ignoring the constrain w i ∈ [ α, β ] in the first ( 10), which is what we mentioned in the introduction, that the update of GPTQ is unconstrained.) As for problem 10, there are many mature solutions in the field of convex optimization, such as active-set method, projected gradient descent (PGD), projected coordinate descent and so on (3). We choose PGD because its parallelization is much better than the other two methods. In the experimental part, we will compare the final accuracy of the model via between solving the first level (10) and the second level 11 approximation on small models, while on large models (e.g. lager than 7 billion parameters), we have to choose the second level 11 approximation because the intolerable runtime of solving the first (10). The algorithm is shown in Alg. 1 and Alg. 2.

Algorithm 1: Alternative Iteration to solve problem 6.

Input:

predefined iteration number N .

Result:

w ∗ , s ∗ , z ∗

1 Initialize t = 1 , w 0 , s 0 , z 0 ;

2 while t ≤ N do

3 Freeze ( s t -1 , z t -1 ) , and optimize g ( w ; s t -1 , z t -1 ) to obtain an approximate solution w t via solving 8 via 2;

4

Freeze w t , and solve the unconstraint quadratic equation g ( w t ; s, z ) to obtain an analytic solution for ( s t , z t ) ;

5

t

6 end

7 w ∗ = w N ; s ∗ = s N ; z ∗ = z N

1

Algorithm 2: Approximate solution of 8

Input: predefined iteration number K,M , and the froozen ( s, z ) .

Result: w ∗

if

Approximaton (10) is used

2

then

Ignoring the constraint w i ∈ Z in Eq. 8, and train Eq. 8 with M iterations via PGD;

3 Initialize j = 1 ;

5

round and clip the j -th element of w , then keep the first j elements frozen, and update the remainings via PGD to optimize 10 with K iterations or until converged, or via the method in GPTQ to optimize 11.

6

7

3.4 Initialization of w and ( s, z )

Since the values of w are discrete, a good initialization is very important in order to obtain a more accurate solution to the original problem 6 with a faster convergence. Intuitively, the function g ( w ; s, z ) contains the term w ∗ s , which means that the scales of the initial values of w and s have to be reasonably distributed. For example, in the extreme case when the initial value of ( s, z ) have a very large scale, the first iteration will make most of the entries of w strictly 0, which will make the iteration crash. We start by initializing ( s, z ) . We can use grid search to solve the Eq. 12 for the initial value of ( s, z ) . In Eq. 12, p is a single number, may be different end

w

∗

=

w

for different columns of W , b and b are the minimum and maximum value of b

0 min max respectively.

This step is the same as the previous post-training quantization (19) process. Once the grid search is

=

t

+1

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

finished, we no longer need to concern ourselves with the ( s, z ) inside the ⌊·⌉ function. The point of this step is simply to find an initial value for ( s, z ) for the optimization problem 6.

When solving problem 8 via the first-level approximation ( 10), before entering the for-loop in Alg. 2, we ignore the constraint w i ∈ Z in problem 8 and optimize it via projected gradient decent with M iterations. The purpose of this is to allow the first-level approximation to converge in a small number of iterations, i.e., a small K .

3.5 Block-wise minimization

After solving problem 6, we obtain a solution for the layer-wise minimization stage and a reasonable model accuracy. But minimizing the ℓ 2 loss at the layer level does not necessarily lead to the minimizing the ℓ 2 loss at the block level. We found that the model accuracy can be further improved via optimization 2. BRECQ (18) also shows that block-reconstruction results in a better model accuracy than layer-reconstruction. In this stage, we freeze the integer part ̂ W in the whole block and fine-tuning ( s, z ) and the parameters in norm layer with J epochs.

4 Experiments

In this section, we describe in detail the experimental results of our method in comparison with other methods. Unless otherwise stated, all the experiments are conducted on a single A100-SXM-80GB, and the default experimental setting is as follows:

ResNet: 10240 images in the training dateloader are used as calibration data, with the standard augmentation in Pytorch official code (27), and the pretrained full precision checkpoints are from Torchvision (22). N = 4 , M = 50 ( N and M is defined in refalg1 and refalg2). All the convolution layers and fully-connected layers are quantized into W2 without groups.

Llama-1/2: 128 2048-token segments from C4 (28) are used as calibration data. We choose C4 as calibration dataset instead of WikiText2 (23) to be consistent with GPTQ. If the block-wise minimization is used, we use Adam optimizer (15) to finetune the ( s, z ) and the parameters in norm layer with J = 4 epochs. The learning rate is 1 e -5 , weight decay is 1 e -6 .

4.1 Private Experiments

We applied decoupleQ to our company's two Automatic Speech Recognition models(ASR) (corresponding to task A and task B). Each of the models contain an encoder and an LLM decoder. The input of the models is a speech sequence and some prompt, and the output is the corresponding text. We quantize the LLM decoder to W2A16g64. The decoders of the two models contain 40 transformer blocks with 13 billion parameters and 32 transformer blocks with 7 billion parameters, respectively. Word Error Rate (WER) is used as metric to measure the accuracy of the models (less is better). In this experiments, we use about 8 millions of speech tokens as calibration dataset, and train 3 epoch in each block-wise minimization process. When an input batch contains sequences of varying lengths, we use a mask to make sure that the

1000

006

800

700

600

500

400

300

200

100

0

GEMM 1

GEMM 2

GEMM 3

GEMM 4

BF16

W8A16

W4A16g64

W2A16g64

padding part is not involved in the computation of H and the loss of Eq. 2. In task B, once the whole model is quantized, we also fine-tune all the ( s, z ) and layer norm in the LLM with labeled dataset, while freezing all the integer part ̂ W , with 8 A100-SXM-80GB GPUs. The accuracy is shown in Tab. 1, and the CUDA kernel latency is shown in Fig. 1. The W2A16 CUDA kernel is attached and will be merged into the NVIDIA repo as one of our core contribution.

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

4.2 Public Comparison

As a first comparison, we compare decoupleQ with other methods on ImageNet (5) with ResNet (12), which are standard benchmarks and are efficient to implement. Most importantly, its Top-1 is a strong indicator of model accuracy. Tab. 2 shows the results of decoupleQ and others. The results other than decoupleQ are copied from GPTQ (9) and OBQ (8).

Tab. 3 shows the results on Llama. In this experiment, we have to choose the second level approximation(11) because the intolerable runtime of solving the first(10). For a fair comparison, the calibration dataset contains 128 samples, although a larger calibration dataset will result in stronger results. we can see that decoupleQ outperforms others almost in all settings, although we use a weaker approximation(11) to save time. As for the hype-parameters, we choose { N = 4 , J = 4 } .

4.3 Ablation studies

4.3.1 the two approximations

The soul of decoupleQ is problem 6, but when solving problem 6, we have to take some approximations(10 or 11). Obviously, solving approximation 10 will be much more time consuming than solving approximation 11. But if solving approximation 10 yields better results, the time cost may be worth it. We first evaluate these two approximations from the perspective of model accuracy. In practice, we don't have to wait for approximation 10 to fully converge when we solve it via projected gradient decent, and only need to iterate some steps to get a sub-optimal solution. In Alg. 2, the for-loop takes up the majority of the runtime. So, we first study the influence of the number of iterations K (defined in the for-loop) on the final accuracy of the model. Fig. 2 shows the Top-1 accuracy of ResNet-18 on ImageNet w.r.t the number of iterations K . First of all, in the blue line, we use only the layer-wise minimization of decooupleQ to quantize the model. After the quantization is finished, in the red line, we use the labeled dataset with the common 1.2 millions images to fine-tune all the ( s, z ) and norm layers for one epoch, with the integer part being frozen. In this step, we use SGD optimizer with learning rate 1 e -6 , weight decaying rate 1 e -4 to train for only one epoch. Fig. 2 clearly indicates the following conclusions: 1. As the number of iterations K increases, the model accuracy increases almost monotonically; 2. When K > 4 , model accuracy via the first approximation(10) is better than via the second(11). This is to be expected, since the second approximation(11) drops the constraint α ≤ w i ≤ β , leading to a looser approximation; 3. By the supervised fine-tuning (sft), the model accuracy is further improved. The same experimental phenomenon also occurs on the ResNet-50 model, which we do not show here.

276

277

278

279

280

281

282

66

64

62

acc.

60

op-1

58

10

56

w/osft

54

w/ sft

52

0

1

3

4

51020

50

numberofiterationsK

24.0

23.5

23.0

PP

13.5

13.0

12.5

0

1

2.3

4

5.10

15

20

numberofiterationsK

In the experiment shown in 3, we randomly select 512 2048-token segments from C4 (28). We chose 512 segments here instead of the common 128 in order to reduce the effect of overfitting and thus compare the two approximations more objectively. In this experiment, we take N = 2 , and quantize Llama-7B into W2A16 without groups, and only the layer-wise minimization is used to exclude the interference of other factors. The PPL decrease almost monotonically as the number of iterations K increases. It shows that, when K > 1 , solving approximation 10 yields better model accuracy than approximation 11.

However, when block-wise minimization is introduced in addition to the experiment in 3, the situation 283 becomes a little more elusive. The results are shown in 4. The model's best PPL is where K = 1 , 284 and then fluctuates within a range as K continues to increase. But all PPLs are inferior to when 285 the second-level approximation (11) is used. We also plot the loss, defined in 2, of the first block 286 between pre-and post quantization on the right vertical axis. As K increases, the loss decreases 287 strictly monotonically, and when K > 2 , the loss falls below the case when the approximation 11 is 288 used. This suggests that the correlation between PPL and loss is perhaps weak, and we will investigate 289 this in the future. 290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

12.0

PPL

4.5

PPL of WikiText2

11.5

blockloss

4.0

11.0

5

（e-

3.5

block loss

10.5

3.0

10.0

2.5

9.5

2.0

9.0

0

2

3

4

5101520

numberofiterationsK

4.3.2 the size of calibration dataset

The solution of problem 6 is dependent on H and thus on the the calibration dataset, as does Eq. 2. Fig. 5 shows the relationship between dataset size and PPL. In this experiment, Llama-7B is quantized into W2A16g64. We use the second-level approximation (11) to save time, and { N = 4 , J = 4 }. For runtime reference, when the number of segments is 128/2048, the experiment took 4.3/19.5 hours.

4.3.3 the necessity of block-wise minimization

Tab. 4 shows that block-wise minimization(2) can further improve the model accuracy. In this experiment, we choose N = 4 and the approximation 11 for the layer-wise minimization, and J = 4 if block-wise minimization is used.

5 Conclusion and Discussion

deocupleQ decouples the model parameters into the integer part and a floating point part, and then optimizes them alternately. This optimization process contains two stages. In the layer-wise minimization, we transform the quantization problem into the purely mathematical constrained optimization problem refdecoupleQ; while in the block-wise minimization, we freeze the integer part and then finetune the floating point part.

The risks of decoupleQ include the following: 1. How much the minimization of the ℓ 2 loss of the layer's or block's output correlates with the accuracy of the model; 2. decoupleQ is prone to overfitting the calibration dataset; 3. The runtime of the quantization process is longer than others.

For the first risk, we find experimentally that the correlation between Top-1 and the loss is strong in the Imagenet classification task; however, the correlation between PPL and the loss is slightly weaker in LLM. This could be mainly because of an inherent bias between the loss and the accuracy of the model, or because PPL is not a good indicator of the accuracy of LLM, or for other reasons. For the second risk, when H in Eq. 7 is an underdetermined matrix, the risk of overfitting rises sharply. In this case, the possibility of H being underdetermined can be reduced either by enhancing the diagonal element values of H or by increasing the amount of calibration data. In our practice, we found that the accuracy of quantization models can rise monotonically with the increase of the size of the calibration dataset, especially in W2 quantization, but the runtime of quantization rise as well. In addition, due to time constraints, we do not provide a wealth of public comparisons. However, we believe that the novelty of a method may outweigh the number of experiments.

The idea of decoupleQ is helpful for the adaptation of large model to downstream sub-task. We can quantize a large foundation model via decoupleQ, then freeze the integer part of the model, and finetune the floating-point part with labeled dataset from downstream sub-task. Tab. 1 and Tab. 2 show that the model accuracy can be further improved by end-to-end supervised learning.

WikiText2

10.6

8.75

C4

10.4

8.50

10.2

8.25

10.0

8.00

9.8

7.75

9.6

64

128

256

512

10242048

#ofsegmentsforcalibration

328

329

330

331

332

333

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023.

Sébastien Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends® in 334 Machine Learning , 8(3-4):231-357, 2015. 335

Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. Quip: 2-bit quantization of large 336 language models with guarantees. arXiv preprint arXiv:2307.13304 , 2023. 337

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical 338 image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. 339 Ieee, 2009. 340

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication 341 for transformers at scale. arXiv preprint arXiv:2208.07339 , 2022. 342

Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, 343 Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized representation for 344 near-lossless llm weight compression. arXiv preprint arXiv:2306.03078 , 2023. 345

Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training 346 quantization and pruning. Advances in Neural Information Processing Systems , 35:4475-4488, 2022. 347

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization for 348 generative pre-trained transformers. In The Eleventh International Conference on Learning Representations , 349 2022. 350

Yi Guo, Yiqian He, Xiaoyang Li, Haotong Qin, Van Tung Pham, Yang Zhang, and Shouda Liu. Rdimkd: 351 Generic distillation paradigm by dimensionality reduction. arXiv preprint arXiv:2312.08700 , 2023. 352

353

354

355

Yi Guo, Huan Yuan, Jianchao Tan, Zhangyang Wang, Sen Yang, and Ji Liu. Gdp: Stabilized neural network pruning via gates with differentiable polarization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5239-5250, 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 356 In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016. 357

Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training 358 quantization with small calibration sets. In International Conference on Machine Learning , pages 4466359 4475. PMLR, 2021. 360

Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, 361 and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629 , 2023. 362

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 363 arXiv:1412.6980 , 2014. 364

Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. 365 arXiv preprint arXiv:1806.08342 , 2018. 366

Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Lessons learned from 367 activation outliers for weight quantization in large language models. arXiv preprint arXiv:2306.02272 , 368 2023. 369

Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi 370 Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint 371 arXiv:2102.05426 , 2021. 372

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware 373 weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978 , 2023. 374

Zechun Liu, Kwang-Ting Cheng, Dong Huang, Eric P Xing, and Zhiqiang Shen. Nonuniform-to-uniform 375 quantization: Towards accurate quantization via generalized straight-through estimation. In Proceedings of 376 the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4942-4952, 2022. 377

Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, 378 Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large 379 language models. arXiv preprint arXiv:2305.17888 , 2023. 380

Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In Proceedings 381 of the 18th ACM international conference on Multimedia , pages 1485-1488, 2010. 382

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. 383 arXiv preprint arXiv:1609.07843 , 2016. 384

Katta G Murty and Feng-Tien Yu. Linear complementarity, linear and nonlinear programming , volume 3. 385 Heldermann Berlin, 1988. 386

Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or 387 down? adaptive rounding for post-training quantization. In International Conference on Machine Learning , 388 pages 7197-7206. PMLR, 2020. 389

Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, Alex M Bronstein, and 390 Avi Mendelson. Loss aware post-training quantization. Machine Learning , 110(11-12):3245-3262, 2021. 391

392

[27]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen,

Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep 393

394

learning library. Advances in neural information processing systems , 32, 2019.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, 395 Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. 396 Journal of machine learning research , 21(140):1-67, 2020. 397

Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng 398 Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language 399 models. arXiv preprint arXiv:2308.13137 , 2023. 400

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, 401 Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation 402 language models. arXiv preprint arXiv:2302.13971 , 2023. 403

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay 404 Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and 405 fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023. 406

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz 407 Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 408 30, 2017. 409

Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong 410 Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal 411 shifting and scaling. arXiv preprint arXiv:2304.09145 , 2023. 412

Stephen J Wright. Numerical optimization . 2006. 413

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: 414 Accurate and efficient post-training quantization for large language models. In International Conference 415 on Machine Learning , pages 38087-38099. PMLR, 2023. 416

Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhensu Chen, Xiaopeng 417 Zhang, and Qi Tian. Qa-lora: Quantization-aware low-rank adaptation of large language models. arXiv 418 preprint arXiv:2309.14717 , 2023. 419

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher 420 Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. 421 arXiv preprint arXiv:2205.01068 , 2022. 422

Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, 423 Vera Axelrod, Gary Wang, et al. Google usm: Scaling automatic speech recognition beyond 100 languages. 424 arXiv preprint arXiv:2303.01037 , 2023. 425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our claims and justification include:

a. Our results are higher than others in very low bit (2-bit) quantization.( This is justified in Tab. 3.);

b. decoupleQ has achieved comparable accuracy as fp16/bf16 for 2-bit quantization of large speech models in our company. (This is justified in Tab. 1, and the W2 CUDA kernel used in our company are attached.);

c. decoupleQ gets rid of any tricks for dealing with outliers, sensitive channels, etc. (This is justified in the Problem 6, we do not use any tricks, such as scaling factor (19; 29), mixed-precision quantization (6), etc., to deal with outliers and sensitive channels.)

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper has discussed the three limitations of decoupleQ in the last section, Conclusion and Discussion , and the risk overall that we did not provide as many public comparison experiments as other work due to time constraints.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer:[NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: At the beginning of the section Experiments , we provide details of the experimental parameters; specifically for each experiment, we also provide the key experimental parameters.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code (including W2 CUDA kernels) is attached in supplementary material, and can reproduce the results in the public experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: At the beginning of the section Experiments , we provide details of the experimental parameters; specifically for each experiment, we also provide the key experimental parameters. The code is attached in supplementary material and will be made public.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The cost of the experiment is high.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have reported that most of the experiments are conducted in one single A100-SXM-80GB, except for the sft process. And we also reported the time of execution.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a work for accelerating the inference of deep models, where the social impact is determined by the function of the model, not by how the inference is accelerated.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This work does not release models or datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper has cited all related works, and included the relevant license.

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide the source code, and a readme and license file are alongside.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces decoupleQ, a novel method that decouples model parameters into integer and floating-point parts. This approach transforms the quantization problem into a mathematical constrained optimization problem, avoiding the limitations of traditional heuristic quantization methods. DecoupleQ achieves a significant improvement over existing methods in LLM., especially at extreme low bits (2-bit) and also release the W2A16 CUDA kernel.\n\nSTRENGTHS:\n1. DecoupleQ eliminates the need for ad-hoc techniques to handle outliers and sensitive channels, focusing solely on optimizing model accuracy under extreme low-bit quantization.\n2. DecoupleQ achieves a notable advancement over existing methods in LLM, particularly at extremely low bit. And the W2A16 CUDA kernel has been released.\n3. DecoupleQ approach can be readily extended to supervised fine-tuning (SFT) to enhance model accuracy, or adapted for downstream sub-tasks.\n\nWEAKNESSES:\n1. Please correct me if I am wrong. It seems that decoupleQ combines several existing approaches. Specifically, it uses Adaround to get the integer part in ResNets and GPTQ to get the integer part in LLMs. Additionally, it integrates PTQ and QAT by applying PTQ to the integer part while using supervised training for the floating-point part.\n2. Regarding your point from lines 58-61, I believe GPTQ clearly outlines how to calculate scale and zero point in their code. Moreover, GPTQ can be seen as a constrained optimization problem, where the constraints align with yours: each integer weight is confined within [$\\alpha$, $\\beta$], which is a default constraint in GPTQ.\n3. Further experiments on LLMs are essential. For example, evaluating decoupleQ's performance in multi-task settings and within the LlaMa 3 family would provide valuable insights.\n4. Could you provide more ablation studies in the second stage, such as experiments without training norm layers?\n5. There is a typo in line 125. The first letter of 'decoupleQ' should be capitalized.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nPlease refer to the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

comments: SUMMARY:
This paper proposes a linear and uniform quantization method, decoupleQ, which abandons the traditional heuristic quantization paradigm and decouples the model parameters into integer and floating-point parts, then transforming the quantization problem into integer and floating-point part. Experiments show decoupleQ achieves comparable acc as fp16/bf16 on 2-bit weight quantization setting.

STRENGTHS:
Experiments show decoupleQ achieves comparable acc as fp16/bf16 on 2-bit weight quantization setting.

WEAKNESSES:
1. Experiments are based on W2A16, lower activation bitwidth(<=8bit) should be experimented.
2. The novelty is limited. The core idea of decoupleQ is similar to Normalization(Batch-Norm or Layer-Norm). The learnable floating part of decoupleQ equals to a learnable Normalization parameters.
3. More existing Quantization methods should be compared, such as, NWQ[1], PD-Quant[2]

[1] Leveraging Inter-Layer Dependency for Post -Training Quantization 
[2] PD-Quant: Post-Training Quantization based on Prediction Difference Metric.

QUESTIONS:
1. What's the gain for lower activation bitwidth(<=8bit) of decoupleQ?
2. Experiment comparison over more existing Quantization methods.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents decoupleQ, a post-training quantization method that improves the accuracy of quantized models, particularly at very low bit-widths (2-bit). It achieves this by separating model parameters into integer and floating-point components and formulating the quantization process as a constrained optimization problem. This approach eliminates the need for traditional quantization techniques like outlier handling and focuses on optimizing the core objective.\n\nSTRENGTHS:\n1. The paper introduces a fresh perspective on quantization by abandoning traditional methods and reframing it as a constrained optimization problem.\n2. decoupleQ demonstrates impressive results in 2-bit quantization, achieving accuracy comparable to higher precision formats like fp16/bf16 in large speech models.\n3. The quantization process is linear and uniform, making it easier to implement in hardware compared to non-uniform methods.\n\nWEAKNESSES:\n1. The paper's writing lacks cohesion and clarity regarding its ultimate goal. The paper also has several spelling mistakes.\n2. The authors claim to separate the model parameters into integers and floating-point components. However, as far as I understand, this practice is not a novel contribution but rather a common approach in quantization.\n3. They address a portion of the optimization problem using GPTQ and another portion similar to BRECQ.\n4. The authors acknowledge that their solution may not be optimal. \n5. The quantization process in decoupleQ can be more time-consuming than other methods.\n\nQUESTIONS:\n1. The paper mentions achieving state-of-the-art accuracy in Llama-1/2. It would be helpful to see a more detailed comparison with other state-of-the-art quantization methods on this specific model.\n2. The authors could elaborate on the potential impact of the weak correlation between PPL (perplexity) and loss in LLMs. How might this affect the practical application of decoupleQ for LLM quantization?\n3. The paper briefly mentions the risk of overfitting. Providing more insights into mitigation strategies for this risk would be beneficial, especially when dealing with underdetermined matrices.\n4. Given the longer runtime compared to other methods, it would be interesting to see a more comprehensive analysis of the trade-off between quantization time and model accuracy.\n5. The authors could consider adding a section on future work, outlining potential directions for further research and improvement of the decoupleQ method.\n\nLIMITATIONS:\nYes, however, it will be beneficial to divide the current Discussion section into separate Conclusion and Limitations sections.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel post-training quantization method to achieve 2-bit uniform quantization on large language and speech models. The proposed method decouples the quantized values into integer and floating-point parts, which are then optimized via a constrained optimization problem that can be solved with off-the-shelf solutions. The proposed method allows uniform quantization down to extreme bits.\n\nSTRENGTHS:\n1. This paper proposes a novel optimization-based method to conduct PTQ on large models. The proposed method is solid and unique from previous methods.\n2. The proposed method achieves good performance with only uniform quantization, without special procedure for outliers etc., providing direct benefit to the runtime of the quantized model on general hardware.\n3. The limitations and future directions are clearly discussed in the paper.\n\nWEAKNESSES:\n1. The distinction between the proposed decoupleQ and the traditional quantization methods are not clearly derived in Sec. 3.2. The statement that \"(s,z) lost the traditional meaning\" on line 138 is not clear. My understanding is that W, s, and z are now totally independent of the original weight w0 in the optimization process, as long as the final output error is minimized? I think adding a comparison with the optimization objective/procedure of the traditional quantization here will help.\n2. The proposed method appears to be sensitive to the size of the calibration set, so that the calibration size reported in the experiments are much larger than that of the previous baselines. As it is understandable that the optimization process may require more data to avoid overfitting, it would be more fair if the baseline methods are also calibrated with the same dataset/training cost.\n3. For the LLM experiments, only ppl is used as metric. However, the ppl has been shown to be an inaccurate metric to reflect the utility of the LLM after compression. More evaluations such as zero-shot performance on downstream tasks and the instruction following ability etc., as in SqueezeLLM and OmniQuant papers, would be helpful to see if the quantized model still retains the ability as the FP one.\n\nQUESTIONS:\nPlease see the weakness section.\n\nLIMITATIONS:\nThe limitations and potential social impacts are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}