Title: Towards Next-Level Post-Training Quantization of Hyper-Scale Transformers
Venue: NeurIPS 2024
Accepted: yes
Keywords: Hyper-scale

Compression

Quantization

Transformers

LLM
Abstract: With the increasing complexity of generative AI models, post-training quantization (PTQ) has emerged as a promising solution for deploying hyper-scale models on edge devices such as mobile and TVs.
Existing PTQ schemes, however, consume considerable time and resources, which could be a bottleneck in real situations where frequent model updates and multiple hyperparameter tunings are required.
As a cost-effective alternative, learning-free PTQ schemes have been proposed. 
However, the performance is somewhat limited because they cannot consider the inter-layer dependency within the attention module, which is a significant feature of Transformers.
In this paper, we thus propose a novel PTQ algorithm that balances accuracy and efficiency.
The key idea of the proposed algorithm called aespa is to perform quantization layer-wise for efficiency while targeting attention-wise reconstruction to consider the cross-layer dependency.
Through extensive experiments on various language models and complexity analysis, we demonstrate that aespa is accurate and efficient in quantizing Transformer models. The code will be available at https: //github.com/SamsungLabs/aespa.

Towards Next-Level Post-Training Quantization of Hyper-Scale Transformers

Abstract

With the increasing complexity of generative AI models, post-training quantization (PTQ) has emerged as a promising solution for deploying hyper-scale models on edge devices such as mobile and TVs. Existing PTQ schemes, however, consume considerable time and resources, which could be a bottleneck in real situations where frequent model updates and multiple hyperparameter tunings are required. As a cost-effective alternative, learning-free PTQ schemes have been proposed. However, the performance is somewhat limited because they cannot consider the inter-layer dependency within the attention module, which is a significant feature of Transformers. In this paper, we thus propose a novel PTQ algorithm that balances accuracy and efficiency. The key idea of the proposed algorithm called aespa is to perform quantization layer-wise for efficiency while targeting attention-wise reconstruction to consider the cross-layer dependency. Through extensive experiments on various language models and complexity analysis, we demonstrate that aespa is accurate and efficient in quantizing Transformer models. The code will be available at https://github.com/SamsungLabs/aespa .

1 Introduction

Model size has been gradually growing, resulting in deep generative models such as diffusion [25] and large-scale language models (LLMs) [29, 35] becoming more mainstream; the trend of AI is transitioning from discriminative models to generative models with numerous parameters in trillions. With the explosive growth in model complexity (parameters), the performance of AI models has been advancing and is now approaching or even exceeding human intelligence levels. However, this growth in scale has resulted in a corresponding increase in computational costs, which necessitates the efficient processing and compression of AI models. Interestingly, one attempts to expand the complexity of AI models to scale up performance, whereas the other aims to compress models to reduce cost.

Quantization is a promising solution and indispensable procedure facilitating the efficient deployment of AI models on devices that mainly support fixed-point arithmetic. By reducing the precision of weights, the memory bandwidth requirements can be relieved, and the embarrassing parallelism of quantized models can be SIMDified using highly efficient vector processing units such as NPU. To minimize the inevitable performance degradation caused by quantization, we can choose one of two approaches: quantization-aware training (QAT) [5, 14] and post-training quantization (PTQ) [23, 18]. Considering the model complexity and required resources ( e.g., training costs and available datasets), QAT is not practical for compressing models with billions of parameters. Consequently, recent quantization studies on hyper-scale Transformer [31] models have focused more on PTQ.

Although existing PTQ schemes have successfully quantized relatively small-scale models ( e.g., ResNet) [23, 10, 18, 6, 11], they have difficulty handling large-scale models because of their time and space complexity. As a cost-effective alternative, learning-free algorithms have been proposed recently [7, 13, 19], but their performance is somewhat limited because they do not consider the inter-layer dependency and are reliant on the nearest rounding. There is an accuracy-efficiency trade-off; thus, we aim to bridge the gap toward next-level quantization of hyper-scale Transformer models.

In this paper, we propose a novel PTQ algorithm, called aespa , 2 that pursues both accuracy and efficiency. The key idea of aespa is to perform quantization layer-wise for efficiency while targeting the attention-wise reconstruction to consider the cross-layer dependency.

Our contributions are summarized as follows:

We propose a new quantization strategy that balances accuracy and efficiency. Our scheme aims to reconstruct the attention output to consider the cross-layer dependency while quantizing models layer-wise to pursue efficiency.

To accelerate the quantization process, we propose refined quantization objectives for the attention module. Through a complexity analysis, we demonstrate that quantization that is approximately 10 times faster than existing block-wise approaches can be achieved by exploiting the proposed objectives.

From extensive experiments on language models, we demonstrate that our approach outperforms conventional schemes by a significant margin, particularly for low-bit precision (INT2).

2 Background

2.1 Classic PTQ methods

Recent studies on PTQ have mostly attempted to minimize the increase in the task loss incurred by quantization rather than the quantization error itself ( ∆ W ). Consider a pre-trained neural network parameterized by weights W . If we assume the well-convergence of the network, the problem of quantizing weights W to minimize the loss degradation can be formulated as [16, 23]

where H ( w ) is the Hessian related to the flattened weight w . Because computing and storing H ( w ) is infeasible, further assumptions have been made to simplify (1). In [23], for example, layer-wise independence has been assumed, relaxing (1) into the layer-wise reconstruction problem:

where W ( ℓ ) denotes the weights of the ℓ -th layer, X is the input, and Q is a quantization function. For a uniform quantization, if the nearest-rounding is used to assign integer weights, Q is defined as

where s , z , and n are the scale, zero-point, and bit-width, respectively, and ⌊·⌉ represents the round-off.

Early studies on PTQ focused on optimizing the weight-rounding policy [23, 10, 18, 11, 12]. These studies have attempted to assign each weight to a 'proper' grid (instead of an adjacent grid), such that the loss degradation could be minimized. In [23], a learning-based weight-rounding optimization algorithm, called AdaRound, has been proposed to solve the layer-wise reconstruction problem in (2). In [18], AdaRound has been extended to the following block-wise reconstruction problem:

where W ( ℓ ) denotes the weights of the ℓ -th block f ( ℓ ) ( e.g., ResNet or Transformer block). By considering the dependency between layers inside the block, this algorithm, termed BRECQ, not only performs better than AdaRound, but also exhibits robust performance for a low bit-width ( e.g., INT2).

2.2 PTQ for LLMs

Although AdaRound and BRECQ have been successful in quantizing small-scale networks ( e.g., ResNet), scaling those learning-based schemes to LLMs with billions of parameters is challenging. In fact, BRECQ requires more than 20 GPU hours to quantize relatively small-sized language models ( e.g., OPT-2.7B; see Appendix K), which would not be suitable for the real-world deployment of LLMs where models to be deployed are frequently updated.

Owing to the excessive time and memory costs of classic PTQ schemes, recent studies have focused on developing cost-effective alternatives for quantizing LLMs. In OPTQ [7], a one-shot PTQ scheme that optimizes a weight-rounding policy without relying on learning, has been proposed. In addition, PTQ schemes that enhance the performance of the nearest-rounding, rather than optimizing the weight-rounding policy, have been proposed. These schemes use additional 'foldable' parameters 3 to suppress activation outliers or quantize weights more precisely [33, 19, 13, 27, 20].

Although previous studies have mitigated the computational overhead of classic PTQ methods, they often sacrifice the low-bit quantization performance or suffer from an unstable quantization process. The main reason for this unsatisfactory performance is that all the schemes mentioned above, except OPTQ, rely on nearest-rounding and do not optimize the weight-rounding policy. Moreover, most of them target layer-wise reconstruction in (2), not block-wise reconstruction in (4), thus ignoring the cross-layer dependency within the attention module. Although [27, 20] target block-wise reconstruction via learning, they need to approximate gradients for a non-differentiable quantization function, which results in an unstable training process (see Table 1 in Section 4) [19].

Thus, we propose a novel PTQ scheme that balances accuracy and efficiency. In contrast to conventional LLM quantization methods, our scheme optimizes a weight-rounding policy while targeting block-wise reconstruction to consider the cross-layer dependency. The key difference over classic block-wise weight-rounding optimization is that we quantize models layer-wise for scalability, whereas layers are jointly quantized in the existing methods. Furthermore, we present an efficient precomputation-based method for the computation of the block-wise objective in (4), which significantly reduces the computational overhead caused by repeated attention operations.

3 Method

3.1 Motivation

To gain insight into our approach, we first consider the objective of the layer-wise reconstruction in (2). Let ∆ W ( ℓ ) = Q ( W ( ℓ ) ) -W ( ℓ ) , then the reconstruction error can be expressed as

(a)

Layer

-Wise,

Layer

Output

(b)

Block

-Wise,

Block

Output

(c)

Layer

-Wise,

Block

Output (Proposed)

Consequently, the layer-wise quantization problem can be recast as follows:

The new form of the quantization objective in (6) implies that if E [ XX T ] is pre-computed and stored before quantization, we can measure the reconstruction error over the entire calibration dataset with a single matrix multiplication and element-wise multiplication. 4 This is in contrast to the original formulation in (2) which requires the computation of Q ( W ) X or ∆ WX for every input X .

A natural question that arises from this finding is 'Can we also measure the block reconstruction error efficiently based on such a pre-computation?' . In the following subsections, we describe our main strategy to simplify block-wise quantization and then present a refined objective for the attention module, where the objective can be computed efficiently with certain pre-computed values.

3.2 Quantization strategy of aespa

When quantizing the attention module using conventional block-wise reconstruction methods (Figure 1(a)), the query, key, and value projections have been jointly optimized such that

where the output of attention module SA( Q , K , V ) is defined as

In such a case, we need to compute SA( ̂ Q , ̂ K , ̂ V ) for every batch sequence in each iteration, which is computationally heavy and time-consuming (see Section 3.5 for details on complexity).

To overcome this computational overhead, we quantize each projection separately in a divide-andconquer manner. For example, when quantizing the query projection W Q , we fix W K and W V with full-precision (Figure 1(b)), which facilitates the factoring out of common terms affected by W K and W V (see Section 3.3 for details). We emphasize that this strategy differs from conventional layer-wise quantization schemes ( e.g., AdaRound and OPTQ) in that we aim to minimize the reconstruction error for the attention module, not the reconstruction error for each layer.

We conduct experiments to demonstrate the importance of targeting attention-wise reconstruction and validity of the proposed quantization strategy. In our experiments, we set the loss function for each projection as the attention reconstruction error in (7) but quantize each projection separately (see Figure 2(c)). Table 5 in Appendix B summarizes the performance of AdaRound, BRECQ, and our approach. As evident, our approach uniformly outperforms AdaRound for all bit-widths, although both methods quantize models layer-wise. This is because we can consider cross-layer dependency ( i.e., relationship between the query, key, and value) by targeting attention-wise reconstruction, which is different from AdaRound wherein layers are considered independent. Furthermore, once we target attention-wise reconstruction, separate layer-wise quantization does not incur severe performance degradation compared to the joint quantization method (BRECQ). In fact, our approach causes only a marginal performance degradation for 2-bit and exhibits comparable performance for 3-bit and 4-bit. For further discussion on the proposed strategy, see Appendix B.

3.3 Refined quantization objectives for aespa

One might ask whether our strategy incurs more computational cost than that required by the joint quantization because we update only one layer at a time (see Figure 1(b)). This is in contrast

to existing methods, in which the layers inside the attention module are updated simultaneously (Figure 1(a)). To reduce this additional cost, we refine the quantization objective in (7) for each projection.

Value projection When quantizing the value projection W V , the query and key projections are fixed with full-precision. In this case, by factoring out the common term influenced by Q and K , we can simplify the attention reconstruction error ∆SA V as follows:

Thus, the problem to quantize W V to minimize the attention reconstruction error can be recast as

Query projection When the key and value projections are fixed with full-precision, the attention reconstruction error ∆SA Q caused by ∆ W Q is expressed as

where ∆ A = softmax( ̂ QK T / √ d ) -softmax( QK T / √ d ) . To avoid the computational overhead of repetitive softmax operations, we approximate ∆ A with its first-order Taylor series as

where J softmax is the Jacobian of the softmax function. By combining (11) and (12), we obtain

Although we can circumvent conducting attention operations using the modified form in (13), a large amount of memory is required to store the Jacobian J softmax ( e.g., more than 100 GB of memory for OPT-125M). 5 As a cost-effective alternative, we build an upper bound of (13) and then employ it as a surrogate of ∆SA Q when quantizing W Q . Specifically, by noting that

and the term ∥ V T J softmax ∥ 2 F is fixed in the quantization process, we minimize ∥ K ∆ W Q X ∥ 2 F with the hope that ∆SA Q also decreases. In other words, our quantization objective for W Q is

Key projection By taking similar steps, the quantization objective for the key projection W K can be formulated as (see Appendix C for the detailed derivation)

3.4 Algorithm description

The proposed aespa consists of two main steps. Specifically, aespa first determines the quantization parameters ( i.e., scale and zero-point) and then optimizes an integer weight W int for each weight.

Note that we only used the definition of the attention operation when developing the refined objectives in (10), (15), and (16). Thus, our objectives can be integrated into any layer-wise quantization scheme without effort. For example, we can compute the quantization parameters by combining existing parameter initialization algorithms ( e.g., AWQ[19] and Z-FOLD [13]) with the proposed objectives. We can also optimize a weight-rounding policy using conventional methods ( e.g., AdaRound [23])

together with our objectives (see Appendix F for details). In the proposed aespa , we use Z-FOLD in computing the quantization parameters and employ AdaRound in optimizing a weight-rounding policy. In Algorithm 1 (see Appendix A), we summarize the proposed aespa . 6

To accelerate the weight-rounding learning process, we further modify the objective functions such that the value can be computed efficiently via pre-computation, as in (5).

Modified objective for (10) The proposed objective for the value projection can be recast as

The modified objective allows us to perform each iteration of the weight-rounding learning efficiently. Specifically, by computing E [ XA T AX T ] before quantization and reusing it in the quantization process 7 , we can avoid the overhead of computing ∥ ∥ ∆ W V XA T ∥ ∥ 2 F for every input X and compute the loss with one simple matrix multiplication and a single element-wise multiplication (see Footnote 4).

Another intriguing feature of this modification is that it facilitates a more reliable update of ∆ W V than the original objective in (10). Specifically, because E [ XA T AX T ] is pre-computed using all calibration data, the loss computed with (17) considers the entire calibration dataset ( i.e., the batch size is the total number of data). Thus, a better estimate of the true gradient can be obtained without any memory issues, which could lead to more consistent updates of ∆ W V and faster convergence [28].

The modified objective in (17) also implies that the Hessian H V for each row of W V is

We note that the proposed Hessian H V differs from H = 2 E [ XX T ] , which has been commonly used as an approximated Hessian in conventional methods [6, 7, 13, 3]. The key reason for the difference is that we consider the dependency between W Q , W K , and W V by targeting attentionwise reconstruction, whereas the previous methods assumed independence. To observe the effect of considering the cross-layer dependency, we use different Hessians ( i.e., H V and H ) when quantizing language models and then compare the performance of the quantized models (see Appendix D). Evidently, the quantization performance is much better when the proposed Hessian H V is used, which demonstrates the importance of considering the cross-layer dependency.

Modified objectives for (15) and (16) If we denote the vectorized representation of ∆ W Q as ∆ w Q , the proposed objective in (15) can be expressed as (see Appendix E for the derivation)

where ⊗ is the Kronecker product operation. To reduce the memory cost of storing the Kronecker product term E [ XX T ⊗ K T K ] , we approximate it as [2]

By combining (19) and (20), we obtain

where the proof of (a) is provided in Appendix E. By taking similar steps, the objective for the key projection can be recast as

The modified objectives in (21) and (22) imply that the loss over the total calibration dataset can be calculated efficiently by computing E [ K T K ] , E [ Q T Q ] , and E [ XX T ] in advance.

3.5 Complexity analysis for aespa

We discuss the computational complexity of aespa . Specifically, we analyze the number of floatingpoint operations (flops) required to perform one iteration for weight-rounding optimization (line 6 in Algorithm 1). For each projection, the required number of flops is summarized as follows.

Value : By reusing the pre-computed E [ XA T AX T ] , the loss value in (17) can be computed with one matrix multiplication and one element-wise multiplication/addition (see Footnote 4). The associated cost is 2 d h d 2 + d h d -1 flops, where d is the hidden size and d h is the input dimension for each attention head.

Query/key : Once E [ K T K ] , E [ Q T Q ] , and E [ XX T ] have been computed in advance, the loss values in (21) and (22) can be computed by performing two matrix multiplications and one element-wise multiplication/addition. This requires 2 d h d 2 +2 d 2 h d -1 flops for each projection.

To summarize, the total number of flops required in each iteration of the proposed aespa is

We emphasize that regardless of the amount of calibration data, the number of flops to compute the loss considering the entire dataset is fixed as C aespa .

We now compare the complexities of aespa and conventional block-wise quantization methods. It can be easily verified that the existing methods require the following number of flops for handling B input sequences of length L (see Appendix G):

Table 7 in Appendix G summarizes the computational costs for different sizes of OPT models. For the conventional methods, we report the cost of using four sequences in each iteration ( B = 4 ). We observe that the computational cost of aespa is considerably lower than that of conventional methods. In particular, for small-scale models, aespa performs ten times fewer number of flops. It can be observed that the gap between C aespa and C exist decreases as the model size increases. This is because the hidden size d exceeds the sequence length L (which is fixed for all models) for large models. Nevertheless, aespa still incurs a lower computational cost, and the gap increases if conventional methods use larger batch sizes.

4 Experimental results

4.1 Experimental setup

Wequantize publicly available LLMs ( e.g., OPT[35], BLOOM [26], LLaMA [29], and LLaMA2 [30]) using the proposed aespa . When implementing aespa , we compute the quantization parameters with Z-FOLD [13] and optimize a weight-rounding policy via AdaRound [23], where the proposed rowwise Hessians and loss functions (see Table 4 in Appendix A) are utilized instead of the existing ones. When computing the quantization parameters, we follow the stopping criterion introduced by [13]. Before optimizing a weight-rounding policy, we update the full-precision weights via OPTQ [7], which empirically reduces the number of iterations required for weight-rounding optimization. When optimizing a weight-rounding policy, we set the number of iterations, learning rate, and weight of the rounding loss (see λ in (28)) to 2,000, 0.015, and 1.5, respectively.

When constructing the calibration dataset, we randomly sample 128 segments consisting of 2048 tokens from the C4 dataset [24] as in [7, 13, 3]. In our experiments, we quantize only weights and retain activations in full-precision because activations are not a significant bottleneck for LLMs [7] and the inference of LLMs can be accelerated sufficiently by reducing memory movement through weight quantization [15]. We evaluate the performance of the quantized models using benchmark datasets ( e.g., WikiText-2 [22], C4 [24], and PTB [21]) and zero-shot tasks. Except for the experiments on the LLaMA2 models, which were performed using an NVIDIA H100 GPU, we conducted all experiments using a single NVIDIA A100 GPU (80 GB).

4.2 Comparison with prior arts

Comparison with block-wise PTQ schemes We compare the proposed aespa with conventional block-wise PTQ methods, among which BRECQ is a classic weight-rounding optimization method,

(a) WikiText-2

(b) C4

* 'NaN' means that loss diverges in the quantization process.

* 'OOM' means that out-of-memory issues occur when quantizing models with a single A100 GPU.

* Results for high bit-widths are provided in Appendix H due to the page limitation.

and OmniQuant and AffineQuant are LLM quantization methods that mitigate the computational overhead of BRECQ by learning only a few quantization and foldable parameters [27, 20]. For OmniQuant and AffineQuant, we ran the official codes 8 provided by the authors. For both methods, we activated the learnable equivalent transformation (LET) and learnable weight clipping (LWC) options and reported the obtained results. When implementing BRECQ, we employed the hyperparameter settings provided in [18]. In this comparison, the BLOOM models and OPT-350M were excluded because they are not supported by OmniQuant and AffineQuant.

As Table 1 shows, aespa uniformly outperforms OmniQuant/AffineQuant. 9 In particular, the performance gap is significant for 2-bit; while OmniQuant/AffineQuant suffer from instability ( i.e., loss diverges) or collapse (perplexity (PPL) > 10 3 ), aespa exhibits reasonable PPL. The outstanding performance is attributed to the fact that aespa optimizes a weight-rounding policy after determining the quantization parameters (lines 5-8 in Algorithm 1), whereas OmniQuant/AffineQuant rely on the naive nearest rounding and approximate gradients for the non-differentiable quantization function.

Although BRECQ performs best for the 2-bit quantization of OPT-125M, it lacks scalability; BRECQ requires approximately 20 GPU hours for a relatively small-scale OPT-2.7B (see Table 14 in Appendix K). Even for OPT-125M, BRECQ requires approximately 2 GPU hours, whereas the proposed aespa completes quantization in 5 minutes. One might wonder why the performance of BRECQ worsens as the model size increases. We assume that this is attributable to the choice of hyperparameters ( e.g., learning rate and weight of rounding loss). In fact, the hyperparameters presented in [18] have been optimized for ImageNet, but not for LLMs. It is expected that we can obtain better performance for BRECQ via deliberate hyperparameter tuning; however, this would not be feasible for real-world deployment because it requires considerable time (see Table 14 in Appendix K).

Comparison with layer-wise PTQ schemes We compare the proposed aespa with conventional layer-wise PTQ schemes, among which RTN is the method that naively assigns the nearest grid, OPTQ is a backpropagation-free weight-rounding optimization algorithm [7], and Z-FOLD is the

*

Results for high bit-widths and other language models ( e.g., OPT, LLaMA, and LLaMA2) are provided in Appendix I.

method exploiting additional foldable parameters to quantize weights more elaborately [13]. Table 2 and Tables 9-12 (see Appendix I) summarize the results for the OPT, BLOOM, LLaMA, and LLaMA2 models of various sizes. Evidently, aespa uniformly outperforms conventional schemes, regardless of the size and type of LLMs. In particular, for 2-bit, there exists a significant performance gap between aespa and existing methods; the PPL obtained by aespa is twice as low as those of conventional methods for small-scale models ( e.g., OPT-125M). The key factors leading to such an outstanding performance are: 1) the consideration of the cross-layer dependency achieved by targeting attentionwise reconstruction, and 2) efficient weight-rounding optimization based on pre-computations.

Zero-shot task performance We evaluate the reasoning performance of quantized models using zero-shot tasks ( e.g., ARC [4], HellaSwag [34], and MMLU [8]). We note that the zero-shot setting was ensured in our experiments because we used excerpts from randomly crawled websites (not task-specific data) as a calibration dataset. From the zero-shot results in Table 3 and Table 13 (see Appendix J), we observe that the proposed aespa performs the best in almost all cases, and the performance gap between aespa and the existing methods is large for 2-bit.

Time cost We summarize the processing times of the different quantization algorithms in Appendix K. We note that the processing time of aespa includes the time required for pre-computations (lines 2-4 in Algorithm 1). As expected, aespa completes quantization much faster than BRECQ. For example, while BRECQ requires more than 10 GPU hours for OPT-1.3B, aespa completes quantization in 1.24 hours, which demonstrates the effectiveness of the proposed pre-computation-based loss computation strategy. Although other block-wise methods (OmniQuant/AffineQuant) perform quantization faster than aespa for hyper-scale models, they suffer from unstable training processes or exhibit poor PPL performance ( e.g., PPL of OmniQuant is larger than 10 3 for OPT-6.7B; see Table 1). In addition, we observe that OPTQ performs quantization quickly, but its 2-bit performance collapses regardless of the model size (see Table 9 in Appendix I). Except for aespa , Z-FOLD is the only method that shows both reasonable performance and processing time.

Discussion In real situations, when one needs to preserve the performance of the original model as much as possible, the proposed aespa would be an intriguing solution. In particular, when deploying LLMs on resource-constrained platforms where up to 7B models are commonly employed ( e.g., mobile devices), aespa would be a good fit. Even when fast quantization of hyper-scale models is required, aespa can be used with a slight modification. Specifically, in time-limited cases, one can skip the weight-rounding optimization (lines 5-8 in Algorithm 1) and simply determine the quantization parameters using the proposed Hessian that considers the cross-layer dependency (line 4 in Algorithm 1). In doing so, we can not only save the time required to optimize a weightrounding mechanism, but also save the memory required to store pre-computed values ( E [ K T K ] and E [ Q T Q ] ). Indeed, when performing only quantization parameter computation, we achieved a significant reduction in the processing time (see Table 15 in Appendix K) while still exhibiting better performance than conventional methods (see Table 6 in Appendix D).

5 Conclusion

We proposed a next-level PTQ scheme for Transformers, called aespa . By targeting the attentionwise reconstruction while quantizing Transformers layer-wise, we could consider the cross-layer dependency within the attention module and complete the quantization much faster than the existing

* 'NaN' means that loss diverges in the quantization process.

* Results for high bit-widths are provided in Appendix J due to the page limitation.

approach for block-wise reconstruction ( i.e., BRECQ). Extensive experiments on language models have demonstrated the outstanding performance of aespa .

Limitations and future work While we focused on the attention output, the output of the entire Transformer block (containing fully connected layers) can be used to consider the dependencies between more layers. However, in this case, the objective functions would be more complicated than those in (13) and (25) due to nonlinear activation functions ( e.g., SiLU for LLaMA models), normalization layers, and weights of larger dimensions. Enhancing the quantization performance by developing an efficient form of the reconstruction error for the Transformer block would be an interesting future work. Furthermore, while we focused on weight-only quantization, activations may need to be quantized to deploy AI models on integer-only arithmetic hardware ( e.g., NPU). Extending the proposed aespa for weight-activation quantization by integrating existing techniques to suppress activation outliers [33, 1] is also an interesting research direction. Finally, while we verified the performance of aespa with LLMs, we believe that aespa can also be used for the quantization of diffusion models. To that end, we may need to incorporate some diffusion-specific quantization strategies to overcome output distribution discrepancies over different time steps ( e.g., grouping of time-steps with similar distributions [32], temporal feature preservation [9], and separate quantization for shortcuts in U-Net [17]), which will be considered in our future studies.

References

Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. QuaRot: Outlier-free 4-bit inference in rotated LLMs. arXiv:2404.00456 , 2024.

Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical Gauss-Newton optimisation for deep learning. In International Conference on Machine Learning , pages 557-565. PMLR, 2017.

Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. QuIP: 2-bit quantization of large language models with guarantees. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv:1803.05457v1 , 2018.

Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. Learned step size quantization. In International Conference on Learning Representations (ICLR) , 2019.

Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate posttraining quantization and pruning. Advances in Neural Information Processing Systems , 35:44754488, 2022.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained Transformers. In The Eleventh International Conference on Learning Representations , 2023.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations , 2020.

Yushi Huang, Ruihao Gong, Jing Liu, Tianlong Chen, and Xianglong Liu. TFMQ-DM: Temporal feature maintenance quantization for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7362-7371, 2024.

Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration sets. In International Conference on Machine Learning , pages 4466-4475. PMLR, 2021.

Yongkweon Jeon, Chungman Lee, Eulrang Cho, and Yeonju Ro. Mr. BiQ: Post-training nonuniform quantization based on minimizing the reconstruction error. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12329-12338, 2022.

Yongkweon Jeon, Chungman Lee, and Ho-young Kim. GENIE: show me the data for quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12064-12073, 2023.

Yongkweon Jeon, Chungman Lee, Kyungphil Park, and Ho-young Kim. A frustratingly easy post-training quantization scheme for llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 14446-14461, 2023.

Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4350-4359, 2019.

Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. SqueezeLLM: Dense-and-sparse quantization. arXiv:2306.07629 , 2023.

Yann LeCun, John S Denker, Sara A Solla, Richard E Howard, and Lawrence D Jackel. Optimal brain damage. In Advances in Neural Information Processing Systems (NIPS) , volume 2, pages 598-605, 1989.

Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-Diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 17535-17545, 2023.

Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. BRECQ: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations (ICLR) , 2021.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for llm compression and acceleration. In MLSys , 2024.

Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. AffineQuant: Affine transformation quantization for large language models. arXiv:2403.12544 , 2024.

Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. 1993.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv:1609.07843 , 2016.

Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? Adaptive rounding for post-training quantization. In International Conference on Machine Learning (ICML) , pages 7197-7206, 2020.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(1):5485-5551, 2020.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684-10695, June 2022.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. BLOOM: A 176B-parameter open-access multilingual language model. arXiv:2211.05100 , 2022.

Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. OmniQuant: Omnidirectionally calibrated quantization for large language models. arXiv:2308.13137 , 2023.

Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations , 2018.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv:2302.13971 , 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288 , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Changyuan Wang, Ziwei Wang, Xiuwei Xu, Yansong Tang, Jie Zhou, and Jiwen Lu. Towards accurate post-training quantization for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16026-16035, 2024.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning , pages 38087-38099. PMLR, 2023.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4791-4800, 2019.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models. arXiv:2205.01068 , 2022.

Appendices

A Pseudo-code for the proposed aespa

In this appendix, we provide the pseudo-code for the proposed aespa excluded in the main text due to the page limitation.

As mentioned, the proposed aespa consists of two main steps; aespa first determines the quantization parameters ( i.e., scale s and zero-point z in (3)) together with foldable parameters, as in [19, 13, 27, 20] (see line 4 in Algorithm 1), and then optimizes an integer weight W int for each weight (see lines 5-8 in Algorithm 1). We emphasize that each iteration for the integer weight optimization can be performed efficiently based on pre-computed values ( i.e., E [ XX T ] , E [ XA T AX T ] , E [ K T K ] , and E [ Q T Q ] in Table 4). We also note that while we have used Z-FOLD in computing the quantization parameters and used AdaRound in optimizing integer weights, our refined objectives in (17), (21), and (22) can be integrated into any layer-wise quantization scheme without effort because we only used the definition of the attention operation in our derivation.

B Validity of the proposed separate quantization strategy

We conduct experiments to demonstrate the importance of targeting attention-wise reconstruction and the validity of the proposed separate quantization strategy. In our experiments, we learn a weightrounding policy using conventional AdaRound, but we set the loss function for each projection as the attention reconstruction error in (7) (not the layer reconstruction error; see Figure 2(c)).

Table 5 summarizes the quantization performance of AdaRound, BRECQ, and our approach on the OPT-125M model. As evident, our approach uniformly outperforms AdaRound for all bitwidths, although both methods quantize models layer-wise. This is because we can consider the cross-layer dependency ( i.e., the relationship between the query, key, and value) by targeting attentionwise reconstruction, which differs from AdaRound wherein layers are considered independent. Furthermore, once we target attention-wise reconstruction, separate layer-wise quantization does not incur severe performance degradation compared to the joint quantization method (BRECQ). Indeed, our approach causes only a marginal performance degradation for 2-bit and exhibits comparable performance for 3-bit and 4-bit.

One might wonder about the strategy of quantizing more than one layer jointly while maintaining remaining weights with full-precision, e.g., simultaneous quantization of the query and key projections while fixing the value projection with full-precision. To say the conclusion first, in this case, we cannot use the proposed pre-computation-based loss computation strategy (see Section 3.4), resulting in a much longer quantization processing time. Specifically, when quantizing W Q and W K simultaneously, the attention reconstruction error is expressed as

where

Then, by taking similar steps as in Section 3.3 ( i.e., approximating ∆ A with its first-order Taylor series and constructing an upper bound of ∆SA Q,K ), we can obtain the following objective:

Obviously, the objective becomes much more complex than the proposed ones in (15) and (16), and it would be difficult to simplify and accelerate the loss computation by exploiting pre-computed values as in aespa . In fact, without the proposed pre-computation-based loss computation, the simultaneous quantization of W Q and W K requires 3.5 hours for the quantization of OPT-125M, which is about 44 times longer than the proposed aespa and even 1.9 times longer than BRECQ.

C Refined quantization objective (16) for the key projection

When quantizing the key projection W K , we fix the query and value projections with full-precision. In this case, the attention reconstruction error ∆SA K can be expressed as

where

To avoid the computational overhead of repetitive softmax operation, we approximate ∆ A with its first-order Taylor series, which leads to

Furthermore, to reduce the huge memory cost required to store the Jacobian J softmax having L 3 elements (see Footnote 5), we establish an upper bound of (25) and then use it as a surrogate of ∆SA K . Specifically, we separate the term ∥ Q ∆ W K X J T softmax V ∥ 2 F into two components as follows:

Noting that the term ∥ ∥ J T softmax V ∥ ∥ 2 F is not affected by the quantization of W K and thus fixed in the quantization process, we minimize ∥ Q ∆ W K X ∥ 2 F to enforce ∆SA K to be small, which leads to the proposed objective in (16).

D Effectiveness of the proposed Hessian in (18)

We recall from Section 3.4 that the proposed quantization objective for the value projection is

which implies that the Hessian H V for each row of W V is

We note that the proposed Hessian H V differs from

which has been commonly used as an approximated Hessian in existing methods [6, 7, 3, 13]. The key reason for the difference is that we consider the dependency between the query, key, and value projections by targeting attention-wise reconstruction, whereas the previous methods assumed independence.

To observe the effect of considering the cross-layer dependency, we use different Hessians ( i.e., H V and H ) when quantizing language models via Z-FOLD and then compare the performance of the quantized models. As Table 6 shows, the quantization performance is much better when the proposed Hessian H V is used, which demonstrates the importance of considering the cross-layer dependency.

(b) PTB

(c) C4

E Proof of (19) and (21)

Note that E [ ∥ ∥ K ∆ W Q X ∥ ∥ 2 F ] = E [ ∥ ∥ vec ( K ∆ W Q X )∥ ∥ 2 2 ] , where vec( · ) denotes the vectorization operation. Then, by exploiting the following properties of Kronecker product

we have

which is the desired result in (19).

We now prove (21). By combining (19) and (20), we have

Note that since E [ XX T ] and E [ K T K ] are symmetric, there exist G X and G K such that

Then, by following the steps used to derive (26) in the reverse order, we have

which completes the proof of (21).

F Integration of proposed loss functions into existing PTQ schemes

We recall that we only utilized the definition of the attention operation when developing the proposed loss functions for the attention output reconstruction. Therefore, our loss functions can be integrated into any PTQ schemes based on layer-wise reconstruction and used to enhance their performance. In this section, we describe how to combine our loss functions with existing quantization schemes by taking AdaRound as an example.

In short, AdaRound learns a weight-rounding mechanism by solving the following optimization problem [23]:

where B is the continuous variable to be learned, h is the rectified sigmoid function, and ˜ W is the soft-quantized weights defined as

One can see that the loss function of AdaRound consists of two components, layer-wise reconstruction error and weight-rounding loss.

To consider the cross-layer dependency between W Q , W K , and W V in the learning process, we integrate the proposed loss functions developed for the attention output reconstruction into (27). In other words, we replace the layer-wise reconstruction error in (27) with our loss functions in (17), (21), and (22). For example, when learning the rounding policy for the query projection matrix W Q , the objective of the proposed aespa is expressed as

where ∆ W Q = W Q -˜ W Q .

G Complexity analysis for conventional block-wise quantization schemes

Recall from (7) that conventional block-wise quantization schemes require to compute SA( ̂ Q , ̂ K , ̂ V ) in each iteration. This means that for each input sequence, one needs to perform

forward pass for ̂ Q , ̂ K , and ̂ V : 3 d h L (2 d -1) flops

matrix multiplications for computing ̂ Q ̂ K T and ̂ A ̂ V : 4 d h L 2 -d h L -L 2 flops

softmax operation with additional scaling ( i.e., softmax( ̂ Q ̂ K T √ d h ) ): 3 L 2 + d h L -L flops

final computation of reconstruction error: 3 d h L -1 flops

If B input sequences are used in each quantization iteration, then the total number of flops required in conventional methods is

Comparison of C aespa and C exist We now compare the complexities of aespa and conventional block-wise quantization methods in terms of the number of flops. Table 7 summarizes the computational costs required to quantize different sizes of OPT models. For conventional methods, we report the cost of using four sequences in each iteration ( B = 4 ). We observe that the computational cost of aespa is considerably lower than that of conventional methods. In particular, for small-scale models ( e.g., OPT-125M, OPT-350M, and OPT-1.3B), aespa performs ten times fewer number of flops. One can notice that the gap between C aespa and C exist decreases as the model size increases. This is because the hidden size d exceeds the sequence length L (which is fixed for all models) as the model size increases. Nevertheless, aespa still incurs a lower computational cost, and the gap increases if conventional methods use larger batch sizes.

H Comparison with block-wise PTQ schemes

We provide experimental results excluded from the main text due to page limitations.

(b) Performance on PTB

OPT

125M

38.99

41.04

42.34

42.99

40.50

46.93

56.88

51.47

44.96

90.22

NaN

177.8

Precision

FP16

INT3

INT2

Method

Baseline

BRECQ [18]

OmniQuant [27]

AffineQuant [20]

aespa

BRECQ [18]

OmniQuant [27]

AffineQuant [20]

aespa

BRECQ [18]

OmniQuant [27]

AffineQuant [20]

1.3B

20.29

20.97

21.32

21.26

20.78

23.41

25.11

24.38

22.35

344.9

377.9

75.25

2.7B

17.97

18.41

18.70

18.49

18.30

19.82

22.59

21.03

19.48

282.0

2.0e3

47.07

aespa

99.12

37.19

32.57

6.7B

15.77

OOM

16.04

16.02

15.84

OOM

18.33

17.40

16.28

OOM

7.7e3

37.90

22.80

7B

10.12

OOM

10.57

10.53

10.43

OOM

11.98

11.92

11.45

OOM

33.51

29.33

19.83

LLaMA

13B

9.081

OOM

9.330

9.325

9.277

OOM

10.24

10.24

9.818

OOM

NaN

18.58

15.65

30B

8.159

OOM

8.354

8.355

8.283

OOM

9.065

8.998

8.684

OOM

17.38

NaN

12.98

I Comparison with layer-wise PTQ schemes

We provide experimental results excluded from the main text due to page limitations.

I.1 Results on OPT models

(a) WikiText-2

I.2 Results on BLOOM models

I.3 Results on LLaMA models

I.4 Results on LLaMA2 models

J Results for zero-shot tasks

We provide INT3 zero-shot performance results that are excluded from the main text due to page limitations.

K Time and memory cost comparison

(a) INT2 quantization processing time

*

'OOM' means that out-of-memory issues occur when quantizing models with a single NVIDIA A100 GPU.

Table 14 summarizes the processing time and memory cost of different quantization algorithms. We note that the processing time of the proposed aespa includes the time required for pre-computations (lines 2-4 in Algorithm 1).

As expected, aespa completes quantization much faster than BRECQ. For example, while BRECQ requires more than 10 hours to quantize OPT-1.3B, aespa completes quantization in 1.24 hours, which demonstrates the effectiveness of the proposed objectives and pre-computation-based loss computation strategy. Although other block-wise PTQ methods (OmniQuant/AffineQuant) perform quantization faster than aespa for hyper-scale models, they suffer from unstable training process or exhibit poor PPL performance ( e.g., PPL of OmniQuant is larger than 10 3 for OPT-6.7B; see Table 1). We also observe that OPTQ performs quantization very fast, but its PPL performance collapses completely regardless of the model size (see Table 9). Except aespa , Z-FOLD is the only method that shows both reasonable performance and processing time.

In real situations, when one needs to preserve the performance of the original model as much as possible, the proposed aespa would be an intriguing solution. In particular, when deploying LLMs on resource-constrained platforms where up to 7B models are commonly employed ( e.g., mobile devices), aespa would be a good fit. Even when fast quantization of hyper-scale models is needed, aespa can be used with a slight modification. Specifically, in time-limited cases, one can skip weight-rounding optimization (lines 5-8 in Algorithm 1) and simply perform the quantization parameter computation (line 4 in Algorithm 1) using the proposed Hessian that considers the cross-layer dependency (see (18)). In doing so, we can not only save the time required to perform weight-rounding learning, but also save the memory required to store pre-computed values ( E [ K T K ] and E [ Q T Q ] ). Indeed, when performing only quantization parameter computation, we achieved a significant reduction in the processing time (see Table 15 below) while still exhibiting better performance than conventional methods (see Table 6 in Appendix D).

L Experimental results for different calibration datasets

One might wonder why the PPL performances of OmniQuant summarized in Table 1 are much worse than those reported in the original paper [27]; INT2 PPL performances of quantized LLaMA models are 18.18, NaN, and 10.15 for WikiText-2 in Table 1, which are worse than the values (15.47, 13.21, and 8.71) reported in [27]. This is because we used a different calibration dataset for quantization. Specifically, we used C4 when constructing a calibration dataset, while [27] used WikiText-2.

Additionally, we evaluate the performance of the proposed aespa using WikiText-2 as a calibration dataset. From Table 16, we observe that when calibration data are sampled from WikiText-2, our results for OmniQuant are comparable with those reported in the original paper [27]. While it has been reported that the performance variance of OmniQuant across different calibration datasets is low for INT3 and INT4 (see [27, Table A10]), such low variance does not hold for INT2. Furthermore, we observe that the proposed aespa outperforms OmniQuant regardless of the type of the calibration dataset.

*

Test dataset: WikiText-2

M Quantization performance of aespa for high bit-widths

While previous results demonstrate that the proposed aespa is very competitive for low-bit quantization ( e.g., INT2 and INT3), one might wonder whether aespa can preserve the performance of the original full-precision model at high bit-widths. We thus evaluate INT4 and INT6 quantization performances of aespa with LLaMA models. From Table 17, we observe that aespa almost preserves the performance of the original full-precision model for the INT6 quantization. Even for the INT4 quantization, the performance degradation is very marginal ( e.g., less than 1% degradation for 13B and 30B models).

N Experimental results for different seeds

We recall that when constructing a calibration dataset, we randomly draw 128 sequences from the C4 dataset [24]. By changing the seed for the sampling, different calibration datasets can be constructed, which leads to different quantization results. In this appendix, we report the corresponding results and overall statistics.

Table 18: Quantization performance (PPL ↓ ) of aespa on OPT models for different seeds.

(a) WikiText-2

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state our main contribution (proposal of a next-level quantization algorithms that balance accuracy and efficiency for the quantization of hyper-scale Transformer models).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations together with some future research directions in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have included detailed derivations and complete proofs of the proposed quantization loss functions in Sections 3.3, 3.4, 3.5 and Appendices C, E, F, G.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have included detailed instructions, e.g., pseudo-code for the proposed algorithm (see Algorithm 1), hyperparameter settings, and stopping criterion (see Section 4), to reproduce the main experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code will be available at https://github.com/SamsungLabs/aespa .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have specified all the training details including hyperparameter settings and stopping criterion in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have reported the results accompanied by statistics (e.g., mean and standard deviation) in Appendix N.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have mentioned that we used a single NVIDIA A100 GPU (80 GB) in our experiments (see the last sentence in Section 4.1).

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research have been conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed potential positive societal impacts of our work in Section 1, Section 4, and Section 5. As evident from our experiments, the proposed algorithm can serve as a useful quantization solution that pursues both efficiency and accuracy when deploying LLMs on resource-constrained devices.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no risks for misuse (e.g., pretrained language models, image generators, or scraped datasets).

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the original paper and the codes provided by the authors of the original paper as in Footnote 8.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript proposed a layer-wise post-training quantization (PTQ) algorithm called aespa aiming at quantizing large transformer models, and refined quantization objectives for the attention layer which can accelerate the quantization process by pre-computation.\nThe authors gave clearly descriptions for the formulas which are used as quantization objectives and showed analysis results that aespa has a lower computational cost. The authors also showed stable experiment results comparing aespa with other quantization methods. Overall, I would suggest an Accept.\n\nSTRENGTHS:\n1. This PTQ algorithm is proved by mathematical methods, so it is easy to be applied to other transformer models.\n2. The proposed algorithm considers the cross-layer dependency by targeting attention modules. In detail, it quantizes query, key, and value in an attention module separately: when quantizing one projection, the other 2 projections are fixed with full precision, decreasing the heavy computation of block-wise quantization.\n3. Compared with conventional block-wise quantization and layer-wise PTQ schemes, the perplexity of aespa showed good results\n\nWEAKNESSES:\n1. This quantization algorithm only focuses on the attention module, targeting the cross-layer dependency within an attention module, not cross-layer dependency within the full model.\n2. In Experiments and Appendices, most tables compare the performance (perplexity), time, and memory cost. There is only one table (Table 10) showing the accuracy of zero-shot performance. There is no explicit result showing how aespa is more accurate than other methods. How much is the decrease in the accuracy compared with the original floating point models?\n\nQUESTIONS:\n1. When quantizing a transformer model with the proposed algorithm, will all weights and activations stay the same quantization precision? Or only the weight will be quantized?\n2. Will it be possible to discuss more on the accuracy of aespa compared to other methods?\n\nLIMITATIONS:\nThe authors dicusse dthe limitation of aespa: this algorithm only focused on the attention module as a block.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

comments: SUMMARY:
The paper introduces 'aespa,' a novel post-training quantization (PTQ) algorithm designed to balance accuracy and efficiency in quantizing hyper-scale Transformer models. This method quantizes layer-wise for efficiency and employs attention-wise reconstruction to account for cross-layer dependencies. Extensive experiments and complexity analysis demonstrate that aespa outperforms existing PTQ methods, especially in low-bit quantization scenarios.

STRENGTHS:
1. Aespa, which combines layer-wise quantization with attention-wise reconstruction, is a significant advancement in the field of model quantization.
2. This paper emphasizes a pre-computation-based loss computation strategy, demonstrating its significant speed advantage over traditional block-wise PTQ methods like BRECQ.
3. Extensive experiments on diverse language models and datasets provide a comprehensive evaluation of aespa's performance.
4. Enhancing the credibility of this method would benefit from conducting thorough ablation studies.

WEAKNESSES:
1. Further experiments on LLMs are crucial, particularly with the LlaMa 2 & 3 family.
2. How were other layers updated within a transformer block, apart from the attention module? Were they done layer-wise or block-wise?
3. The implementation details require clarification. Specifically, in the attention module, Q, K, and V are updated separately. How many iterations are necessary for each layer? Does the order of updating Q, K, and V impact performance?

QUESTIONS:
please refer to Weakness.

LIMITATIONS:
A more in-depth discussion on implementation details would be beneficial.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAs a cost-effective alternative, learning-free PTQ schemes have been proposed for LLMs. Still, the performance is somewhat limited because they cannot consider inter-layer dependency within the attention module, a significant feature of Transformers. This paper propose a PTQ algorithm that balances accuracy and efficiency. The key idea of the proposed algorithm called aespa is to perform quantization layer-wise for efficiency while targeting attention-wise reconstruction to consider the cross-layer dependency. Through extensive  experiments on various language models and complexity analysis, it demonstrates that aespa is accurate and efficient in quantizing Transformer models.\n\nSTRENGTHS:\nThe scheme aims to reconstruct the attention output to consider cross-layer dependency while quantizing models layer-wise to pursue efficiency. It is different from previous layer-wise error minimization. \n\nFrom extensive experiments on language models, it demonstrates that the approach outperforms conventional schemes by a significant margin, particularly for low-bit precision (INT2).  \n\nTo accelerate the quantization process, it proposes refined quantization objectives for the attention module. Through a complexity analysis, it demonstrates that about 10 times faster quantization than existing block-wise approaches can be achieved by exploiting the proposed objectives.\n\nWEAKNESSES:\nI am a bit confused about the results of baselines. For OmniQuant, in the original paper, the perplexity on wikitext2 for 2bit weight quantization on llama-7B, 13B, and 30B are 15.47, 13.21, 8.71, respectively. However, in this paper, they become 24.28, 22.85,13.04. The differences are too large. Though the authors mention that the difference is due to different calibration dataset, the ablation study in OmniQuant demonstrates that different calibration datasets (wikitext2, C4, and pile) do not lead to significant perplexity difference with very small variance, see table A10 of OmniQuant.  So the performance of OmniQuant using another calibration dataset such as C4, should also achieve a similar performance close to the original paper. However, the results in this paper are significantly different from the original paper. It is better to provide more discussions about this problem.\n\n\nThe OmniQuant and AffineQuant methods demonstrate the performance of quantization in groups. Smaller groupsize leads to better performance. It is better to include some ablation study with different groupsizes. \n\n\nLLaMA3 is published. It is better to have some results for recent models such as LLaMA2.\n\n\nMost of the results are shown in the appendix. It is better to organize the experimental part better to include the main results in the main paper.  Currently there are only block-wise perplexity comparison in the main paper. It may be better to put zero-shot evaluation in the main paper.\n\nQUESTIONS:\nSee the weakness.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

comments: SUMMARY:
This paper propose a new quantization strategy that balances accuracy and efficiency that reconstruct the attention output to consider cross-layer dependency. To accelerate the quantization process, this paper proposes refined quantization objectives for the attention module. This approach gets better results in low-bit precision.

STRENGTHS:
1 This article does a lot of experiments, which are also in the appendix section.

2 This paper obtains higher accuracy at ultra-low bit than any other work.

WEAKNESSES:
1 Formula derivation is too redundant.

2 Paper does not make clear the trade-off between accuracy and efficiency in introduction.

QUESTIONS:
1 In Figure 2, what is Layer, what is Block?

2 Following the method in paper, will the training time longer than ordinary PTQ?

LIMITATIONS:
I did not identify any issues related to limitations and potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-18

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a post-training quantization approach for transformers. The proposed algorithm finds a better balance between accuracy and efficiency. The method is to perform quantization in a layer-wise way for efficiency and the optimization objective is to maintain the reconstruction of the quantized model to obtain good accuracy. Experimental results on 3 datasets over 2 transformers (OPT and LLaMA) are reported at the low bit widths (3 Bits and 2 Bits). The proposed approach outperforms other baselines at the low bit widths.\n\nSTRENGTHS:\n- This paper is well written and organized in general.\n\n- The mathematical formulation and derivation are clear and easy to follow.\n\n- Results on low bit widths outperform several baselines including BRECQ, OmniQuant, and AffineQuant.\n\n- Experimental results on two sota LLMs (OPT and LLaMA) are reported.\n\nWEAKNESSES:\n- The novelty in this paper is limited, as similar formulation and optimization methods have been widely used in prior papers. \n\n- Experiments are not solid. Some important results and comparisons are missing.\n\n- The paper lacks analysis and results to support the claims of the contributions.\n\nPlease check the questions section for details.\n\nQUESTIONS:\n- The idea of reducing reconstruction error for quantization is not new - several prior works have already proposed this idea and adopted it in their works. The optimization method (iterative optimization - updating one variable and fixing others) is not new. \n\n- Actually, this paper only considers the reconstruction of the current quantization layer, which is sub-optimal. A better way is to optimize the reconstruction of the last layer (the reconstruction of the whole network). Because output errors could accumulate, small output errors at the first layers may change to large ones when going to the last layers. Only the reconstruction of the last layer's output can reflect the impact on the network.\n\n- This paper only reports results at low bit widths (4, 3, and 2 Bits). Although it outperforms the three baselines (BRECQ, OmniQuant, AffineQuant), the accuracy loss is noticeable at low bit widths. How about the results at higher bit widths (6 bits or 8 bits)? Is that possible to maintain the accuracy at higher bit widths? People want to see quantization to a certain number of bits without losing accuracy.\n\n- One important contribution claimed by this paper is the capability to handle situations where frequent model updates and multiple hyperparameter tunings are required. However, we haven't found practical results about the analysis and comparison of such situations. Although the authors gave the time complexity of the algorithm and provided the time to run the algorithm, I think this is different with the real situation when weights are frequently updated. Moreover, some baselines also show faster speed (OPTQ, Z-FOLD) than the proposed method.\n\n- I suggest that the authors cite the latest related post-training quantization works for generative models or diffusion models [1][2][3][4][5], discuss the differences with them, and compare with them if possible.  \n\n[1] Towards Accurate Post-training Quantization for Diffusion Models\n\n[2] TFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Models\n\n[3] Post-training Quantization on Diffusion Models\n\n[4] Q-Diffusion: Quantizing Diffusion Models\n\n[5] PTQD: Accurate Post-Training Quantization for Diffusion Models\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-29"}