Attention-aware Post-training Quantization without Backpropagation

Authors: Junhan Kim, Ho-young Kim, Eulrang Cho, Chungman Lee, Joonyoung Kim, Yongkweon Jeon
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Quantization, Hyper-scale LLMs, Attention, Hessian

Abstract:
Quantization offers a promising solution for deploying large-scale language models (LLMs) on resource-constrained devices. However, early quantization methods, developed for smaller networks like ResNet, rely on gradient-based optimization, which becomes impractical for hyper-scale LLMs with billions of parameters. While recently proposed backpropagation-free post-training quantization (PTQ) methods alleviate this issue, their performance is limited by a lack of inter-layer dependency consideration. In this paper, we introduce a novel PTQ algorithm that incorporates inter-layer dependencies without relying on backpropagation. The key innovation is the development of attention-aware Hessian matrices that capture inter-layer interactions within the attention module. Extensive experiments demonstrate that our approach significantly outperforms conventional PTQ methods, particularly at low bit-widths.

Decision: reject
Reviewer scores: 6, 3, 3, 3

Reviews and discussion:
<thread>
creation date:
1730205505002

thread_type:
official_review

<thread_object>
creator: reviewer_kfpf

<content>
summary:
This paper presents a training-free post-training quantization method based on GPTQ. It introduces inter-layer interaction by calculating Hessian matrices using an attention module instead of a simple linear module in LLMs. Additionally, the paper proposes techniques to improve the efficiency of Hessian matrix calculations.

strengths:
1. Introducing inter-layer interaction in a training-free manner is innovative.
2. The paper is well-written.

weaknesses:
1. The experimental setup is somewhat outdated. Additional experiments on newer models, such as LLama-2 and LLama-3, are needed.
2. Although the paper introduces a training-free PTQ method, it may be slower than training-based methods. For example, Table 2 shows that BOA takes 1 hour to quantize 2.7B models, while GPTQ quantizes larger 13B models in only 21 minutes. OmniQuant, a training-based method, requires only ~1.1 hours for 7B models. The paper should provide comprehensive comparisons of quantization times to demonstrate the proposed method's effectiveness.
3. The paper focuses on 2-bit per-channel quantization and mentions that "group-wise parameters result in additional memory costs and processing time during inference." However, weight-only quantization aims to alleviate memory constraints during the decoding stage. Group-wise quantization introduces negligible overhead but significantly improves performance and is a common practice in existing inference engines. Therefore, the paper should include results for group-wise quantization.

questions:
Please refer weaknesses for details.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730601454521

thread_type:
official_review

<thread_object>
creator: reviewer_rmbt

<content>
summary:
This paper presents a post-training quantization method called BOA that incorporates inter-layer dependencies without relying on backpropagation. BOA leverages attention-aware Hessian matrices to capture dependencies within the attention module, a relatively rare approach in existing PTQ methods. Additionally, BOA demonstrates compatibility with techniques like SmoothQuant and Z-FOLD, allowing for further enhancements in quantization performance. However, despite these strengths, BOA does not show sufficient memory and processing time benefits compared to existing PTQ methods. The experiments are conducted on outdated models, and the comparison methods lack recent advancements. Adding more experiments with up-to-date models and techniques would strengthen the paper.

strengths:
1.	The paper introduces an innovative PTQ method that cleverly captures inter-layer dependencies within attention modules through attention-aware Hessian matrices while avoiding backpropagation overhead. 
2.	BOA is compatible with other techniques, such as SmoothQuant and Z-FOLD, enabling further improvements in quantization accuracy by integrating different quantization strategies.

weaknesses:
1.	The experiments are primarily conducted on BLOOM, LLaMA1, and OPT models, which are somewhat outdated compared to current state-of-the-art models. The paper lacks validation on more recent models, such as the LLaMA3 series.
2.	Although the paper introduces various techniques to reduce computational overhead and claims to use a Hessian-based strategy to avoid time-consuming gradient-based optimization, as shown in Table 13, BOA’s actual overhead in terms of memory and processing time is greater than GPTQ. Additionally, in Tables 3, 4, and 5, even under 2-bit quantization, BOA's improvement over GPTQ is marginal. For Table 6, it’s worth noting that GPTQ can also integrate certain quantization algorithms, like QuaRot [1] and SpinQuant [2], to achieve better results. Including comparisons with these methods is recommended.	

[1] Ashkboos S, Mohtashami A, Croci M L, et al. Quarot: Outlier-free 4-bit inference in rotated LLMs. arXiv preprint arXiv:2404.00456, 2024.
[2] Liu Z, Zhao C, Fedorov I, et al. SpinQuant—LLM quantization with learned rotations. arXiv preprint arXiv:2405.16406, 2024.

questions:
1.How does the performance of BOA compare when tested on more advanced models, such as the LLaMA3 series, instead of the relatively outdated models used in the paper?
2.How does BOA's accuracy compare to more recent quantization methods, such as QuaRot and SpinQuant?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730607930036

thread_type:
official_review

<thread_object>
creator: reviewer_xkim

<content>
summary:
The paper introduced the BOA post-training quantization algorithm designed for LLMs that overcomes the limitations of traditional quantization methods, which struggle with inter-layer dependencies and backpropagation requirements in LLMs. BOA leveraged attention-aware Hessian matrices to better capture inter-layer interactions within the attention module, enhancing performance, especially at low bit-widths. The algorithm employed Hessian relaxation and head-wise simultaneous quantization, to attempt to reduce computational and memory costs, making it feasible for quantizing LLMs without backpropagation.

strengths:
The topic of this paper is of significant importance and represents one of the most active and rapidly evolving research areas in the field. As LLMs grow increasingly complex, their deployment on resource-constrained devices requires innovative solutions to reduce computational and memory demands. Quantization, as a compression technique, has gained considerable traction for enabling efficient deployment of LLMs without sacrificing model accuracy.

weaknesses:
The technical approach of this paper is relatively straightforward, lacking intricate or highly novel methodologies. Additionally, certain English terminology within the paper is used imprecisely, which may affect clarity and readability. The comparison methods are somewhat limited, providing a narrow benchmark for evaluating the proposed technique. Moreover, while the experimental results demonstrate some improvements, the advantage over existing methods is not substantial, suggesting the need for further validation, such as, SmoothQuant, LLMC,QuIP etc.

questions:
The advantage over existing methods is not substantial, suggesting the need for further validation, such as, SmoothQuant, LLMC,QuIP etc.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709889518

thread_type:
official_review

<thread_object>
creator: reviewer_gh6w

<content>
summary:
This paper presents a novel post-training quantization (PTQ) method, termed BOA (Backpropagation-free Optimization for Attention-aware PTQ), targeting large language models (LLMs) without relying on backpropagation. The approach introduces attention-aware Hessian matrices that capture inter-layer dependencies within the attention module, aiming to improve quantization accuracy, especially at low bit-widths (e.g., INT2). BOA incorporates techniques like Hessian relaxation and efficient computation of inverse Hessians to mitigate the high computational costs. The method is benchmarked against existing PTQ approaches on LLMs, demonstrating improved performance in terms of perplexity and zero-shot task accuracy.

strengths:
1. The proposed BOA consider inter-layer dependencies within the attention module when optimize a weight-rounding mechanism. It is beneficial to maintain higher quantization accuracy, especially at low-bit precision.

2. The proposed BOA method demonstrates impressive results, especially in the low-bit regime (e.g., INT2 quantization).

3. The paper includes extensive experiments across multiple model types and sizes, demonstrating  scalability across LLMs of different parameter counts.

weaknesses:
1. Novelty Limitations: The primary contribution, the attention-aware Hessian matrix, is an incremental improvement over existing Hessian-based PTQ methods. While capturing inter-layer dependencies within the attention module is beneficial, the idea is not a novel quantization paradigm. 

2. The authors introduce optimizations approaches like Hessian relaxation and efficient computation of inverse Hessians, but the results did not show the effect of these optimization methods.

questions:
Refer to 2 in weakness. What is the effectiveness of proposed approaches in terms of efficiency?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732159232012

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Look forward to further discussions

comment:
Dear reviewers,

We sincerely appreciate the reviewers' valuable feedback and constructive suggestions. 

We have made a thorough effort to address all the concerns raised. 

Below is a summary of the key points in our responses.

**1. Additional Experimental Results**
 
 - We have included 
   - results for recent language models such as LLaMA2 and LLaMA3
   - integration results with the recent transformation method QuaRot [1]
   - group-wise quantization results
 - Overall, we have achieved **at least 8% improvement** in accuracy across almost all quantization configurations, which we believe is a significant advancement.

**2. Emphasis on Our Contribution**
 - While two reviewers acknowledged the innovation of our method, two others expressed concerns about its novelty.
 - The core novelty of our work lies in capturing inter-layer dependencies **without relying on backpropagation**, unlike conventional methods (e.g., BRECQ [2], OmniQuant [3], AffineQuant [4]) that depend on time-intensive, gradient-based optimization.

**3. Comprehensive Comparisons of Quantization Processing Times**
 - We have compared the quantization processing times of the proposed method, the conventional training-free method (GPTQ [5]), and training-based methods (OmniQuant [3] and AffineQuant [4]).
 - Compared to existing training-based approaches, the proposed method enables **faster** quantization while delivering **significantly better** performance.
 - Although the proposed method requires more processing time than GPTQ due to the consideration of inter-layer dependencies, this consideration results in significantly improved performance, with **over 8% improvement**.

For more details, we kindly invite the reviewers to refer to our point-by-point responses.

Once again, we are grateful to the reviewers for their time to read and review our paper, and we look forward to further constructive discussions.

[1] S. Ashkboos et. al., "QuaRot: Outlier-free 4-bit inference in rotated LLMs," NeurIPS 2024.

[2] Y. Li et. al., "BRECQ: Pushing the limit of post-training quantization by block reconstruction," ICLR 2021.

[3] W. Shao et. al., "OmniQuant: Omnidirectionally calibrated quantization for large language models," ICLR 2024.

[4] Y. Ma et. al., "AffineQuant: Affine transformation quantization for large language models," ICLR 2024.

[5] E. Frantar et. al., "GPTQ: Accurate post-training quantization for generative pre-trained Transformers," ICLR 2023.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces BOA, a post-training quantization (PTQ) method for large language models (LLMs) that avoids backpropagation by leveraging attention-aware Hessian matrices to capture inter-layer dependencies within the attention module. BOA demonstrates improved quantization accuracy, particularly at low bit-widths (e.g., INT2), and incorporates techniques like Hessian relaxation and head-wise simultaneous quantization to reduce computational overhead. While reviewers commend the paper's focus on resource-efficient quantization for LLMs and its compatibility with other techniques like SmoothQuant and Z-FOLD, they find the contributions incremental and the novelty limited. The experiments are primarily conducted on outdated models (e.g., BLOOM, LLaMA1, OPT) and lack validation on state-of-the-art models or comparisons with recent quantization methods (e.g., QuaRot, SpinQuant). Additionally, BOA’s processing and memory overheads remain higher than some existing methods, while its performance improvements are marginal. Overall, reviewers recognize the importance of the topic but suggest that stronger experimental validation and comparisons are needed to make BOA a more compelling contribution.