Kati8/ICLR_2023_2025 huggingface dataset:
authors: Xin Ding
authors: Xiaoyu Liu
authors: Yehui Tang
authors: Zhiwei Xiong
authors: Baoqun Yin
authors: Yunhe Wang
reviews: 
Comments: This work introduces Cross-Block Quantization (CBQ), a novel post-training quantization (PTQ) framework targeting large language models (LLMs). CBQ's core innovation is its cross-block dependency mechanism, which introduces dependencies across multiple blocks to mitigate error accumulation. Additionally, a coarse-to-fine preprocessing strategy and LoRA-Rounding are proposed to handle weight and activation outliers. The authors validate CBQ on various LLMs and demonstrate that it achieves state-of-the-art performance in low-bit settings, like W4A4, across a range of datasets.
Strengths: Innovation: The idea of using cross-block dependencies to reduce quantization errors is both intuitive and impactful, as it tackles a recognized issue in PTQ for LLMs.

Experimental Scope: CBQ is tested extensively across multiple LLM architectures and low-bit settings, showing clear performance gains over prior methods.

Significance: Reducing computational overhead for LLMs without retraining is valuable for real-world applications, and CBQ provides a viable solution for efficient model deployment.
Weaknesses: Baseline Comparisons: Although the results on LLAMA models are promising, the exclusion of more recent PTQ methods like BiE[1] limits the impact of the results. Including such baselines could provide a more complete picture of CBQ’s effectiveness.

Broader Applicability: The focus on language models limits the generalizability of CBQ. Expanding evaluations to other architectures, such as vision transformers and VLMs (vision language models), could strengthen the paper’s applicability.

[1] BiE: Bi-Exponent Block Floating-Point for Large Language Models Quantization. ICML 2024
Recommendation: 6
Reviewer confidence: 5
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: The authors find that optimizing for inter-layer dependencies is crucial, especially for lower-bit quantization. To address this challenge, they propose the use of cross-block loss where the loss is taken over a window of blocks and not just a single block. For the cross-block loss, in addition to using the L2-distance, the authors use KL-divergence between the activations. Inspired by AdaRound, the authors use LoRA parameters for accurate rounding. For outlier detection, the authors present a two step process. They use a quantile-based thresholding to identify an initial set of outliers following which they divide these coarse outliers into two sets by maximizing the inter-set variance and minimizing the intra-set variance.
Strengths: - The paper clearly motivates the proposed cross-block quantization.
- CBQ shows substantial performance improvements over OmniQuant, especially in the 2-bit setting.
- The authors provide a thorough ablation study on several aspects of the proposed objective.
Weaknesses: - The addition of LoRA parameters should take some hit in latency. The authors should present the latency numbers, especially for W2A16 and W4A4.
- Given that the paper focuses on two-bit quantization, comparison to several recent rotation-based methods such as FrameQuant [1] and QuIP [2] are omitted from the related works section and the results. The authors should compare against these methods for 2-bit quantization.
- In practice, most quantization frameworks use sub channel quantization for weights, the authors should report weight-only quantization numbers with sub-channel quantization for all bit-widths.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper proposes Cross-Block Quantization (CBQ), a novel method designed to optimize post-training quantization (PTQ) for large language models (LLMs) by addressing inter-block dependencies and introducing a coarse-to-fine outlier handling strategy. CBQ's approach aims to preserve model accuracy at low-bit configurations by leveraging cross-block reconstructions and refining outlier quantization through LoRA-Rounding. The authors present experimental results demonstrating CBQ’s strong performance across different quantization bit widths and various LLM architectures, with promising efficiency for real-world deployment.
Strengths: 1. By focusing on inter-block dependencies, CBQ takes a proactive approach to minimize error accumulation, which is often a critical challenge in low-bit quantization. This dependency handling shows clear improvements in model accuracy.
2. The design of CBQ, particularly the coarse-to-fine outlier preprocessing and adaptive rounding, ensures flexibility across different LLM sizes, making it a practical choice for varied deployment needs.
3. The authors have included a wide range of experiments showing CBQ’s effectiveness in diverse quantization configurations, including settings that emphasize computational efficiency, which makes it relevant for both research and applied contexts.
Weaknesses: 1. While the paper demonstrates that overlapping blocks in CBQ contributes to performance, it lacks an in-depth analysis of how varying the overlap size impacts memory efficiency, latency, and overall quantization stability. Providing such details would clarify practical deployment considerations.
2. The coarse-to-fine preprocessing for outliers appears effective; however, an assessment of its necessity relative to simpler methods could be useful. It is unclear whether this specific strategy is essential or if comparable results could be achieved with simpler preprocessing.
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper introduced CBQ  post-training quantization PTQ technique designed for compressing large language models under low-bit precision. The authors identified the primary challenge in ultra-low-bit quantization: the dependencies within and between layers that amplify quantization errors, especially as model size and parameter counts increase. To address this, CBQ incorporates a cross-block reconstruction strategy, which leverages both intra-layer and inter-layer dependencies by optimizing multiple transformer blocks within a sliding window approach. Additionally, the method employs a LoRA-Rounding technique to manage intra-layer dependencies and reduce computational costs, while an adaptive coarse-to-fine preprocessing strategy effectively handles outliers in weights and activations.
Strengths: I have carefully read this work, I think the result of this work was convincing, and the approach was solid.
Weaknesses: N/A
Recommendation: 10
Reviewer confidence: 5
Soundness/correctness: 4
Clarity: 4
reviews: 
Comments: The paper introduces Cross-Block Quantization (CBQ), a novel post-training quantization (PTQ) technique developed for large language models (LLMs). CBQ tackles critical inter- and intra-layer dependencies that compromise quantization accuracy at ultra-low bit settings, establishing long-range dependencies through cross-block reconstruction and managing intra-layer dependencies via adaptive LoRA-Rounding. The approach also incorporates a coarse-to-fine preprocessing method to optimize handling of weights and activations.
Strengths: 1. The paper offers an interesting insight: full model quantization introduces inter-layer correlations.

2. Experimental results on LLAMA1, LLAMA2, and OPT demonstrate substantial improvements in W2 and W4 settings, alongside a significant reduction in PTQ scenario.
Weaknesses: 1. The writing could be improved, especially regarding clarity in notation. While the concepts are interesting, it is difficult to follow due to unclear notation. For example, symbols like \(i\), \(j\), and \(K\) in Equation 3, as well as \(V\) in Equation 11, are not clearly defined when first introduced. It would help if each symbol had an explicit definition upon first use. Additionally, the term "scales" in the phrase "comparisons of the scales between adjacent layers..." lacks clarity. Specifying what "scales" refers to in the context of adjacent layers would enhance readability.

2. The range of models tested is limited. While the authors include results on LLAMA1, LLAMA2, and OPT, these are relatively dated models, and results on newer models like LLAMA3, Mistral, and Falcon should be included.

3. While the authors suggest that LoRA-Rounding was introduced to reduce computation, they do not evaluate this aspect in their experiments. To make this claim more compelling, the authors could include specific metrics or experiments comparing the computational efficiency of LoRA-Rounding to alternative approaches.

4. The authors should compare their method with other W4 approaches, such as AWQ. Comparing with AWQ would be valuable because of its relevant strengths or similarities, which could provide a more comprehensive evaluation of the method’s performance in a competitive context.
Recommendation: 8
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This work presents CBQ, a novel post-training quantization approach for compressing large language models at ultra-low bit widths. The authors effectively identified a critical limitation in ultra-low-bit quantization schemes: the propagation of quantization errors through intra-layer and inter-layer dependencies, which becomes increasingly problematic as model scale increases. To overcome this issue, CBQ employs a cross-block reconstruction mechanism that captures both within-layer and across-layer relationships through an optimized sliding window strategy applied to transformer blocks. The method further integrates a LoRA-Rounding component to address intra-layer dependencies while reducing computational overhead, complemented by an adaptive hierarchical preprocessing approach that effectively mitigates outliers present in both weights and activations.
Strengths: Upon thorough examination of this work, the empirical results demonstrate strong validity and the proposed methodology is principled and well-executed.
Recommendation: 10
Reviewer confidence: 5
Soundness/correctness: 4
Clarity: 4