AWCQ: Activation-Weight Co-Quantization for Large Language Models
Authors: Lin Zhao, Jiangshan Fan, Wei Luo, Xinjun Niu, Zhaofeng Zhang, Ke Wang, Hongwei Meng, Fan Jiang, Ashish Sirasao


Reviewer paper summaries: The paper introduces a new post-training quantization method, named AWCQ, for large language models (LLMs). AWCQ extends the previous GPTQ approach by quantizing also the activation and incorporating the quantization errors of the activations into the GPTQ formulation. Experimental results show that AWCQ achieves superior performance on conventional zero-shot NLP benchmarks, compared to prior works like SmoothQuant, OmniQuant.
Summary of strengths: i) Topic. the paper studies an important and timely problem, i.e. post-training weight-activation quantization for LLMs, which would be of interest to the community.

ii) Method. The paper extends GPTQ to weight-activation quantization, which is technically interesting.

iii) Evaluation. The proposed AWCQ method demonstrates strong performances on standard NLP benchmarks, e.g. PIQA, ARC, BoolQ. AWCQ also outperforms prior works (e.g. SmoothQuant, OmniQuant) with per-tensor dynamic activation quantization.
Summary of weaknesses: i) The reviewer wonders if there is any specific reason to consider only per-tensor dynamic activation quantization, rather than per-token dynamic activation quantization. This is the main concern from the reviewer. The reviewer asks this question because the latter is the default setting for baselines like SmoothQuant and OmniQuant, and in this setting, the baselines perform significantly better, with near lossless performance compared to the unquantized models, while in Table 1, all methods suffered from huge performance drops.

The two quantization settings both require computing the activation range and quantizing the activation on-the-fly, which could be the main latency bottleneck. Compared to the per-tensor quantization, the per-token setting may have extra overhead by representing the quantization parameters as vectors instead of scalars. However, it is unclear to the reviewer if the impact is large. Also, there are already public W4A8 kernels for per-token dynamic activation quantization (https://github.com/mit-han-lab/qserve/tree/main/kernels/csrc/qgemm) which are shown to be efficient. 

In order to demonstrate the advantages of AWQC, the authors are encouraged to either show that AWQC outperforms the baselines in their default settings (i.e. per-token dynamic activation quantization), or justify the advantages of using per-tensor quantization over per-token quantization, e.g. in latency.


ii)The authors are also encouraged to include more evaluations, e.g. the perplexity on WikiText2. Compared to the classification accuracy in tasks like PIQA, ARC, the perplexity score aligns better with the quantization errors. The perplexity score is also widely used in previou quantization literature.
Comments, suggestions and typos: L219, take into consideration
Overall assessment scores: 2