Towards Next-Level Post-Training Quantization of Hyper-Scale
  Transformers
Authors: name: Junhan Kim; name: Kyungphil Park; name: Chungman Lee; name: Ho-young Kim; name: Joonyoung Kim; name: Yongkweon Jeon
Published: 2024-02-14 05:58:43

Abstract: With the increasing complexity of generative AI models, post-training
quantization (PTQ) has emerged as a promising solution for deploying
hyper-scale models on edge devices such as mobile devices and TVs. Existing PTQ
schemes, however, consume considerable time and resources, which could be a
bottleneck in real situations where frequent model updates and multiple
hyper-parameter tunings are required. As a cost-effective alternative, one-shot
PTQ schemes have been proposed. Still, the performance is somewhat limited
because they cannot consider the inter-layer dependency within the attention
module, which is a very important feature of Transformers. In this paper, we
thus propose a novel PTQ algorithm that balances accuracy and efficiency. The
key idea of the proposed algorithm called aespa is to perform quantization
layer-wise for efficiency while considering cross-layer dependency to preserve
the attention score. Through extensive experiments on various language models
and complexity analysis, we demonstrate that aespa is accurate and efficient in
quantizing Transformer models.