다음 영어를 한글로 자연스럽게 번역해주세요.
첨
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

Abstract

We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from the hidden state without changing the output, making quantization easier. This computational invariance is applied to the hidden state (residual) of the LLM, as well as to the activations of the feed-forward components, aspects of the attention mechanism and to the KV cache. The result is a quantized model where all matrix multiplications are performed in 4-bits, without any channels identified for retention in higher precision. Our quantized Llama 
2-70B
 model has losses of at most 0.29 WikiText-2 perplexity and retains 99% of the zero-shot performance. Code is available at: https://github.com/spcl/QuaRot.

1 Introduction

Large language models (LLMs) have become increasingly important due to their countless applications. However, their inference requires a significant amount of computations, memory, and energy. LLM inference has two main phases: during the prefill phase, the prompt is processed and cached in each layer, and then in the decoding phase the output tokens are generated one-by-one. The prefill phase is usually compute-bound while the decoding phase is known to be memory-bound.

Quantization is among the most important techniques to solve both memory and compute issues during LLM inference. Joint quantization aims to reduce the precision of parameters and KV cache (which results in lower memory usage) as well as inputs (known as activations) and compute the forward pass in low precision. However, quantizing the activations is hard as they have large outlier elements (see Figure 1 for an illustrative example) with much larger values, making activation quantization more difficult than weight quantization, especially for the 4-bit case. Previous work relies on characterizing outlier features using a calibration set and keeping them in higher precision during the inference.

In this work, we address the issue of outlier features by rotating the inputs of the model using randomized Hadamard transformations. We do this using the computational invariance idea (Ashkboos et al., 2024) and fuse Hadamard transformations into the weight matrices, resulting in an equivalent network without outlier features. This enables the weights, activations, and KV caches to be quantized to 4 bits with minimal accuracy drop. Our main contributions are:

• Using computational invariance from SliceGPT (Ashkboos et al., 2024), we apply randomized Hadamard transformations to the weight matrices without changing the model. This completely eliminates the outlier features and makes the activations easy to quantize without changing the output of the model.
• We also apply online Hadamard transformations to the attention module to remove outlier features in keys and values, enabling the KV cache to be quantized.
• Using the above modifications, QuaRot enables 4-bit LLM inference by quantizing all the weights, activations, and KV caches using integer quantization. On Llama 2-7B model, QuaRot achieves up to 2.16x prefill speedups, and 3.39x memory saving during the decoding stage, with at most 0.63 WikiText-2 perplexity loss. QuaRot preserves 99% of the accuracy of zero-shot tasks. We show that our 6 and 8-bit quantization (weights and activations) is lossless.