LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior
  Accuracy Preservation
Authors: name: Han Chen, name: Zicong Jiang, name: Zining Zhang, name: Bingsheng He, name: Pingyi Luo, name: Mian Lu, name: Yuqiang Chen

We introduce LogQuant, a groundbreaking 2-bit quantization technique for KV
Cache in large language model (LLM) inference, delivering substantial memory
savings while preserving superior performance. Previous methods either assume
that later tokens are more important or attempt to predict important tokens
based on earlier attention patterns. Both approaches, however, can result in
performance bottlenecks or frequent mispredictions.
  LogQuant takes a different approach. By applying a log-based filtering
mechanism, it selectively compresses the KV Cache across the entire context,
achieving better performance with the same or even reduced memory footprint
compared to existing methods. In benchmark tests, it enhances throughput by 25%
and boosts batch size by 60% without increasing memory consumption. For
challenging tasks such as Math and Code Completion, LogQuant improves accuracy
by 40% to 200% at the same compression ratio, outperforming comparable
techniques.LogQuant integrates effortlessly with popular inference frameworks
like Python's transformers library. Implementation can be available in
https://github.com/Concyclics/LogQuantKV.