ZipVL: Efficient Large Vision-Language Models with Dynamic Token
  Sparsification and KV Cache Compression
Authors: name: Yefei He; name: Feng Chen; name: Jing Liu; name: Wenqi Shao; name: Hong Zhou; name: Kaipeng Zhang; name: Bohan Zhuang
Published: 2024-10-11 07:24:21

Abstract: The efficiency of large vision-language models (LVLMs) is constrained by the
computational bottleneck of the attention mechanism during the prefill phase
and the memory bottleneck of fetching the key-value (KV) cache in the decoding
phase, particularly in scenarios involving high-resolution images or videos.
Visual content often exhibits substantial redundancy, resulting in highly
sparse attention maps within LVLMs. This sparsity can be leveraged to
accelerate attention computation or compress the KV cache through various
approaches. However, most studies focus on addressing only one of these
bottlenecks and do not adequately support dynamic adjustment of sparsity
concerning distinct layers or tasks. In this paper, we present ZipVL, an
efficient inference framework designed for LVLMs that resolves both computation
and memory bottlenecks through a dynamic ratio allocation strategy of important
tokens. This ratio is adaptively determined based on the layer-specific
distribution of attention scores, rather than fixed hyper-parameters, thereby
improving efficiency for less complex tasks while maintaining high performance
for more challenging ones. Then we select important tokens based on their
normalized attention scores and perform attention mechanism solely on those
important tokens to accelerate the prefill phase. To mitigate the memory
bottleneck in the decoding phase, we employ mixed-precision quantization to the
KV cache, where high-bit quantization is used for caches of important tokens,
while low-bit quantization is applied to those of less importance. Our
experiments demonstrate that ZipVL can accelerate the prefill phase by
2.6times and reduce GPU memory usage by 50.0%, with a minimal accuracy
reduction of only 0.2% on Video-MME benchmark over LongVA-7B model, effectively
enhancing the generation efficiency of LVLMs.