LongVU: Spatiotemporal Adaptive Compression for Long Video-Language
  Understanding
Authors: name: Xiaoqian Shen; name: Yunyang Xiong; name: Changsheng Zhao; name: Lemeng Wu; name: Jun Chen; name: Chenchen Zhu; name: Zechun Liu; name: Fanyi Xiao; name: Balakrishnan Varadarajan; name: Florian Bordes; name: Zhuang Liu; name: Hu Xu; name: Hyunwoo J. Kim; name: Bilge Soran; name: Raghuraman Krishnamoorthi; name: Mohamed Elhoseiny; name: Vikas Chandra
Published: 2024-10-22 21:21:37

Abstract: Multimodal Large Language Models (MLLMs) have shown promising progress in
understanding and analyzing video content. However, processing long videos
remains a significant challenge constrained by LLM's context size. To address
this limitation, we propose LongVU, a spatiotemporal adaptive compression
mechanism thats reduces the number of video tokens while preserving visual
details of long videos. Our idea is based on leveraging cross-modal query and
inter-frame dependencies to adaptively reduce temporal and spatial redundancy
in videos. Specifically, we leverage DINOv2 features to remove redundant frames
that exhibit high similarity. Then we utilize text-guided cross-modal query for
selective frame feature reduction. Further, we perform spatial token reduction
across frames based on their temporal dependencies. Our adaptive compression
strategy effectively processes a large number of frames with little visual
information loss within given context length. Our LongVU consistently surpass
existing methods across a variety of video understanding benchmarks, especially
on hour-long video understanding tasks such as VideoMME and MLVU. Given a
light-weight LLM, our LongVU also scales effectively into a smaller size with
state-of-the-art video understanding performance.