LongVU: Spatiotemporal Adaptive Compression for Long Video-Language
  Understanding
Authors: name: Xiaoqian Shen, name: Yunyang Xiong, name: Changsheng Zhao, name: Lemeng Wu, name: Jun Chen, name: Chenchen Zhu, name: Zechun Liu, name: Fanyi Xiao, name: Balakrishnan Varadarajan, name: Florian Bordes, name: Zhuang Liu, name: Hu Xu, name: Hyunwoo J. Kim, name: Bilge Soran, name: Raghuraman Krishnamoorthi, name: Mohamed Elhoseiny, name: Vikas Chandra

Multimodal Large Language Models (MLLMs) have shown promising progress in
understanding and analyzing video content. However, processing long videos
remains a significant challenge constrained by LLM's context size. To address
this limitation, we propose LongVU, a spatiotemporal adaptive compression
mechanism thats reduces the number of video tokens while preserving visual
details of long videos. Our idea is based on leveraging cross-modal query and
inter-frame dependencies to adaptively reduce temporal and spatial redundancy
in videos. Specifically, we leverage DINOv2 features to remove redundant frames
that exhibit high similarity. Then we utilize text-guided cross-modal query for
selective frame feature reduction. Further, we perform spatial token reduction
across frames based on their temporal dependencies. Our adaptive compression
strategy effectively processes a large number of frames with little visual
information loss within given context length. Our LongVU consistently surpass
existing methods across a variety of video understanding benchmarks, especially
on hour-long video understanding tasks such as VideoMME and MLVU. Given a
light-weight LLM, our LongVU also scales effectively into a smaller size with
state-of-the-art video understanding performance.