Long Context Transfer from Language to Vision
Authors: name: Peiyuan Zhang; name: Kaichen Zhang; name: Bo Li; name: Guangtao Zeng; name: Jingkang Yang; name: Yuanhan Zhang; name: Ziyue Wang; name: Haoran Tan; name: Chunyuan Li; name: Ziwei Liu
Published: 2024-06-24 17:58:06

Abstract: Video sequences offer valuable temporal information, but existing large
multimodal models (LMMs) fall short in understanding extremely long videos.
Many works address this by reducing the number of visual tokens using visual
resamplers. Alternatively, in this paper, we approach this problem from the
perspective of the language model. By simply extrapolating the context length
of the language backbone, we enable LMMs to comprehend orders of magnitude more
visual tokens without any video training. We call this phenomenon long context
transfer and carefully ablate its properties. To effectively measure LMMs'
ability to generalize to long contexts in the vision modality, we develop
V-NIAH (Visual Needle-In-A-Haystack), a purely synthetic long vision benchmark
inspired by the language model's NIAH test. Our proposed Long Video Assistant
(LongVA) can process 2000 frames or over 200K visual tokens without additional
complexities. With its extended context length, LongVA achieves
state-of-the-art performance on Video-MME among 7B-scale models by densely
sampling more input frames. Our work is open-sourced at
https://github.com/EvolvingLMMs-Lab/LongVA.