MMLongBench: Benchmarking Long-Context Vision-Language Models
  Effectively and Thoroughly
Authors: name: Zhaowei Wang, name: Wenhao Yu, name: Xiyu Ren, name: Jipeng Zhang, name: Yu Zhao, name: Rohit Saxena, name: Liang Cheng, name: Ginny Wong, name: Simon See, name: Pasquale Minervini, name: Yangqiu Song, name: Mark Steedman

The rapid extension of context windows in large vision-language models has
given rise to long-context vision-language models (LCVLMs), which are capable
of handling hundreds of images with interleaved text tokens in a single forward
pass. In this work, we introduce MMLongBench, the first benchmark covering a
diverse set of long-context vision-language tasks, to evaluate LCVLMs
effectively and thoroughly. MMLongBench is composed of 13,331 examples spanning
five different categories of downstream tasks, such as Visual RAG and Many-Shot
ICL. It also provides broad coverage of image types, including various natural
and synthetic images. To assess the robustness of the models to different input
lengths, all examples are delivered at five standardized input lengths (8K-128K
tokens) via a cross-modal tokenization scheme that combines vision patches and
text tokens. Through a thorough benchmarking of 46 closed-source and
open-source LCVLMs, we provide a comprehensive analysis of the current models'
vision-language long-context ability. Our results show that: i) performance on
a single task is a weak proxy for overall long-context capability; ii) both
closed-source and open-source models face challenges in long-context
vision-language tasks, indicating substantial room for future improvement; iii)
models with stronger reasoning ability tend to exhibit better long-context
performance. By offering wide task coverage, various image types, and rigorous
length control, MMLongBench provides the missing foundation for diagnosing and
advancing the next generation of LCVLMs.