LEOPARD : A Vision Language Model For Text-Rich Multi-Image Tasks
Authors: name: Mengzhao Jia, name: Wenhao Yu, name: Kaixin Ma, name: Tianqing Fang, name: Zhihan Zhang, name: Siru Ouyang, name: Hongming Zhang, name: Meng Jiang, name: Dong Yu
Organization: University of Notre Dame DM2 Lab

Text-rich images, where text serves as the central visual element guiding the
overall understanding, are prevalent in real-world applications, such as
presentation slides, scanned documents, and webpage snapshots. Tasks involving
multiple text-rich images are especially challenging, as they require not only
understanding the content of individual images but reasoning about
inter-relationships and logical flows across multiple visual inputs. Despite
the importance of these scenarios, current multimodal large language models
(MLLMs) struggle to handle such tasks due to two key challenges: (1) the
scarcity of high-quality instruction tuning datasets for text-rich multi-image
scenarios, and (2) the difficulty in balancing image resolution with visual
feature sequence length. To address these challenges, we propose \OurMethod, a
MLLM designed specifically for handling vision-language tasks involving
multiple text-rich images. First, we curated about one million high-quality
multimodal instruction-tuning data, tailored to text-rich, multi-image
scenarios. Second, we developed an adaptive high-resolution multi-image
encoding module to dynamically optimize the allocation of visual sequence
length based on the original aspect ratios and resolutions of the input images.
Experiments across a wide range of benchmarks demonstrate our model's superior
capabilities in text-rich, multi-image evaluations and competitive performance
in general domain evaluations.