LEOPARD : A Vision Language Model For Text-Rich Multi-Image Tasks
Authors: name: Mengzhao Jia; name: Wenhao Yu; name: Kaixin Ma; name: Tianqing Fang; name: Zhihan Zhang; name: Siru Ouyang; name: Hongming Zhang; name: Meng Jiang; name: Dong Yu
Published: 2024-10-02 16:55:01

Abstract: Text-rich images, where text serves as the central visual element guiding the
overall understanding, are prevalent in real-world applications, such as
presentation slides, scanned documents, and webpage snapshots. Tasks involving
multiple text-rich images are especially challenging, as they require not only
understanding the content of individual images but reasoning about
inter-relationships and logical flows across multiple visual inputs. Despite
the importance of these scenarios, current multimodal large language models
(MLLMs) struggle to handle such tasks due to two key challenges: (1) the
scarcity of high-quality instruction tuning datasets for text-rich multi-image
scenarios, and (2) the difficulty in balancing image resolution with visual
feature sequence length. To address these challenges, we propose \OurMethod, a
MLLM designed specifically for handling vision-language tasks involving
multiple text-rich images. First, we curated about one million high-quality
multimodal instruction-tuning data, tailored to text-rich, multi-image
scenarios. Second, we developed an adaptive high-resolution multi-image
encoding module to dynamically optimize the allocation of visual sequence
length based on the original aspect ratios and resolutions of the input images.
Experiments across a wide range of benchmarks demonstrate our model's superior
capabilities in text-rich, multi-image evaluations and competitive performance
in general domain evaluations.