Are Vision-Language Models Truly Understanding Multi-vision Sensor?
Authors: name: Sangyun Chung; name: Youngjoon Yu; name: Youngchae Chee; name: Se Yeon Kim; name: Byung-Kwan Lee; name: Yong Man Ro
Published: 2024-12-30 06:44:25

Abstract: Large-scale Vision-Language Models (VLMs) have advanced by aligning vision
inputs with text, significantly improving performance in computer vision tasks.
Moreover, for VLMs to be effectively utilized in real-world applications, an
understanding of diverse multi-vision sensor data, such as thermal, depth, and
X-ray information, is essential. However, we find that current VLMs process
multi-vision sensor images without deep understanding of sensor information,
disregarding each sensor's unique physical properties. This limitation
restricts their capacity to interpret and respond to complex questions
requiring multi-vision sensor reasoning. To address this, we propose a novel
Multi-vision Sensor Perception and Reasoning (MS-PR) benchmark, assessing VLMs
on their capacity for sensor-specific reasoning. Moreover, we introduce Diverse
Negative Attributes (DNA) optimization to enable VLMs to perform deep reasoning
on multi-vision sensor tasks, helping to bridge the core information gap
between images and sensor data. Extensive experimental results validate that
the proposed DNA method can significantly improve the multi-vision sensor
reasoning for VLMs.