Visual Haystacks: Answering Harder Questions About Sets of Images
Authors: name: Tsung-Han Wu; name: Giscard Biamby; name: Jerome Quenum; name: Ritwik Gupta; name: Joseph E. Gonzalez; name: Trevor Darrell; name: David M. Chan
Published: 2024-07-18 17:59:30

Abstract: Recent advancements in Large Multimodal Models (LMMs) have made significant
progress in the field of single-image visual question answering. However, these
models face substantial challenges when tasked with queries that span extensive
collections of images, similar to real-world scenarios like searching through
large photo albums, finding specific information across the internet, or
monitoring environmental changes through satellite imagery. This paper explores
the task of Multi-Image Visual Question Answering (MIQA): given a large set of
images and a natural language query, the task is to generate a relevant and
grounded response. We propose a new public benchmark, dubbed "Visual Haystacks
(VHs)," specifically designed to evaluate LMMs' capabilities in visual
retrieval and reasoning over sets of unrelated images, where we perform
comprehensive evaluations demonstrating that even robust closed-source models
struggle significantly. Towards addressing these shortcomings, we introduce
MIRAGE (Multi-Image Retrieval Augmented Generation), a novel retrieval/QA
framework tailored for LMMs that confronts the challenges of MIQA with marked
efficiency and accuracy improvements over baseline methods. Our evaluation
shows that MIRAGE surpasses closed-source GPT-4o models by up to 11% on the VHs
benchmark and offers up to 3.4x improvements in efficiency over text-focused
multi-stage approaches.