Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark

Authors: Tsung-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E. Gonzalez, Trevor Darrell, David Chan
Primary area: foundation or frontier models, including LLMs
Keywords: Large Multimodal Models, Visual Question Answering

Abstract:
Large Multimodal Models (LMMs) have made significant strides in visual question-answering for single images. Recent advancements like long-context LMMs have allowed them to ingest larger, or even multiple, images. However, the ability to process a large number of visual tokens does not guarantee effective retrieval and reasoning for multi-image question answering (MIQA), especially in real-world applications like photo album searches or satellite imagery analysis. In this work, we first assess the limitations of current benchmarks for long-context LMMs. We address these limitations by introducing a new vision-centric, long-context benchmark, "Visual Haystacks (VHs)". We comprehensively evaluate both open-source and proprietary models on VHs, and demonstrate that these models struggle when reasoning across potentially unrelated images, perform poorly on cross-image reasoning, as well as exhibit biases based on the placement of key information within the context window. Towards a solution, we introduce MIRAGE (Multi-Image Retrieval Augmented Generation), an open-source, lightweight visual-RAG framework that processes up to 10k images on a single 40G A100 GPU—far surpassing the 1k-image limit of contemporary models. MIRAGE demonstrates up to 13% performance improvement over existing open-source LMMs on VHs, sets a new state-of-the-art on the RetVQA multi-image QA benchmark, and achieves competitive performance on single-image QA with state-of-the-art LMMs. Our dataset, model, and code are available at: https://visual-haystacks.github.io.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730498195933

thread_type:
official_review

<thread_object>
creator: reviewer_io8n

<content>
summary:
This paper introduces a long context, visual needle in a haystack benchmark which composed of 1k yes/no questions changeling the model to reasoning and find the target object in the images. It evaluated on both open-source and closed-source LMMs and reveal several critical findings such as susceptibility to visual distractors, difficulty in multi-image reasoning, and a bias in image positioning. It introduces a new baseline called MIRAGE (Multi-Image Retrieval Augmented Generation) for better handling of VH tasks.

strengths:
1. This paper introduced a new visual needle in a haystack benchmark which composed of 1k yes/no questions. 
2. Evaluated on both open-source and close-source models and gained three insightful findings. 
3. Introduced a new baseline called MIRAGE for better handling of visual haystack tasks.

weaknesses:
1. The questions are only limited to yes/no questions. 
2. The question template are very limited, seems only three. 
3. MIRAGE has a significant performance drop in 4 out of 7 general VQA tasks. 
4. The approach of MIRAGE, deselecting unrelated (distracting) images somehow circumvents the VH challenge, as the this challenge lies in how model can reasoning in long context.  
5. The task of finding a target object seems still not simulating a real world scenario of long context visual reasoning task.

questions:
1. I'm confused about the difference between the MIRAGE model in Table 1 and the Q-former Model in Table. Doesn't MIRAGE utilize Q-former. 
2. See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730508989402

thread_type:
official_review

<thread_object>
creator: reviewer_euwk

<content>
summary:
This paper addresses the limitations of Large Multimodal Models (LMMs) in multi-image question answering, where handling large visual contexts does not ensure effective retrieval and reasoning across images. Current benchmarks reveal biases and challenges in MIQA, such as poor cross-image reasoning and sensitivity to information placement. To overcome these, the authors propose "Visual Haystacks (VHs)," a vision-centric benchmark that tests retrieval and reasoning over multiple images, highlighting models' struggles with visual distractors and multi-image reasoning. They also introduce MIRAGE, an open-source Multi-Image Retrieval Augmented Generation framework capable of handling up to 10,000 images on a single GPU, achieving significant improvements over existing models and setting new standards in MIQA benchmarks like RetVQA. Key contributions include VHs, systematic LMM evaluation, and MIRAGE's scalable MIQA capabilities.

strengths:
- I generally feel the direction is important to our community where design meaningful Visual Haystack benchmark for evaluating VLM. 
- Some interesting points are discovered when evaluating models on the proposed benchmark. Since random guess could achieve 50% accuracy in the proposed benchmark, some open-sourced VLMs performance significantly drop even the Haystack size is very small. However, those models maintain high scores in some public evaluation-datasets. 
- Some detailed experiments are conducted such as needle position and running time. 
- The proposed benchmark are made publicly available under MIT license, which is good for community.

weaknesses:
- Benchmark construction is still mainly centered around recognition tasks, based on benchmark design principles listed in Line129~138. Basically, it requires a strong recognition among all the input images, rather than true visual reasoning. 
- Based on the Figure 2 and 3, certain models, such as Gemini, GPT and the proposed MIRAGE, consistently perform better on the proposed multi-needle challenges compared to single-needle tasks. However, the multi-needle challenges are intentionally designed to be more difficult, as they demand additional reasoning across multiple images. Does this mean failure in designing the benchmark?
- Since the benchmark is constructed in way of examining recognition, therefore the proposed method contain ad-hoc modules, such as "a retriever module then calculates relevance scores, ensuring that only the most relevant images are passed to the LLM for final reasoning." Does this design hold for general visual reasoning tasks? For example, many of the tested single image dataset used in this paper, do not need this retriever module at all. 
- The proposed framework achieved not-very-good performance on some of the tested datasets. Also, there are many datasets that not being tested such as SEED, MME, and CHAIR.

questions:
- Could you please address the points raised in the above weakness?
- Could you please add some randomly sampled failure cases made by GPT or Gemini? Sometimes failure cases can tell more than good cases. 
- Could you please address the ethics concerns around the code license?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692932301

thread_type:
official_review

<thread_object>
creator: reviewer_kpnk

<content>
summary:
The authors presents Visual Haystacks (VHs), a new vision centric benchmark designed to assess the performance of Large Multimodal Models (LMMs) in the multi-image question answering (QA) task. In addition, the author proposes a new visaul-RAG framework, MIRAGE, to enhance the task performance.

strengths:
* Novel Multi-Image QA Benchmark: The authors introduce an interesting multi-image QA benchmark, Visual Haystacks, designed around a vision-centric "needle-in-a-haystack" scenario, providing a fresh and challenging setting for the LMM evaluation.

* Comprehensive Model Evaluation:  The paper conducts a thorough evaluation of LMMs on the VHs benchmark, uncovering important insights into current models, such as vulnerability to visual distractors, challenges with multi-image understanding, and tendencies toward positional visual bias.

* Novel Visual RAG Framework: The authors introduce a novel visual RAG framework that combines a compressor and a retriever. The compressor efficiently processes up to 10,000 images on a single 40GB A100 GPU, while the retriever identifies the top-k most relevant images for a given question, enhancing the framework’s scalability and efficiency.

weaknesses:
* Limited Object Diversity: The authors constructed the VHs benchmark using objects from the COCO dataset, which contains only 80 object categories. This limited selection may restrict the diversity and comprehensiveness of the benchmark, potentially affecting its ability to evaluate models across a broader range of visual scenarios.

* Restricted Question Diversity: The authors appear to rely on a few simple templates to generate questions, which may restrict the variety of question types in the benchmark.

* More like Object Detection than QA Reasoning: Many questions in the benchmark (e.g., "For the image with a truck, is there a dog?") seem to primarily assess the model’s object detection abilities rather than its visual QA reasoning skills. It is questionable if the benchmark requires the advanced visual QA reasoning skills from the models.

* Missing Related Work: The paper does not reference several recent multi-image QA benchmarks, for example: 
 1. CompBench: A Comparative Reasoning Benchmark for Multimodal LLMs
 2. MANTIS: Interleaved Multi-Image Instruction Tuning
 3. MUIRBENCH: A Comprehensive Benchmark for Robust Multi-Image Understanding. 

Additionally, a similar multi-image retrieval approach was introduced in "ColPali: Efficient Document Retrieval with Vision Language Models", but this work was also not cited.

questions:
Please see the weakeness. In addition,
* How many templates were used to generate questions?
* What advantages does the VHs benchmark offer compared to recent multi-image QA benchmarks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732252064590

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Follow-Up on Rebuttal for "Visual Haystacks"

comment:
Dear Reviewers,

We're following up on the rebuttal for our paper, "Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark." We appreciate the time and effort you've invested in reviewing our work.

In our rebuttal and the updated paper, we've thoroughly addressed the concerns and suggestions raised in the initial reviews. If you find that all your questions have been resolved, we kindly ask you to consider reflecting this in the scores. Should you have any additional questions or require further clarification, we are eager to engage in more discussion during the official ICLR discussion period.

Thank you once again for your thoughtful feedback and consideration.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission introduces a new benchmark "Visual Haystacks" for measuring the quality of large multimodal models (LMMs) at the task of multi-image reasoning. Along with the benchmark dataset, it also introduces a RAG framework (MIRAGE) that can process a magnitude larger number of images compared to prior work. This uses a retrieval-based relevance filter as well as compression of image features to fit into the context length of the LMMs.

After the initial round of reviews, this submission received scores of 6, 6, 6. The reviewers did not find sufficient reason to reject this submission during the discussion, and the consensus remained positive. The AC recommends acceptance, and requests the authors to use the constructive feedback from reviewers to update the submission.