Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark

## Abstract

Abstract Large Multimodal Models (LMMs) have made significant strides in visual question-answering for single images. Recent advancements like long-context LMMs have allowed them to ingest larger, or even multiple, images. However, the ability to process a large number of visual tokens does not guarantee effective retrieval and reasoning for multi-image question answering (MIQA), especially in real-world applications like photo album searches or satellite imagery analysis. In this work, we first assess the limitations of current benchmarks for long-context LMMs. We address these limitations by introducing a new vision-centric, long-context benchmark, “Visual Haystacks (VHs)”. We comprehensively evaluate both open-source and proprietary models on VHs, and demonstrate that these models struggle when reasoning across potentially unrelated images, perform poorly on cross-image reasoning, as well as exhibit biases based on the placement of key information within the context window. Towards a solution, we introduce MIRAGE (Multi-Image Retrieval Augmented Generation), an open-source, lightweight visual-RAG framework that processes up to 10k images on a single 40G A100 GPU—far surpassing the 1k-image limit of contemporary models. MIRAGE demonstrates up to 13% performance improvement over existing open-source LMMs on VHs, sets a new state-of-the-art on the RetVQA multi-image QA benchmark, and achieves competitive performance on single-image QA with state-of-the-art LMMs. Our dataset, model, and code are available at: https://visual-haystacks.github.io .

## 1 Introduction

Large Multimodal Models (LMMs) have demonstrated remarkable success in visual question-answering tasks on single images. Recent advancements, such as long-context LMMs, now allow these models to ingest multiple images simultaneously (Liu et al., ; Chen et al., ; Ye et al., ). However, fitting more visual tokens within a context window does not inherently translate to improved performance in more complex, multi-image question-answering (MIQA) scenarios. In fact, we demonstrate in Sections 2.1 and 3 that long-context LMMs are unable to accurately retrieve or reason across thousands of images, thereby debilitating for real-world applications such as searching through photo albums or analyzing medical and satellite imagery. This motivates the development of rigorous benchmarks that evaluate these capabilities, and methods that can appropriately retrieve and reason, in large-scale settings.

The Needle-In-A-Haystack (NIAH) benchmark (Kamradt, ) is a popular method for evaluating long-context models. The task is simple: a specific set of words (a “needle”) is inserted into a large set of text (a “haystack”). Models are then assessed on their ability to find the needle in the haystack. In natural language processing (NLP), this simple evaluation has revealed intriguing behaviors, such as the “lost-in-the-middle” phenomenon in which needles placed in the middle of the haystack are harder to find than those placed at the beginning or end (Liu et al., ). The utility of this diagnostic has proven difficult to replicate in the vision domain; existing visual NIAH benchmarks (Reid et al., ) showcase near-perfect performance when evaluated on a variety of models, leading to limited insights into the models’ capabilities. This performance saturation is likely due to the fact that these benchmarks simulate unrealistic scenarios, such as when the “visual” needles are simply out-of-distribution domains with overlaid text serving as the needle (Figure 1 (A)), testing a model’s optical character recognition (OCR) capabilities rather than useful object retrieval or reasoning capabilities.

We address these shortcomings by introducing “Visual Haystacks (VHs),” a realistic, simple-to-understand, vision-centric NIAH benchmark that challenges models to find truly “visual needles” and reason about them among natural, visually distinctive objects. Unlike previous benchmarks, LMMs must integrate and reason about visual information spread across multiple images to perform well on VHs. For example, in Figure 1 (A), a query such as “For the image containing a truck, is there a dog?” requires LMMs not only to retrieve the relevant image containing a truck but reason about the presence of a dog in this image to answer correctly.

Our evaluation of both open-source and proprietary LMMs on VHs yielded three notable findings: (1) While most LMMs can process up to 100 images, they exhibit up to a 40% performance drop on VHs compared to non-retrieval QA tasks, implying that LMMs struggle with retrieval and are prone to visual distractors. (2) This performance decline is even more severe when multiple images are used as needles, suggesting that LMMs struggle to reason across multiple images. (3) We observed significant positional bias in LMMs, reminiscent of the “lost-in-the-middle” pathology found in NLP tasks (Liu et al., ). Models’ sensitivity to the placement of key information within the context window often results in sub-optimal performance when the needle is positioned unfavorably.

Figure: Figure 1: (A) Unlike existing visual Needle-In-A-Haystack (NIAH) challenges (Reid et al., ) that overlay needle information as text onto an image, our “Visual Haystacks” (VHs) benchmark is vision-centric, requiring the model to first retrieve the needle image(s) from the haystack and then reason about the image(s) to answer the question. (B) We benchmark existing LMMs under different NIAH settings where only one needle image is present among ten images. While traditional visual NIAH challenges overemphasize text retrieval, which can be easily hacked by state-of-the-art models with strong OCR capabilities, they are unable to solve the simple visual questions in VHs.
Refer to caption: extracted/5917889/figures/visual_haystack_v6.png

In support of developing models which can solve long-context MIQA tasks, we further introduce MIRAGE (Multi-Image Retrieval Augmented Generation), a visual-RAG framework that gives LMMs the capacity to solve large-scale MIQA tasks such as VHs. Built on the LLaVA architecture (Liu et al., ), MIRAGE builds on the strengths of several methods to mitigate context length limitations found in contemporary LMMs, ultimately enabling MIQA on over 10k images. First, we introduce a compressive image encoding strategy to make efficient use of a fixed context window. Second, we implement a query-aware, retrieval-based relevance filter focuses the model on pertinent content, facilitating the processing of larger image sets. Finally, we constructed a 1.2M-image instruction-tuning dataset for MIQA tasks using a mixture of synthetic and real-world data. Evaluations on VHs and other MIQA benchmarks, such as RetVQA (Khattar et al., ), demonstrate that MIRAGE outperforms existing retrieval-augmented methods and LMM-based approaches in most tasks. Additionally, MIRAGE shows competitive performance on conventional single-image QA tasks compared to other state-of-the-art models. In sum, our contributions are as follows:

- •
We introduce a new benchmark, “Visual Haystacks (VHs)” which explicitly tests MIQA models on their ability to retrieve and integrate visual information.
- •
We systematically assess existing open and closed-source LMMs on the VHs benchmark, and reveal three key findings: susceptibility to visual distractors, difficulty in multi-image reasoning, and a bias in image positioning.
- •
We introduce a novel baseline for VHs, MIRAGE (Multi-Image Retrieval Augmented Generation), which is the first open-source visual-RAG framework capable of scaling to over 10k images.

## 2 The Visual Haystacks Benchmark (VHs)

The Needle-In-A-Haystack (NIAH) evaluation Kamradt () has recently become a widely adopted unit test for evaluating LLM/LMM systems’ ability to uniformly process long-context inputs. In the vision domain, however, no visual NIAH benchmark (Reid et al., ) tests whether a model can locate key visual information within a large pool of images and subsequently use that information to answer a specific query.

The predominant “visual” NIAH benchmarks have several limitations as illustrated in Figure 1 (A). For example, Gemini-v1.5’s demo creates “visual” needles by overlaying text reading “The secret is Needle” on a specific frame in a long video, then asking the model to retrieve this text. While this setup technically evaluates a model’s ability to handle long-context inputs, it does so by placing disproportionate emphasis on textual/OCR tasks, thereby underplaying the importance of image-based retrieval and reasoning. Additionally, the benchmark includes only a single test case—with a single needle in the video—limiting its ability to comprehensively evaluate models across varied and complex visual scenarios reflective of real-world applications.

We address these shortcomings in the Visual Haystacks (VHs) benchmark which emphasizes realistic, visually grounded needles and mirrors real-world visual long-context learning tasks. The design of VHs is based on two core principles: realism and reliability. The needles in VHs are real objects within natural images, overcoming the artificial inclusion of overlaid text or image patches (Reid et al., ; Wang et al., ) that previous visual NIAH benchmarks suffer from. VHs utilizes in-domain images and straightforward questions with human-annotated ground truth. Compared to recent LLM and LMM datasets (Gema et al., ; Zhu et al., ) which face noises, biases, and out-of-distribution (OOD) challenges due to their data labeling methodology, VHs is much more reliable. Finally, VH is diverse and large-scale; haystack sizes are as large as 10,000 images, haystacks can feature one or more needles, and each needle can draw from over 50 distinct objects. With over 1,000 examples of each setting, VHs provides 97k images total for comprehensive visual NIAH evaluation.

### 2.1 Benchmark Construction

We construct the VHs dataset from the COCO dataset (Lin et al., ) which has accurate, object-level annotations. To generate a question/answer pair, we first select two objects from COCO’s label set to serve as an anchor and target, respectively. These objects then seed question generation in two settings: a single-needle setting with the template ‘‘For the image with anchor object, is there target object?’’ and a multi-needle setting with either ‘‘For all images with anchor object, do all of them contain target object?’’ (requiring the model to retrieve and look at all relevant anchor images) or ‘‘For all images with anchor object, do any of them contain target object?’’ as templates. The answers are binary (yes/no). We curate the dataset such that guessing or relying on common sense reasoning without viewing the image results in a 50% accuracy rate. This design ensures that the anchor object serves as the key for image retrieval, while the target object forms the basis of the actual question during image reasoning.

After selecting the object pairs, setting the questions, and determining the answers, we then compile corresponding image haystacks by first curating the needle images, which contain the “anchor object” as indicated by the paired answer. We then accompany these images with multiple negative distractors to form haystacks of varying sizes, ranging from 1 to 10,000 haystack images. We select distractor images in a manner such that no distrator contains any anchor object (following COCO’s object annotations). However, some of them contain the target objects so as to create meaningful distractors. The single-needle setting contains only one needle image in the haystack, while the multi-needle setting includes either two and three needle images. VHs consists of 1000 question-answer pairs for both single- and multi-needle settings, with an explicit small subset $\text{VHs}_{\text{small}}$ consisting of 100 questions, which is helpful for economical evaluation of expensive closed-source models. Appendix A illustrates several examples and statistics of single- and multi-needle benchmarks.

As shown in Figure 1, the VHs dataset aligns more closely with real-world scenarios compared to previous visual NIAH challenges and challenges contemporary LMMs (section 3). While developing a comprehensive MIQA benchmark with more diverse images and questions beyond COCO is a valuable direction for future research, given the consistently low performance of current LMMs on the VHs benchmark, we believe that it is eminently important to establish this simple foundational unit test for evaluating long-context visual retrieval and reasoning.

Figure: Figure 2: Experimental results on the VHs single-needle challenge. All LMMs experience significant falloff as the size of the haystack (N) increases, indicating that existing approaches are not robust to complex visual-linguistic processing over long visual contexts. Note the non-linear x-axis in this plot.
Refer to caption: x1.png

## 3 Can LMMs handle Long-Context Visual Inputs?

We evaluated a range of state-of-the-art open-source and proprietary LMMs on the VHs benchmark. Specifically, this includes GPT-4o (OpenAI, ), Gemini 1.5 Pro (Reid et al., ), Qwen2-VL (Wang et al., ), Phi-3 Vision (Abdin et al., ), mPLUG-Owl3 (Ye et al., ), and InternVL2 (Chen et al., ).

We also implement two non-LMM baselines to further contextualize LMM performance. The first baseline, termed “caption aggregation,” follows a two-stage process in which images are first captioned using Qwen2-VL, and subsequently, Llama-v3.1 (Dubey et al., ) is used to answer questions based on the aggregated captions. This is a query-unaware baseline given that this method generates captions without conditioning on the questions being asked, capturing a sub-optimal, yet informative, comparison point. The second baseline, the “detector oracle,” serves as an upper-bound baseline by utilizing OWLv2 (Minderer et al., ) as an object detection model to first find all images with the anchor object which are then each checked for the target object. Detailed descriptions of both baselines are available in subsection B.1.

In both single-needle and multi-needle settings, we conducted experiments using the full VHs dataset where the haystack size was 100 images or fewer, and switched to the $\text{VHs}_{\text{small}}$ subset with larger haystacks to mitigate computational costs. We report the results as bootstrapped averages, with standard deviations indicated as error bars to ensure robustness. The outcomes of the single- and multi-needle challenges are presented in Figure 2 and Figure 3. Figure 4 explores LMM behavior when the needle image is located in different positions within the input image set. Comprehensive results are shown in subsection B.2.

Our analysis yielded three key insights into LMM behavior on long-context visual retrieval and reasoning tasks: (1) a susceptibility to visual distractors, (2) difficulty in reasoning across multiple images, and (3) a bias in relative image positioning. These findings are not possible to conclude using prior visual NIAH benchmarks that utilize artificial needle generation, further highlighting the significance and contribution of our realistic, vision-centric benchmark. We elaborate on each of these observations below.

### 3.1 Susceptibility to Visual Distractors

As shown in Figure 2, the single-needle challenge reveals that LMMs are significantly impacted by visual distractors. When presented with only one image, general-purpose LMMs such as InternVL2-8B and Gemini v1.5 Pro perform comparably to specialized detectors like OWLv2, indicating that these models can handle simple visual reasoning tasks in isolation. However, as the number of input images increases, a notable drop in performance is observed compared to an oracle baseline that combines a detector with a language parser. This degradation, which does not appear in prior artificial OCR-based NIAH benchmarks as shown in Figure 1 (B), suggests that the main limitation of current LMMs lies in their ability to retrieve relevant visual information from large sets of images containing distractors. This challenge is particularly pronounced in open-source models like InternVL2-8B, which excel at single-image tasks but struggle as the visual context grows in complexity.

An intriguing outcome arises from the caption aggregation baseline: despite being an inherently sub-optimal approach due to the lack of context-specificity when generating captions, it begins to match or even surpass the performance of open-source LMMs when the number of images reaches 20. This indicates that while language models can exhibit robustness against irrelevant text (captions in this case), general LMMs nowadays are still impacted by irrelevant images. Nevertheless, the caption aggregation method is impractical for general multi-image question answering due to its high computational overhead and limited context length. For instance, processing 100 images for a single question can take over three minutes (see Figure 6 for runtime analysis), and 1,000 detailed captions would exceed Llama-v3.1’s 128k context length limit.

Beyond the issue of distractors, context length limitations remain a fundamental bottleneck for all tested LMMs. Qwen2-VL, e.g., is constrained by its shorter context windows and can ingest no more than 50 images. While Phi-3 theoretically offers a higher context capacity, it exhausted the memory of four 40GB A100 GPUs when processing 100 images. API-based models such as Gemini v1.5 Pro and GPT-4o can process up to 100 images but cannot solve larger haystacks (1,000+ images) due to API limitations.

Figure: Figure 3: Experimental results on the VHs multi-needle challenge reveal insightful outcomes. (A) The oracle experiment, which uses only needle images as input, demonstrates significant performance degradation in both proprietary and open-source LMMs when required to integrate information across multiple images. (B) In the full multi-needle challenge that includes distractor images, we observed a performance decline of existing LMMs as the size of the haystack (N) increases. Given the same haystack size, the performance deteriorates considerably compared to the single-needle challenge across all models in most scenarios. These findings indicate that current methodologies struggle with real-world, large-scale multi-image QA tasks that demand both visual retrieval and reasoning across extensive visual contexts.
Refer to caption: x2.png

### 3.2 Difficulty in Reasoning Across Multiple Images

To further examine LMMs’ ability to integrate information across multiple images, we conducted experiments using the VHs multi-needle challenge. This task requires LMMs to first retrieve relevant visual information and then integrate it across images to generate a correct response.

Our initial experiment focused on an oracle cross-image reasoning scenario. In this setting, models were provided only with needle images and a standardized multi-image question template described in subsection 2.1. Interestingly, we observed a sharp performance decline across all LMMs as the number of needle images increased from one to two (Figure 3 (A)). However, as the number of needle images increased from two to three, the performance decline became much smoother or even stabilized. This trend was especially pronounced in open-source LMMs, which exhibited larger performance drops initially. We hypothesize that this may be due to the models being primarily trained on single-image question-answering datasets, with limited exposure to multi-image tasks—an observation corroborated by their technical reports. Overall, these experimental results suggest that LMMs continue to struggle with integrating information across multiple images, even in scenarios where the task of visual retrieval is mostly excluded.

In the full VHs multi-needle challenge, where models must both retrieve visual information and reason across multiple images, the results (shown in Figure 3 (B)) were consistent with those from the single-needle experiments. As the number of input images increased, performance steadily declined across all models. While Gemini v1.5 Pro maintained relatively high accuracy (above 65%, with only a 5% decline from N=5 to N=100), other models experienced steeper drops, with some falling below the caption aggregation baseline. It is particularly noteworthy that GPT-4o, which performed best in the oracle multi-needle experiments with two or three needles, struggled significantly in real-world multi-needle challenges with distractor images, performing worse than Gemini v1.5 Pro by a large margin. Open-source models also demonstrated performance declines from the oracle scenario to the real-world multi-needle challenges, with mPLUG-Owl3 dropping from the top-performing open-source model to the worst among its peers. Overall, these results highlight the persistent challenges that LMMs face in managing complex, multi-image visual retrieval and reasoning tasks.

### 3.3 Positional Biases

In natural language tasks, prior studies have identified the lost-in-the-middle phenomenon in models like GPT-3.5, where performance deteriorates on long-context retrieval and reasoning tasks when key information is positioned near the middle of the context window. However, previous benchmarks in LMMs did not observe this effect, likely because the datasets were too simple, allowing LMMs to easily retrieve relevant information regardless of its position. In VHs, we revisit this issue by systematically evaluating both open-source and proprietary models using the $\text{VHs}_{\text{small}}$ single-needle dataset.

As shown in Figure 4, LMMs are also sensitive to the positional placement of the needle image. Interestingly, this positional bias varies across models. For example, Gemini v1.5 Pro tends to favor images at the beginning of the sequence, while GPT-4o exhibits a clear lost-in-the-middle effect, similar to what has been observed in natural language tasks (Liu et al., ). In contrast, open-source models like Qwen2-VL, Phi-3-vision, and InternVL2 favor images closest to the question (i.e., the last image in the sequence) given a small number of images ($N\leq 5$). We break down the 2D heatmap for each method in Appendix B.

Figure: Figure 4: Plots showing needle position, vs. performance on the VHs benchmark for several image settings. For existing LMMs, the needle position is extremely important, with performance degradation of up to 25% when the needle is not placed in the optimal location in the input context. The gray boxes indicate that these experiments exceed the available context length or the model is unable to execute on 4 A100 GPUs.
Refer to caption: x4.png

## 4 MIRAGE: M ulti- I mage R etrieval A ugmented Ge neration

In the previous section, we discussed several sub-optimal behaviors of current Large Multimodal Models (LMMs) when handling long-context visual retrieval and reasoning tasks. A particularly pressing challenge is the inability of existing methods to efficiently process large-scale visual inputs—those involving over 1,000 or even 10,000 images. To address this limitation, we introduce MIRAGE, an open-source Retrieval-Augmented Generation (RAG) framework for large-scale, long-context visual retrieval and reasoning.

The design of MIRAGE was informed by three guiding principles: (1) Beyond simply extending context length, we aimed to optimize efficiency by reducing the number of tokens per image, enabling the processing of larger sets of images. (2) To minimize computational overhead and reduce vulnerability to irrelevant images, we integrated a lightweight retriever module trained alongside the LMM to filter out distractors—a core RAG approach. (3) Recognizing the scarcity of multi-image QA datasets with realistic visual distractors, we also construct a custom instruction-tuning dataset to enhance the model’s retrieval and reasoning capabilities. The following subsections detail these components.

Despite its simplicity, MIRAGE is the first framework to enable visual-RAG operations with LMMs. MIRAGE can take over 10,000 images during inference with a single 40GB A100 GPU while maintaining competitive performance across various tasks, as demonstrated in section 5.

### 4.1 Model Architecture

Figure: Figure 5: MIRAGE enables large-scale long-context visual retrieval and reasoning through a combination of key components and a large-scale training set. (A) MIRAGE processes questions and images in several stages: first, image features are encoded using CLIP, followed by compression through our Q-Former. A retriever module then calculates relevance scores, ensuring that only the most relevant images are passed to the LLM for final reasoning. The red dashed line illustrates the key difference between conventional LMMs, like LLaVA, and our visual-RAG approach. (B) To further improve MIRAGE’s performance in long-context retrieval and reasoning, we introduce a large-scale MIQA instruction-tuning dataset, blending both synthetic and real-world data.
Refer to caption: extracted/5917889/figures/MIRAGE.png

MIRAGE is a RAG-based solution built on LLaVA-v1.5-7B (Liu et al., ) and Llama-v3.1-8B (Dubey et al., ). The model input consists of a series of image tokens followed by a textual question, formatted as ‘‘<img_token> <img_token> ... \n <Actual Question>’’. Each image is first passed through a frozen CLIP image encoder to extract patch-level features (576 tokens). To reduce the computational and context-length burden, we apply an image compressor to reduce the number of tokens per image. Specifically, we employ a Q-Former (Li et al., ), a lightweight transformer with $K=32$ learned query vectors that cross-attend to the full 576 patch features, reducing the tokens per image from 576 to 32—a significant 18x reduction in token intensity. Following this, the output is passed through two MLP layers with GELU activation functions (Hendrycks & Gimpel, ) to align with the LLM’s input dimensions. MIRAGE is visualized in Figure 5. Additional comparisons of this compression technique against alternative methods can be found in Table 2.

Despite token compression allowing MIRAGE to handle over 1,000 images, processing 10,000 images remains inefficient if a significant portion of those tokens are irrelevant to the query. To address this, we implemented a retriever module that performs a lightweight relevance-based filtering of images, ensuring that only pertinent images are processed by the LMM. Although many forms of retrieval can be used (e.g., CLIP similarity thresholds), we found it more effective to use a query-aware retrieval model trained alongside the next-token prediction task (shown in Figure 6, and discussed in section 5). Formally, given a set of images, $I$ made up of image tokens $\{i_{0},\dots,i_{n}\}$, it is our goal to retain the minimal subset $I_{\text{min}}\subset I$ such that we can still accurately answer the query $Q$. Let the image encoding module as a function $\psi(I)$, we can get:

$$ $F_{i},\ R_{i}=\psi(I_{i},\ Q),$ (1) $$

where $F_{i}$ represents image features and $R_{i}$ represents the relevance score of the image of Image $I$. In practice, our retriever module consists of several transformer blocks on the query and compressed image features, followed by a sigmoid activation to predict a 0/1 relevance score.

After token reduction and retrieval filtering, the LLM processes the resulting set of visual features along with the encoded question. Like other LMMs, MIRAGE concatenates aligned image features and text features together and passes to the downstream LLM, going through next-token prediction to generate textual output.

### 4.2 Model Training

Training Data: Due to the limited availability of multi-image QA (MIQA) datasets, we created a training dataset for MIRAGE by combining two main sources: (1) existing MIQA datasets, and (2) synthetic MIQA data derived from single-image QA datasets. We first included all publicly available MIQA training sets, including RetVQA (Penamakuri et al., ), SlideVQA (Tanaka et al., ), and WebQA (Chang et al., ). RetVQA, derived from Visual Genome, contains 377K questions but focuses on narrow domains such as object attributes, relationships, and counting. SlideVQA and WebQA add some diversity but are limited in size, containing only 19K and 12K questions, respectively. These datasets typically include one or two relevant images out of a set of 15-30 images.

To supplement the training data, we adapted LLaVA’s single-image QA data into a multi-image format. Instead of simply adding random distractors, which risks diluting the relevance of questions, we employed a keyword-based filtering method to cluster questions with similar content, followed by random sampling of two to ten distractor images from unrelated subsets. This process results in an instruction-tuning dataset containing around 600K samples. We further shuffle the images within each instruction-tuning pair during training to ensure the relevant images can appear in any position, forcing the model to remain insensitive to the positional context of relevant images. Figure 5 (B) shows the composition of the assembled dataset.

Training Procedure: MIRAGE’s training follows a two-stage process: pre-training and fine-tuning. During pre-training, the CLIP visual encoder and LLM backbone are frozen, with only the Q-Former and MLP layers trained on the next-token prediction task. For pre-training, we utilized a combination of ShareGPT captioning data (Chen et al., ), alongside large-scale datasets like CC3M/12M (Changpinyo et al., ), LAION-400M (Schuhmann et al., ), and COCO (Lin et al., ), using a data annealing approach (Li et al., ) to emphasize high-quality data in the later stage of the training. During fine-tuning, the retriever module is activated, and the entire model (except the CLIP encoder) is trained on both real and synthetic MIQA data. In addition to the next-token prediction task, we co-trained the retriever using the binary cross-entropy loss, assigning a higher weight (5.0) to positive samples to address data imbalance and prioritize recall. The instruction tuning was completed in two days using 16 A100 GPUs, with the first 60% of the training focused on passing only relevant images to the LLM. In the remaining 40%, several distractor images were added to improve robustness, following recommendations from (Zhang et al., ).

## 5 Results & Discussion

**Table 1: Comparative performance of methods on multi-image and single-image QA tasks. MIRAGE shows strong performance in multi-image QA and competitive performance in single-image QA compared to existing proprietary and open-source long-context models. Built primarily on LLaVA-v1.5-7B, MIRAGE-8.3B matches its performance on single-image QA while excelling in multi-image QA, achieving this with a reduced number of tokens per image.**
| Method | RetVQA (MIQA) | VQAv2 | GQA | TextVQA | POPE | MMB | MMB-CN | MM-Vet |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GPT-4o (OpenAI, ) | 34.6 | 77.2 | - | 78.0 | - | - | - | - |
| Gemini-v1.5 (Reid et al., ) | 32.2 | 73.2 | - | 73.5 | - | - | - | - |
| LWM (Liu et al., ) | - | 55.8 | 44.8 | 18.8 | 75.2 | - | - | 9.6 |
| LLaVA-v1.5-7B Liu et al. () | 30.6 | 78.5 | 62.0 | 58.2 | 85.9 | 64.3 | 58.3 | 31.1 |
| MIRAGE-8.3B (Ours) | 67.6 | 76.6 | 59.1 | 56.2 | 85.4 | 69.2 | 66.9 | 33.4 |

**Table 2: Exploration of various token reduction methods. The Q-former demonstrates the most efficiency in reducing token count while maintaining the majority of general QA performance. All experiments were conducted using the official LLaVA-v1.5-7B model with Vicuna-v1.5 as the LLM and LLaVA’s instruction tuning data. Note that the Q-former variant shows lower performance compared to our final model MIRAGE-8.3B, which leverages better training data and a more powerful LLM, Llama-v3.1-8B.**
| Method | Tokens/Img | VQAv2 | GQA | Vizwiz | TextVQA | POPE | MMB | MMB-CN | MM-Vet |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Original LLaVA | 576 | 78.5 | 62.0 | 50.0 | 58.2 | 85.9 | 64.3 | 58.3 | 31.1 |
| 3x3 Max-Pooling | 64 | 68.7 | 56.2 | 41.3 | 48.5 | 83.0 | 59.2 | 49.3 | 24.3 |
| Global Avg. Pooling | 1 | 62.5 | 51.3 | 37.7 | 45.5 | 79.6 | 55.0 | 45.5 | 18.9 |
| Q-Former (Ours) | 32 | 72.8 | 56.6 | 48.0 | 47.1 | 83.9 | 61.5 | 55.0 | 27.3 |

Visual Haystacks: MIRAGE’s performance on single- and multi-needle VHs challenge is presented in Figure 2 and Figure 3. All baselines and MIRAGE are tested in an identical, zero-shot setting in which they are not trained on these kinds of questions. In the single-needle challenge, MIRAGE demonstrates competitive oracle (N=1) performance against existing approaches. Notably, MIRAGE is the only method that scales to 10k input images, outperforming all open-source models when processing more than three images, and surpassing both Gemini v1.5 Pro and GPT-4o when using over 50 images.

In the multi-needle setting, in addition to handling a significantly larger number of input images, MIRAGE outperforms open-source LMMs in most cases and approaches GPT-4o’s performance when using 100 images. However, like other open-source models, MIRAGE exhibits a drop in performance when transitioning from single-needle to multi-needle tasks, especially compared to Gemini v1.5 Pro. These results highlight the need for further advancements in the cross-image reasoning capabilities of these models. We hypothesize that the degraded performance stems from current training datasets which typically focus on two-image inputs and lack diverse question types. This limits MIRAGE’s ability to effectively handle tasks requiring reasoning across more than three images with varied queries. Future enhancements to MIQA datasets could address these challenges.

Conventional VQA Tasks: We also compare MIRAGE with other methods on conventional VQA tasks, including the RetVQA multi-image task (Penamakuri et al., ) and several single-image QA tasks, VQA-v2 (Goyal et al., ), GQA (Hudson & Manning, ), TextVQA (Singh et al., ), POPE (Li et al., ), MMB (Liu et al., ), MMB-CN (Liu et al., ), and MM-Vet (Yu et al., ). In all cases, we follow standard evaluation procedures from either RetVQA (Penamakuri et al., ) or LLaVA (Liu et al., ). As shown in Table 1, MIRAGE is the only method capable of excelling in both multi-image and single-image QA tasks. Open-source models like LLaVA-v1.5-7B (Liu et al., ) struggle with multi-image QA, likely due to insufficient context length and lack of specific training for this task. Similarly, commercial long-context LMMs also perform poorly on multi-image tasks containing a lot of distracted images.

Image Token Compression Scheme: MIRAGE aims to facilitate future research into MIQA tasks, with a critical focus on reducing the context length required per image. To investigate this, we compared various token compression schemes in the LLaVA-v1.5-7B model. These schemes included max-pooling and average-pooling compression, reducing an image to a single token. All models were trained using LLaVA’s single-image instruction fine-tuning dataset, with the retriever disabled for this experiment. The results, presented in Table 2, indicate that while previous work has suggested that Q-Former is sub-optimal (Fan et al., ), our findings show that Q-Former retains reasonable performance while significantly reducing context length.

MIRAGE retriever vs. CLIP: While MIRAGE trains the retrieval model inline, it is also possible to use an external retrieval module such as CLIP (Radford et al., ) to perform lookup prior to passing images to the downstream LLM. To explore this question, we compare the single-stage MIRAGE retriever with using CLIP as the retriever on the VHs benchmark in Figure 6. This experiment demonstrates that while CLIP is somewhat faster, it struggles with recall, a critical metric for the MIRAGE system, as poor recall leads to necessary information being dropped before even making it to the LLM for analysis. Additionally, we include the performance of CLIP paired with LLaVA-v1.5-7B on single-needle VHs in Table C.1, where it underperforms compared to our MIRAGE-8.3B for almost all cases.

Figure: Figure 6: Comparison between MIRAGE retriever and CLIP retriever across both performance and runtime. We can see that while CLIP is slightly faster, MIRAGE has significantly higher recall, which impacts downstream performance. Both MIRAGE approaches with a retrieval component are significantly more efficient than GPT-4o and Caption-based baselines.
Refer to caption: x5.png

## 6 Related Work

Ours is not the first work to address the MIQA problem. Similar to our work, Bansal et al. (Bansal et al., ) introduce a method for multi-image question answering, however in their approach, all of the images (up to 10) are relevant to the question, and their method does not contain a retrieval component. Penamakur et al. (Penamakuri et al., ) propose a dataset and method for retrieval-based visual question answering (RetVQA), and share with our work the fact that answers must be gleaned from a set of both relevant and irrelevant images. However, their work differs from ours in two key ways: (1) RetVQA question contexts contain at most two relevant images, meaning that models do not have to reason across many images, largely sidestepping the problem of limited context length, and (2) their approach only allows for small image worlds (up to 30), meaning that they can use a pairwise encoder for each image, and they do not need to search over a large dataset of images potentially containing distractors, limiting the efficiency of their approach. Finally, they are only able to perform question answering over the images, whereas we pursue a method that can additionally perform more complex reasoning tasks given the images in the dataset.

Similar to our MIRAGE approach is Chen et al. (Chen et al., ) and Yasunaga et al. (Yasunaga et al., ) which both retrieve multiple visual documents to answer queries. In the case of Chen et al. (Chen et al., ), the queries are open-ended question answering, and thus, are not grounded within a particular set of context images. Yasunaga et al. (Yasunaga et al., ) focuses on one and few-shot classification and image generation, and does not use their multi-image retrieval to answer aggregated questions about those images. In single-image QA, image retrieval in large multimodal models (LMMs) has been explored using “retrieval-tokens” (Koh et al., ), however, it is unclear how such an approach would scale to multi-image QA problems with multiple relevant images. Complementary to our approach, several methods have focused on true long-context LMMs by introducing down-sampling techniques or state-space models (Li et al., ; Chen et al., ; Abdin et al., ; Wang et al., ) - and while these methods cannot yet handle 10K images well, we expect in the future for this to be a promising direction of further research. In addition to general QA, several other works contain domain-specific multi-image QA problems, including Slide VQA (Tanaka et al., ), Multimodal QA (Talmor et al., ), WebQA (Chang et al., ) and Document VQA (Tito et al., ), however, none are explicitly designed to answer integration questions across multiple images from a subset of large, unrelated images (like VHs).

Outside of VQA, In traditional NLP, retrieving small passages or single documents from large-scale corpuses has proven effective. Zhang et al. (Zhang et al., ) introduces a method that fine-tunes LLMs on both relevant and irrelevant documents to support better RAG performance, but does not train an explicit query-aware filter or compression module. ATLAS (Izacard et al., ) treats documents as latent variables during training, allowing for efficient retrieval, but requires a complex joint-training setup. Similarly, several methods (Shi et al., ; Ram et al., ; Borgeaud et al., ; Khandelwal et al., ) have demonstrated success with zero or few-shot augmentation of standard LLM contexts with retrieved documents. Beyond context augmentation, several traditional NLP approaches (Lin et al., ; Wang et al., ; Xu et al., ; Liu et al., ; Xiang et al., ; Xu et al., ) have demonstrated that fine-tuning LLMs to be robust to noisy RAG outputs can lead to performance improvements.

Contemporary to VHs, Wang et al. () introduce a synthetic benchmark to evaluate LMMs on multi-image question answering (up to 36 images per haystack). Similarly, Zhao et al. () propose a video-centric benchmark to test video LMMs on spatiotemporal understanding. Unlike VHs which relies on object labels in real images to build questions about up to 10K images, both benchmarks focus heavily on synthetic content with short-range retrieval problems (<100 frames/images). Moreover, similar to traditional visual NIAH benchmarks, they incorporate out-of-distribution content (frames, patches) into the image or video as the visual “needle.” This contrasts sharply with VHs, which employs realistic and in-distribution visual content, thereby posing a more complex challenge that is less susceptible to straightforward solutions.

MIQA vs. Video-QA: While several methods have been developed for video question answering (such as Video-LLaVA (Lin et al., )), the task of answering questions over video is fundamentally different from MIQA. While models that can solve MIQA can often solve video tasks, the inverse is not true, as video frames contain many frame-wise inter-dependencies that are exploited by encoder models (such as MAG-ViT (Yu et al., ) which use temporal blocks, or frame-subsampling, which drops closely related frames). MIQA has independent images, rendering most video models inadequate for this task. It is an interesting direction for future work to explore how to connect MIQA and Video-QA, particularly across the data dimension, wherein Video-QA datasets could provide useful training data for MIQA models.

## 7 Conclusion

In this work, we introduce “Visual Haystacks (VHs)”, a benchmark specifically designed to challenge LMMs in multi-image question answering (MIQA) by testing their ability to both retrieve relevant images from large collections of unrelated inputs and reason across them. Our evaluation shows that VHs provides significantly more realistic and challenging tasks compared to existing visual NIAH benchmarks which effectively measure OCR or text-based performance. With VHs, we then identified three key limitations across both open-source and proprietary models: susceptibility to visual distractors, difficulty reasoning across multiple images, and biases due to the positional placement of key information within the context window. Additionally, we found that contemporary LMMs are limited to processing no more than 100 images simultaneously. To address these limitations, we introduced MIRAGE, an open-source visual-RAG framework capable of handling up to 10,000 images on a single 40GB A100 GPU. MIRAGE showed superior performance over existing open-source LMMs on VHs and set a new state-of-the-art on the RetVQA benchmark while maintaining reasonable single-image QA performance.

Both VHs and MIRAGE represent some of the first concrete steps towards large models capable of answering questions over thousands or tens of thousands of unrelated images. While they represent strong steps forward, the problem is far from solved, and we hope that this benchmark and framework will inspire continued research in models and the MIQA problem.

#### Acknowledgments

We thank Lisa Dunlap, Boyi Li, and Xudong Wang for their invaluable feedback during the discussions. This project was primarily supported by a grant from the National Geospatial-Intelligence Agency (Grant No. HM0476-22-1-2001). Authors, as part of their
affiliation with UC Berkeley, were supported in part by the National Science Foundation, US Department of Defense, and/or the Berkeley
Artificial Intelligence Research (BAIR) industrial
alliance program, as well as gifts from Anyscale, Astronomer, Google, IBM, Intel, Lacework, Microsoft, Mohamed Bin Zayed University of Artificial Intelligence, Samsung SDS, Uber, and VMware. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NGA, DoD, or the US government. The paper is approved for public release in accordance with NGA-U-2024-01397.

## Appendix

In this appendix, we include several additional discussions:

- •
Appendix A includes additional analyses of the VHs benchmark, discussing the dataset distribution and key characteristics, along with visual examples of single- and multi-needle tasks.
- •
Appendix B describes the implementation of the baseline models used in the VHs experiments and presents comprehensive results from the VHs experiments, including detailed performance metrics for single- and multi-needle tasks.
- •
Appendix C explores the potential limitations of the VHs benchmark and the MIRAGE model, along with their societal impacts.

## Appendix A More Analyses on VHs Benchmark

In subsection 2.1, we previously discussed the construction of the VHs benchmark dataset. Here, we take a closer look at its key aspects to provide a more detailed understanding. Figure A.1 presents the distribution of anchor and target objects within the VHs dataset.
Our dataset exhibits a diverse range of objects, and the question/answer pairs are well-balanced. Additionally, we include a Language-Only baseline in Table C.1 to emphasize that the benchmark task cannot be easily solved without leveraging the visual content. These statistics support that our benchmark is both representative and free from bias.

Furthermore, we visualize real single- and multi-needle examples in Figure A.2 and Figure A.3, respectively. These examples highlight the realism and vision-centric nature of our VHs dataset, establishing it as a robust and reliable visual NIAH benchmark.

Figure: Figure A.1: We visualize the distribution of the anchor (needle) and target objects in the VHs dataset, highlighting its diversity and comprehensive coverage of common objects from COCO.
Refer to caption: x7.png

Figure: Figure A.2: Real examples of the single-needle VHs challenges (10-image sub-task shown). These visual needles are more realistic and challenging compared to the copy-and-paste approaches in (Reid et al., ; Wang et al., ). Models must retrieve the correct image containing the visual needle and answer the associated question. Distractor images are intentionally included to increase the task difficulty.
Refer to caption: extracted/5917889/figures/single_needle_examples.png

Figure: Figure A.3: Real examples of the multi-needle VHs challenges (10-image subtask shown). This task is significantly more difficult than the single-needle task, as it requires models to integrate information across multiple images in addition to handling distractors.
Refer to caption: extracted/5917889/figures/multi_needle_examples.png

## Appendix B Details on the VHs Experiments

### B.1 Baseline Implementation

In section 3, we benchmarked several LMMs and included two non-LMM baselines. Their implementations are detailed below:

Detector Oracle: This baseline assumes access to a specialized object detector, specifically the OWLv2 open-vocabulary detector, and an ideal language parser capable of perfectly extracting the anchor object (the visual needle) and the target object. For the single-needle challenge, given a set of images, OWLv2 first detects whether each image contains the anchor object. From the image with the highest detection confidence, we then verify if it also contains the target object and return a yes/no answer. For the multi-needle challenge, we apply a detection confidence threshold and check whether images identified as containing the anchor object also contain the target object. Depending on the question format—whether asking for all or any of the objects—simple control flow determines the final yes/no answer. This setup, leveraging a specialized detector rather than a general LMM and assuming a perfect query understanding, serves as an upper-bound baseline.

Caption Aggregation: Another interesting baseline is a non-query-aware approach where we first caption each image and then use an LLM to aggregate these captions to answer the question. In practice, we prompt Qwen2-VL with the instruction, "Please provide detailed and concrete captions of the image," to generate high-quality image descriptions. These captions are then passed to Llama-v3.1-8B, prompted in a specific format (showing in Figure B.1) to integrate the information and provide an answer.

LMMs: For proprietary models, we used the APIs for “gpt-4o-2024-05-13” and “gemini-1.5-pro-001.” For open-source models, we employed their official Huggingface versions, with details provided in Table B.1. Each model was prompted with: “You are given a set of images. Please answer the following question in Yes or No: {real question}.”

Figure: Figure B.1: The prompt for Llama-v3.1 (8B) used in the caption aggregation baseline.

**Table B.1: Open-source models and their corresponding HuggingFace links used in the evaluation.**
| Model Name | Hugging Face Model Link |
| --- | --- |
| OWLv2 (Minderer et al., ) | [https://huggingface.co/openai/clip-vit-base-patch32](https://huggingface.co/openai/clip-vit-base-patch32) |
| LongVILA-8B-1024Frames (Xue et al., ) | [https://huggingface.co/Efficient-Large-Model/Llama-3-LongVILA-8B-1024Frames](https://huggingface.co/Efficient-Large-Model/Llama-3-LongVILA-8B-1024Frames) |
| Qwen2-VL-7B-Instruct (Wang et al., ) | [https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct) |
| Phi3-vision-128k-instruct-4.2B (Abdin et al., ) | [https://huggingface.co/microsoft/Phi-3-vision-128k-instruct](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct) |
| InternVL2-8B (Chen et al., ) | [https://huggingface.co/OpenGVLab/InternVL2-8B](https://huggingface.co/OpenGVLab/InternVL2-8B) |
| mPLUG-OWL3-8B (Ye et al., ) | [https://huggingface.co/mPLUG/mPLUG-Owl3-7B-240728](https://huggingface.co/mPLUG/mPLUG-Owl3-7B-240728) |
| Idefics3-8B-Llama3 (Laurençon et al., ) | [https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3](https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3) |
| LLaMA-v3.1 (Dubey et al., ) | [https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) |

### B.2 Comprehensive Results

In addition to the line charts in Figure 2 and Figure 3, we provide raw data and more baseline comparisons in the appendix. The complete results for the single-needle and multi-needle challenges are presented in Table C.1 and Table C.2, respectively. To better illustrate the positional bias pathology highlighted in Figure 4, we include multiple 1-D plots in Figure C.1 for clearer interpretation. For proprietary models, we observe that Gemini v1.5 Pro tends to favor needle images at the beginning of the sequence, whereas GPT-4o shows a preference for images that are not in the middle. In contrast, for open-source models with a small number of images ($N\leq 5$), LMMs perform significantly better when the image is placed near the question (at the end of the image sequence). However, as the number of images increases, models like Qwen2-VL shift their preference towards images at the beginning, while other models still favor images at the end. We believe this intriguing behavior is influenced by the training data, but further investigation is needed to confirm and address this issue in future work.

## Appendix C Potential Limitations and Societal Impact

VHs: While VHs is the first benchmark to investigate how LMMs can retrieve and reason across sets of images, it has some notable limitations. The primary limitation of the benchmark is the scope: since it is based on MS-COCO images, it inherits the implicit biases in the dataset including notable gender, race, and location biases (Hendricks et al., ; Bhargava & Forsyth, ; Hirota et al., ; Wang et al., ). It is important to work in the future toward developing benchmarks that do not favor models that prefer data having such biases. Beyond such implicit biases, COCO objects are also limited to 80 categories - strong performance on the VHs benchmark does not imply that the model will generalize well to all MIQA problems. Finally, the VHs benchmark is primarily template-based, which means that it does not evaluate the language reasoning capabilities of the LLM. A more complex benchmark would require making more detailed inferences, and require multi-hop reasoning across a wider range of open-domain object sets.

MIRAGE: MIRAGE is significantly more efficient than its LLaVA base, while simultaneously performing better on many MIQA benchmarks. To perform well on MIQA benchmarks, however, we note that MIRAGE sacrifices some single-image performance, likely due to inefficiencies in the multi-task training setup. It remains interesting and necessary for future work to explore how such approaches can retain single-image performance while improving multi-image capabilities. It is also important to recognize that as a large multi-modal model, the potential for misuse of the model exists. Many of the impacts of such models are well studied in other related works (Bender et al., ; Liu et al., ; ). Recognizing this, MIRAGE inherits the safety mechanisms from the LLaVA code-base (Liu et al., ), and includes relevant training details in a fully public code release.

Figure: Table C.1: Performance on VHs for single-needle questions. We can observe that as the number of input images (N) increases, LMMs’ performance degrades rapidly, specifically for open-source models. This indicates their difficulty in retrieving the correct visual needle from large-scale visual inputs. “E” denotes either exceeding context length or failure to execute on 4 × 80GB A100 GPUs for open-source models, or an API error for proprietary models. Additionally, we observed that the Idefics3 model demonstrated low compliance, often failing to provide a Yes/No response to our questions.

Figure: Table C.2: As in the single-needle experiments, the performance of LMMs degrades rapidly as the number of input images (N) increases. The low performance of most LMMs at the beginning (N=5) highlights their difficulty in both retrieving the correct images and integrating information across frames. The definition of "E" is the same as in Table C.1.

Figure: Figure C.1: Visualization of the positional bias pathology across various models.
Refer to caption: extracted/5917889/figures/positional_biases_gemini-v1.5-pro.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the weakeness. In addition,
* How many templates were used to generate questions?
* What advantages does the VHs benchmark offer compared to recent multi-image QA benchmarks?
soundness: 3
strengths: * Novel Multi-Image QA Benchmark: The authors introduce an interesting multi-image QA benchmark, Visual Haystacks, designed around a vision-centric "needle-in-a-haystack" scenario, providing a fresh and challenging setting for the LMM evaluation.

* Comprehensive Model Evaluation:  The paper conducts a thorough evaluation of LMMs on the VHs benchmark, uncovering important insights into current models, such as vulnerability to visual distractors, challenges with multi-image understanding, and tendencies toward positional visual bias.

* Novel Visual RAG Framework: The authors introduce a novel visual RAG framework that combines a compressor and a retriever. The compressor efficiently processes up to 10,000 images on a single 40GB A100 GPU, while the retriever identifies the top-k most relevant images for a given question, enhancing the framework’s scalability and efficiency.
summary: The authors presents Visual Haystacks (VHs), a new vision centric benchmark designed to assess the performance of Large Multimodal Models (LMMs) in the multi-image question answering (QA) task. In addition, the author proposes a new visaul-RAG framework, MIRAGE, to enhance the task performance.
weaknesses: * Limited Object Diversity: The authors constructed the VHs benchmark using objects from the COCO dataset, which contains only 80 object categories. This limited selection may restrict the diversity and comprehensiveness of the benchmark, potentially affecting its ability to evaluate models across a broader range of visual scenarios.

* Restricted Question Diversity: The authors appear to rely on a few simple templates to generate questions, which may restrict the variety of question types in the benchmark.

* More like Object Detection than QA Reasoning: Many questions in the benchmark (e.g., "For the image with a truck, is there a dog?") seem to primarily assess the model’s object detection abilities rather than its visual QA reasoning skills. It is questionable if the benchmark requires the advanced visual QA reasoning skills from the models.

* Missing Related Work: The paper does not reference several recent multi-image QA benchmarks, for example: 
 1. CompBench: A Comparative Reasoning Benchmark for Multimodal LLMs
 2. MANTIS: Interleaved Multi-Image Instruction Tuning
 3. MUIRBENCH: A Comprehensive Benchmark for Robust Multi-Image Understanding. 

Additionally, a similar multi-image retrieval approach was introduced in "ColPali: Efficient Document Retrieval with Vision Language Models", but this work was also not cited.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: - Could you please address the points raised in the above weakness?
- Could you please add some randomly sampled failure cases made by GPT or Gemini? Sometimes failure cases can tell more than good cases. 
- Could you please address the ethics concerns around the code license?
soundness: 3
strengths: - I generally feel the direction is important to our community where design meaningful Visual Haystack benchmark for evaluating VLM. 
- Some interesting points are discovered when evaluating models on the proposed benchmark. Since random guess could achieve 50% accuracy in the proposed benchmark, some open-sourced VLMs performance significantly drop even the Haystack size is very small. However, those models maintain high scores in some public evaluation-datasets. 
- Some detailed experiments are conducted such as needle position and running time. 
- The proposed benchmark are made publicly available under MIT license, which is good for community.
summary: This paper addresses the limitations of Large Multimodal Models (LMMs) in multi-image question answering, where handling large visual contexts does not ensure effective retrieval and reasoning across images. Current benchmarks reveal biases and challenges in MIQA, such as poor cross-image reasoning and sensitivity to information placement. To overcome these, the authors propose "Visual Haystacks (VHs)," a vision-centric benchmark that tests retrieval and reasoning over multiple images, highlighting models' struggles with visual distractors and multi-image reasoning. They also introduce MIRAGE, an open-source Multi-Image Retrieval Augmented Generation framework capable of handling up to 10,000 images on a single GPU, achieving significant improvements over existing models and setting new standards in MIQA benchmarks like RetVQA. Key contributions include VHs, systematic LMM evaluation, and MIRAGE's scalable MIQA capabilities.
weaknesses: - Benchmark construction is still mainly centered around recognition tasks, based on benchmark design principles listed in Line129~138. Basically, it requires a strong recognition among all the input images, rather than true visual reasoning. 
- Based on the Figure 2 and 3, certain models, such as Gemini, GPT and the proposed MIRAGE, consistently perform better on the proposed multi-needle challenges compared to single-needle tasks. However, the multi-needle challenges are intentionally designed to be more difficult, as they demand additional reasoning across multiple images. Does this mean failure in designing the benchmark?
- Since the benchmark is constructed in way of examining recognition, therefore the proposed method contain ad-hoc modules, such as "a retriever module then calculates relevance scores, ensuring that only the most relevant images are passed to the LLM for final reasoning." Does this design hold for general visual reasoning tasks? For example, many of the tested single image dataset used in this paper, do not need this retriever module at all. 
- The proposed framework achieved not-very-good performance on some of the tested datasets. Also, there are many datasets that not being tested such as SEED, MME, and CHAIR.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. I'm confused about the difference between the MIRAGE model in Table 1 and the Q-former Model in Table. Doesn't MIRAGE utilize Q-former. 
2. See above.
soundness: 3
strengths: 1. This paper introduced a new visual needle in a haystack benchmark which composed of 1k yes/no questions. 
2. Evaluated on both open-source and close-source models and gained three insightful findings. 
3. Introduced a new baseline called MIRAGE for better handling of visual haystack tasks.
summary: This paper introduces a long context, visual needle in a haystack benchmark which composed of 1k yes/no questions changeling the model to reasoning and find the target object in the images. It evaluated on both open-source and closed-source LMMs and reveal several critical findings such as susceptibility to visual distractors, difficulty in multi-image reasoning, and a bias in image positioning. It introduces a new baseline called MIRAGE (Multi-Image Retrieval Augmented Generation) for better handling of VH tasks.
weaknesses: 1. The questions are only limited to yes/no questions. 
2. The question template are very limited, seems only three. 
3. MIRAGE has a significant performance drop in 4 out of 7 general VQA tasks. 
4. The approach of MIRAGE, deselecting unrelated (distracting) images somehow circumvents the VH challenge, as the this challenge lies in how model can reasoning in long context.  
5. The task of finding a target object seems still not simulating a real world scenario of long context visual reasoning task.

Meta-review: The submission introduces a new benchmark "Visual Haystacks" for measuring the quality of large multimodal models (LMMs) at the task of multi-image reasoning. Along with the benchmark dataset, it also introduces a RAG framework (MIRAGE) that can process a magnitude larger number of images compared to prior work. This uses a retrieval-based relevance filter as well as compression of image features to fit into the context length of the LMMs.

After the initial round of reviews, this submission received scores of 6, 6, 6. The reviewers did not find sufficient reason to reject this submission during the discussion, and the consensus remained positive. The AC recommends acceptance, and requests the authors to use the constructive feedback from reviewers to update the submission.

Decision: Accept (Poster)