MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models

## Abstract

Abstract Existing multimodal retrieval benchmarks primarily focus on evaluating whether models can retrieve and utilize external textual knowledge for question answering. However, there are scenarios where retrieving visual information is either more beneficial or easier to access than textual data. In this paper, we introduce a m ultimodal r etrieval- a ugmented g eneration benchmark, MRAG-Bench , in which we systematically identify and categorize scenarios where visually augmented knowledge is better than textual knowledge, for instance, more images from varying viewpoints. MRAG-Bench consists of 16,130 images and 1,353 human-annotated multiple-choice questions
across 9 distinct scenarios. With MRAG-Bench , we conduct an evaluation of 10 open-source and 4 proprietary large vision-language models (LVLMs). Our results show that all LVLMs exhibit greater improvements when augmented with images compared to textual knowledge, confirming that MRAG-Bench is vision-centric.
Additionally, we conduct extensive analysis with MRAG-Bench , which offers valuable insights into retrieval-augmented LVLMs. Notably, the top-performing model, GPT-4o, faces challenges in effectively leveraging retrieved knowledge, achieving only a 5.82% improvement with ground-truth information, in contrast to a 33.16% improvement observed in human participants.
These findings highlight the importance of MRAG-Bench in encouraging the community to enhance LVLMs’ ability to utilize retrieved visual knowledge more effectively.

## 1 Introduction

Retrieval-augmented generation (RAG) has emerged as a promising direction in large vision-language models (LVLMs) (OpenAI, ; Liu et al., ; Bai et al., ; Huang et al., ; Chen et al., ; Hu et al., ; Chen et al., ; Tong et al., ; McKinzie et al., ). By incorporating external knowledge during generation, models such as Wiki-LLaVA (Caffagni et al., ) have demonstrated improved performance in knowledge-intensive question answering tasks.

There are several existing benchmarks evaluating retrieval-augmented LVLMs. For example, OK-VQA (Marino et al., ) focused on scenarios where the image content alone is insufficient to answer the questions. A-OKVQA (Schwenk et al., ) further extended this dataset to incorporate additional types of world knowledge. More recent works (Chang et al., ; Chen et al., ; Mensink et al., ) further expanded and curated large-scale knowledge base data to evaluate pre-trained vision and language models in knowledge-intensive and information-seeking visual questions.
However, as shown in Table 1, these benchmarks remain text-centric, as their questions can often be resolved with related external textual knowledge. In contrast, retrieving visual information is sometimes more beneficial than retrieving text, as humans often gain greater insights from it.
Specifically, we illustrate examples in Figure 1 where retrieving correct textual knowledge can be *hard* and retrieved textual knowledge can be *useless*, while retrieving additional images is helpful.
For instance, when presented with a top-down view of a car, humans may struggle to accurately identify it; however, with a front-facing view, they can quickly recognize the vehicle and effectively leverage the visual information.

**Table 1: Compared with previous works, MRAG-Bench focuses on evaluating LVLMs in utilizing vision-centric retrieval-augmented multimodal knowledge. “Diverse scenarios” refers to whether a benchmark categorized different scenarios during evaluation.: Web, : ImageNet (Russakovsky et al., ), : Flowers102 (Nilsback & Zisserman, ), : StanfordCars (Krause et al., ).**
| Benchmarks | Knowledge | Knowledge | Multi-Image | Diverse |
| --- | --- | --- | --- | --- |
| Modality | Source | Input | Scenarios |  |
| K-VQA (Shah et al., ) | Text | Wikipedia | ✗ | ✗ |
| OK-VQA (Marino et al., ) | Text | Wikipedia | ✗ | ✗ |
| MultiModalQA (Talmor et al., ) | Text | Wikipedia | ✗ | ✗ |
| ManyModalQA (Hannan et al., ) | Text | Wikipedia | ✗ | ✓ |
| A-OKVQA (Schwenk et al., ) | Text | Common/World | ✗ | ✗ |
| ViQuAE (Lerner et al., ) | Text | Wikipedia | ✗ | ✗ |
| WebQA (Chang et al., ) | Text/Caption | Wikipedia | ✗ | ✗ |
| Encyclopedia VQA (Mensink et al., ) | Text | Wikipedia | ✗ | ✗ |
| InfoSeek (Chen et al., ) | Text | Wikipedia | ✗ | ✗ |
| MRAG-Bench (Ours) | Image |  | ✓ | ✓ |

In this paper, we introduce MRAG-Bench, a benchmark specifically designed for vision-centric evaluation for retrieval-augmented multimodal models, with visual questions typically benefit more from retrieving visual knowledge than textual information. MRAG-Bench consists of 16,130 images and 1,353 human-annotated multi-choice questions spanning 9 distinctive scenarios. Focusing on utilizing visually augmented knowledge in real-world scenarios, we divide our benchmark into two aspects:
*perspective*, where changes in visual entity’s perspective requiring visually augmented knowledge; and *transformative*, where the visual entity undergoes transformative change physically thus requiring visually augmented knowledge. Specifically, MRAG-Bench requires models to reason about visual entities that undergo perspective changes, such as *angle*, *partial*, *scope* and *occlusion*, as well as transformative changes, such as *temporal*, *incomplete*, *biological* and *deformations*. Additionally, MRAG-Bench includes 9,673 human-selected images, which serves as the ground-truth image knowledge corpus for model evaluation.

We conduct extensive experiments on MRAG-Bench to evaluate 10 open-source and 4 proprietary LVLMs.
The results confirm that MRAG-Bench is vision-centric, as all LVLMs show greater improvements when augmented with images compared to textual knowledge.
Our results indicate that the best-performing GPT-4o model only achieve 68.68% and 74.5% of accuracy without RAG knowledge and with ground-truth (GT) RAG knowledge, respectively. This substantially outperforms the best open-source model LLaVA-OneVision by 15.39% and 15.52%, respectively. Notably, we observe while all models improve with GT knowledge, only proprietary models are able to effectively utilize noisy retrieved multimodal knowledge. This indicates the gap between open-source and close-source models still exists. Open-source models are falling short on their parametric knowledge and the ability to distinguish between high-quality and poor-quality retrieved visually augmented examples. In comparison to humans, GPT-4o achieves only a 5.82% improvement when augmented with GT knowledge and 0.28% with retrieved knowledge, whereas humans demonstrate a 33.16% and 22.91% improvement, respectively. These results highlight the importance of MRAG-Bench in encouraging the community to develop LVLMs better utilizing of visually augmented knowledge.

## 2 MRAG-Bench

### 2.1 Benchmark Overview

Figure: Table 2: Key statistics of MRAG-Bench.
Refer to caption: x2.png

Our benchmark is designed for systematic evaluation of LVLM’s vision-centric multimodal RAG abilities.
To achieve this, we focus on evaluating the model’s understanding of image objects that are not commonly associated with its knowledge base, while the collected ground-truth images can help incentivize specific visual concepts within LVLMs’ memory.
Therefore, we divide our benchmark into two main aspects, as illustrated in the examples in Figure 1:

- •
*perspective*, refers to the challenges in visual recognition and reasoning that arise when a visual entity is presented from varying viewpoints, scopes, or levels of visibility.
- •
*transformative*, refers to the challenges that arise when a visual entity undergoes fine-grained physical transformations, making it unfamiliar or not easily associated with the model’s prior knowledge.

MRAG-Bench consists of 16,130 images and 1,353 multiple choice questions, with key statistics shown in Table 2.
MRAG-Bench adheres to the following design principles: (1) it focuses on real-world scenarios where visually augmented information is useful; (2) it incorporates 9 diverse multimodal RAG scenarios covering various types of image objects; (3) it features cleaned ground-truth images for each question that align with human knowledge; and (4) it provides robust evaluation settings for deterministic evaluations. Unlike previous works focus on retrieving textual knowledge, evaluation on MRAG-Bench focuses on retrieving vision-centric knowledge, which can be formulated as follows: Given a query tuple $\mathbf{Q}$ composed of (query image, textual question), the multimodal retriever $\mathcal{R}$ returns a set of relevant images $\mathbf{I}$ ($[\mathbf{i}_{1},\mathbf{i}_{2},...,\mathbf{i}_{N}]$), then the LVLM $\mathcal{M}$ take the input ($\mathbf{Q}$, $\mathbf{I}$) and output the final answer.

### 2.2 Benchmark Composition

MRAG-Bench provides a systematic evaluation across 9 distinctive multimodal RAG scenarios, with four scenarios focused on the *perspective* understanding of visual entities, four on *transformative* understanding, and one categorized as “others”. As illustrated in Figure 2, each scenario comprises 7.5% to 23.8% of the whole benchmark. The selected examples of each scenario is shown in Figure 3. The details of each scenario are introduced as follows.

#### Perspective understanding aspect.

First, we have *perspective* aspect comprising [Angle], [Partial], [Scope], and [Occlusion] dimensions.

- •
[Angle] evaluates the ability of models to utilize visual knowledge of common shooting angles to identify and reason about less common, long-tailed viewpoints of visual entities.
- •
[Partial] evaluates the ability of models to use complete appearance knowledge to identify and reason when only a partial image of the visual entities is available.
- •
[Scope] evaluates the ability of models to leverage high-resolution, detailed images for identifying and reasoning about visual entities in longer-scoped, low-resolution images.
- •
[Occlusion] evaluates the ability of models to use ground-truth image knowledge to identify and reason when visual entities are occluded or partially hidden in natural scenes.

Figure: Figure 3: Qualitative examples on MRAG-Bench. For each scenario, we show the result of GPT-4o (OpenAI, ), Gemini Pro (Team et al., ), LLaVA-Next-Interleave (Li et al., ) and Mantis-8B-Siglip (Jiang et al., ). The ground-truth answer is in blue.
Refer to caption: x3.png

#### Transformative understanding aspect.

On the other hand, the *transformative* understanding scenarios cover [Temporal], [Deformation], [Incomplete], and [Biological] dimensions.

- •
[Temporal] evaluates the ability of models to use familiar image knowledge to identify and reason about visual entities undergoing temporal changes that may not be represented in the model’s knowledge base.
- •
[Deformation] evaluates the ability of models to use intact physical appearance knowledge to identify and reason when visual entities undergo deformation not captured in the model’s knowledge base.
- •
[Incomplete] evaluates the ability of models to compare and contrast the complete layout and structure of image knowledge to identify and reason about missing parts and the correct layout of visual entities.
- •
[Biological] evaluates the ability of models to utilize image knowledge after biological transformations of the visual entities.

[Others] aims to evaluate the ability of models to leverage geographic image knowledge to accurately identify and reason about the correct regions of origin for the visual entities of interest. All these scenarios work in tandem to comprehensively evaluate LVLMs’ abilities of leveraging visually augmented knowledge.

### 2.3 Data Collection

As the guidelines discussed in $\S$ 2.1, our benchmark collection involves a clean ground-truth image corpus that can resonate with model’s internal knowledge and a query question and image that challenge model’s memory according to our definition of 9 diverse scenarios. To collect a dataset for systematic evaluation of vision-centric multimodal RAG scenarios, we manually annotate all multiple-choice question answering (MCQA) data while sourcing images from either publicly available datasets or manually scraping them from the web.

#### Collection of perspective aspect.

To collect diverse image objects and knowledge that are not extensively represented in LVLMs’ memories (Zhang et al., ),
we considered three sources of data, ImageNet (Russakovsky et al., ), Oxford Flowers102 (Nilsback & Zisserman, ), and StanfordCars (Krause et al., ). To construct a high quality image corpus, for each of the image class that we included in our benchmark, we examined the validation set and excluded the unqualified images which can’t provide sufficient visual information for the recognition of this class. Among the selected corpus, we further humanly picked five representative examples covering the diverse aspects of each class object, as the five ground-truth examples in our experimental results (See $\S$3). For constructing the query images, we adhered to our scenario definitions and manually selected qualified images for the [Angle], [Scope], and [Occlusion] scenarios. For the [Partial] scenario, we randomly cropped images by 50% in both height and width. Then we performed another human inspection to ensure the quality of the cropped images, filtering out examples where the visual object did not occupy the dominant area of the image. We repeated the random cropping process until satisfactory images were obtained, filtering to 20.4 GT images per question on average.

#### Collection of transformative aspect.

We chose to manually scrape images from the web based on the definitions of the *transformative* aspect. To construct the image corpus, we employed Bing Image Search for each of the image object keyword predefined by us, please refer to Appendix A.1 for more details. We filtered out image objects that did not form a clear transformative pair between the query image and the ground-truth image, retaining approximately 74% of the keyword names in the process. For ground-truth image examples, we employed automatic scripts to download the top 15 images related to its keyword names and human filtered out the unqualified image. On average, this results to 5.9 images per question and the five ground-truth images used during our evaluation are manually selected same as in *perspective* aspect.

According to our guidelines, additional related image object knowledge from the same geographic region can assist in identifying that region more effectively. For the [Others] scenario, we source the data from the GeoDE dataset (Ramaswamy et al., ). For each distinct image object category, we randomly sampled 3 out of 6 regions to serve as the answers for each question and selected the corresponding image as the query image.

#### Quality control.

After constructing the entire benchmark, we implemented two quality control procedures: an automatic check with predefined rules and a manual examination of each instance. The automatic check verifies the correct MCQA format, assesses image validity and filters out redundant images in the corpus, more details are presented in Appendix A.1. The manual examination is conducted by two experts in this field, who checked the correspondence between query images and ground-truth image examples, and filtered or revised ambiguous questions and uncorrelated query image and ground-truth images.

**Table 3: Accuracy scores on MRAG-Bench. The highest scores for open-source models in each section and proprietary models are highlighted in blue and red, respectively. Both Retrieved RAG and GT RAG employ top-5 image examples (except for the incomplete scenario, where a single example is intuitively sufficient). The relative difference in performance compared to the score without RAG is shown in subscript, with blue indicating performance drops and red indicating improvements.**
| Model | Overall | Perspective | Transformative | Others |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Angle | Partial | Scope | Occlusion | Temporal | Deformation | Incomplete | Biological |  |  |  |
| Random chance | 24.83 | 27.64 | 23.98 | 24.51 | 19.44 | 22.15 | 25.49 | 29.41 | 25.49 | 22.5 |
| Human performance | 38.47 | 25.16 | 34.96 | 31.37 | 41.67 | 21.48 | 24.51 | 58.82 | 54.9 | 53.33 |
| + Retrieved RAG | 61.38${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>2.91}}}$ | 62.42${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>7.26}}}$ | 60.16${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>5.2}}}$ | 58.82${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>7.45}}}$ | 62.96${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>1.29}}}$ | 54.36${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>2.88}}}$ | 49.02${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>4.51}}}$ | 78.43${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>9.61}}}$ | 63.73${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+8<br>.83}}}$ | 62.5${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+9<br>.17}}}$ |
| + GT RAG | 71.63${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>3.16}}}$ | 83.85${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>8.69}}}$ | 70.33${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>5.37}}}$ | 66.67${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>5.3}}}$ | 69.44${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>7.77}}}$ | 59.73${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>8.25}}}$ | 68.63${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>4.12}}}$ | 83.33${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>4.51}}}$ | 73.53${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>8.63}}}$ | 69.17${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>5.84}}}$ |
| Open-Source LVLMs |  |  |  |  |  |  |  |  |  |  |
| OpenFlamingo-v2-9B | 26.83 | 27.95 | 26.02 | 31.37 | 30.56 | 29.53 | 34.31 | 20.59 | 17.65 | 21.67 |
| + Retrieved RAG | 28.31${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>.48}}}$ | 29.5${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>.55}}}$ | 28.86${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.84}}}$ | 28.43${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.94}}}$ | 30.56${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.0}}}$ | 34.23${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.7}}}$ | 31.37${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.94}}}$ | 22.55${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>.96}}}$ | 21.57${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.92}}}$ | 22.5${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.83}}}$ |
| + GT RAG | 28.90${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.07}}}$ | 26.71${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>.24}}}$ | 33.74${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.72}}}$ | 28.43${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.94}}}$ | 33.33${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.77}}}$ | 35.57${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+6<br>.04}}}$ | 27.45${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-6<br>.86}}}$ | 27.45${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+6<br>.86}}}$ | 25.49${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.84}}}$ | 18.33${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-3<br>.34}}}$ |
| Idefics2-8B | 31.04 | 31.06 | 33.33 | 31.37 | 38.89 | 30.2 | 35.29 | 25.49 | 24.51 | 26.67 |
| + Retrieved RAG | 30.16${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-0<br>.88}}}$ | 29.81${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>.25}}}$ | 27.64${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-5<br>.69}}}$ | 29.41${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>.96}}}$ | 36.11${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.78}}}$ | 36.24${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+6<br>.04}}}$ | 28.43${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-6<br>.86}}}$ | 27.45${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>.96}}}$ | 32.35${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.84}}}$ | 25.83${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-0<br>.84}}}$ |
| + GT RAG | 37.03${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.99}}}$ | 36.34${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.28}}}$ | 35.37${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.04}}}$ | 38.24${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+6<br>.87}}}$ | 54.63${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>5.74}}}$ | 47.65${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>7.45}}}$ | 36.27${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.98}}}$ | 24.51${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-0<br>.98}}}$ | 34.31${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+9<br>.8}}}$ | 25.83${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-0<br>.84}}}$ |
| VILA1.5-13B | 43.68 | 45.34 | 41.87 | 52.94 | 48.15 | 50.34 | 38.24 | 21.57 | 30.39 | 57.5 |
| + Retrieved RAG | 35.48${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-8<br>.2}}}$ | 33.54${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>1.8}}}$ | 28.86${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>3.01}}}$ | 29.41${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>3.53}}}$ | 40.74${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-7<br>.41}}}$ | 47.65${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.69}}}$ | 33.33${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-4<br>.91}}}$ | 22.55${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.98}}}$ | 33.33${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.94}}}$ | 54.17${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-3<br>.33}}}$ |
| + GT RAG | 47.01${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.33}}}$ | 45.65${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.31}}}$ | 46.75${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.88}}}$ | 39.22${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>3.72}}}$ | 51.85${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.7}}}$ | 53.69${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.35}}}$ | 43.14${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.9}}}$ | 25.49${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.92}}}$ | 44.12${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>3.73}}}$ | 69.17${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>1.67}}}$ |
| Mantis-8B-clip-llama3 | 40.8 | 45.03 | 39.43 | 42.16 | 49.07 | 49.66 | 36.27 | 28.43 | 19.61 | 45.0 |
| + Retrieved RAG | 36.88${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-3<br>.92}}}$ | 36.65${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-8<br>.38}}}$ | 34.96${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-4<br>.47}}}$ | 42.16${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}0.<br>0}}}$ | 47.22${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>.85}}}$ | 50.34${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.68}}}$ | 33.33${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.94}}}$ | 18.63${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-9<br>.8}}}$ | 21.57${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>.96}}}$ | 42.5${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.5}}}$ |
| + GT RAG | 44.72${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.92}}}$ | 48.14${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.11}}}$ | 46.75${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.32}}}$ | 43.14${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.98}}}$ | 54.63${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.56}}}$ | 57.05${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.39}}}$ | 45.1${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+8<br>.83}}}$ | 19.61${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-8<br>.82}}}$ | 18.63${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-0<br>.98}}}$ | 51.67${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+6<br>.67}}}$ |
| Mantis-8B-siglip-llama3 | 45.01 | 46.89 | 45.12 | 57.84 | 58.33 | 45.64 | 45.1 | 26.47 | 29.41 | 45.0 |
| + Retrieved RAG | 39.62${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-5<br>.39}}}$ | 42.55${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-4<br>.34}}}$ | 35.37${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-9<br>.75}}}$ | 47.06${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>0.78}}}$ | 47.22${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>1.11}}}$ | 42.95${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.69}}}$ | 45.1${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}0.<br>0}}}$ | 23.53${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.94}}}$ | 29.41${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}0.<br>0}}}$ | 40.83${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-4<br>.17}}}$ |
| + GT RAG | 48.85${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.84}}}$ | 54.66${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.77}}}$ | 52.85${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.73}}}$ | 51.96${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-5<br>.88}}}$ | 58.33${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}0.<br>0}}}$ | 48.99${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.35}}}$ | 50.0${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.9}}}$ | 21.57${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-4<br>.9}}}$ | 33.33${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.92}}}$ | 49.17${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.17}}}$ |
| Deepseek-VL-7B-chat | 43.39 | 45.34 | 47.56 | 47.06 | 45.37 | 46.31 | 48.04 | 28.43 | 20.59 | 49.17 |
| + Retrieved RAG | 34.66${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-8<br>.73}}}$ | 33.54${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>1.8}}}$ | 32.11${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>5.45}}}$ | 33.33${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>3.73}}}$ | 37.04${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-8<br>.33}}}$ | 43.62${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.69}}}$ | 40.2${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-7<br>.84}}}$ | 20.59${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-7<br>.84}}}$ | 26.47${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.88}}}$ | 45.0${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-4<br>.17}}}$ |
| + GT RAG | 50.33${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+6<br>.94}}}$ | 54.04${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+8<br>.7}}}$ | 56.5${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+8<br>.94}}}$ | 50.98${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.92}}}$ | 56.48${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>1.11}}}$ | 57.05${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>0.74}}}$ | 50.0${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>.96}}}$ | 21.57${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-6<br>.86}}}$ | 23.53${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.94}}}$ | 60.83${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>1.66}}}$ |
| LLaVA-NeXT-Interleave-7B | 43.46 | 44.41 | 43.5 | 40.2 | 64.81 | 44.97 | 44.12 | 32.35 | 26.47 | 45.83 |
| + Retrieved RAG | 40.35${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-3<br>.11}}}$ | 40.06${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-4<br>.35}}}$ | 33.33${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>0.17}}}$ | 39.22${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-0<br>.98}}}$ | 56.48${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-8<br>.33}}}$ | 43.62${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>.35}}}$ | 44.12${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.0}}}$ | 27.45${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-4<br>.9}}}$ | 36.27${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+9<br>.8}}}$ | 49.17${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.34}}}$ |
| + GT RAG | 52.99${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+9<br>.53}}}$ | 54.97${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>0.56}}}$ | 54.88${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>1.38}}}$ | 49.02${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+8<br>.82}}}$ | 62.04${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.77}}}$ | 52.35${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.38}}}$ | 47.06${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.94}}}$ | 38.24${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.89}}}$ | 48.04${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>1.57}}}$ | 61.67${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>5.84}}}$ |
| mPLUG-Owl3-7B | 49.74 | 48.45 | 50.81 | 54.9 | 58.33 | 54.36 | 51.96 | 30.39 | 45.1 | 51.67 |
| + Retrieved RAG | 41.83${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-7<br>.91}}}$ | 40.06${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-8<br>.39}}}$ | 36.59${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>4.22}}}$ | 40.2${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>4.7}}}$ | 50.0${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-8<br>.33}}}$ | 50.34${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-4<br>.02}}}$ | 46.08${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-5<br>.88}}}$ | 20.59${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-9<br>.8}}}$ | 51.96${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+6<br>.86}}}$ | 46.67${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-5<br>.0}}}$ |
| + GT RAG | 56.32${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+6<br>.58}}}$ | 58.39${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+9<br>.94}}}$ | 58.94${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+8<br>.13}}}$ | 58.82${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.92}}}$ | 62.96${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.63}}}$ | 61.74${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.38}}}$ | 59.8${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.84}}}$ | 26.47${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-3<br>.92}}}$ | 50.0${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.9}}}$ | 58.33${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+6<br>.66}}}$ |
| LLaVA-OneVision | 53.29 | 58.39 | 56.1 | 49.02 | 60.19 | 47.65 | 53.92 | 37.25 | 52.94 | 51.67 |
| + Retrieved RAG | 50.11${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-3<br>.18}}}$ | 50.93${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-7<br>.46}}}$ | 48.78${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-7<br>.32}}}$ | 50.0${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.98}}}$ | 60.19${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.0}}}$ | 50.34${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.69}}}$ | 48.04${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-5<br>.88}}}$ | 33.33${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-3<br>.92}}}$ | 53.92${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.98}}}$ | 54.17${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.5}}}$ |
| + GT RAG | 58.98${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.69}}}$ | 62.42${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.03}}}$ | 63.82${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.72}}}$ | 59.8${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>0.78}}}$ | 66.67${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+6<br>.48}}}$ | 59.73${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>2.08}}}$ | 53.92${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.0}}}$ | 30.39${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-6<br>.86}}}$ | 57.84${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.9}}}$ | 60.83${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+9<br>.16}}}$ |
| Pixtral-12B | 47.97 | 52.48 | 45.53 | 58.82 | 50.0 | 51.68 | 49.02 | 38.24 | 42.16 | 37.5 |
| + Retrieved RAG | 45.97${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.0}}}$ | 51.86${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-0<br>.62}}}$ | 40.24${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-5<br>.29}}}$ | 53.92${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-4<br>.9}}}$ | 50.93${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.93}}}$ | 49.66${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.02}}}$ | 47.06${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>.96}}}$ | 19.61${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>8.63}}}$ | 47.06${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.9}}}$ | 46.67${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+9<br>.17}}}$ |
| + GT RAG | 59.28${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>1.31}}}$ | 63.04${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>0.56}}}$ | 63.41${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>7.88}}}$ | 65.69${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+6<br>.87}}}$ | 66.67${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>6.67}}}$ | 61.74${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>0.06}}}$ | 59.8${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>0.78}}}$ | 20.59${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>7.65}}}$ | 50.98${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+8<br>.82}}}$ | 65.0${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>7.5}}}$ |
| Proprietary LVLMs |  |  |  |  |  |  |  |  |  |  |
| GPT-4-Turbo | 57.21 | 64.29 | 59.35 | 54.9 | 56.48 | 62.42 | 47.06 | 41.18 | 59.8 | 50.0 |
| + Retrieved RAG | 58.95${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>.74}}}$ | 66.53${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.24}}}$ | 59.94${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.59}}}$ | 53.94${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-0<br>.96}}}$ | 66.74${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>0.26}}}$ | 59.73${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.69}}}$ | 49.06${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.0}}}$ | 38.27${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.91}}}$ | 62.78${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.98}}}$ | 58.83${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+8<br>.83}}}$ |
| + GT RAG | 62.85${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.64}}}$ | 68.94${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.65}}}$ | 69.51${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>0.16}}}$ | 60.78${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.88}}}$ | 67.59${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>1.11}}}$ | 63.33${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.91}}}$ | 51.96${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.9}}}$ | 38.24${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.94}}}$ | 59.8${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.0}}}$ | 62.5${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>2.5}}}$ |
| Gemini Pro | 61.71 | 68.01 | 69.92 | 73.53 | 71.3 | 70.47 | 42.16 | 39.22 | 53.92 | 40.83 |
| + Retrieved RAG | 65.93${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.22}}}$ | 73.29${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.28}}}$ | 69.92${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.0}}}$ | 69.61${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-3<br>.92}}}$ | 73.15${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>.85}}}$ | 75.84${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.37}}}$ | 49.02${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+6<br>.86}}}$ | 34.31${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-4<br>.91}}}$ | 56.86${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.94}}}$ | 65.0${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>4.17}}}$ |
| + GT RAG | 71.40${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+9<br>.69}}}$ | 77.33${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+9<br>.32}}}$ | 79.27${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+9<br>.35}}}$ | 78.43${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.9}}}$ | 75.93${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.63}}}$ | 78.52${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+8<br>.05}}}$ | 54.9${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>2.74}}}$ | 36.27${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.95}}}$ | 61.76${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.84}}}$ | 72.5${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>1.67}}}$ |
| Claude 3.5 Sonnet | 59.87 | 70.19 | 57.72 | 56.86 | 57.41 | 68.46 | 48.04 | 49.02 | 62.75 | 47.5 |
| + Retrieved RAG | 63.56${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.69}}}$ | 73.91${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.72}}}$ | 70.73${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>3.01}}}$ | 56.86${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.0}}}$ | 62.96${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.55}}}$ | 70.47${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.01}}}$ | 55.88${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.84}}}$ | 31.37${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>7.65}}}$ | 62.75${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.0}}}$ | 53.33${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.83}}}$ |
| + GT RAG | 71.10${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>1.23}}}$ | 78.88${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+8<br>.69}}}$ | 80.49${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>2.77}}}$ | 76.47${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>9.61}}}$ | 70.37${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>2.96}}}$ | 75.17${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+6<br>.71}}}$ | 67.65${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>9.61}}}$ | 36.27${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>2.75}}}$ | 65.69${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.94}}}$ | 59.17${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>1.67}}}$ |
| GPT-4o | 68.68 | 76.09 | 70.42 | 69.61 | 74.07 | 73.82 | 61.21 | 47.62 | 58.82 | 65.83 |
| + Retrieved RAG | 68.96${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.28}}}$ | 77.95${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>.86}}}$ | 78.86${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+8<br>.44}}}$ | 69.61${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.0}}}$ | 75.0${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.93}}}$ | 73.83${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.01}}}$ | 54.9${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.28}}}$ | 26.47${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-3<br>4.74}}}$ | 59.8${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.98}}}$ | 68.33${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.5}}}$ |
| + GT RAG | 74.50${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.82}}}$ | 84.47${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+8<br>.38}}}$ | 77.46${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.04}}}$ | 82.35${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>2.74}}}$ | 79.63${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.56}}}$ | 77.18${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.36}}}$ | 68.62${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.41}}}$ | 30.95${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>6.67}}}$ | 62.75${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.93}}}$ | 80.0${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>4.17}}}$ |

## 3 Experiments

In this section, we first introduce the experimental setup and evaluation metric ($\S~{}\ref{sec: experimental setup}$). Then, we present a comprehensive evaluation of 14 recent LVLMs ($\S~{}\ref{sec: main results}$). We demonstrate the importance of visual knowledge and discuss the critical findings revealed by the results from MRAG-Bench.

### 3.1 Experimental Setup

We evaluate
14 popular LVLMs on MRAG-Bench, including 4 proprietary models and 10 open-sourced models that can accept multi-image inputs:

- •
Proprietary models: GPT-4o (0513) (OpenAI, ), GPT-4-Turbo (OpenAI, ), Gemini Pro (Team et al., ), and Claude 3.5 Sonnet (Anthropic, ).
- •
Open-source models: OpenFlamingo (v2-9B) (Awadalla et al., ), Idefics (v2-8B) (Laurençon et al., ), VILA (v1.5-13B) (Lin et al., ), LLaVA-NeXT-Interleave-7B (Li et al., ), LLaVA-OneVision (Li et al., ), Mantis (clip-llama3, and siglip-llama3 versions; 8B) (Jiang et al., ), mPLUG-Owl3-7B (Ye et al., ), Deepseek-VL-7B-chat (Lu et al., ), and Pixtral-12B (Team, ).

#### Evaluation setup.

We follow standard MCQA evaluation setup and employ accuracy score as our metric. We adopt default generation hyper-parameters selected by each model. Following Lu et al. (), we employ GPT-3.5-turbo to extract the multiple choice answer in rare cases where our pre-defined automatic extraction rules failed. We refer the readers to Appendix A.1 and B for more details on evaluation prompts for both without multimodal RAG and with multimodal RAG scenarios, answer extraction prompt and human performance evaluation protocol.

### 3.2 Main Results

As shown in Table 3, the average performance of the most advanced LVLMs is not better than 68.68% without multimodal RAG knowlege, and 74.5% with ground-truth knowledge, which demonstrates MRAG-Bench to be a challenging benchmark. The mean accuracies of open-source LVLMs are between 26.83% and 53.29% without RAG knowledge and between 28.90% and 59.28% with ground-truth knowledge, which fall behind from advanced proprietary LVLMs. Notably, MRAG-Bench proves to be knowledge-intensive as average humans achieved 38.47% without RAG knowledge, while proprietary LVLMs generally perform well, suggesting that their extensive training data equips them with a broader knowledge base.
However, when provided with either retrieved or ground-truth knowledge, humans achieve the most significant improvements of 22.91% and 33.16%, respectively. This underscore the need of LVLMs to better utilize visually augmented information like humans.

#### Can LVLMs utilize retrieved and ground-truth image knowledge well?

As illustrated in Table 3, all models demonstrate improvement when ground-truth image RAG knowledge is provided. Among the open-source models, they achieve improvements ranging from 2.07% to 11.31% when using ground-truth RAG knowledge, whereas 5.64% to 9.69% improvements are observed from proprietary LVLMs. Interestingly, when images from the multimodal retriever is provided, almost all open-source LVLMs on average demonstrate a declined performance while proprietary models can still gain improvement. This indicate proprietary models possess emerging abilities to distinguish between good and bad image knowledge sources, which is a critical skill in the multimodal RAG domain. We further conducted a qualitative analysis to investigate the reasons behind this, as detailed in the following paragraphs.

#### Fine-grained results.

We also report fine-grained scores across 9 scenarios on MRAG-Bench in Table 3. Remarkably, GPT-4o surpasses most other baselines in various categories, with exceptions in problems related to partial, incomplete and biological scenarios. Notably, GPT-4o outperforms human performance on all perspective aspect as well as on temporal and deformation scenarios within the transformative aspect. We conjecture that incomplete and biological scenarios are less likely to be included in the training knowledge. Interestingly, all models exhibit a decline in performance on incomplete scenarios, with only a few exceptions, while humans find this task relatively easy, achieving 58.82% and 83.33% scores with ground-truth knowledge. This further highlights the importance of leveraging retrieved visually augmented knowledge to address questions that do not directly incentivize knowledge stored in the models’ memories.

#### Why can proprietary models better utilize retrieved images?

We conduct an error analysis on an open-source model (LLaVA-Next-Interleave) and a proprietary model (Gemini Pro). For a fair comparison, we filtered results where LLaVA-Next-Interleave answered correctly without or with GT knowledge but was misled to wrong answer with retrieved examples. One example is illustrated in Figure 4, the retrieved images contain two correct examples and three false examples. While Gemini Pro is able to utilize all retrieved images, LLaVA-Next-Interleave leverages bad examples and makes wrong prediction. This example helps explain why do almost all open-source models have lower performance with retrieved knowledge.

## 4 Analysis

In this section, we conduct quantitative analysis addressing three important questions: 1) To what extent can LVLMs benefit more from visual knowledge than from textual knowledge on MRAG-Bench? ($\S$ 4.1) 2) How does the performance of LVLMs vary with examples retrieved from different retrievers? ($\S$ 4.2) 3) How many ground-truth visual knowledge examples are required for LVLMs to continue benefiting? ($\S$ 4.3)

### 4.1 How much can visual knowledge benefit more than textual knowledge?

Figure: Figure 4: Qualitative Example of Proprietary model (Gemini Pro) identifies and utilizes correct examples, while open-source model (LLaVA-Next-Interleave) is misled by noisy retrieved information, resulting in incorrect answers.
Refer to caption: x4.png

**Table 4: LVLMs performance on MRAG-Bench with textual knowledge v.s visual knowledge. Both the open-source and proprietary model benefit more from image knowledge.**
| Model | Overall | Perspective | Transformative | Others |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Angle | Partial | Scope | Occlusion | Temporal | Deformation | Incomplete | Biological |  |  |  |
| LLaVA-NeXT-Interleave-7B | 43.46 | 44.41 | 43.5 | 40.2 | 64.81 | 44.97 | 44.12 | 32.35 | 26.47 | 45.83 |
| + Retrieved Text RAG | 37.99${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-5<br>.47}}}$ | 37.58${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-6<br>.83}}}$ | 34.96${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-8<br>.54}}}$ | 33.33${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-6<br>.87}}}$ | 50.0${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>4.81}}}$ | 41.61${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-3<br>.36}}}$ | 35.29${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-8<br>.83}}}$ | 30.39${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>.96}}}$ | 27.45${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.98}}}$ | 51.67${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.84}}}$ |
| + Retrieved Image RAG | 40.35${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-3<br>.11}}}$ | 40.06${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-4<br>.35}}}$ | 33.33${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>0.17}}}$ | 39.22${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-0<br>.98}}}$ | 56.48${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-8<br>.33}}}$ | 43.62${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>.35}}}$ | 44.12${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.0}}}$ | 27.45${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-4<br>.9}}}$ | 36.27${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+9<br>.8}}}$ | 49.17${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.34}}}$ |
| + GT Text RAG | 41.09${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.37}}}$ | 41.93${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.48}}}$ | 39.02${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-4<br>.48}}}$ | 38.24${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>.96}}}$ | 56.48${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-8<br>.33}}}$ | 44.97${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.0}}}$ | 43.14${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-0<br>.98}}}$ | 30.39${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>.96}}}$ | 21.57${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-4<br>.9}}}$ | 50.83${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.0}}}$ |
| + GT Image RAG | 52.99${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+9<br>.53}}}$ | 54.97${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>0.56}}}$ | 54.88${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>1.38}}}$ | 49.02${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+8<br>.82}}}$ | 62.04${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.77}}}$ | 52.35${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.38}}}$ | 47.06${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.94}}}$ | 38.24${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.89}}}$ | 48.04${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>1.57}}}$ | 61.67${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>5.84}}}$ |
| + GT Image & Text RAG | 47.82${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.36}}}$ | 47.83${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.42}}}$ | 48.78${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.28}}}$ | 44.12${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.92}}}$ | 58.33${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-6<br>.48}}}$ | 49.66${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.69}}}$ | 48.04${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.92}}}$ | 30.39${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>.96}}}$ | 35.29${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+8<br>.82}}}$ | 62.5${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>6.67}}}$ |
| GPT-4-Turbo | 57.21 | 64.29 | 59.35 | 54.9 | 56.48 | 62.42 | 47.06 | 41.18 | 59.8 | 50.0 |
| + Retrieved Text RAG | 56.61${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-0<br>.6}}}$ | 61.8${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.49}}}$ | 59.35${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.0}}}$ | 59.8${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.9}}}$ | 58.33${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>.85}}}$ | 59.06${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-3<br>.36}}}$ | 49.02${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>.96}}}$ | 33.33${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-7<br>.85}}}$ | 60.78${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.98}}}$ | 52.5${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.5}}}$ |
| + Retrieved Image RAG | 58.95${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>.74}}}$ | 66.53${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.24}}}$ | 59.94${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.59}}}$ | 53.94${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-0<br>.96}}}$ | 66.74${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>0.26}}}$ | 59.73${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.69}}}$ | 49.06${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.0}}}$ | 38.27${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.91}}}$ | 62.78${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+2<br>.98}}}$ | 58.83${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+8<br>.83}}}$ |
| + GT Text RAG | 58.98${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>.77}}}$ | 68.01${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+3<br>.72}}}$ | 63.41${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.06}}}$ | 65.69${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>0.79}}}$ | 63.89${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.41}}}$ | 59.73${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.69}}}$ | 38.24${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-8<br>.82}}}$ | 37.25${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-3<br>.93}}}$ | 58.82${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-0<br>.98}}}$ | 50.83${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.83}}}$ |
| + GT Image RAG | 62.85${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.64}}}$ | 68.94${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.65}}}$ | 69.51${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>0.16}}}$ | 60.78${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+5<br>.88}}}$ | 67.59${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>1.11}}}$ | 63.33${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.91}}}$ | 51.96${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+4<br>.9}}}$ | 38.24${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-2<br>.94}}}$ | 59.8${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.0}}}$ | 62.5${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>2.5}}}$ |
| + GT Image & Text RAG | 65.11${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.9}}}$ | 72.05${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.76}}}$ | 72.76${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>3.41}}}$ | 67.65${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>2.75}}}$ | 70.37${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+1<br>3.89}}}$ | 71.81${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+9<br>.39}}}$ | 46.08${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-0<br>.98}}}$ | 39.22${}_{\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-1<br>.96}}}$ | 60.78${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+0<br>.98}}}$ | 57.5${}_{\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+7<br>.5}}}$ |

We used the Wikipedia corpus as of 2023/07/01 as our text knowledge corpus(^1^11https://www.kaggle.com/datasets/jjinho/wikipedia-20230701). To ensure a fair comparison, we employed the same multimodal retriever (CLIP) for retrieving either text or image knowledge. The top-5 ranked documents or images are used for augmenting the input.
We selected one open-source (LLaVA-Next-Interleave) and one proprietary (GPT-4-Turbo) LVLM to examine their preference for textual knowledge versus image knowledge on MRAG-Bench. As shown in Table 4, when both models utilized retrieved knowledge, LLaVA-Next-Interleave demonstrated a 2.36% improvement with image knowledge over text knowledge, while GPT-4-Turbo showed a 2.34% improvement. When using GT knowledge, LLaVA-Next-Interleave exhibited an 11.09% improvement with image knowledge over text knowledge, compared to a 3.87% improvement for GPT-4-Turbo.
Interestingly, when both GT image and text knowledge are provided, LLaVA-Next-Interleave indicated less improvement than with GT image alone whereas GPT-4-Turbo further pushed its performance.
All these results demonstrate that retrieving visual knowledge is more helpful than retrieving text on MRAG-Bench.

### 4.2 How does retriever performance affect LVLMs?

We picked four recent best-performing multimodal retrievers, including CLIP (Radford et al., ), MagicLens (Zhang et al., ), E5-V (Jiang et al., ), VISTA (Zhou et al., ) and evaluated their performance (Recall@5). The detailed retriever performance can be found at Table 6 in Appendix C. We selected LLaVA-Next-Interleave as the end model to assess its performance. As shown in Figure 5, when retrievers achieve higher Recall@5 scores (i.e., better retrieved examples), the LVLM’s accuracy tends to improve, demonstrating a strong 95% positive correlation. Interestingly, despite similar Recall@5 scores from CLIP and VISTA retrievers, LLaVA-Next-Interleave demonstrated a 2.07% gap in overall accuracy. We conjecture that the order of the correctly retrieved examples may also impact the model’s final performance. The sensitivity to the order of retrieved examples is a common issue that persists across various models.
Although this phenomenon, known as position bias, has been examined in text-based RAG (Lu et al., ; Wang et al., ), its impact on visual RAG remains unexplored, presenting a promising direction for future research.

### 4.3 How many ground-truth image examples are needed?

For simplicity, all our experiments used five retrieved or ground-truth image examples. However, it is worth exploring how many examples LVLMs can effectively leverage. As noted in $\S$ 2.3, the perspective aspect of our benchmark includes an average of 20.4 ground-truth examples. To investigate further, we perform an analysis focusing on the perspective and others aspects, covering a total of 892 questions. As shown in Figure 5, we evaluated LLaVA-Next-Interleave using 1, 2, 3, 5, 10, 20 GT examples, averaging the results across three random seeds for sampling the GT examples. LLaVA-Next-Interleave saw the greatest improvement of 5.64% with just one GT example. Performance continued to increase steadily, reaching a peak at 10 GT examples, which was 0.29% higher than with 20 GT examples. One possible explanation could be LLaVA-Next-Interleave may not able to better leverage visually augmented knowledge in long context scenarios. Moreover, the complexity of questions affects the number of images needed too, one ground-truth example sometimes help the model the most on MRAG-Bench. We encourage the research on adaptatively deciding the number of necessary images based on the complexity of questions.

Figure: Figure 5: Left: LLaVA-Next-Interleave results with 4 different multimodal retrievers. Its performance using retrieved images correlates 95% with retriever’s Recall@5 scores. Right: Average results of three random seed runs. Improve the number of ground-truth RAG examples shows steady increase of model’s performance, reaches the maximum with 10 examples.
Refer to caption: extracted/5917422/files/analysis_new.png

## 5 Related Work

We overview three lines of related work: 1) multimodal retrieval-augmented generation benchmarks ($\S$ 5.1), 2) large vision language models ($\S$ 5.2), and 3) retrieval-augmented mutlimodal large language models ($\S$ 5.3).

### 5.1 Multimodal Retrieval-Augmented Generation Benchmarks

A number of recent benchmarks have been developed to comprehensively assess the capabilities of LVLMs (Lu et al., ; ; Li et al., ; Liu et al., ; Lu et al., ; Yue et al., ; Zhang et al., ; Ying et al., ). There are several benchmarks well-suited for evaluating retrieval-augmented LVLMs. For instance, OK-VQA (Marino et al., ) and A-OKVQA (Schwenk et al., ) both focus on scenarios where external textual knowledge is required to answer visual questions. More recent works (Chang et al., ; Chen et al., ; Mensink et al., ) have curated large-scale knowledge bases to evaluate models on knowledge-intensive and information-seeking visual questions. In contrast, MRAG-Bench focus on scenarios where retrieving visual information is more helpful than retrieving text.

### 5.2 Large Vision Language Models

Large Vision Language Models (LVLMs) (Liu et al., ; Zhu et al., ; Dai et al., ; Yin et al., ; Hu et al., ) have showcased promising results on a wide variety of vision-language tasks. Many works, such as Flamingo (Alayrac et al., ), Emu (Sun et al., ), Idefics (Laurençon et al., ), and VILA (Lin et al., ), have demonstrated in-context learning capabilities, where multiple image examples can be leveraged to improve text generation. Recent works start training LVLMs with interleaved image-text corpora, such as MMC4 (Zhu et al., ) and OBELICS (Laurençon et al., ), for pretraining, as well as high-quality instruction tuning in models like Mantis-Instruct (Jiang et al., ), LLaVA-Next-Interleave (Li et al., ), and LLaVA-OneVision (Li et al., ), enabling models to process and understand information from multiple images. Naturally, evaluating the ability of LVLMs to effectively leverage visually augmented knowledge becomes an important task, which is the primary focus of MRAG-Bench.

### 5.3 Retrieval-Augmented Multimodal Large Language Models

Retrieval-augmented generation (RAG) has emerged as a potential solution to overcome limitations in language models by incorporating external knowledge retrieval during the generation process (Lewis et al., ; Shi et al., ). Reasonably, several works have focused on using multimodal knowledge to enhance the generation capabilities of Large Language Models (LLMs) (Yasunaga et al., ; Chen et al., ; Zhao et al., ; Cui et al., ). Recently, more works (Caffagni et al., ; Xuan et al., ; Du et al., ) has incorporated external knowledge to improve LVLMs’ general generation abilities and the comprehensiveness of their reasoning. Although some works (Chen et al., ; Yuan et al., ) have proposed directly using image information from the web, a systematic vision-centric benchmark to evaluate LVLMs’ abilities to leverage visually augmented knowledge is lacking, which is the focus of our work.

## 6 Conclusion

In this work, we introduce MRAG-Bench, a benchmark specifically designed for vision-centric
evaluation for retrieval-augmented multimodal models. Our evaluation of 14 LVLMs highlights that visually augmented knowledge brings more improvements on MRAG-Bench compared to textual knowledge. Moreover, the top-performing model, GPT-4o, struggles to effectively utilize the retrieved knowledge, achieving only a 5.82% improvement when augmented with relevant information, compared to a 33.16% improvement demonstrated by human participants. We further conduct extensive analysis and propose several promising directions for future research. Our findings underscore the significance of MRAG-Bench in motivating the community to develop LVLMs that better utilize retrieved visual knowledge.

## Appendix A MRAG-Bench Details

### A.1 Dataset Curation Details

#### Dataset collection of transformative aspect

We chose to manually scrape images from the web based on the definitions of the transformative aspect. To construct the image corpus, we employed Bing Image Search for each of the image object keyword predefined by us. We filtered some of the search results where the image objects do not have a clear pair of query image and ground-truth image example, around 74% keyword names were kept during this process. Here we listed all the keywords that are already filtered and used for search of query image except in biological scenario, it’s for search of ground-truth image example. Each search keyword is composed of an “image object” and a “condition”. For example, “A young kitten image of Himalayan Cat”, here Himalayan Cat is the image object and a young kitten is the condition. For each of keyword listed below, we searched again for its ground-truth examples (except for biological scenario, it’s for query images), in which only “image object” is kept and “conditon” is removed. All searched results are further picked and downloaded by humans to ensure quality. Here is a list of the filtered keywords for transformative aspect:

Transformative: Temporal
- A young kitten image of Himalayan Cat
- A young kitten image of Chartreux
- A young kitten image of Burmese
- A young kitten image of Turkish Van
- A young kitten image of American Shorthair
- A young kitten image of British Shorthair
- A young kitten image of Maine Coon
- A young kitten image of Burma (Myanmar)
- A young kitten image of Selkirk Rex
- A young kitten image of Siberian
- A young kitten image of Persian
- A young kitten image of Manx
- A young kitten image of Ocicat
- A young kitten image of Russian Blue
- A young kitten image of Bengal Cat
- A young kitten image of Devon Rex
- A young kitten image of American Bobtail
- A young kitten image of Balinese
- A young kitten image of LaPerm
- A young kitten image of Egyptian Mau
- A young kitten image of Japanese Bobtail
- A young kitten image of Ragdoll
- A young kitten image of Abyssinian
- A young kitten image of American Wirehair
- A young kitten image of Oriental Shorthair
- A young kitten image of Cornish Rex
- A young kitten image of Kurilian Bobtail
- A young kitten image of Singapura Cat
- A young kitten image of Birman
- A young kitten image of Burmilla
- A young kitten image of Korat
- A young kitten image of Tonkinese
- A young kitten image of Somali Cat
- A young kitten image of Norwegian Forest Cat
- A young kitten image of Turkish Angora
- A young kitten image of Siamese
- A picture of Sainte-Chapelle under construction
- A picture of Washington Monument under construction
- A picture of Hearst Castle under construction
- A picture of Time Square under construction
- A picture of Wrigley Building under construction
- A picture of Eiffel Tower under construction
- A picture of The Arc de Triomphe under construction
- A picture of Golden Gate Bridge under construction
- A picture of White House under construction
- A picture of Palace of Versailles under construction
- A picture of Opéra Garnier under construction
- A picture of San Simeon under construction
- A picture of The Louvre under construction
- A picture of Cathédrale Notre-Dame de Paris under construction
- A picture of Sacré-Cœur Basilica under construction
- A picture of Brooklyn Bridge under construction
- A picture of Panthéon under construction
- A picture of Capitol Building under construction
- A picture of Independence Hall under construction
- A picture of Mont Saint-Michel under construction
- A picture of St Patrick’s Cathedral under construction
- A picture of Space Needle under construction
- A picture of Château de Chambord under construction
- A picture of Versailles under construction
Transformative: Deformation
- An image of Toyota Camry damaged
- An image of Ford F-150 damaged
- An image of Ferrari 458 damaged
- An image of Audi Q5 damaged
- An image of Lamborghini LP640 damaged
- An image of McLaren 675LT damaged
- An image of Mercedes SLC damaged
- An image of Lamborghini Aventador damaged
- An image of Lamborghini LP570 damaged
- An image of Porsche 911 GT3 RS damaged
- An image of Audi A6 damaged
- An image of Audi A4 damaged
- An image of Lamborghini Aventador SV damaged
- An image of GMC Sierra 2500 HD damaged
- An image of Infiniti G37 damaged
- An image of GMC Yukon damaged
- An image of Honda Accord damaged
- An image of Infiniti FX35 damaged
- An image of Tesla Model 3 damaged
- An image of Acura RDX 2020 damaged
- An image of BMW 7 Series damaged
- An image of Audi A5 Sportback damaged
- An image of Hyundai IX35 damaged
- An image of Cadillac XTS damaged
- An image of BMW M3 damaged
- An image of Acura MDX damaged
- An image of Audi A3 damaged
- An image of BMW X3 damaged
- An image of Porsche Boxster damaged
- An image of Mercedes CLA45 AMG damaged
- An image of Jaguar XJ damaged
Transformative: Incomplete
- MacBook Keyboard missing keys
- Windows Keyboard missing keys
- Laptop Keyboards (Generic) missing keys
- Mechanical Keyboard missing keys
- Ergonomic Keyboard missing keys
- Compact Keyboard missing keys
- Gaming Keyboard missing keys
- Chiclet Keyboard missing keys
- Tenkeyless (TKL) Keyboard missing keys
- Virtual Keyboard (On-screen) missing keys
- Numeric Keypad missing keys
- ISO Keyboard Layout missing keys
- ANSI Keyboard Layout missing keys
- Ortholinear Keyboard missing keys
- Bluetooth/Wireless Keyboard missing keys
Transformative: Biological
- An image of Lime after oxidation
- An image of breadfruit after oxidation
- An image of dragonfruit after oxidation
- An image of starfruit after oxidation
- An image of Raspberry after oxidation
- An image of Zucchini after oxidation
- An image of Pear after oxidation
- An image of passionfruit after oxidation
- An image of Blackberry after oxidation
- An image of durian after oxidation
- An image of persimmon after oxidation
- An image of Apple after oxidation
- An image of bell pepper after oxidation
- An image of olive after oxidation
- An image of Mango after oxidation
- An image of nectarine after oxidation
- An image of tomato after oxidation
- An image of quince after oxidation
- An image of coconut after oxidation
- An image of soursop after oxidation
- An image of Kiwi after oxidation
- An image of cucumber after oxidation
- An image of apricot after oxidation
- An image of Honeydew after oxidation
- An image of Peach after oxidation
- An image of pomegranate after oxidation
- An image of carrot after oxidation
- An image of fig after oxidation
- An image of Papaya after oxidation
- An image of Blueberry after oxidation
- An image of Banana after oxidation
- An image of jackfruit after oxidation
- An image of Lemon after oxidation
- An image of tamarind after oxidation
- An image of lychee after oxidation
- An image of Pineapple after oxidation
- An image of Cantaloupe after oxidation
- An image of Orange after oxidation
- An image of Rambutan after oxidation
- An image of guava after oxidation
- An image of sweet potato after oxidation
- An image of Plum after oxidation
- An image of Avocado after oxidation
- An image of Watermelon after oxidation
- An image of potato after oxidation
- An image of Grapefruit after oxidation
- An image of Grapes after oxidation
- An image of pumpkin after oxidation
- An image of Cherry after oxidation
- An image of Strawberry after oxidation
- An image of custard apple after oxidation

#### Quality control

We employ two types of quality control throughout the annotation process: an automatic check with predefined rules and a manual examination of each instance.
The automatic check verifies correct MCQA format in which each question should only have one correct answer, metadata values, assesses image validity (checking the accessibility of each image) and filters out redundant images in the corpus (images that are repetitively downloaded).
The manual examination
is conducted by two experts in this field, who checked the correspondence between query images and
ground-truth image examples, and filtered or revised ambiguous questions and uncorrelated query
image and ground-truth images.

### A.2 Human Evaluation Protocol

Three human annotators in domain conducted the human evaluation. The interface for human evaluation without RAG knowledge and with RAG knowledge are shown in Figure 6 and Figure 7.

Figure: Figure 6: Human evaluation interface without RAG examples
Refer to caption: extracted/5917422/files/human_eval_norag.png

Figure: Figure 7: Human evaluation interface with ground-truth RAG examples
Refer to caption: extracted/5917422/files/human_eval_rag.png

## Appendix B Experiment Setting Details

### B.1 Model Prompts

Following Lu et al. () and Liu et al. () our prompt consists of four parts, the instruction, question, options, and a prefix of the answer. For images, we insert them into the text to form a coherent prompt as the image placeholder ({Image}) indicated below. The complete prompt is as follows:

### B.2 Evaluation Tool

Following Lu et al. (), we first use a rule-based automatic tool to extract the exact answer. First, the tool detects if a valid option index appears in the model output. If no direct answer is found, the tool matches the output to the content of each option. If there is still no match, we employ GPT-3.5-turbo to automatically extract the answer following our prompts in Table 5. If GPT-3.5-turbo finds there is still no match, we will randomly select an option as the answer.

**Table 5: Prompt template to extract multiple choice answer from model’s response. {In-context examples} are in-context examples.**
| Prompt |
| --- |
| Please read the following example. Then extract the multiple choice letter with the answer corresponding to the choice list from the model response and type it at the end of the prompt. You should only output either A, B, C, or D. |
| {In-context examples} |
| Question: {QUESTION} |
| Choice List: (A) {OPTION_A} (B) {OPTION_B} (C) {OPTION_C} (D) {OPTION_D} |
| Model Response: {Response}<br>Extracted answer: |

## Appendix C More Results

We present the Recall@5 scores per each scenarios on 4 multimodal retreivers as shown in Table 6 and LLaVA-Next-Interleave’s accuracy score affected by these retrievers in Table 7.

**Table 6: Recall@5 scores with 4 retriever models on MRAG-Bench.**
| Model | Overall | Perspective | Transformative | Others |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Angle | Partial | Scope | Occlusion | Temporal | Deformation | Incomplete | Biological |  |  |  |
| MagicLens | 37.03 | 41.61 | 33.33 | 36.27 | 36.11 | 12.75 | 10.78 | 79.41 | 29.41 | 56.67 |
| E5-V | 54.92 | 49.69 | 48.78 | 61.76 | 66.67 | 38.93 | 22.55 | 73.53 | 71.57 | 82.50 |
| VISTA | 59.65 | 66.15 | 67.48 | 64.71 | 63.89 | 38.26 | 8.82 | 33.33 | 94.12 | 80.83 |
| CLIP | 60.46 | 70.19 | 54.47 | 71.57 | 73.15 | 44.30 | 31.37 | 67.65 | 40.2 | 81.67 |

**Table 7: LLaVA-Next-Interleave accuracy scores on MRAG-Bench with 4 different retrievers.**
| Model | Overall | Perspective | Transformative | Others |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Angle | Partial | Scope | Occlusion | Temporal | Deformation | Incomplete | Biological |  |  |  |
| MagicLens | 35.18 | 34.78 | 29.67 | 30.39 | 34.26 | 40.94 | 36.27 | 27.45 | 49.02 | 39.17 |
| E5-V | 40.06 | 38.82 | 39.84 | 41.18 | 46.3 | 38.93 | 41.18 | 27.45 | 48.04 | 41.67 |
| VISTA | 42.42 | 40.37 | 35.77 | 40.2 | 52.78 | 45.64 | 42.16 | 36.27 | 50.98 | 48.33 |
| CLIP | 40.35 | 40.06 | 33.33 | 39.22 | 56.48 | 43.62 | 44.12 | 27.45 | 36.27 | 49.17 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Could you elaborate on your motivation for creating this benchmark? Please include examples of real-world applications where this benchmark could offer substantial value.

2. Using this benchmark in real-world applications could be challenging, as it would require a comprehensive image repository encompassing a wide range of objects globally. This need introduces various limitations, such as legal and computational restrictions and the necessity for thorough curation to prevent biased or intrusive content. Could you discuss potential strategies for making this benchmark feasible in real-world scenarios, as well as approaches to mitigate these limitations?

3. Increasing context (whether visual or textual) means that large language models (LLMs) would need to process longer input sequences, which would raise inference times. Since images are typically represented as lengthy sequences, adding more images could substantially impact both inference time and associated costs, especially in the case of proprietary models, which gain the most from visual augmentation. Could you provide a detailed analysis of the trade-offs between performance gains, increased costs, and extended inference times?
soundness: 2
strengths: The paper is well-structured and accessible, presenting a benchmark for evaluating and analyzing the relevance of visually augmented samples across various scenarios. For example, it considers cases where the main object in an image is partially occluded or where the image depicts an outdated or modified version of the object. The authors conduct extensive experiments with both proprietary and open-source methods, demonstrating that visual augmentation offers greater utility than textual information for large vision-language models (LVLMs).
summary: The paper presents a Multimodal Retrieval-Augmented Generation Benchmark (MRAG-Bench) to analyze and identify scenarios where visually augmented knowledge is better than textual one. MRAG-Bench comprises 16,130 images and 1,353 human-annotated multiple-choice questions, covering nine distinct scenarios. The authors analyze the performance of 10 open-source and 4 proprietary large vision-language models (LVLMs). Their results show that all the LVLMs exhibit performance improvement when augmented with visual data rather than textual. Moreover, the proprietary models can better leverage the retrieved visual information.
weaknesses: While the paper is easy to follow, the primary motivation behind the task remains unclear. In a real-world scenario, how would this pipeline function operationally? Given the findings that visually augmented samples offer significant value, an extensive repository of curated and labeled images for retrieval would be essential. However, this may be impractical due to legal and computational restrictions. Additionally, I would be interested in analyzing how the incorporation of augmented images affects response time and costs.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How to conduct the process of human evaluation? For example, how many annotators were involved in evaluating each question, and what measures were taken to ensure consistency and accuracy in their assessments?
2. In Table 3, there are two human evaluation settings. How do you ensure that human annotators do not rely on their own knowledge when answering questions, and what steps have you taken to mitigate any potential biases in the human evaluation process
3. Whether retrieving additional visual information also enhance performance in text-intensive scenarios? 
4. The paper does not report the performance of GPT-o1 on MRAG-BENCH. How does GPT-o1 perform on this benchmark?
5. In Section 4.3, the paper shows that further increasing the number of retrieved images does not improve performance. Could the authors explain the reasons behind this effect, and what insights does this provide for future research on retrieval-augmented multimodal models?
6. Will you release this benchmark?
soundness: 3
strengths: 1. The paper is well-written and easy to follow.
2. The paper offers a different visual-centric perspective and addresses a gap in the current text-centric benchmarks.
3. The comprehensive evaluation of multiple LVLMs provides valuable insights, such as how visual knowledge benefits LVLMs over textual knowledge, and the examination of retriever performance and its impact on LVLM accuracy.
summary: This paper introduces a benchmark (MRAG-BENCH) designed to evaluate the performance of large vision-language models (LVLMs) in scenarios where visual information retrieval is more beneficial than textual data retrieval. First, this paper establishes the first benchmark focusing on visual information superiority over textual data. Second, this paper conducts an extensive analysis of LVLMs, and makes detailed analyses. Third, this paper points out a way that LVLMs could be evaluated and improved by emphasizing the importance of visual information in retrieval-augmented multimodal tasks.
weaknesses: 1. The process of selecting ground-truth images and query images may introduce bias if not carefully controlled. The paper could be strengthened by discussing potential biases in the data collection process and how they were mitigated.
2. The paper focuses on a specific set of scenarios, which may not scale well to other types of visual queries or more complex tasks. For example, what if there are many objects in the question? 
3. The paper could further discuss the implications of the findings, such as how they might influence the development of future LVLMs.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - What is the meaning of "Unique number" in Table 2?

- How were the nine scenarios in MRAG-BENCH selected?

- Regarding the "Others" example in Figure 3, why is option A marked as the correct answer? This raises questions about the rule for obtaining question-answer pairs. How is the correct answer determined for each question?

- Could the authors clarify how CLIP is used for text retrieval? Is it a direct approach based on finding the sentence with the highest similarity to the current question, or does it involve a method similar to INFOSEEK [1], where visual entity recognition is performed first, followed by retrieval?

- Concerning Table 3:

  - If human performance on this task is low, how can the quality of GT images be assured during data collection?

  - How is "Human performance" in Table 3 measured?

  - Most open-source LVLMs exhibit performance declines when using Retrieved RAG, suggesting that retrieval effectiveness is crucial to answer quality. If retrieval consistently underperforms, is the entire process reasonable and valid?

- Concerning Table 4:

  - How were the five comparison methods implemented?

  - What is the meaning of GT Text RAG, and how is GT Text collected? Additionally, it seems counterintuitive that LLaVA-Next-Interleave performs worse with GT Text RAG. If adding GT Text RAG degrades model performance, does this suggest that the quality of GT Text might be low?

  - Why does the performance improvement for LLaVA-Next-Interleave, when provided with both GT images and textual knowledge, fall short compared to using GT images alone?

- There appears to be a minor error in Section 4.1, where it states, "LLaVA-Next-Interleave exhibited an 11.09% improvement with image knowledge over text knowledge." Shouldn't this be 11.90%?
soundness: 2
strengths: - The authors conduct an investigation into the capability of models to retrieve and utilize external visual knowledge for question answering, which is a crucial and direct exploration within the LVLM field.

- The paper introduces a comprehensive multimodal retrieval-augmented generation benchmark that provides a foundation for the community to further explore and advance in this area.

- Figure 1 is well-designed, with well-chosen examples to illustrate the benchmark's scenarios and model evaluations effectively.
summary: This paper introduces MRAG-BENCH, a multimodal retrieval-augmented generation benchmark comprising 16,130 images and 1,353 human-annotated multiple-choice questions across 9 distinct scenarios. The authors evaluate the performance of 10 open-source and 4 proprietary large vision-language models (LVLMs) on this benchmark.
weaknesses: - The data collection process is unclear. Adding a figure to illustrate the entire process could clarify the methodology, especially regarding the acquisition of ground truth (GT) text and the source of question-answer pairs.

- The experimental setup lacks sufficient detail. The authors should provide a complete subsection dedicated to describing the experimental setup, placed within Section 3, instead of scattering brief descriptions across various sections. This current layout diminishes the reliability of the experimental conclusions. For example, in Table 3 (Retrieved RAG), it’s unclear which model was used as the retriever, what the retriever’s input and output were, etc. Additional concerns regarding the experimental setup are outlined in the Questions section below.

- The analysis of the question, "How much does visual knowledge provide additional benefits over textual knowledge?" is insufficiently supported. Additional experimental analysis would strengthen the authors’ conclusions.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see weakness.
soundness: 3
strengths: ### Originality
- **New Benchmark**: The introduction of MRAG-BENCH is a significant contribution, focusing on vision-centric retrieval-augmented generation, which is less explored compared to text-centric benchmarks. The paper identifies and categorizes scenarios where visual knowledge is more beneficial than textual knowledge, such as perspective and transformative changes, which is a fresh approach in the field.

### Quality
- **Comprehensive Evaluation**: The paper evaluates 10 open-source and 4 proprietary LVLMs using a large dataset of 16,130 images and 1,353 questions, providing robust and extensive analysis.
- **Detailed Analysis**: The results include fine-grained performance metrics across different scenarios, highlighting the strengths and weaknesses of various models.

### Clarity
- **Well-Structured**: The paper is well-organized, with clear sections detailing the benchmark's design, data collection, and evaluation methodology.
- **Illustrative Examples**: Figures and tables, such as the scenario distributions and qualitative examples, effectively illustrate the concepts and findings, making the paper easy to follow.

Overall, the paper makes a substantial contribution to the field of multimodal models by introducing a novel benchmark that highlights the importance of visual knowledge, backed by thorough evaluation and clear presentation.
summary: - **Introduction of MRAG-BENCH**: The paper introduces **MRAG-BENCH**, a benchmark designed for evaluating retrieval-augmented multimodal models, focusing on scenarios where visual information is more beneficial than textual data. MRAG-BENCH consists of **16,130 images** and **1,353 human-annotated multiple-choice questions** across **9 distinct scenarios**.

- **Evaluation and Findings**: The paper evaluates **10 open-source** and **4 proprietary large vision-language models (LVLMs)**, showing that models exhibit greater improvements when augmented with images compared to textual knowledge.

- **Insights and Challenges**: The findings highlight the challenges LVLMs face in effectively leveraging retrieved visual knowledge, emphasizing the need for improved utilization of visually augmented information.
weaknesses: - The motivation is unclear. In Figure 1, do the authors mean that retrieving from the image corpus is not difficult? While I agree that retrieving correct textual knowledge is challenging, why is retrieving from the image corpus considered easier? Additionally, I do not understand why retrieved correct textual knowledge is deemed not useful, which is confusing[^1^][1]. Please clearly state your motivation.

- How is the image knowledge database handled in the experiments? It is unclear how the authors retrieve the images.

- In my view, visual RAG is more difficult than text RAG. It would be valuable to explore CoT + Visual RAG. For example, in Figure 1, if VLLMs are unsure about the model of the car, they could consider some candidate answers and then use Visual RAG to validate and confirm the final answer. Directly retrieving images is not easy with a large image knowledge database and may sometimes hurt performance when incorrect images are selected.

- I cannot understand why GT Image RAG is better than GT Text RAG in Table 4. Can the authors provide more clarifications and illustrations?

I recommend that authors submit a link to the anonymous repo. It is very important for benchmark to release ASAP. After that, reviewers can check the data quality.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Suggestions: Maybe the authors can fix the ground truth images and add retrieved images from different retrieval systems (i.e. search engine, different multimodal knowledge graph), so that this benchmark can also be used to evaluate multimodal retrieval systems.
soundness: 3
strengths: Multimodal RAG is surely an important problem, along the improvement of multi-image understanding ability in current LVLMs. Many cases in the benchmark are close to real world scenarios. The settings of noisy retrieved information is very meaningful and necessary, since in real world multimodal RAG, errors or failures in the retrieval process are inevitable.
summary: The authors propose a new multimodal retrieval-augmented generation benchmark, MRAG-BENCH. In this benchmark the LVLMs are given the input of query image, textual question and retrieved relevant images, and required to answer the multiple choice questions. Evaluations on MRAG-BENCH show the gap between current leading LVLMs and human beings. The authors also have an interesting findings that current open-source LVLMs lack the ability to select useful information from the retrieved images while ignore the irrelevant or misleading retrieved images.
weaknesses: Some cases in the benchmark might be too ideal for current retrieval systems, for example ,the Angle case and the Deformation case in Figure3. It is unlikely that a multimodal retrieval system (i.e., search engine, multimodal knowledge graph) can retrieve such accurate ground truth image given the query image.

Meta-review: The AC acknowledges the contributions of this work and significance of the problem domain, which explores the integration of retrieval systems to address challenges such as missing details or unclear input in vision-language tasks. While the approach shows promise, concerns remain about its practical implications, like,  there is a risk that the model may overly depend on retrieved visual information, potentially disregarding the original input image, which is an issue particularly problematic when the question pertains directly to the input. 
Despite these limitations, the AC recognizes that the paper provides a thoughtful examination of a relevant problem and offers a foundation for further exploration. While the work has good amount of room for improvement, its contributions deserve a weak acceptance.

Decision: Accept (Poster)