MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models

Authors: Wenbo Hu, Jia-Chen Gu, Zi-Yi Dou, Mohsen Fayyaz, Pan Lu, Kai-Wei Chang, Nanyun Peng
Primary area: datasets and benchmarks
Keywords: multimodal, rag, vision-language, visual knowledge

Abstract:
Existing multimodal retrieval benchmarks primarily focus on evaluating whether models can retrieve and utilize external textual knowledge for question answering. However, there are scenarios where retrieving visual information is either more beneficial or easier to access than textual data. 
In this paper, we introduce a multimodal retrieval-augmented generation benchmark, MRAG-Bench, in which we systematically identify and categorize scenarios where visually augmented knowledge is better than textual knowledge, for instance, more images from varying viewpoints.
MRAG-Bench consists of 16,130 images and 1,353 human-annotated multiple-choice questions across 9 distinct scenarios. With MRAG-Bench, we conduct an evaluation of 10 open-source and 4 proprietary large vision-language models (LVLMs). Our results show that all LVLMs exhibit greater improvements when augmented with images compared to textual knowledge, confirming that MRAG-Bench is vision-centric. Additionally, we conduct extensive analysis with MRAG-Bench, which offers valuable insights into retrieval-augmented LVLMs. Notably, the top-performing model, GPT-4o, faces challenges in effectively leveraging retrieved knowledge, achieving only a 5.82\% improvement with ground-truth information, in contrast to a 33.16\% improvement observed in human participants. These findings highlight the importance of MRAG-Bench in encouraging the community to enhance LVLMs' ability to utilize retrieved visual knowledge more effectively.

Decision: accept (poster)
Reviewer scores: 5, 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730196370069

thread_type:
official_review

<thread_object>
creator: reviewer_jirs

<content>
summary:
The authors propose a new multimodal retrieval-augmented generation benchmark, MRAG-BENCH. In this benchmark the LVLMs are given the input of query image, textual question and retrieved relevant images, and required to answer the multiple choice questions. Evaluations on MRAG-BENCH show the gap between current leading LVLMs and human beings. The authors also have an interesting findings that current open-source LVLMs lack the ability to select useful information from the retrieved images while ignore the irrelevant or misleading retrieved images.

strengths:
Multimodal RAG is surely an important problem, along the improvement of multi-image understanding ability in current LVLMs. Many cases in the benchmark are close to real world scenarios. The settings of noisy retrieved information is very meaningful and necessary, since in real world multimodal RAG, errors or failures in the retrieval process are inevitable.

weaknesses:
Some cases in the benchmark might be too ideal for current retrieval systems, for example ,the Angle case and the Deformation case in Figure3. It is unlikely that a multimodal retrieval system (i.e., search engine, multimodal knowledge graph) can retrieve such accurate ground truth image given the query image.

questions:
Suggestions: Maybe the authors can fix the ground truth images and add retrieved images from different retrieval systems (i.e. search engine, different multimodal knowledge graph), so that this benchmark can also be used to evaluate multimodal retrieval systems.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730307201224

thread_type:
official_review

<thread_object>
creator: reviewer_tc9p

<content>
summary:
- **Introduction of MRAG-BENCH**: The paper introduces **MRAG-BENCH**, a benchmark designed for evaluating retrieval-augmented multimodal models, focusing on scenarios where visual information is more beneficial than textual data. MRAG-BENCH consists of **16,130 images** and **1,353 human-annotated multiple-choice questions** across **9 distinct scenarios**.

- **Evaluation and Findings**: The paper evaluates **10 open-source** and **4 proprietary large vision-language models (LVLMs)**, showing that models exhibit greater improvements when augmented with images compared to textual knowledge.

- **Insights and Challenges**: The findings highlight the challenges LVLMs face in effectively leveraging retrieved visual knowledge, emphasizing the need for improved utilization of visually augmented information.

strengths:
### Originality
- **New Benchmark**: The introduction of MRAG-BENCH is a significant contribution, focusing on vision-centric retrieval-augmented generation, which is less explored compared to text-centric benchmarks. The paper identifies and categorizes scenarios where visual knowledge is more beneficial than textual knowledge, such as perspective and transformative changes, which is a fresh approach in the field.

### Quality
- **Comprehensive Evaluation**: The paper evaluates 10 open-source and 4 proprietary LVLMs using a large dataset of 16,130 images and 1,353 questions, providing robust and extensive analysis.
- **Detailed Analysis**: The results include fine-grained performance metrics across different scenarios, highlighting the strengths and weaknesses of various models.

### Clarity
- **Well-Structured**: The paper is well-organized, with clear sections detailing the benchmark's design, data collection, and evaluation methodology.
- **Illustrative Examples**: Figures and tables, such as the scenario distributions and qualitative examples, effectively illustrate the concepts and findings, making the paper easy to follow.

Overall, the paper makes a substantial contribution to the field of multimodal models by introducing a novel benchmark that highlights the importance of visual knowledge, backed by thorough evaluation and clear presentation.

weaknesses:
- The motivation is unclear. In Figure 1, do the authors mean that retrieving from the image corpus is not difficult? While I agree that retrieving correct textual knowledge is challenging, why is retrieving from the image corpus considered easier? Additionally, I do not understand why retrieved correct textual knowledge is deemed not useful, which is confusing[^1^][1]. Please clearly state your motivation.

- How is the image knowledge database handled in the experiments? It is unclear how the authors retrieve the images.

- In my view, visual RAG is more difficult than text RAG. It would be valuable to explore CoT + Visual RAG. For example, in Figure 1, if VLLMs are unsure about the model of the car, they could consider some candidate answers and then use Visual RAG to validate and confirm the final answer. Directly retrieving images is not easy with a large image knowledge database and may sometimes hurt performance when incorrect images are selected.

- I cannot understand why GT Image RAG is better than GT Text RAG in Table 4. Can the authors provide more clarifications and illustrations?

I recommend that authors submit a link to the anonymous repo. It is very important for benchmark to release ASAP. After that, reviewers can check the data quality.

questions:
Please see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730565992626

thread_type:
official_review

<thread_object>
creator: reviewer_xa2r

<content>
summary:
This paper introduces MRAG-BENCH, a multimodal retrieval-augmented generation benchmark comprising 16,130 images and 1,353 human-annotated multiple-choice questions across 9 distinct scenarios. The authors evaluate the performance of 10 open-source and 4 proprietary large vision-language models (LVLMs) on this benchmark.

strengths:
- The authors conduct an investigation into the capability of models to retrieve and utilize external visual knowledge for question answering, which is a crucial and direct exploration within the LVLM field.

- The paper introduces a comprehensive multimodal retrieval-augmented generation benchmark that provides a foundation for the community to further explore and advance in this area.

- Figure 1 is well-designed, with well-chosen examples to illustrate the benchmark's scenarios and model evaluations effectively.

weaknesses:
- The data collection process is unclear. Adding a figure to illustrate the entire process could clarify the methodology, especially regarding the acquisition of ground truth (GT) text and the source of question-answer pairs.

- The experimental setup lacks sufficient detail. The authors should provide a complete subsection dedicated to describing the experimental setup, placed within Section 3, instead of scattering brief descriptions across various sections. This current layout diminishes the reliability of the experimental conclusions. For example, in Table 3 (Retrieved RAG), it’s unclear which model was used as the retriever, what the retriever’s input and output were, etc. Additional concerns regarding the experimental setup are outlined in the Questions section below.

- The analysis of the question, "How much does visual knowledge provide additional benefits over textual knowledge?" is insufficiently supported. Additional experimental analysis would strengthen the authors’ conclusions.

questions:
- What is the meaning of "Unique number" in Table 2?

- How were the nine scenarios in MRAG-BENCH selected?

- Regarding the "Others" example in Figure 3, why is option A marked as the correct answer? This raises questions about the rule for obtaining question-answer pairs. How is the correct answer determined for each question?

- Could the authors clarify how CLIP is used for text retrieval? Is it a direct approach based on finding the sentence with the highest similarity to the current question, or does it involve a method similar to INFOSEEK [1], where visual entity recognition is performed first, followed by retrieval?

- Concerning Table 3:

  - If human performance on this task is low, how can the quality of GT images be assured during data collection?

  - How is "Human performance" in Table 3 measured?

  - Most open-source LVLMs exhibit performance declines when using Retrieved RAG, suggesting that retrieval effectiveness is crucial to answer quality. If retrieval consistently underperforms, is the entire process reasonable and valid?

- Concerning Table 4:

  - How were the five comparison methods implemented?

  - What is the meaning of GT Text RAG, and how is GT Text collected? Additionally, it seems counterintuitive that LLaVA-Next-Interleave performs worse with GT Text RAG. If adding GT Text RAG degrades model performance, does this suggest that the quality of GT Text might be low?

  - Why does the performance improvement for LLaVA-Next-Interleave, when provided with both GT images and textual knowledge, fall short compared to using GT images alone?

- There appears to be a minor error in Section 4.1, where it states, "LLaVA-Next-Interleave exhibited an 11.09% improvement with image knowledge over text knowledge." Shouldn't this be 11.90%?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649207984

thread_type:
official_review

<thread_object>
creator: reviewer_pyuc

<content>
summary:
This paper introduces a benchmark (MRAG-BENCH) designed to evaluate the performance of large vision-language models (LVLMs) in scenarios where visual information retrieval is more beneficial than textual data retrieval. First, this paper establishes the first benchmark focusing on visual information superiority over textual data. Second, this paper conducts an extensive analysis of LVLMs, and makes detailed analyses. Third, this paper points out a way that LVLMs could be evaluated and improved by emphasizing the importance of visual information in retrieval-augmented multimodal tasks.

strengths:
1. The paper is well-written and easy to follow.
2. The paper offers a different visual-centric perspective and addresses a gap in the current text-centric benchmarks.
3. The comprehensive evaluation of multiple LVLMs provides valuable insights, such as how visual knowledge benefits LVLMs over textual knowledge, and the examination of retriever performance and its impact on LVLM accuracy.

weaknesses:
1. The process of selecting ground-truth images and query images may introduce bias if not carefully controlled. The paper could be strengthened by discussing potential biases in the data collection process and how they were mitigated.
2. The paper focuses on a specific set of scenarios, which may not scale well to other types of visual queries or more complex tasks. For example, what if there are many objects in the question? 
3. The paper could further discuss the implications of the findings, such as how they might influence the development of future LVLMs.

questions:
1. How to conduct the process of human evaluation? For example, how many annotators were involved in evaluating each question, and what measures were taken to ensure consistency and accuracy in their assessments?
2. In Table 3, there are two human evaluation settings. How do you ensure that human annotators do not rely on their own knowledge when answering questions, and what steps have you taken to mitigate any potential biases in the human evaluation process
3. Whether retrieving additional visual information also enhance performance in text-intensive scenarios? 
4. The paper does not report the performance of GPT-o1 on MRAG-BENCH. How does GPT-o1 perform on this benchmark?
5. In Section 4.3, the paper shows that further increasing the number of retrieved images does not improve performance. Could the authors explain the reasons behind this effect, and what insights does this provide for future research on retrieval-augmented multimodal models?
6. Will you release this benchmark?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659812475

thread_type:
official_review

<thread_object>
creator: reviewer_u78r

<content>
summary:
The paper presents a Multimodal Retrieval-Augmented Generation Benchmark (MRAG-Bench) to analyze and identify scenarios where visually augmented knowledge is better than textual one. MRAG-Bench comprises 16,130 images and 1,353 human-annotated multiple-choice questions, covering nine distinct scenarios. The authors analyze the performance of 10 open-source and 4 proprietary large vision-language models (LVLMs). Their results show that all the LVLMs exhibit performance improvement when augmented with visual data rather than textual. Moreover, the proprietary models can better leverage the retrieved visual information.

strengths:
The paper is well-structured and accessible, presenting a benchmark for evaluating and analyzing the relevance of visually augmented samples across various scenarios. For example, it considers cases where the main object in an image is partially occluded or where the image depicts an outdated or modified version of the object. The authors conduct extensive experiments with both proprietary and open-source methods, demonstrating that visual augmentation offers greater utility than textual information for large vision-language models (LVLMs).

weaknesses:
While the paper is easy to follow, the primary motivation behind the task remains unclear. In a real-world scenario, how would this pipeline function operationally? Given the findings that visually augmented samples offer significant value, an extensive repository of curated and labeled images for retrieval would be essential. However, this may be impractical due to legal and computational restrictions. Additionally, I would be interested in analyzing how the incorporation of augmented images affects response time and costs.

questions:
1. Could you elaborate on your motivation for creating this benchmark? Please include examples of real-world applications where this benchmark could offer substantial value.

2. Using this benchmark in real-world applications could be challenging, as it would require a comprehensive image repository encompassing a wide range of objects globally. This need introduces various limitations, such as legal and computational restrictions and the necessity for thorough curation to prevent biased or intrusive content. Could you discuss potential strategies for making this benchmark feasible in real-world scenarios, as well as approaches to mitigate these limitations?

3. Increasing context (whether visual or textual) means that large language models (LLMs) would need to process longer input sequences, which would raise inference times. Since images are typically represented as lengthy sequences, adding more images could substantially impact both inference time and associated costs, especially in the case of proprietary models, which gain the most from visual augmentation. Could you provide a detailed analysis of the trade-offs between performance gains, increased costs, and extended inference times?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732172648738

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We sincerely thank all the reviewers for their time and their constructive reviews and suggestions. We are encouraged that the reviewers find that:

 * Our work is **crucial** (Reviewer XA2r) and explores an **important** problem (Reviewer JirS), with **significant** contribution and a **fresh approach** in the field (Reviewer tC9p). 
 * Our work demonstrates visual augmentation offers greater utility than textual information (Reviewer U78R), and **addresses a gap** in the current text-centric benchmarks (Reviewer Pyuc). It builds the foundation for the community to further explore and advance in this area (Reviewer XA2r).
Our benchmark is **comprehensive** (Reviewers Pyuc, XA2r, tC9p) and the evaluation of multiple LVLMs provides **valuable insights** (Reviewer Pyuc). Many cases in the benchmark are close to **real world scenarios** (Reviewer JirS). The experiments are extensive (Reviewer U78R) and offer detailed analysis (Reviewer tC9p). 
* Our paper is **well-structured**, **well-written** and easy to follow (Reviewers U78R, Pyuc, tC9p), with well-chosen and **illustrative** examples to illustrate the benchmark's scenarios and model evaluations effectively (Reviewers XA2r, tC9p). The paper has **clear sections** detailing the benchmark's design, data collection, and evaluation methodology (Reviewer tC9p). 


Thank all the reviewers for their valuable reviews! We believe the comments and revisions have made the paper stronger. Here is a general response to common questions and please find individual responses to your questions below. 

**[Motivation]**

We would like to reiterate the motivation of this paper. As we outlined in our abstract (lines 11-15) stems from the observation that existing research primarily evaluates whether models can retrieve and utilize external **textual** knowledge. However, there are scenarios where retrieving **visual information** is either more beneficial or easier to access than textual data. Our benchmark targeted **real world scenarios** and systematically categorized into 9 distinct scenarios (see lines 91–98), each illustrated with explicit examples in Figure 3. These images encompass a diverse range of objects, and across all scenarios, we have demonstrated the critical role of incorporating visual knowledge in **real-world visual question answering**.

**[Why is image knowledge useful?]**

A simple explanation is that images capture details, contexts, and nuances that cannot always be directly translated to or recorded in  text corpora.  Besides all 9 scenarios outlined in our paper’s Figure 3 and comparisons given in Figure 1, we provide a concrete example here. 
For instance, consider the example in **Figure 4** of our paper. In this case, the question asks for the exact car model and make. The **GT text** includes the context of the ground truth answer (e.g., "Chevrolet Silverado 1500 Extended Cab") and additional details such as the car's year of manufacture and engine specifications, etc. Although all these information described the information of this car model, **none of them includes visual appearance** of the car. Since this example represent a difficult Angle scenario, the textual information alone is insufficient—the model **cannot recognize** the car visually due to the challenging perspective in the image. This highlights why **GT image knowledge is invaluable** in such scenarios.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732493289533

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Follow-Up on Rebuttal for "MRAG-Bench"

comment:
Dear Reviewers,

We're following up on the rebuttal for our paper, "MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models" We appreciate the time and effort you've invested in reviewing our work.

In our rebuttal, we've thoroughly addressed the concerns and suggestions raised in the initial reviews. If you find that all your questions have been resolved, we kindly ask you to consider reflecting this in the scores. Should you have any additional questions or require further clarification, we are eager to engage in more discussion during the official ICLR discussion period.

Thank you once again for your thoughtful feedback and consideration.

Best regards,

The Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733291126006

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Rebuttal and Discussion

comment:
Dear SACs and ACs,

Thank you for your time and effort in handling our manuscript. We also deeply appreciate the thorough reviews provided by all reviewers.

Reviewer XA2r’s primary concerns focus on the clarity of our work and we have made efforts to address their concerns with detailed explanations and clear references to the information in our appendix. 

For reviewer U78R, their concern is on further real world application of our benchmark. We explained how our benchmark provides valuable insights into real-world scenarios, as supported by Reviewer JirS.
We emphasize our primary focus has been to ensure high-quality examples and **realistic scenarios in the benchmark**, with all experiments validating these contributions. After our explanation, Reviewer U78R followed up with a question about "what if the model depends more heavily on the retrieved visual information." This question is **not a weakness** but instead aligns directly with one of the key findings of our benchmark, which underscores the need for future work to better handle noisy retrieved images. Despite addressing this concern comprehensively, Reviewer U78R became unresponsive thereafter. 

For other reviewers Pyuc, tC9p, and JirS, we believe our experiments and explanations could effectively address their concerns. We emphasize improving the performance of retrieval models is one of the key areas of future work that our benchmark aims to inspire. And we have experimentally demonstrated the quality of retrieved examples with significant improved performance. **However, this aspect has been noted as a weakness by several reviewers.** 

Unfortunately, despite our time and effort, the reviewers did not actively participate in the discussion throughout the rebuttal process, and therefore they did not have a deeper understanding of the contribution of this paper. 

We would like to re-emphasizes the contributions of this paper: 

- We introduce the first visual centric RAG benchmark that focuses on utilization of visual information, unlike previous benchmarks focusing on retrieving and utilizing external textual knowledge for question answering.

- Our benchmark consists of 16,130 images and 1,353 human-annotated multiple-choice questions across 9 distinct real-world scenarios, evaluated on 14 large vision-language models (LVLMs).

- Extensive experiments demonstrated that visual augmentation provides greater utility than textual information in our benchmark, offering **valuable insights** such as 1) how retrieved visual knowledge benefits LVLMs, 2) GPT-4o lags significantly behind human performance in visual information utilization, and 3) Open-source models are more susceptible to noisy retrieved examples than proprietary models.

- As mentioned by many reviewers, our benchmark **lays a foundation for the community** to further explore and advance in this area encouraging exploration of open questions such as reducing the effects of noisy examples, determining the optimal number of retrieved examples per task, understanding the quality and positional bias of examples, and more.

We genuinely thank all reviewers and ACs for their efforts and time in reviewing our paper, as well as their constructive suggestions that contribute to the improvement of our paper. 

We sincerely hope you will take the above into consideration.

Bests,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The AC acknowledges the contributions of this work and significance of the problem domain, which explores the integration of retrieval systems to address challenges such as missing details or unclear input in vision-language tasks. While the approach shows promise, concerns remain about its practical implications, like,  there is a risk that the model may overly depend on retrieved visual information, potentially disregarding the original input image, which is an issue particularly problematic when the question pertains directly to the input. 
Despite these limitations, the AC recognizes that the paper provides a thoughtful examination of a relevant problem and offers a foundation for further exploration. While the work has good amount of room for improvement, its contributions deserve a weak acceptance.