MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models
Authors: Wenbo Hu
Jia-Chen Gu
Zi-Yi Dou
Mohsen Fayyaz
Pan Lu
Kai-Wei Chang
Nanyun Peng
Venue: ICLR 2025 Poster (2025)
Keywords: multimodal
rag
vision-language
visual knowledge
Primary area: datasets and benchmarks

Abstract: Existing multimodal retrieval benchmarks primarily focus on evaluating whether models can retrieve and utilize external textual knowledge for question answering. However, there are scenarios where retrieving visual information is either more beneficial or easier to access than textual data. 
In this paper, we introduce a multimodal retrieval-augmented generation benchmark, MRAG-Bench, in which we systematically identify and categorize scenarios where visually augmented knowledge is better than textual knowledge, for instance, more images from varying viewpoints.
MRAG-Bench consists of 16,130 images and 1,353 human-annotated multiple-choice questions across 9 distinct scenarios. With MRAG-Bench, we conduct an evaluation of 10 open-source and 4 proprietary large vision-language models (LVLMs). Our results show that all LVLMs exhibit greater improvements when augmented with images compared to textual knowledge, confirming that MRAG-Bench is vision-centric. Additionally, we conduct extensive analysis with MRAG-Bench, which offers valuable insights into retrieval-augmented LVLMs. Notably, the top-performing model, GPT-4o, faces challenges in effectively leveraging retrieved knowledge, achieving only a 5.82\% improvement with ground-truth information, in contrast to a 33.16\% improvement observed in human participants. These findings highlight the importance of MRAG-Bench in encouraging the community to enhance LVLMs' ability to utilize retrieved visual knowledge more effectively.

Published as a conference paper at ICLR 2025
MRAG-BENCH: VISION-CENTRIC EVALUATION FOR RETRIEVAL-AUGMENTD MULTIMODAL MODELS
Wenbo Hu', JiaaCChnn Gu', Zi-Yi Dou', Mohsen Fayyaz', Pan Lu",
Kai-Wei Chang', Nanyun Peng' UUCA,, 2Stanford University
{wenbohu, guje, zdou)eucla.edu
https://mraggennch..g io
ABSTRACT
Existing multimodal retrieval benchmarks primarily focus on evaluating whether models can retrieve and utilize external textual knowledge for question answering. However, there are scenarios where retrieving visual information either more beneficial casier access than textual data. In this paper, introduce a multimodal retrieval-augmenned generation benchmark, MRAG-BENCI, which we systematically identify and categorize scenarios where visually aug- mented knowledge is better than textual knowledge, for instance, more images from varying viewpoints. MRAG-BENCH consists of 16,130 images and 1,353 human-annotated multiple-choice questions across distinct scenarios. With MRAG-BENCH, we conduct an evaluation of 10 open-source and 4 proprictary large vision-language models (LVLMs). Our results show that all LVLMs exhibit greater improvements when augmented with images compared to textual knowledge, confirming that MRAG-BENCE is vision-centrie. Additionally, we conduct extensive analysis with MRAG- which offers valuable insights into retrieval-augmeente LVLMs. Notably, the top-performing model, GPT- 4o, faces challenges in effectively leveraging retrieved knowledge, achieving only 5.82% improvement with ground-truth information, in contrast to 33.16% improvement observed in human participants. These findings highlight the importance of MRAG-BENCH in encouraging the community enhance LVLMs' ability to utilize retrieved visual knowledge more effectively.
Previous Benchmarks Retrieve toxtuol-intenn -ittnssive knowledge
We focus on scenarios where
1.Retrieving correct textual knowledge is hard
R.terieeed textuol knowledge useful
Inforeek
Excycoopoiic FQ4
from (Perspective: Angle)
Example from (Transfomative Biological)
Whadyb to an aayrio
Q: Can you identify the specific model of this car?
What not likely to happen when this fruit gows bad?
corpus:
Ratrieve from text corpus:
Hard to be identified and correlated with test. No related knowladge can be found. X
"Strawberry a widely grown hybrid spacies of the genua Fragaria in the rose family .... X
Retrieve from image corpus (Our focus): foundi
Retrieve from image corpus (Our focus):
More angles easily
images strawberry goes
founc!
Figure 1: Example scenarios from MRAG-BENCE Previous benchmarks (Chang et al., 2022; Mensink et al., 2023; Chen et al., 2023b) mainly focused on retrieving from textual knowledge. However, there are scenarios where retrieving correct textual knowledge is hard and sometimes not as useful as visual knowledge.

---

Published as a conference paper at ICLR 2025
Knowledge
Knowledge
Multi-Image
Diverse
Benchmarks
Modality
Source
Input
Scenarios
K-VQA (Shah et al., 2019)
Text
Wikipedia
X
OK-VQA (Marino et al., 2019)
Text
Wikipedia
X
MultiModalQA (Talmor et al., 2021)
Text
Wikipedia
X
ManyModalQA (Hannan et al. 2020)
Text
Wikipedia
X
A-OKVQA (Schwenk et al., 2022)
Text
Common/ World
X
X
ViQuAE (Lerner et al., 2022)
Text
Wikipedia
X
X
WebQA (Chang et al.. 2022)
Text/Caption
Wikipedia
X
Encyclopedia VQA (Mensink et al., 2023)
Text
Wikipedia
X
InfoSeek (Chen et al., 2023b)
Text
Wikipedia
X
X
MRAG-BENCE (Ours)
Image
Table I: Compared with previous works, MRAG-BENCH focuses evaluating LVLMs in utilizing vision-centriy -augeete- multimodal knowledge. "Diverse scenarios" refers to whether
benchmark categorized different scenarios during evaluation.
Web, ImageNet (Russakovsky
al., 2015), Flowers 102 (Nilsback & Zisserman, 2008), StanfordCars (Krause al., 2013).
INTRODUCTION
Retrieval-augmented generation (RAG) has emerged as promising direction in large vision- language models (LVLMs) (OpenAI, 2023; Liu et al., 2023a; Bai et al., 2023; Huang et al., 2023; Chen et al., 2023a; Hu et al., 2024b; Chen et al., 2024; Tong et al., 2024; McKinzie et al., 2024). By incorporating external knowledge during generation, models such as Wiki-LLaVA (Caffagni et al., 2024) have demonstrated improved performance in knowledge-intensiv question answering tasks.
There are several existing benchmarks evaluating retrieval-augmented LVLMs. For example, OK- VQA (Marino et al., 2019) focused scenarios where the image content alone is insufficient to answer the questions. A-OKVQA (Schwenk et al., 2022) further extended this dataset to incorporate additional types of world knowledge. More recent works (Chang al., 2022; Chen et al., 2023b; Mensink et al., 2023) further expanded and curated large-scale knowledge base data evaluate pre-trained vision and language models knowledge-intensive and information-seeking visual questions. However, as shown in Table 1, these benchmarks remain text- text-centric, as their questions can often be resolved with related external textual knowledge. In contrast, retrieving visual information is sometimes more beneficial than retrieving text, as humans often gain greater insights from it. Specifically, we illustrate examples Figure where retrieving correct textual knowledge can be hard and retrieved textual knowledge can be useless, while retrieving additional images is helpful. For instance, when presented with a top-down view of a car, humans may struggle to accurately identify it; however, with front-facing view, they can quickly recognize the vehicle and effectively leverage the visual information.
In this paper, centric evaluation for retrieval-auge augmented multimodal models, with visual questions typically benefit more from retrieving visual knowledge than textual information. MRAG-BENCH consists of 16,130 images and 1,353 human-annotated multi-choice questions spanning distinctive scenarios. Focusing on utilizing visually augmented knowledge in real-world scenarios, we divide our benchmark into two aspects: perspective, where changes in visual entity's 's perspective requiring visually augmented knowledge; and transformative, where the visual entity undergoes transformative change physically thus requiring visually augmented knowledge. Specifically, MRAG- GEGGH requires models reason about visual entities that undergo perspective changes, such as angle, partial, scope and occlusion, well as transformative changes, such temporul, incomplete, biological and deformations. Additionally, MRAG-BENCH includes 9,673 human- selected images, which serves as the ground-truth image knowledge corpus for model evaluation.
introduce MRAG- AAGBHEN benchmark specifically designed for vision-
We conduct extensive experiments on MRAG- BENCH to evaluate 10 open-source and proprictary LVLMs. The results confirm that MRAG-BENCH is vision-centrie, all LVLMs show greater improvements when augmented with images compared to textual knowledge. Our results indicate that the berf-perg GPT-40 model only achieve 68.68% and 74.5% of accuracy without RAG knowledge and with ground-truth (GT) RAG knowledge, respectively. This substantially outperforms the best open-source model Veee-n-iiiiioon by 15.39% and 15.52%, respectively.
2

---

Published as a conference paper at ICLR 2025
Statistie
Number
Total questions
1,353
Others 8.9%
Multipl--hhoice questions 1.353 (100%)
Biological 7.5%
Angle 23.8%
Questions newly annotated 1,353 (100%)
Total Scenarios
9
Unique number of questions
375
Incomplete 7.5%
Unique number of answers
663
Total number of images
16,130
Transformatie
Perspective 57.5%
Unique number of images
16,130
Deform 7.5%
Human selected images
9,673
Average image size (px) 1076 x 851
Partial 18.2%
Maximum question length
20
Temporal 11.0%
Maximum answer length
9
Average question length
8.03
Occlude
Scope 7.3%
Average answer length
2.16
Averuge choice number
4
Table 2: Key statistics MRAG-BENCH.
Figure 2: Scenarios distribution of MRAG-BENCH.
Notably, we observe while all models improve with GT knowledge, only proprietary models are able to effectively utilize noisy retrieved multimodal knowledge. This indicates the gap between open- source and close-source models still exists. Open-source models are falling short their parametric knowledge and the ability to distinguish between high-quality and poor- poor-quality retrieved visually augmented examples. In comparison to humans, GPT-4o achieves only a 5.82% improvement when augmented with GT knowledge and 0.28% with retrieved knowledge, whereas humans demonstrate 1.3.16% 16% and 22.91% improvement, respectively. These results highlight the importance of MRAG- BENCH in encouraging the community to develop LVLMs better utilizing of visually augmented knowledge.
MRAG-BENCH
2.1 BENCHMARK OVERVIEW
Our benchmark is designed for systematic evaluation of LVLM's vision-centric multimodal RAG abilities. To achieve this, focus evaluating the model's understanding of image objects that not commonly associated with its knowledge base, while the collected ground- images can help incentivize specific visual concepts within LVLMs' memory. Therefore, we divide our benchmark into two main aspects, as illustrated in the examples in Figure 1:
perspective, refers to the challenges in visual recognition and reasoning that arise when visual entity presented from varying viewpoints, scopes, or levels of visibility.
transformative, refers the challenges that arise when visual entity undergoes fine-grained physical transformations, making it unfamiliar or not easily associated with the model's prior knowledge.
KRRAG-BENCH consists of 16.130 images and 1,353 multiple choice questions, with key statistics shown in Table 2. MRAG-BENCE adheres to the following design principles: (1) it focuses on real-world scenarios where visually augmented information is useful; (2) it incorporates diverse multimodal RAG scenarios covering various types of image objects; (3) it features cleaned ground-truth images for each question that align with human knowledge; and (4) it provides robust evaluation settings for deterministic evaluations. Unlike previous works focus on retrieving textual knowledge, evaluation on MRAG-B BENCH focuses on retrieving vision-centric knowledge, which can be formulated as follows: Given query tuple Q composed of (query image, textual question), the multimodal retriever R returns a set relevant images I (ii, i,, then the LVLM M take the input (Q, I) and output the final answer.
3

---

Published as conference paper at ICLR 2025
Perspective: Angle
Perspective: Partial
Q How petals does this flower
Can you identity the specific car?
(4)
(u): DMW Senes Coupe 2012
(b):
2010
te)
[e) Series Wagon
(d:
Query mage
GT Example
(d): DMW M3 Coups 2012
Query Image
GT Example
Perspective: Scope
Perspective: Occlusion
Q you identify which
Q
this?
this
(a): Appenzelier Sennenhand V
(b): Greater Swiss Mountain Dog
(b) Sorrel V
Hemese Mountain Dog
(e): Bay
(d) Farl:Rucher
Dun
Query Image
Example
Exampee
Transformative: Temporal Buildings)
Transformative: Temporal (Animals
Q What the name of stricture
Q What broed this cnt?
construction finished?
Kol: Devon Rex X
Panihöun $
(b) Sphynx
(b): St. Paul's Cathedral
Xel: Conish Ros
United Capitol
(d): Oriental Shorthazr
(d): Invalides
Quely Image
GI Exampla
Transformative Deformation
Which Transformative Incomplete
ideniiy the
(e): 'F'
Punshe 911 Turbo
[b): 1
[ck: TT
(e): Posseli 911 GT3 RS
(d):
(d) Porsche 911 GT2
Query
GT Example
Transformative Biological
Others
Q What feature this fruit unlisely
Q: From which region is object
to show after oxidation?
joiginased!
Kw) Growth of dark blue muld
(u): Americas
$ TV:
Xel: Growth fuzzy white mold VV
changes
(ck: ErstAsia $
[b):
(d): Sho and fummarion of wrinkles
(d): WestAsia
Query
GT Exemple
GPT-40
Gemini
Intteeeee
Mantis
Ground Truth
Figure 3: Qualitative examples on MRAG-BENCH, For each scenario, we show the result GPT- 4o (OpenAI, 2023), Gemini Pro (Team et al., 2023), LLaaVA-NA -et-Nee-Intterlaae (Li et al., 2024b) and Mantis-8B Mantis-8B-Siglin 8B-Siglip (Jiang et al., 2024a). The ground-truth answer is in blue.
2.2 BENCHMARK COMPOSITION
MRAG-BERT BEGCHE provides a systematic evaluation across 9 distinctive multimodal RAG scenarios, with four scenarios focused the perspective understanding of visual entities, four transfor- mative understanding, and one categorized as "others". As illustrated in Figure 2, each scenario comprises 7.5% to 23.8% of the whole benchmark. The selected examples of each scenario is shown in Figure 3. The details of each scenario are introduced as follows.
Perspective understanding aspect. First, we have perspective aspect comprising [ANGLE], [PARTIAL], [SCOPE], and [OCcLuSION] dimensions.
[ANGLE] evaluates the ability of models utilize visual knowledge of common shooting angles to identify and reason about less common, long-tailed viewpoints of visual entities.
[PARTiAL] evaluates the ability of models and reason when only a partial image the visual entities is available.
complete appearance knowledge identify
[SCOPE] evaluates the ability of models to leverage high-resolution, detailed images for identifying and reasoning about visual entities in longer-scoped, low-resolution images.
[OCCI.USTO]] evaluates the ability models to use ground-truth image knowledge to identify and reason when visual entities are occluded or partially hidden in natural scenes.

---

Published as a conference paper at ICLR 2025
Transformative understanding aspect. On the other hand, the transformative understanding scenarios cover [TEMPORAL]. [DEFORMATION] [INCOMPLETE], and [BIOLOGICAL] dimensions.
[TEMPORAL] evaluates the ability of models to use familiar image knowledge to identify and reason about visual entities undergoing temporal changes that may not be represented in the model's knowledge base.
[DEFORMATION] evaluates the ability of models to use intact physical appearance knowledge identify and reason when visual entities undergo deformation not captured in the model's knowledge base.
[INCOMPLETE] evaluates the ability of models compare and contrast the complete layout and structure of image knowledge to identify and reason about missing parts and the correct layout of visual entities.
[Bro1.oGICAL] evaluates the ability of models to utilize image knowledge after biological transformations of the visual entities.
[OTHERS] aims to evaluate the ability of models to leverage geographic image knowledge to accurately identify and reason about the correct regions of origin for the visual entities of interest. All these scenarios work in tandem to comprehensively evaluate LVLMs' abilities of leveraging visually augmented knowledge.
2.3 DATA COLLECTION
As the guidelines discussed in $ 2.1, our benchmark collection involves clean ground-truth image corpus that can resonate with model's internal knowledge and a query question and image that challenge model's memory according to our definition of 9 diverse scenarios. To collect a dataset for systematic evaluation of vision-centric multimodal RAG scenarios, we manually annotate all multiple-choice question answering (MCQA) data while sourcing images from either publicly available datasets or manually scraping them from the web.
Collection of perspective aspect. To collect diverse image objects and knowledge that are not extensively represented in LVLMs' memories (Zhang et al., 2024c), we considered three sources of data, ImageNet (Russakovsky et al., 2015), Oxford Flowers102 (Nilsback & Zisserman, 2008), and StanfordCars (Krause et al., 2013). the image class that we included in our benchmark, we examined the validation set and excluded the unqualified images which can't 't provide sufficient visual information for the recognition this class. Among the selected corpus, we further humanly picked five representative examples covering the diverse aspects of each class object, as the five ground-truth examples in our experimental results (See {3). For constructing the query images, manually selected qualified images for the [ANGLE], [SCOPE], and [OCCLUSION] scenarios. For the [PARTIAL] scenario, we randomly cropped images by 50% in both height and width. Then we performed another human inspection to ensure the quality of the cropped images, filtering out examples where the visual object did not occupy the dominant area of the image. We repeated the random cropping process until satisfactory images were obtained, filtering to 20.4 GT images per question on average.
construct a high quality image corpus, for each of
adhered to our scenario definitions and
Collection of transformative aspect. We chose manually scrape images from the web based on the definitions of the transformative aspect. To construct the image corpus, we employed Bing Image Search for each of the image object keyword predefined by us, please refer to Appendix A. for more details. We filtered out image objects that did not form a clear transformative pair between the query image and the ground-truth image, retaining approximately 74% of the keyword names in the process. For ground-truth image examples, 15 images related to its keyword names and human filtered out the unqualified image. On average, this results 5.9 images per question and the five ground-truth images used during our evaluation are manually selected same as in perspective aspect.
employed automatic scripts to download the top
According to guidelines, additional related image object knowledge from the same geographic region can assist in identifying that region more effectively. For the [OTHERS] scenario, we source the data from the GeoDE dataset (Ramaswamy et al., 2023). For each distinct image object category,
5

---

Published as a conference paper at ICLR 2025
Model
Overall
Penspective
Transformative
Others
Angle
Partial
Scope
Occlusion
Temporal
Deformation Incommlete Biological
Rundom chance
24.83
27.64
23.98
24.51
19.44
22.15
29.41
Human performance
38.47
25.16
34.96
31.37
41.67
21.48
24.51
58.82
54.9
53,33
+ Retrieved RAG
61.38.2m
62.42
58.82 60.16.2s2
62.96.
51.36.
49.02.24x.
78.13 78.13.mm1
62.5 63.73
GT RAG
71.63
83,85
70.33.
66.67
69.44.20m
59.73.
68.63
83,33
v3.53
Sownn-Swrce LVLMs
OpenFlaningo- 2-98
26.83
27.95
26.02
31.37
30.56
29.53
34,31
20.59
17.65
21.67
+ Retrieved RAG
28.31.1-8
29.5...5
2X 81
28.43
90.56.a
34.23.13
31.37
22.55.1%
21.57.vv
22.5
GT RAG
28.90,am
26.71.
33.74,13
28.43,
33.33,335
35,57.43
47.45.44
27.45
25,49,2
18.33
Idefics2-8B
31,01
31,06
33.33
31.37
38.89
30.2
35,29
25.19
21.51
26.67
1 Reerrieeed RAG
c0.
29.81
27.64.4.
29.41-36
36. 2m
36.24...4
28.43 4ss
27.45.1.66
52.35,254
25.83 c.44
GT RAG
37.03.50m
36.34.3.3
25.37.204
38.24
54.63.155
47.65 11745
36.27
24.5
34,31v0s
25.83..ox
VILA11.5-338
43.68
45.34
41.87
52.94
15
50.34
38,24
30.39
Reerieved RAG
35.48 x.
33.54.1
13m1
29.41 33:53
40.74 2:11
47.65 3.0
cM
22,55.00m
54, 33.33
1.33
GT RAG
47.01
45.65 (331
46.75.111
39.22.0132
51.85.17
53.69.335
43.14,49
25.49,500
44, 12.175
69.17.1147
Mantis Sontis-SB-uuee -B---Bp--lllaaaa
40.8
45.03
39.43
42.16
49.07
49.66
36.27
28:43
19.61
45.0
Retrioved RAG
36,88 so
36.65
1.
.0.34...s
33.33
vs
21.57,1
RAG
44,72,
14.14
46 46.75.73
43.14.0.s
54.63.55
57.05.1
45.1.a33
19.61 sso
18.63 om6
51.67.607
Mantis SB -i---B---g--laaaan Hama3
45.01
46.89
15.12
$7.84
58.35
45.61
45.1
26.17
29.41
45.0
Retriowed RAG
39.62...
42.55
35.37
47.22 32.
42.95.zm
45.
23.53,2
29.4
40.83...1
48.85.
54
52.85,13
$1,96..a
58:35
48.99.135
50.0.19
21.57
33.33,
49. 49.17
Deepscek-VL.. BLoo-VV--BBB-hha + Retrrived RAG
43.39
45.34
47.56
47.06
45.37
46.31
48.04
28.43
20.59
49.17
31.66,332
33.51..13
32. 11,4e
33.33
37.04
43.62
10.2.1m
20.59.124
26.47..ss
45.04.17
GT RAG
50.33.6ol
54,04.67
56.5vast
$0.98...0
56.48.
17.08.
50.0...0
21.57 ca6
23.53,29
60..3311.6
LLaVA-NcXT--meelcawee
43/46
44,41
44.97
26.47
+ Restrived RAG
40.35 (
40.06 15
33, m1
39.22.c.
56.48 36.48.22 am
1.36
12.00
27.45 19
36, 27
17.17.
+ GTT RAG
52.99 55
14.97.10%
$4.88.
49.02.
62.04 377
52.35.
47.06....9
38.24,
48.04, 57
61.67.1150
49.74
48.45
50.81
54.9
58.35
51.36
51,96
30.39
45.1
51.67
-Retrieved RAG
41.83..m
36.59 40.06..35
40.2.
50,0
50.34
16.08...
20.59
46.67.g 41.96.4
+ GTT RAG
56.32 s
58.39
19 is
58.82.10
62.96.
61.74.) 3s
59.8% 59.8.1
26.47.vv vm
50.0.2o
S8. 38.33...0
LLaVA-One Vision
53,29
58.39
$6.1
49.02
60.19
47.65
53.92
37.25
$2.94
$1.67
RRtrrived RAG
50.1
50.93.<e
48.78...
50.0.acs
60.19.aa
50.34.299
48.04.a3
33.33.39
s3.92
54,17
58.98..
62.42.41e
59.8 is
19.73.12m
53.92.u
30.39
57,84
60.83
Pixtral-128 + Reerieved RAG
47,97
52.48
45.53
58.82
50.0
51.68
49.02
38.24
42. 16
37.5
45.97
51,86.ca2
40.24
53.9249
50.93
49 2v
47,00 47.06...s
19.61.1943
47.06...
46.67
GT RAG
59.28.1131
63.04..02e
63.4
65, 65.m
66.67
61.74.000
59.8..3.m
20.59.
50.98.ss2
65.0.an
Proprictury LVLMs
GPT---Thibb
57.21
64.29
59.3
$4.9
56.48
62.42
47.06
41.18
59.8
50.0
Retrieved RAG
18.95.17:
66.53.23
53.94 59.94.2
66.74..03
59,73
49,0 49.06.25
38.27
62. 62.78,294
58.83
RAG
62.85.54
68.94.
69.51.aa
60.78...a
67.59..1.1
63.33.
51,96.49
38.24.304
59.8,am
62.5.125
Gemini Pro + Retrieeed RAG
61.71
68.01
69.92
73.53
71.3
70.47
42.16
39.23
53.92
40.83
65.93..
78.29. ss
69.92.ae
69.61 140
73. 15
49 49.02..0s
34.31 49)
56.86,
65.0..m
GT RAG
71.40...
77.33.
39.27.u35
78.43.19
75.93.163
78.52...5
54.9.
36,27
61.76,
72.5
Claudo Sonnet
59.87
70.19
$7.72
$6.86
57.41
68.46
62.75
+ Rettrioed RAG
63.56,se
.5.9
70.73.
km
86.86.00
55.96.555
70.47-2a
55 88 m1
31.37.
62. 75.
$3, 33,3
+ +GT RAG
71.10.
78.88
80 m9
76.47.
70.37.
75.17..27
67. 67.65.146
36.27 13
65.09
59.17.11
GPT-4o
68.68
76.09
70.42
69.61
74.07
73.82
61.21
47.62
58.82
65,83
-Retrieved RAG
68.96.uus
77.95.13e
.4
69.6 Lau
75.0wass
w3.83
$4.99.2
26.47.3434
Sams
68,33,23
+ GT RAG
84.47 a.3
77.46 ea
82.35
79.63.539
77.18 35
68.62
30.95.
62.7 62.75
80.0.
Table 3: Accuracy scores on MRAG- -ENCH. The highest scores for open-source models in
each section and proprietary models are highlighted in blue and red, respectively. CLIP retriever consistently used across all models. Both Retrieved RAG and GT RAG employ top-5 image examples (except for the incomplete scenario, where a single example is intuitively sufficient). The relative difference in performance compared to the score without RAG is shown in subscript, with blue indicating performance drops and red indicating improvements.
we randomly sampled 3 out of 6 regions to serve as the answers for each question and selected the corresponding image as the query image.
Quality control. After constructing the entire benchmark, we implemented two quality control procedures: an automatic check with predefined rules and a manual examination of each instance. The automatic check verifies the correct MCQA format, assesses image validity and filters out redundant images in the corpus, more details are presented in Appendix A. The manual examination is conducted by two experts in this field, who checked the correspondence between query images and ground-truth image examples, and filtered or revised ambiguous questions and uncorrelated query image and ground-truth images.
6

---

Published as a conference paper at ICLR 2025
3 EXPERIMENTS
In this section, we first introduce the experimental setup and evaluation metric (5 3.1). Then, we present comprehensive evaluation of 14 recent LVLMs (3 3.2). We demonstrate the importance of visual knowledge and discuss the critical findings revealed by the results from MRAG-BENCI.
3.1 EXPERIMENTAL SETUP
We evaluate 14 popular LVLMs on MRAG-BENCH, including 4 proprictary models and 10 open- sourced models that can accept multi-image inputs:
Proprietary models: GPT-4o (0513) (OpenAI, 2023), GPT-4-Turbo (OpenAI, 2023), Gemini Pro (Team et al., 2023), and Claude 3.5 Sonnet (Anthropic, 2024).
Open-source models: OpenFlamingo (v2-9B) (Awadalla et al., 2023), Idefics (v2- 8B) (Laurençon et al., 2024), VILA (v1.5-13B) (Lin et al., 2023), LLaVA-NeeTT-mterllaaee 7B (Li et al., 2024b), LLaVA-OneVision (Li et al., 2024a), Mantis (clip-Ilama3, and 1saip-llamaa3 versions; 8B) (Jiang et al., 2024a), mPLUG-OW13-7B (Ye et al., 2024), Brepseek-VL-7B-oee (Lu et al.. 2024a), and Pixtral- 12B (Team, 2024).
Evaluation setup. We follow standard MCQA evaluation setup and employ accuracy score as our metric. adopt default generation hyper-parameters selected by each model. Following et al. (2024b), employ GPT-3.5-turbe to extract the multiple choice answer in rare cases where our pre-defined automatic extraction rules failed. We refer the readers to Appendix A. and for more details on evaluation prompts for both without multimodal RAG and with multimodal RAG scenarios, answer extraction prompt and human performance evaluation protocol.
3.2 MAIN RESULTS
As shown in Table 3. the average performance of the most advanced LVLMs not better than 68.68% without multimodal RAG knowlege, and 74.5% with ground-truth knowledge, which demonstrates MRAG-BENCH to be a challenging benchmark. The mean accuracies of open- source LVLMs are between 26.83% and 53.29% without RAG knowledge and between 28.909 and 59.28% with ground-truth knowledge, which fall behind from advanced proprictary LVLMs. Notably, DIRAG-BENCE proves to be cowwlede-inttnsivve without RAG knowledge, while proprietary LVLMs generally perform well, suggesting that their extensive training data equips them with broader knowledge base. However, when provided with either retrieved or ground-truth knowledge, humans achieve the most significant improvements of 22.91% % and 36.,6%, respectively. This underscore the need of LVLMs to better utilize visually augmented information like humans.
average humans achieved 38.47%
Can LVLMs utilize retrieved and ground-truth image knowledge well? As illustrated in Table 3, all models demonstrate improvement when ground-truth image RAG knowledge is provided. Among the open-source models, they achieve improvements ranging from 2.07% to 11.31% when using ground-truth RAG knowledge, whereas 5.64% to 9.69% improvements observed from proprictary LVLMs. Interestingly, when images from the multimodal retriever is provided, almost all open-source LVLMs average demonstrate declined performance while proprietary models can still gain improvement. This indicate proprietary models possess emerging abilities to distinguish between good and bad image knowledge sources, which is a critical skill in the multimodal RAG domain. We further conducted a qualitative analysis to investigate the reasons behind this, detailed in the following paragraphs.
Fine-grained results. We also report fine- ine-grained scores across 9 scenarios BRAA--BENCE in Table Remarkably, GPT-4o surpasses most other baselines in various categories, with exceptions in problems related to partial, incomplete and biological scenarios. Notably, GPT-4o outperforms human performance on all perspective aspect as well as on temporal and deformation scenarios within the transformative aspect. We conjecture that incomplete and biological scenarios less likely to be included in the training knowledge. Interestingly, all models exhibit a decline in performance incomplete scenarios, with only few exceptions, while humans find this

---

Published as a conference paper at ICLR 2025
Q: Can you identify
model
car?
LLaVe-Next-nntrreeve
Gemini Pro
(o): Chevrolet Silvendo 1500 Classic Extendee Cab 2007
Without RAG: (c)
Without RAG: (d)
(b) Chevrolet Silverado 2500HD Regular Cab 2012
With Retrieved
Retrieved RAG: (c)
(c): Chevrokt Silverado Extended Cab 2012
With Ground Truth RAG: (6)
With Ground Truth RAG: (c)
(d) Chevrolet Silverado 1500 Regalor Cab 2012
Query Image:
Retrieved Images:
(e): 1500 Extended
(b): 2500HD
(4): 1500 Regular
(c):1500 Extended
(a) 1500 Classic Fatended
Figure 4:
Qualitative Example of Proprietary model (Gemini Pro) identifies and utilizes is misled by noisy retrieved
correct examples, while open-source model information, resulting incorrect answers.
Model
Ovenll
Perspective
Transformative
Others
Angle
Scope
Occlusion
Temponi
Deformation
Incomplete Biological
V.L.VA-NNXTTTT Net.Neeeeetttteeeeeeee77
43.46
44.41
43.5
40.2
64.81
44,97
44.12
32.35
26.47
45.83
Retrieved Test RAG
37.99.s4
34.96
33.33
50.0.
41.61.a3s
35.29
30.39.
27.45.::5
$1.67.sp
Retrieved Image RAG
10.35.t11
10.06.m
33.33
39.22
56.48.ss
43.62.
44.12.a8
27.45.3
36.27.1a
49.17
Text RAG
41.09
11.93.2
39.02..m
38.24.
56.48.cs
44.97mm
43.14us
21.57.29
50.83,.v
.GT Image RAG
$2.99
54 84.88.133 1
49.02.x.
am
52.35.:
47.06.2ss
38.24.s
48 28.09.2..
11164
GTT Image & Text R
47
4
48.78.5
44,12,5v
58 s
49.66...
48.04.1
30.39
35.30 35.29.43
62. 62.5
Gir---Turbo
15.21
64.29
99.35
$4.9
56.48
62.42
47.06
41.18
59.8
50.0
Rerrivvdd Text RAG
56.61
61.8
99.35.co
58.33.
19.06-136
49.02.1.4
33.33:14
60.78.3
$2.5,2.
Retrieved Image RAG
58.95..m4
66.53.254
59.94.css
53.94
66.74...3s
29.73...
49.06.28
38:27
62. 02.78.2
GT Text RAG
58.98. 1m7
68.01..
63.41.ue
65.69.mm
63.89..44
59.73.2u
38.24.a.2
37.25.
58. $8.82au
50.83.uu
.TT Image RAG
62.85
68.94
69.5
60.78
67.59....
63.33
$1.96.11
38.24
59 59.8.a3
62.5 62.5.12
GT Image & Text RAG
65.11.1s
72.05.7%
72.76.114:
67.65.127
70.37.1339
71,81,00
46.08
39.22
60.78.3.1
57.5
Table 4: LVLMs performance on BRAG-BENCHE with textual knowledge v.s visual knowledge. Both the open-source and proprietary model benefit more from image knowledge.
task relatively easy, achieving 58.82% and 83.33% scores with ground-truth knowledge. This further highlights the importance of leveraging retrieved visually augmented knowledge to address questions that do not directly incentivize knowledge stored in the models' memories.
Why can proprictary models better utilize retrieved images? We conduct error analysis on an open-source model comparison, we filtered results where See-V---ex--Intereenne answered correctly without with GT knowledge but was misled to wrong answer with retrieved examples. One example is illustrated in Figure 4, the retrieved images contain two correct examples and three false examples. While Gemini Pro is able to utilize all retrieved images, AlaVA-Next-Interleee and makes wrong prediction. This example helps explain why do almost all open-source models have lower performance with retrieved knowledge.
and proprietary model (Gemini Pro). For a fair
leverages bad examples
4 ANALYSIS
In this section, we conduct quantitative analysis addressing three important questions: 1) To what extent can LVLMs benefit more from visual knowledge than from textual knowledge on MRAG- How does the performance of LVLMs vary with examples retrieved from different retrievers? (5 4.2) 3) How many ground-truth visual knowledge examples are required for LVLMs to continue benefiting? (6 4.3)
BENCH? ( 4.1) 2)
4.1 How MUCH CAN VISUAL KNOWLEDGE BENEFIT MORE THAN TEXTUAL KNOWLEDGE?
We used the Wikipedia corpus as of 2023/07/01 as our text knowledge corpus To ensure a fair comparison, we employed the same multimodal retriever (CLIP) for retrieving either text or image knowledge. The top-5 ranked documents or images are used for augmenting the input. We selected one open-source -L.aVA-Next-Interleeee their preference for textual knowledge versus image knowledge on MRAG-BENCH As shown
and one proprietary (GPT-4-uurb) LVLM to examine
8

---

Published as a conference paper at ICLR 2025
Correlation: 0.95
Correlation: 0.82
Overall Accuracy (96
VISTA
Overall Accuracy
40
E5-V
CLIP
56
39-
37
52
36
MagicLens
RAG
40
50
55
60
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Retriever Models Recall@5 (%)
Number of Ground Truth Examples
Figure 5: Left: LLaVA-Next-Intereawy results with 4 different multimodal retrievers. Its performance using retrieved images correlates 95% with retriever's Recall@5 scores. Right: Average results of three random seed runs. Improve the number of ground-truth RAG examples shows steady increase of model's performance, reaches the maximum with 10 examples.
in Table 4, when both models utilized retrieved knowledge, LLaVA- demonstrated 2.36% improvement with image knowledge over text knowledge, while GPT-4-Turbo showed 2.34% improvement. When using GT knowledge, improvement with image knowledge over text knowledge, compared 3.87% improvement for GPT-4-Turbo. Interestingly, when both GT image and text knowledge are provided, LLaVA-Next- Interleave indicated less improvement than with GT image alone whereas GPT-4-Turbo further pushed its performance. All these results demonstrate that retrieving visual knowledge is more helpful than retrieving text on MRAG-BENCH. BENCH.
M..VA--ext-Interleer exhibited an 11.90%
4.2 HOw DOES RETRIEVER PERFORMANCE AFFECT LVLMs?
We picked four recent perf-perf multimodal retrievers, including CLIP (Radford et al., 2021), MagieLens (Zhang et al., 2024a), E5-V (Jiang et al., 2024b), VISTA (Zhou et al., 2024) and evaluated their performance (Recall @ 5). The detailed retriever performance can be found at Table in Appendix C. We selected LLa LLaVA--ext-Interlaaa As shown in Figure 5, when retrievers achieve higher Recall@5 scores (i.e., better retrieved examples), the LVLM's accuracy tends to improve, demonstrating a strong 95% positive correlation. Interestingly, despite similar Recall@5 scores from CLIP and VISTA retrievers, VLa Interleave demonstrated a 2.07% gap in overall accuracy. We conjecture that the order of the correctly retrieved examples may also impact the model's final performance. The sensitivity to the order of retrieved examples is common issue that persists across various models. Although this phenomenon, known as position bias, has been examined in text-based RAG (Lu al., 2022b; Wang et al., 2023), its impact visual RAG remains unexplored, presenting promising direction for future research.
as the end model to assess its performance.
4.3 HOw MANY GROUND-TRUTII IMAGE EXAMPLES ARE NEEDED?
For simplicity, all our experiments used five retrieved or ground-truth image examples. However, it is worth exploring how many examples LVLMs effectively leverage. As noted in 2.3, the perspective aspect of our benchmark includes an average of 20.4 ground-truth examples. To investigate further, we perform an analysis focusing on the perspective and others aspects, covering a total of 892 questions. As shown in Figure 5, we evaluated
using 1, 2, 3, 5, 10, 20 GT examples, averaging the results across three random seeds for sampling the GT examples. ALaVA-Next-Interte example. Performance continued to increase steadily, reaching a peak at 10 GT examples, which was 0.29% higher than with 20 GT examples. One possible explanation could be LLaVA-Next- Interleave may not able to better leverage visually augmented knowledge in long context scenarios. Moreover, the complexity of questions affects the number of images needed too, ground-truth example sometimes help the model the most on MRAG- -EGNH.. We encourage the research on adaptatively deciding the number of necessary images based the complexity of questions.
LaaV--Next-Interleee
saw the greatest improvement of 5.64% with just one GT

---

Published as a conference paper at ICLR 2025
5 RELATED WORK
We overview three lines of related work: 1) multimodal retrieval-augmeented generation benchmarks (5 5.1), 2) large vision language models ($ 5.2), and 3) retrieval-augmented mutlimodal large language models (3 5.3).
5.1 MULTIMODAL RETRIEVAL-AUGMENTE GENERATION BENCHMARKS
A number of recent benchmarks have been developed to comprehensively assess the capabilities of LVLMs (Lu et al., 2021; 2022a; Li et al., 2023; Liu et al., 2023c; Lu et al., 2024b; Yue et al., 2023; Zhang et al., 2024b; Ying et al., 2024). There are several benchmarks well-suited LVLMs. For instance, OK-VQA (Marino et al., 2019) and
for evaluating eetrievll-uggmente A-OKVQA (Schwenk et al., 2022) both focus on scenarios where external textual knowledge is required answer visual questions. More recent works (Chang al., 2022; Chen et al., 2023b; Mensink et al., 2023) have curated large-scale knowledge bases to evaluate models on knowledge- intensive and information-seeking visual questions. In contrast, MRAG-BENCH focus on scenarios where retrieving visual information is more helpful than retrieving text.
5.2 LARGE VISION LANGUAGE MODELS
Large Vision Language Models (LVLMs) (Liu et al., 2023b; Zhu et al., 2023a; Dai et al., 2023; Yin et al., 2023; Hu et al., 2024a) have showeased promising results on a wide variety of vision- language tasks. Many works, such as Flamingo (Alayrac al., 2022), Emu (Sun al., 2023), Idefics (Laurençon et al., 2023), and VILA (Lin et al., 2023), have demonstrated in-context learning capabilities, where multiple image examples can be leveraged to improve text generation. Recent works start training LVLMs with interleaved image- text corpora, such as MMC4 (Zhu et al., 2023b) and OBELICS (Laurençon al., 2023), for pretraining, as well high-qualit, instruction tuning -eeV---eee--nteeleea
in models like Mantis-Instruct (Jiang et al., 2024a), and LLaVA-One (Li et al., 2024a), enabling models process and understand information from multiple images. Naturally, evaluating the ability of LVLMs to effectively leverage visually augmented knowledge becomes an important task, which is the primary focus of MRAG-BENCH.
(Li et al., 2024b),
5.3 RETRIEVAL-AUGMENTE MULTIMODAL LARGE LANGUAGE MODELS
Retrievallaugmenneed generation (RAG) has emerged as a potential solution to overcome limi- tations in language models by incorporating external knowledge retrieval during the generation process (Lewis et al., 2020; Shi et al., 2024). Reasonably, several works have focused on using multimodal knowledge
enhance the generation capabilities of Large Language Models (LLMs) (Yasunaga et al., 2023; Chen et al., 2022: Zhao et al., 2023; Cui et al., 2024). Recently, more works (Caffagni et al., 2024; Xuan et al., 2024; Du et al., 2024) has incorporated external knowledge to improve LVLMs' general generation abilities and the comprehensiveness of their reasoning. Although some works (Chen et al., 2022; Yuan et al., 2023) have proposed directly using image information from the web, a systematic vision-centric benchmark to evaluate LVLMs' abilities to leverage visually augmented knowledge is lacking, which is the focus of our work.
CONCLUSION
In this work, we introduce MRAG-B -BNNHH, benchmark specifically designed for vision-centric evaluation for retrieval-augmented multimodal models. Our evaluation of 14 LVLMs highlights that visually augmented knowledge brings more improvements on MRAG-BENNEE compared to textual knowledge. Moreover, the top-performing model, GPT-4o, struggles to effectively utilize the retrieved knowledge, achieving only a 5.82% improvement when augmented with relevant information, compared to a 33.16% improvement demonstrated by human participants. We further conduct extensive analysis and propose several promising directions for future research. Our findings underscore the significance of MRAG-BER BAGBBENCH in motivating the community to develop LVL.Ms that better utilize retrieved visual knowledge.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like to thank the reviewers for their valuable reviews, We would also like to thank UCLA- NLP members for their helpful comments and thank PLUSLab members for their proofreading during the paper clinic session. This research is supported in part by the ECOLE program under Cooperative Agreement HR001 12390060 with the US Defense Advanced Research Projects Agency (DARPA), and UCLA-Amazon Science Hub for Humanity and Artificial Intelligence.
REFERENCES
lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: visual language model for few-shot learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neurul Information Processing Systems 35: Annuul Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL htwp: htwpp//apaasssseeaaeeeeeeeeeesea wttpsersssspp...cpppppppeepppppe cc/paper_aeeeee http//aapp////e/e 2202. //ahh// htt.://cccc....aaoooooooooooooon html. 10
Anthropic. Claude 3.5 sonnet.
https: www.:www..........oo comp:pppp.......//./...eee.e
claude-33 3- 5- sonnet. 2024. 7
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openftamingo: An open- source framework for training large autoregressive vision-language models. ArXiv preprint, 03s/2308.01399 2023, URL https: / //a/aaiivvvvro/.. org. abs//aa///// 2308. 01390.
Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. ArXiv preprint, abs/2308. 12966, 2023. URL https:
/ //arxiv.org////eee.. absg///b///a /2088 12966.
Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornaa, Lorenzo Baraldi, and Rita Cucchiara. Wiki-llava: Hierarchical retrievallaugmmened generation for multimodal Ilms, 2024. URL httpss///aaxx wwtp://gg/ab/aa/ao/eoee 15406. 2, 10
Yingshan Chang, Guihong Cao, Mridu Narang. Jianfeng Gao, Hisami Suzuki, and Yonatan Bisk. Webqa: Multihop and multimodal QA. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 16474-16483. IEEE, 2022. doi: 10.1 htWp:CCCPPP222222222..212112 CVPR52688 2022 01 2, 10
URL https: /ttp:///oiiio/oooo http:///t...o..../////o/////
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. ArXiv preprint, 151/2306.15195, 2023a. URL
https:// /ttp::///rrvvv../ org/ als 2305. 15195. 2
Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William Cohen. MuRAG: Multimodal retrieval- augmented generator for open question answering over images and text. In Yoav Goldberg, Zoriissa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5558-5570, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. doi: 10. 666.:///0222..mmpmmmaii ht tttpps:n haa/ccaannttt ha////aaanntoooooyyooooooo 2ttgg/22222 emnlp-maine 375. 10
URL
Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming- Wei Chang. Can trentrained vision and language models answer visual information-sseking questions? In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Naturl Language Processing, pp. 19948,14968, Singapore, 2023b.
11

---

Published as a conference paper at ICLR 2025
Association for Computational Linguisties. doi: 10.18653//1/2023.commm .emplp-emmmppppepmm. wwt.:/eeemnlmnmnn2221112 main.925. URL
http:ss//accaathhlloe org/ 2023. ee..:..mmmppmmaam. 925. 1,2, 10
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open- open-source suites. ArXiv preprint, abs/2404.1682) 2024.
URL https:
//axiviiiiv../////a/aaa///aawava org/ abs/2404 16821.
Wanqing Cui, Keping Bi, Jiafeng Guo, and Xueqi Cheng. MORE: Multi-mOdal REtrieval augmented generative commonsense reasoning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 1178-1192, Bangkok, Thailand and virtual meeting, 2024. Association for Computational Linguistics. doi: 10. 10.18653/v 11223//122024..... 1/2024. finge4iiiininggacc.... URL htt.:tts///cccaacoooooo org/ 2024. Findings-acl. acl. 69. 10
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.). Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA. USA, December 10 16, 2023, 2023. URL attp://papest nips. htcp/apee_files/pppeeeeeeeeeeeee http/p/pee////atatt/ 9a 5a644355 7543557741111 011 aatprr2eee--bbtaaatt--orereeeeee oaaalaa111.55eaaaa022aaaaaaaaaan
html..1 10
Xueying Du, Geng Zheng, Kaixin Wang. Jiayi Feng, Wentai Deng, Mingwei Liu, Bihuan Chen, Xin Peng, Tao Ma. and Yiling Lou. Vul-rag: Enhancing Ilm-based vulnerability detection via
knowledge-level rag, 2024. URL ptttps://aaxiivorrrtt
10
Darryl Hannan, Akshay Jain, and Mohit Bansal. Manymodalqa: Modality disambiguation and qa over diverse inputs. Proceedings of the AAAI Conference Artificial Intelligence, 34(05):7879- 7886, Apr. 2020. doi: 10.1609/aaai, wa../////a....4..0..... php/ AAAI/ /tt//rrrcceevveeeeee /twp.//2244444 6294. 2
URL https: //tps::///////// aaai www.iioorggnn../eeeeeeeee
Wenbo Hu, Zi-Yi Yi Dou. Liunian Harold Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Mqt- Ilava: Matryoshka query transformer for large vision-language models. In The 38th Conference on Neural Information Processing Systems (NeurIPS), 2024a. 10
Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. BLIVA: A simple multimodal LLM for better handling of text-rich visual questions. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.), Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 2256-2264. AAAI Press, 2024b. doi: 10.1609/AAALL.3885..222 v3813. 27999..
URL. https: /oi org.ii.org//ii /10. 20.00///aa...
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma. Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Nils Johan Bertil Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language is not all you need: Aligning perception with language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL htatt:::/ppppe paappers.nips.o www.niissscppaaeeee e e425b7 www.aaae...00000022222222222
wecp/appr_fiilie www.eeileeeeeeeee httpe/ettsssseeeeeeeeeeeeeeeeeee wwtpppappppaaaaaaaaeeppepeeeaeae Abstract Conference html. 2
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Intericaved multi-image instruction tuning. ArXiv preprint, abs/2405.01483, 2024a. URL
https:/ http///rxvvv///////////e www.orggg///////ooo/////////ooe/ abs/ /2055 01483. 4,7,
Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-v: Universal embeddings with multimodal large language /2407 12580.
models, 2024b. URL htt.tttss////iiiii
12

---

Published as a conference paper at ICLR 2025
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-graine categorization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV)
Workshops, 2013. 5,5
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. OBELICS: open web-scale filtered dataset of interleaved image-ted maage-text documents. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, IA, USA, December 10 16, 2023. 2023. URL http: /ett::/ppage httpppaarrssseee nips. cc/paper_filleeeeeeeeeeeeeeeepee Abstract
- Datasets_ant Dbstract-Dataset__ee
Berchmarks html. 10
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building Vision-language models?, 2024.7
Paul Lerner, Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, Jose G. Moreno, and Jesus Lovon. ViQuAE, Dataset for khowledge-baseer Visual Question Answering about Named Entities. In SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 2022. doi: 10. 1145/3477495.5531777 URL http:///nvvvissseeeee parisa paris-i serris-salllee ay hal.aa............. wal.scia .ccience/ ha1-03650618.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Hugo Larochelle, Mare' Aurelio Ranzato. Raia Hadsell, Maria-Floring Balcan, and Hsuan-Tice Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL Bb493330228 (ww.0000000002225557644440000
neurips httccc//aap/e httpp/ppppr/22///// Aat./88111-AAbstaaatteaee html. 10
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024a. URL https:
org/ abs////////// 2408 03326. 6. 7, 10
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed- bench-2: Benchmarking multimodal large language models. ArXiv preprint, abs/2311.17092, 2023. URL https: /twpvvaaivvv////////a///a./aa.aa org/ abs 17311.17099 10
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-nextiintelleawe: Tackling multi-image, video, and 3d in large multimodal models, 2024b.
URL https:://aaxiii org/ awws/////000 http//447000878 07895. 4, 7, 10
Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023. 7,
10
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning, 2023a. URL https: :/ /tta//aa.rrg//aaaaaee httgg//bb/231100 03744. 2, 23
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023b. URL http: / htwpp/apapasss.me n nips cc/paper wiw.ffll/ppoo htless/ppper222233 html. 10
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Bi,,, Songyang Zhang, Wangbo Zhao Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your
model all-around player? ArXiv preprint, abx/2307.0628 2023c. URL https: arxiv. org/ wtggabs///ee 06281.
10
13

---

Published as a conference paper at ICLR 2025
Haoyu Lu. Wen Liu, Bo Zhang. Bingxuan Wang. Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang. Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vl: vl: Towards real-world vision- vision-language understanding, 2024a. 7
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu, Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning. In Chengqing Zong, Fei Xia. Wenjie Li. and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers). pp. 6774-6786, Online, 2021. Association for Computational Linguistics. doi: 10. https: hatpss//accaaathoooooo
URL
org/ 202200211112 acl-loncconnnn 528. 10
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022a. URL http: wwapperr..ips.coom wttppppecpeppee_fiieeeeeeeeoee fitess/apppeee//22///////// /hahh// 11332b www.000000.0aaaaaaaaaaaaaa....a.
html. 10
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-
Wei Chang. Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations, 2024b. 7, 10, 23
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 8086-8098 Dublin, Ireland, 2022b. Association for Computational Linguistics. doi: 10.18653/v1/2022 acl-long.556. URL https::/aalaatttll..e/
org 22022 acl-long--la-ea--aaaaaaaaaaaaaas 9
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual
question answering benchmark requiring external knowledge, In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 3195-3204. Computer Vision Foundation / IEEE, 2019. doi: N0.1109/CVPR.201..0033 URL http: http:///pppaaccccssee thecvf com cowttottnvvveeeeeeeeeee CLW___CP___________ 9//19/ttmmmm /Ma Monino_ OK-VQA_A__iis Vasaalalllleaaa Questttuuuuteeeeeeeeeeee iaweeesttionnnweerreeeee Answer i Bet_seeinn__eehhharrrr Requ__Reuuuu httprrrrrrn___ta_______ External_ Knowledge_cvPR 2013_201230000 papeerrppppp___ html. 2, 10
Brandon McKinzie, Zhe Gan, Jean- Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal Ilm pre-training. ArXiv preprint, abs/2403.094 10,61,, 2024. URL https:// arxiv. www..rr//aaoo abs 2403. 09611.2
Thomas Mensink. Jasper R. R. Uijlings, Lluís Castrejón, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, André Araújo, and Vittorio Ferrari. Encyclopedic VQA: visual questions about detailed properties of fine-grained categories. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 3090-3101. IEEE, 2023. doi: 10.1109/CCV11100001..0 2000000000.00002888 URL https: /tte://doo..ee (erdoo..orggl/ 1 1109/ ICCV51070. 2023. 00289.088800 2, 10
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. Indian Conference on Computer Vision, Graphics and Image Processing, 2008. 2,
OpenAl. Gpt-4 technical report, 2023. 2, 4,7
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina
14

---

Published as a conference paper at ICLR 2025
Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, JCML 2021. 18-24 July 2021, Virtual Event, volume 139 Proceedings of Machine Learning Research, pp. 8748-8763. PMLR. 2021. URL press/v13 /vt///11////////// radford2la html.99
Vikram V. Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. Geode: geographically diverse evaluation dataset for object recognition. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL hash/d08b Benchmarks html. 5
Neural Information
nips. cc http:/ppprr__iie files/ wttps//ieeesseaeeeeeeeee paper/2023// wtw.sssttattt-----ta--------a--a Datasets and
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252 2015,
doi: 10.1007/s1 12677711126-----008816 2,5
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pp. 146-162. Springer, 2022. 2, 10
Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. Kvqa: Knowledge- aware visual question answering. Proceedings of the AAAI Conference Artificial Intelligence, 33(01))8876-8884, Jul. 2019. doi: 10. 100.1609/aaiivv3331 v33101.330103333 aaai wwt.orr///ineee. index.ndxx.//// php/ AAAI /ttp////aaaaeeeeeeeeeeeeee 4915.
URL https://ojs.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. REPLUG: Retrieval-augmeened black-box language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pupers), pp. 8371-8384, Mexico City, Mexico, 2024. Association for Computational Linguistics. URL https: / acl antaaaahhhoooo org/222
naacl-long 463. 10
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al. Generative multimodal models in-context learners. ArXiv preprint, 1bs/2312.1132 132886 2023. URL 13286. 10
org/ abs/2312.
Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannanch Hajishirzi, and Jonathan Berant. Multimodal(qa): complex question answering over text, tables and images. International Conference on Learning Representations, 2021. URL https: wpw.review..e.eeeee/ee / wwt.:nnutttttatoeteeee 6W5UUus
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptie Cean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. ArXiv preprint, abs/2312. 1885, 2023. URL https://arxiv. org/abs 2312.22... 11805. 4,7
Mistral AI Team. Announcing pixtral 12b. https///tistrrallaa////////tttt 2024.7 7
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang. Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal 1lms, 2024. 2
Xinyi Wang. Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models latent variable models: Explaining and finding good
15

---

Published as a conference paper at ICLR 2025
demonstrations for in-context learning. In Alice Oh, Tristan Naumann, Amir Glober- son, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Anmual Conference
Neural Information Pro- cessing Systems 2023, NewrlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http:://ppppee www.:.ccpppp.ppeeee.ee.pppp wwtpppppppfiliieeeeee /at.ee//ppprr/222//M -bstract Confereeneeeeecceeeeeeeeeeeeeeee htm1.9
Keyang Xuan, Li Yi, Fan Yang. Ruochen Wu, Yi R. Fung, and Heng Ji. Lemma: Towards Ivlm- enhanced multimodal misinformation detection with external knowledge augmentation, 2024.
URL httttt////aaaet org/ aww//402200 11943. 10
Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-Tau Yih. Retrieval-augmentt multimodal language modeling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Homolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. v202 / 2ald2//ysungga23e3 html. 10
PMLR, 2023. URL https: http::::/ppocceedineee mmw....r.... ..t..ress..//p//..mmmmm..
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow13: Towards long image-sequence understanding in multi-modal large language models, 2024. URL https::////........ / wwwii.org//bbaee 2408. 04 04840. 7
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. ArXiv preprint, abs/2306. 13549, 2023. URL https:
/ //arxiv. org/ abs 2023331613220 10
Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi. ArXiv preprint, abs/2404. 16006, 2024. URL https:///arkit org/ abs/ 2404 1144..6000 16006. 10
Zheng Yuan, Qiao Jin, Chuanqi Tan, Zhengyun Zhao, Hongyi Yuan, Fei Huang. and Songfang Huang. Ramm: Retrieval-augmented biomedical visual question answering with multi-modal pre-training. Proceedings of the 31st ACM International Conference on Multimedia, 2023. 10
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. ArXiv preprint, abs/2311.16502, 2023.
URL https: //arxiiivv.///////////aa/////vv/ org/ abs ////22111122 16002. 10
Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming- Ming-Wei Chang. Magiclens: Self-supervised image retrieval with open-ended instructions. In The Forty- first International Conference on Machine Learning (ICML), pp. to appear, 2024a. 9
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kei-Wei Chang, Peng Gao, and Hongsheng Li. Mathverse: Does your multi-modal I1m truly see the diagrams visual math problems? ArXiv preprint, abs/2403.14624, 2024b. URL ht https:/ / /aaxivv.v../ org/ alss 2 2203. 220.146244..... 10
Yuhui Zhang, Alyssa Unell, Xiaohan Wang, Dhruba Ghosh, Yuchang Su, Ludwig Schmidt, and Serena Yeung-Levy. Why are visually language models bad at image classification? ArXiv preprint, abs/2405. 18415, 2024c. URL https: //tp////aiiivaii org/ abs / /2405 18415.
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do. Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, and Shafiq Joty. Retrieving multimodal information for augmented generation: A survey. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 4736- 4756, Singapore. 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1/2023, mndings-emmlp.31 URL https: https/////aaathhlgggy.o.o .rgg2023.3.2......oo..22 findinge emmlp. 31 4. 10
16

---

Published as a conference paper at ICLR 2025
Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, and Yongping Xiong. Vista: Visualized text embedding for universal multi-modal retrieval, 2024. URL https: /ttp/aaaiiv/.////////ee////aalla org/ abs/ 2406. 04292. 9
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. ArXiv preprint, abs/2304.10592 2023a. URL https: arxiv..vvv. org/ abs abs/2304.... 10592. 10
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal C4: an open, billion-scale corpus of images interleaved with text. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA,, USA, December 10 16, 2023, 2023b. URL http:/ http://papers.ntee nips. ccc wwtpppereppeeeeeeeeeeeeeeeeee somper_fils/// passspppp//2233 -absrrcrabb-bbcaatttttcc Datasets_and.
Benchmarks html. 10
17

---

Published as a conference paper at ICLR 2025
CONTENTS
1
Introduction
2
MRAG-BENCH
2.1
Benchmark Overview
2.2
Benchmark Composition
2.3
Data Collection
3
Experiments
3.1
Experimental Setup
3.2 Main Results
Analysis
4.1
How much can visual knowledge benefit more than textual knowledge?
8
4.2
How does retriever performance affect LVLMs?
4.3
How many ground-truth image examples are needed?
5
Related Work
10
5.1
Multimodal Retrieval-Augmented Generation Benchmarks
10
5.2
Large Vision Language Models
10
5.3
Retrieval-Augmented Multimodal Large Language Models
10
6
Conclusion
10
MRAG-BENCH Details
18
A.1 Dataset Curation Details
18
A.2 Human Evaluation Protocol
22
Experiment Setting Details
23
B.1 Model Prompts
23
B.2 Evaluation Tool
23
C More Results
24
A MRAG-BENCH DETAILS
A.I DATASET CURATION DETAILS
Dataset collection of transformative aspect We chose to manually scrape images from the web based on the definitions the transformative aspect. To construct the image corpus, we employed Bing Image Search for each of the image object keyword predefined by We filtered some of the search results where the image objects do not have a clear pair of query image and ground-truth image example, around 74% keyword names were kept during this process. Here listed all the keywords that are already filtered and used for search of query image except in biological scenario, it's for search of gruthd-truth image example. Each search keyword composed of "image
18

---

Published as a conference paper at ICLR 2025
object" and "condition". For example, "A young kitten image of Himalayan Cat", here Himalayan Cat is the image object and a young kitten is the condition. For each of keyword listed below, we searched again for its ground-truth examples (except for biological scenario, it's for query images), which only "image object" is kept and "conditon" is removed. All searched results are further picked and downloaded by humans ensure quality. Here is list of the filtered keywords for transformative aspect:
Transformative: Temporal
A young kitten image of Himalayan Cat young kitten image of Chartreux
A young kitten image of Burmese young kitten image Turkish Van
A young kitten image of American Shorthair young kitten image British Shorthair
A young kitten image of Maine Coon
young kitten image Burma (Myanmar)
A young kitten image of Selkirk Rex
young kitten image Siberian
young kitten image of Persian
young kitten image Manx
young kitten image Ocicat
A young kitten image of Russian Blue
young kitten image Bengal Cat
A young kitten image of Devon Rex
young kitten image American Bobtail
A young kitten image of Balinese
young kitten image LaPerm
A young kitten image of Egyptian Mau young kitten image Japanese Bobtail
A young kitten image of Ragdoll
young kitten image Abyssinian
A young kitten image of American Wirehair
A young kitten image of Oriental Shorthair
young kitten image of Cornish Rex
A young kitten image of Kurilian Bobtail
young kitten image Singapura Cat
A young kitten image of Birman young kitten image Burmilla
A young kitten image of Korat young kitten image Tonkinese
A young kitten image of Somali Cat
young kitten image Norwegian Forest Cat
A young kitten image of Turkish Angora
A young kitten image of Siamese
picture of Sainte- Sainte-Chapelle under construction
A picture of Washington Monument under construction
picture Hearst Castle under construction
A picture of Time Square under construction
picture of Wrigley Building under construction
A picture of Eiffel Tower under construction
picture The Arc Triomphe under construction
A picture of Golden Gate Bridge under construction
picture White House under construction
A picture of Palace of Versailles under construction
picture Opéra Garnier under construction
picture of San Simeon under construction
picture The Louvre under construction
picture of Cathédrale Notre-Dame de Paris under construction
A picture of Sacré-Coeus Basilica under construction
19

---

Published as a conference paper at ICLR 2025
A picture of Brooklyn Bridge under construction
picture Panthéon under construction
A picture of Capitol Building under construction
picture Independence Hall under construction
picture of Mont Saint-Michel under construction
A picture of St Patrick's Cathedral under construction
picture of Space Needle under construction
A picture of Château de Chambord under construction
picture of Versailles under construction
Transformative: Deformation
An image of Toyota Camry damaged
An image of Ford F-150 damaged
An image of Ferrari 458 damaged
An image of Audi Q5 damaged
An image of Lamborghini LP640 damaged
An image of McLaren 675LT damaged
An image of Mercedes SLC damaged
An image of Lamborghini Aventador damaged
An image of Lamborghini LP570 damaged
An image of Porsche 911 GT3 RS damaged
image of Audi A6 damaged
An image of Audi A4 damaged
An image of Lamborghini Aventador SV damaged
An image of GMC Sierra 2500 HD damaged
An image of Infiniti G37 damaged
An image of GMC Yukon damaged
An image of Honda Accord damaged
An image of Infiniti FX35 damaged
An image of Tesla Model damaged
An image of Acura RDX 2020 damaged
An image of BMW 7 Series damaged
image of Audi AS Sportback damaged
An image of Hyundai IX35 damaged
An image of Cadillac XTS damaged
An image of BMW M3 damaged
An image of Acura MDX damaged
An image of Audi A3 damaged
An image of BMW X3 damaged
An image of Porsche Boxster damaged
An image of Mercedes CLA45 AMG damaged
An image of Jaguar XJ damaged
Transformative: Incomplete
MacBook Keyboard missing keys
Windows Keyboard missing keys
Laptop Keyboards (Generic) missing keys
Mechanical Keyboard missing keys
Ergonomic Keyboard missing keys
Compact Keyboard missing keys
Gaming Keyboard missing keys
Chiclet Keyboard missing keys
Tenkeyless (TKL) Keyboard missing keys
Virtual Keyboard (On-screen) missing keys
Numeric Keypad missing keys
ISO Keyboard Layout missing keys
-NNI Keyboard Layout missing keys
Ortholinear Keyboard missing keys
Sinetooth/Wirelese Keyboard missing keys
20

---

Published as a conference paper at ICLR 2025
Transformative: Biological
An image of Lime after oxidation
An image of breadfruit after oxidation
An image of dragonfruit after oxidation
An image of starfruit after oxidation
An image of Raspberry after oxidation
An image of Zucchini after oxidation
An image of Pear after oxidation
An image of passionfruit after oxidation
An image of Blackberry after oxidation
An image of durian after oxidation
An image of persimmon after oxidation
An image of Apple after oxidation
An image of bell pepper after oxidation
An image of olive after oxidation
An image of Mango after oxidation
An image of nectarine after oxidation
An image of tomato after oxidation
An image of quince after oxidation
An image of coconut after oxidation
image of soursop after oxidation
An image of Kiwi after oxidation
An image of cucumber after oxidation
An image of apricot after oxidation
An image of Honeydew after oxidation
An image of Peach after oxidation
An image of pomegranate after oxidation
An image of carrot after oxidation
An image of fig after oxidation
An image of Papaya after oxidation
An image of Blueberry after oxidation
image of Banana after oxidation
An image of jackfruit after oxidation
An image of Lemon after oxidation
An image of tamarind after oxidation
image of lychee after oxidation
An image of Pineapple after oxidation
An image of Cantaloupe after oxidation
An image of Orange after oxidation
An image of Rambutan after oxidation
An image of guava after oxidation
An image of sweet potato after oxidation
An image of Plum after oxidation
An image of Avocado after oxidation
An image of Watermelon after oxidation
An image of potato after oxidation
image of Grapefruit after oxidation
An image of Grapes after oxidation
An image of pumpkin after oxidation
An image of Cherry after oxidation
An image of Strawberry after oxidation An image of custard apple after oxidation
Quality control We employ two types of quality control throughout the annotation process: an automatic check with predefined rules and manual examination of each instance. The automatic check verifies correct MCQA format in which each question should only have correct answer, metadata values, assesses image validity (checking the accessibility of each image) and filters out redundant images in the corpus (images that repetitively downloaded). The manual examination
21

---

Published as a conference paper at ICLR 2025
conducted by two experts in this field, who checked the correspondence between query images and ground tound-truthh image examples, and filtered or revised ambiguous questions and uncorrelated query image and ground-trute images.
A.2 HUMAN EVALUATION PROTOCOL
Three human annotators in domain conducted the human evaluation. The interface for human evaluation without RAG knowledge and with RAG knowledge are shown in Figure 6 and Figure
building construction?
Bitmore Ersate
Mureum Gardens
The Breakers
leant
Figure 6: Human evaluation interface without RAG examples
Figure 7: Human evaluation interface with ground-truth RAG examples
22

---

Published as a conference paper at ICLR 2025
B EXPERIMENT SETTING DETAILS
B.1 MODEL PROMPTS
Following Lu et al. (2024b) and Liu et al. (2023a) our prompt consists of four parts, the instruction, question, options, and prefix of the answer. For images, insert them into the text to form coherent prompt as the image placeholder ([Image]) indicated below. The complete prompt is as follows:
Model Prompts for No RAG Evaluation
Instruction: Answer with the option's 's letter from the given choices directly.
[Image}
Question: {QUESTION}
Choices:
(A) [OPTION_A}
(B) (OPTION_B)
(C) (OPTION_C)
(D) {OPTION D}
Answer:
Model Prompts for RAG Evaluation
Instruction: You will be given one question concerning several images. The first image is the input image, others are retrieved examples to help you. Answer with the option's 's letter from the given choices directly.
[Image]] [Image] H [Image] [Image)
Question: {QUESTION)
Choices:
(A) (OPTION_A)
(B) [OPTION_]]
(C) (OPTION C)
(D) (OPTION D
Answer:
B.2 EVALUATION TOOL
Following Lu et al. (2024b), we first use rule-based automatic tool to extract the exact answer. First, the tool detects valid option index appears the model output. If no direct answer is found, the tool matches the output to the content of each option. If there is still no match, we employ GPT-3.5-turbo to automatically extract the answer following our prompts in Table 5. GPT-3.5-turbo finds there is still match, we will randomly select option as the answer.
Prompt
Please read the following example. Then extract the multiple choice letter with the answer corresponding to the choice list from the model response and type it at the end of the prompt. You should only output either A, B, C, or D.
(In-context examples]
Question: {QUESTION)
Choice List: (A) (OPTION, A} (B) (OPTION B} (C) (OPTION C} (D) (OPTION D)
Model Response: [Response)
Extracted answer:
Table 5: Prompt template to extract multiple choice answer from model's response. (In-context examples) are in-context examples.
23

---

Published as a conference paper at ICLR 2025
C
MORE RESULTS
We present the Recall @5 scores per each scenarios on multimodal retreivers as shown Table and LLaVA-NNex-Ineeriawe'' accuracy score affected by these retrievers in Table 7.
Perspective
Transformative
Model
Overall
Others
Angle
Partial Scope
Occlusion
Temporal
Deformation Incomplete Biological
MagicLens
37.03
41.61
33.33
36.27
36.11
12.75
10.78
79.41
29.41
56.67
E5-V
54.92
49.69
48.78
61.76
66.67
38.93
22.55
73.53
71.57
82.50
VISTA
59.65
66.15
67.48
64.71
63.89
38.26
8.82
33.33
94.12
80.83
CLIP
60.46
70.19
54.47
71.57
73.15
44.30
31,37
67.65
40.2
81.67
Table 6: Recall @5 scores with 4 retriever models on MRAG-BENCIL
Perspective
Transformative
Model
Overall
Others
Angle
Partial Scope
Occlusion
Temporal
Deformation Incomplete Biological
MagicLens
35.18
34.78
29.67
30.39
34.26
40.94
36.27
27.45
49.02
39.17
E5-V
40.06
38.82
39.84
41.18
46.3
38.93
41.18
27.45
48.04
41.67
VISTA
42.42
40.37
35.77
40.2
52.78
45.64
42.16
36.27
50.98
48.33
CLIP
40.35
40.06
33.33
39.22
56.48
43.62
44.12
27.45
36.27
49.17
Table 7: LLaVA-Next-Interleav accuracy scores on TIRAG-BENCHE with 4 different retrievers.
24


Reviews:
[{"review_id": "hELPijXhyF", "reviewer": "Reviewer_U78R", "summary": "The paper presents a Multimodal Retrieval-Augmented Generation Benchmark (MRAG-Bench) to analyze and identify scenarios where visually augmented knowledge is better than textual one. MRAG-Bench comprises 16,130 images and 1,353 human-annotated multiple-choice questions, covering nine distinct scenarios. The authors analyze the performance of 10 open-source and 4 proprietary large vision-language models (LVLMs). Their results show that all the LVLMs exhibit performance improvement when augmented with visual data rather than textual. Moreover, the proprietary models can better leverage the retrieved visual information.", "questions": "1. Could you elaborate on your motivation for creating this benchmark? Please include examples of real-world applications where this benchmark could offer substantial value.\n\n2. Using this benchmark in real-world applications could be challenging, as it would require a comprehensive image repository encompassing a wide range of objects globally. This need introduces various limitations, such as legal and computational restrictions and the necessity for thorough curation to prevent biased or intrusive content. Could you discuss potential strategies for making this benchmark feasible in real-world scenarios, as well as approaches to mitigate these limitations?\n\n3. Increasing context (whether visual or textual) means that large language models (LLMs) would need to process longer input sequences, which would raise inference times. Since images are typically represented as lengthy sequences, adding more images could substantially impact both inference time and associated costs, especially in the case of proprietary models, which gain the most from visual augmentation. Could you provide a detailed analysis of the trade-offs between performance gains, increased costs, and extended inference times?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper is well-structured and accessible, presenting a benchmark for evaluating and analyzing the relevance of visually augmented samples across various scenarios. For example, it considers cases where the main object in an image is partially occluded or where the image depicts an outdated or modified version of the object. The authors conduct extensive experiments with both proprietary and open-source methods, demonstrating that visual augmentation offers greater utility than textual information for large vision-language models (LVLMs).", "weaknesses": "While the paper is easy to follow, the primary motivation behind the task remains unclear. In a real-world scenario, how would this pipeline function operationally? Given the findings that visually augmented samples offer significant value, an extensive repository of curated and labeled images for retrieval would be essential. However, this may be impractical due to legal and computational restrictions. Additionally, I would be interested in analyzing how the incorporation of augmented images affects response time and costs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p91YFaAR9N", "reviewer": "Reviewer_Pyuc", "summary": "This paper introduces a benchmark (MRAG-BENCH) designed to evaluate the performance of large vision-language models (LVLMs) in scenarios where visual information retrieval is more beneficial than textual data retrieval. First, this paper establishes the first benchmark focusing on visual information superiority over textual data. Second, this paper conducts an extensive analysis of LVLMs, and makes detailed analyses. Third, this paper points out a way that LVLMs could be evaluated and improved by emphasizing the importance of visual information in retrieval-augmented multimodal tasks.", "questions": "1. How to conduct the process of human evaluation? For example, how many annotators were involved in evaluating each question, and what measures were taken to ensure consistency and accuracy in their assessments?\n2. In Table 3, there are two human evaluation settings. How do you ensure that human annotators do not rely on their own knowledge when answering questions, and what steps have you taken to mitigate any potential biases in the human evaluation process\n3. Whether retrieving additional visual information also enhance performance in text-intensive scenarios? \n4. The paper does not report the performance of GPT-o1 on MRAG-BENCH. How does GPT-o1 perform on this benchmark?\n5. In Section 4.3, the paper shows that further increasing the number of retrieved images does not improve performance. Could the authors explain the reasons behind this effect, and what insights does this provide for future research on retrieval-augmented multimodal models?\n6. Will you release this benchmark?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The paper offers a different visual-centric perspective and addresses a gap in the current text-centric benchmarks.\n3. The comprehensive evaluation of multiple LVLMs provides valuable insights, such as how visual knowledge benefits LVLMs over textual knowledge, and the examination of retriever performance and its impact on LVLM accuracy.", "weaknesses": "1. The process of selecting ground-truth images and query images may introduce bias if not carefully controlled. The paper could be strengthened by discussing potential biases in the data collection process and how they were mitigated.\n2. The paper focuses on a specific set of scenarios, which may not scale well to other types of visual queries or more complex tasks. For example, what if there are many objects in the question? \n3. The paper could further discuss the implications of the findings, such as how they might influence the development of future LVLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pOzvAtEYFf", "reviewer": "Reviewer_XA2r", "summary": "This paper introduces MRAG-BENCH, a multimodal retrieval-augmented generation benchmark comprising 16,130 images and 1,353 human-annotated multiple-choice questions across 9 distinct scenarios. The authors evaluate the performance of 10 open-source and 4 proprietary large vision-language models (LVLMs) on this benchmark.", "questions": "- What is the meaning of \"Unique number\" in Table 2?\n\n- How were the nine scenarios in MRAG-BENCH selected?\n\n- Regarding the \"Others\" example in Figure 3, why is option A marked as the correct answer? This raises questions about the rule for obtaining question-answer pairs. How is the correct answer determined for each question?\n\n- Could the authors clarify how CLIP is used for text retrieval? Is it a direct approach based on finding the sentence with the highest similarity to the current question, or does it involve a method similar to INFOSEEK [1], where visual entity recognition is performed first, followed by retrieval?\n\n- Concerning Table 3:\n\n  - If human performance on this task is low, how can the quality of GT images be assured during data collection?\n\n  - How is \"Human performance\" in Table 3 measured?\n\n  - Most open-source LVLMs exhibit performance declines when using Retrieved RAG, suggesting that retrieval effectiveness is crucial to answer quality. If retrieval consistently underperforms, is the entire process reasonable and valid?\n\n- Concerning Table 4:\n\n  - How were the five comparison methods implemented?\n\n  - What is the meaning of GT Text RAG, and how is GT Text collected? Additionally, it seems counterintuitive that LLaVA-Next-Interleave performs worse with GT Text RAG. If adding GT Text RAG degrades model performance, does this suggest that the quality of GT Text might be low?\n\n  - Why does the performance improvement for LLaVA-Next-Interleave, when provided with both GT images and textual knowledge, fall short compared to using GT images alone?\n\n- There appears to be a minor error in Section 4.1, where it states, \"LLaVA-Next-Interleave exhibited an 11.09% improvement with image knowledge over text knowledge.\" Shouldn't this be 11.90%?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The authors conduct an investigation into the capability of models to retrieve and utilize external visual knowledge for question answering, which is a crucial and direct exploration within the LVLM field.\n\n- The paper introduces a comprehensive multimodal retrieval-augmented generation benchmark that provides a foundation for the community to further explore and advance in this area.\n\n- Figure 1 is well-designed, with well-chosen examples to illustrate the benchmark's scenarios and model evaluations effectively.", "weaknesses": "- The data collection process is unclear. Adding a figure to illustrate the entire process could clarify the methodology, especially regarding the acquisition of ground truth (GT) text and the source of question-answer pairs.\n\n- The experimental setup lacks sufficient detail. The authors should provide a complete subsection dedicated to describing the experimental setup, placed within Section 3, instead of scattering brief descriptions across various sections. This current layout diminishes the reliability of the experimental conclusions. For example, in Table 3 (Retrieved RAG), it’s unclear which model was used as the retriever, what the retriever’s input and output were, etc. Additional concerns regarding the experimental setup are outlined in the Questions section below.\n\n- The analysis of the question, \"How much does visual knowledge provide additional benefits over textual knowledge?\" is insufficiently supported. Additional experimental analysis would strengthen the authors’ conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sMy6qbwH6C", "reviewer": "Reviewer_tC9p", "summary": "- **Introduction of MRAG-BENCH**: The paper introduces **MRAG-BENCH**, a benchmark designed for evaluating retrieval-augmented multimodal models, focusing on scenarios where visual information is more beneficial than textual data. MRAG-BENCH consists of **16,130 images** and **1,353 human-annotated multiple-choice questions** across **9 distinct scenarios**.\n\n- **Evaluation and Findings**: The paper evaluates **10 open-source** and **4 proprietary large vision-language models (LVLMs)**, showing that models exhibit greater improvements when augmented with images compared to textual knowledge.\n\n- **Insights and Challenges**: The findings highlight the challenges LVLMs face in effectively leveraging retrieved visual knowledge, emphasizing the need for improved utilization of visually augmented information.", "questions": "Please see weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Originality\n- **New Benchmark**: The introduction of MRAG-BENCH is a significant contribution, focusing on vision-centric retrieval-augmented generation, which is less explored compared to text-centric benchmarks. The paper identifies and categorizes scenarios where visual knowledge is more beneficial than textual knowledge, such as perspective and transformative changes, which is a fresh approach in the field.\n\n### Quality\n- **Comprehensive Evaluation**: The paper evaluates 10 open-source and 4 proprietary LVLMs using a large dataset of 16,130 images and 1,353 questions, providing robust and extensive analysis.\n- **Detailed Analysis**: The results include fine-grained performance metrics across different scenarios, highlighting the strengths and weaknesses of various models.\n\n### Clarity\n- **Well-Structured**: The paper is well-organized, with clear sections detailing the benchmark's design, data collection, and evaluation methodology.\n- **Illustrative Examples**: Figures and tables, such as the scenario distributions and qualitative examples, effectively illustrate the concepts and findings, making the paper easy to follow.\n\nOverall, the paper makes a substantial contribution to the field of multimodal models by introducing a novel benchmark that highlights the importance of visual knowledge, backed by thorough evaluation and clear presentation.", "weaknesses": "- The motivation is unclear. In Figure 1, do the authors mean that retrieving from the image corpus is not difficult? While I agree that retrieving correct textual knowledge is challenging, why is retrieving from the image corpus considered easier? Additionally, I do not understand why retrieved correct textual knowledge is deemed not useful, which is confusing[^1^][1]. Please clearly state your motivation.\n\n- How is the image knowledge database handled in the experiments? It is unclear how the authors retrieve the images.\n\n- In my view, visual RAG is more difficult than text RAG. It would be valuable to explore CoT + Visual RAG. For example, in Figure 1, if VLLMs are unsure about the model of the car, they could consider some candidate answers and then use Visual RAG to validate and confirm the final answer. Directly retrieving images is not easy with a large image knowledge database and may sometimes hurt performance when incorrect images are selected.\n\n- I cannot understand why GT Image RAG is better than GT Text RAG in Table 4. Can the authors provide more clarifications and illustrations?\n\nI recommend that authors submit a link to the anonymous repo. It is very important for benchmark to release ASAP. After that, reviewers can check the data quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vs4LlG18Ao", "reviewer": "Reviewer_JirS", "summary": "The authors propose a new multimodal retrieval-augmented generation benchmark, MRAG-BENCH. In this benchmark the LVLMs are given the input of query image, textual question and retrieved relevant images, and required to answer the multiple choice questions. Evaluations on MRAG-BENCH show the gap between current leading LVLMs and human beings. The authors also have an interesting findings that current open-source LVLMs lack the ability to select useful information from the retrieved images while ignore the irrelevant or misleading retrieved images.", "questions": "Suggestions: Maybe the authors can fix the ground truth images and add retrieved images from different retrieval systems (i.e. search engine, different multimodal knowledge graph), so that this benchmark can also be used to evaluate multimodal retrieval systems.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Multimodal RAG is surely an important problem, along the improvement of multi-image understanding ability in current LVLMs. Many cases in the benchmark are close to real world scenarios. The settings of noisy retrieved information is very meaningful and necessary, since in real world multimodal RAG, errors or failures in the retrieval process are inevitable.", "weaknesses": "Some cases in the benchmark might be too ideal for current retrieval systems, for example ,the Angle case and the Deformation case in Figure3. It is unlikely that a multimodal retrieval system (i.e., search engine, multimodal knowledge graph) can retrieve such accurate ground truth image given the query image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)