Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent
Authors: Yangning Li
Yinghui Li
Xinyu Wang
Yong Jiang
Zhen Zhang
Xinran Zheng
Hui Wang
Hai-Tao Zheng
Fei Huang
Jingren Zhou
Philip S. Yu
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model
Multimodal Retrieval Augmented Generation
Knowledge Enhancement
Primary area: generative models

Abstract: Multimodal Retrieval Augmented Generation (mRAG) plays an important role in mitigating the “hallucination” issue inherent in multimodal large language models (MLLMs). Although promising, existing heuristic mRAGs typically predefined fixed retrieval processes, which causes two issues: (1) Non-adaptive Retrieval Queries. (2) Overloaded Retrieval Queries. However, these flaws cannot be adequately reflected by current knowledge-seeking visual question answering (VQA) datasets, since the most required knowledge can be readily obtained with a standard two-step retrieval. To bridge the dataset gap, we first construct Dyn-VQA dataset, consisting of three types of ``dynamic'' questions, which require complex knowledge retrieval strategies variable in query, tool, and time: (1) Questions with rapidly changing answers. (2) Questions requiring multi-modal knowledge. (3) Multi-hop questions. Experiments on Dyn-VQA reveal that existing heuristic mRAGs struggle to provide sufficient and precisely relevant knowledge for dynamic questions due to their rigid retrieval processes. Hence, we further propose the first self-adaptive planning agent for multimodal retrieval, **OmniSearch**. The underlying idea is to emulate the human behavior in question solution which dynamically decomposes complex multimodal questions into sub-question chains with retrieval action. Extensive experiments prove the effectiveness of our OmniSearch, also provide direction for advancing mRAG. Code and dataset will be open-sourced.

Published as a conference paper at ICLR 2025
BENCHMARKING MULTIMODAL RETRIEVAL AUG- MENTED GENERATION WITH DYNAMIC VQA DATASET AND SELF-ADAPTIVE PLANNING AGENT
Yangning L Yinghui Li"", Xinyu Wang", Yong Jiang Zhen Zhang', Xinran Zheng'
Hui Wang", Hai-Tao Zheng!:2;, Fei Huang', Jingren Zhou', Philip S. Yu
'Shenzhen International Graduate School, Tsinghua University Peng Cheng Laboratory TTongyi Lab, Alibaba Group Univeessiy College London University of Illinois Chicago
ABSTRACT
Multimodal Retrieval Augmented Generation (mRAG) plays an important role mitigating the "hallucination" issue inherent in multimodal large language models (MLLMs). Although promising, existing heuristic mRAGS typically predefined fixed retrieval processes, which causes two issues: (1) Non-adaptive Retrieval Queries. (2) Overloaded Retrieval Queries. However, these flaws cannot be ade- quately reflected by current knowledge-seekinn datasets, since the most required knowledge can be readily obtained with a stan- dard two-step retrieval. To bridge the dataset gap, we first construct Dyn-VQA dataset, consisting of three types of "dynamic" questions, which require complex knowledge retrieval strategies variable in query, tool, and time: (1) Questions with rapidly changing answers. (2) Questions requiring multi-modal knowledge. (3) Multi-hop questions. Experiments on Dyn-VQA reveal that existing heuris- tic mR ARAGs struggle to provide sufficient and precisely relevant knowledge for dynamic questions due to their rigid retrieval processes. Hence, we further pro- pose the first self-adaptive planning agent for multimodal retrieval, OmniSearch. The underlying idea is to emulate the human behavior in question solution which dynamically decomposes complex multimodal questions into sub-question chains with retrieval action. Extensive experiments prove the effectiveness of our Om- niSearch, also provide direction for advancing mRAG.
visual question answering (VQA)
INTRODUCTION
Multimodal Retrieval Augmented Generation (mRAG) (Zhao et al. 2024] 2023 Gao et al.
Versafile Retriever
2023) aims to provide more comprehensive, ac- curate and up-to-date knowledge from external sources for Al systems. has emerged as key
O
Planning Agen
Tangke e
Road retriccd information...
technology to mitigate the "hallucination" issue
(Liu et al. 2024a Bai et al., 2024) inherent in
multimodal large language models (MLLMs).
Sub-Question Solver
Fing Jix
Existing heuristic mRAG methods typically predefined fixed retrieval processes that ground all modalities into one primary modality (usu- ally text), then retrieve for single time. Despite the promising results, these retrieval strategies suffer from two issues: (1) Non- adaptive Retrieval Queries refer to the fixed
grasoddaoors moor??
Image Text Tools
Text Retriwer
Figure 1: Bottom: Heuristic mRAG based VQA. Upper: OmniSearch based VQA.
retrieval processes and query structures of heuristic mRAG methods. These inflexible retrieval
strategies fail to adapt to evolving contexts or intermediate findings within a question. hindering the model from re-retrieving to further comprehend, verify, or rethink the question. For example, in
Eqqual Contribution. *Corresponding Author. This work was led by Xinyu Wang Alibaba Group. The code and dataset is open-sourced httpthtttt/tttt com/A AT baba- NI.P/Omn Search

---

Published as a conference paper at ICLR 2025
Common Two-hop Question in provious VQA datasets
Dynamic Questions in Dyn-VQA
Question with Rapidly Changing Anowers
Oantiia: Wikt lactee Eike?
GT Areoi Aakea Buck.s
OT Aususe Sad Lile Thoue (D00g)
tnage
Deninaa Fleet Church
Oltin Marphy
Search: Cilinn Harphy Lotezt
[i]
Opporkaliar [R] Smadl Things Like These, hes
Antario Bortact
Deliger
Tuly 11, 8021
Things
15 Fibruarry 2534
bi Question Roquiring Multinodal Knowledge
<)
MMMutd-bpp Question
Quuste:: What sbruct is lcas al toun?
Queriia: Ma3ic an than grosed ment
lipal Truge
Swage
laage
Bryant
Ia. lang Sien
Win Seards Kobs Bryast rbs har
Web Searth In ling 'Nka Rox deliin
kelken
Uniige dearch with lestaal Ouary LA Lokara icee
Web Search Ting Shes a 'atal Box off approincelly
Figure 2: Dynamic VQA examples that require different kinds of retrieval strategies.
Figure 2 question (a) asks, "What is his (Cillian Murphy's s) latest film?" A fixed retrieval process returns multiple relevant films, but heuristic methods fail to construct further retrieval based on the retrieved content to distinguish between the sequence of different films. (2) Overloaded Retrieval Queries refer to heuristic retrieval methods refer to heuristic mRAG methods merely format a sin- gle query by concatenating textual descriptions of objects in images with input questions. A single query carries multiple retrieval aspects, leading to ambiguous retrieval and influx of superficially relevant knowledge yet not essential to the question solving. For example, in Figure 2 question (c) asks, "Which one of them (two actors, Ling Jia, and Teng Shen) grossed more?" Heuristic methods might generate a single query like "Ling Jia, Teng Shen, Which one of them grossed more?", which contains the intent to retrieve box office information for both actors. This mixed query conversely fails to provide precise knowledge for each individual aspect. Therefore, as shown in Figure [ when faced with real-world questions requiring complex knowledge, current heuristic mR methods fail to provide sufficient and precise knowledge, due to their rigidity issues.
Unfortunately, although several knowledge-seeking visual question answering (VQA) datasets (Chen et al.] [023] Schwenk et al. 2022) are widely utilized as mRAG benchmarks, they cannot adequately reflect the rigidity issues of heuristic mRAGs in acquiring complex knowledge. Since most questions in them merely require textual knowledge within two-hop, which can be readily ob- tained by heuristic mRAGs with standard two-step retrieval process. As illustrated in the upper left of Figure 2 the most common type of question inquires about a certain property of an object.
To bridge the mRAG dataset gap, first construct a challenging dataset, Dyn-VQA, comprising 1, 1,452 dynamic questions that require complex multimodal knowledge retrieval for solution. Dy- namic questions are defined as questions that require the model flexibly provide knowledge retrieval solutions, where the query, tool, and time of retrievals are all variable. These questions cannot be solved by predefined retrieval process. Concretely, there types of dynamic questions are included in VQA: (1) Questions with rapidly changing answers. Since the context knowl- edge of such question updates frequently, the retrieved content may be mixed with outdated and newer knowledge that is difficult to distinguish. This requires mARG methods to flexibly plan addi- tional retrievals based on feedback from current retrieved content for further comprehension, rather than merely one-time retrieval. (2) Questions requiring multi-modal knowledge. The knowl- edge necessary by Dyn-VQA spans various modalities. This demands that mRAG methods retrieve knowledge across diverse modalities with tailored retrieval APIs, differing from most VQA datasets limited in seeking textual knowledge with multimodal questions. (3) Multi-hop questions. Ques- tions in Dyn- VQA necessitate varied reasoning hops for solution, which entails that mRAG methods conduct various retrieval steps. While existing VQA datasets primarily focus on two-hop question, i.e., identifying visual concepts via text and then answering single-hop textual question.
We evaluated the performance of various mRAG methods combined with leading MILMS on Dyn- VQA. Experiments reveal that existing heuristic mRAGs struggle to provide sufficient and precisely relevant knowledge for dynamic questions in Dyn-VQA, due to their rigid retrieval processes.
To address these issues, we further propose a self-adaptive planning agent for multimodal retrieval, OmniSearch" The underlying idea is to emulate the human behavior in question solution which dynamically decomposes complex multimodal questions into sub-question chains with retrieval ac-
We aim for OmniSearch achieve Omnipotent Multimodal Search, solving real-world issues in future.
2

---

Published as a conference paper at ICLR 2025
tion. At each step. OmniSearch flexibly adjusts the next action according to question-solving state and retrieved content, with diverse purposes such as deepening comprehension of retrieved con- tent, modifying retrieval method for current sub-question, proposing the next sub-question, etc. It is noteworthy that OmniSearch can serve as a plug-and-play RAG module, cooperating with arbi- trary MILMS augment their capability in addressing complex dynamic questions. Two different versions of OmniSearch are developed based on closed GPT-4V (Achiam et al. (023) and open-source Owen-VL-Chat (Bai al. (023a)), respectively. As far we know, OmniSearch the first multimodal retrieval agent for VQA tasks with self- adffaadative planning and scalable submodules.
In summary, our main contributions are fourfold:
We reveal that current VQA- mRAG benchmarks don't 't reflect real-world needs for dy- namic knowledge retrieval and propose the Dyn-VQA VQA dataset with three types of dynamic questions.
We benchmark various mRAG methods with leading MLLMs on Dyn- VQA, demonstrating their flaw in providing sufficient and relevant knowledge for dynamic questions.
We propose OmniSearch, a self-adaptive retrieval agent that plans each retrieval action in real- time according to question solution stage and current retrieval content.
Extensive experiments prove the effectiveness of our OmniSearch. Detailed analyses ducted to provide direction for advancing mRAG.
2
RELATED WORKS
2.1
MULTIMODAL RETRIEVAL AUGMENTED GENERATION
The mRAG methods ZZao et al. 2023 2024
Gao et al. 2023) aim to provide MLLMS (Lu al.
2024b Ye et al.] 2024| Liu et al., 2024b Du et al. 2022 Chai et al. 2024a) with more compre- hensive, accurate and up-to-date world knowledge from external sources (Wu et al. 2024 Ji al.,
2024 Hou et al. 2024). They have been empirically proven to be effective on various VQA datasets, which can be categorized into two classes based on the retrieval method.
One category employs visual encoding model for direct image representation, and then retrieves the knowledge from knowledge base with the highest feature similarity. For instance, KAT (Gui et al.]
2022) and Revive ([.in et al. 2022) both use the image encoder of CLIIP (Radford et al. 2021) I) for retrieval. The other category (Hu et al. 2023: Yang et al. 2022 Lin et al.) 2024) first transforms the input image into textual representation utilizing off-the-shelf tools and then performs text retrieval. For example, RA-VQA (Lin & Byrne 2022) and RA-VQA-V (Lin et al., 2024) first use existing object detection and image captioning models convert images to text, and than perform dense passage retrieval to fetch relevant text documents. Several studies Yaoo et al. 2023 Xu et al.) 2023) have preliminarily explored into agentic RAG, but their primary focus was the text domain. Chen et al. (2024a) proposed a causality-enhi enhanncdd agent framework focus on unimodal biases in MLLMs, while lacks plug-anddplay capabilities.
The purpose of OmniSearch aligns with previous works to furnish pertinent and accurate knowledge for MLLMs, but diverging in three aspects: (I) OmniSearch plans multiple retrieval actions for the each question with diverse retrieval tools, supplementing the missing knowledge of each modal- ity. (2) OmniSearch dynamically adjusts subsequent retrieval actions based on retrieved content, diverging from methods formulate query solely with input questions and images, (3) OmniSearch's 's retrieval scope extends to the entire Internet, offering intricate yet more comprehensive knowledge.
DYN-VQA DATASET
In this section, we curated a novel dataset, Dyn-VQA, designed to evaluate the performance of existing mRAG methods in addressing questions requiring dynamic retrieval.
3.1 DYN CONSTRUCTION
To guarantee the dataset quality, we explain the overall goal of the dataset to the annotators, who all professional AI researchers. A straightforward construction strategy might directly request
3

---

Published as a conference paper at ICLR 2025
the annotators to write more visual questions after showing them the various types of VQA cases in Figure 2 However, in the preliminary annotation, we found that this overloaded single-step strategy is quite impractical. Annotators often found themselves in dilemma of searching an image first, then laboriously devising a corresponding question while keeping various criteria, e.g., changing speed, and reasoning steps in their mind. Therefore, we optimize the strategy and divide them into three steps:
Step 1. Textual Question Writing. First, annotators are required to craft textual questions and categorize them using a three-dimensional schema based on answer update frequency (fast, slow, never), whether requiring external visual knowledge (yes, no), and reasoning steps (S 2-hop, 2- hop). The annotation of fast or slow is determined by whether the updating occurs on yearly basis. Whether seeking visual knowledge beyond input images is also considered to separate questions emphasizing different modalities. Meanwhile, multi-hop questions are delineated by a 2-hop cutoff, as previous VQA datasets typically focus on 2-hop. Annotators are prompted to compose questions incorporating newly emerged concepts from the past six months. The annotation difficulty significantly reduced since visual information is not considered. Besides, English QA instances from FreshQA (Vu et al. 2023) are also included.
Step 2. Multimodal Rewriting. The annotator transforms textual questions from the first step into multimodal ones by replacing visual concepts with ro-references (c.g.. "Kobe Bryant" to "this player") and pairing the revised question with a relevant image found on Google. Images sourced from commonly used pre-trained corpus such as Wikipedia and Baidu Encyclopedia are forbidden.
Step 3. Chinese-English Translation. This step aligns Chinese and English parts of the Dyn- VQA for side-by-side language comparison. Chinese and English VQA instances are translated into each other using Google Translate API, followed by manual checks and corrections to guarantee accuracy, especially for proper nouns. Instances that are intractable to translate or not applicable to Chinese/English contexts are filtered. Additionally, each question is annotated with the golden query, which simplifies the question by focusing solely on the last-hop question. This means that references to visual concepts and complex intermediate reasoning are omitted from the questions.
3.2 DYN-VQA ANALYSTS
Table I: Comparison of seoclndge-scckkng VQA datasets.
Dataset
Kaoo Iodge Type
Ama. Change Freq.
Remoning
Step
Exterual Visual
Human Annotation
(L.Q.A)
Lea, of
Q/A
Lang.
Coust.
Year
3 et 2017
nexer changs
S 3-hop
X
614K
6.21.11
2017
fackid
nexes change
24hop
x
14K
5.11.3
SIVQ4 Jam al X01
fackeid
never change
2-hop
7.5K
12.923
m
2001
WQUAE
fixed
change
X
X
3.6%
10,924
O.VVVA
S.hheakkcoo cral
commerefastoi
neves change
24hp
X
24.9K
8.8/1.3
InfoSeck
Ches et al. 2003
fised lib
never change
2 hop
X
1355
9.0016
2023
Dya-VQQ
world
fastiSooiiever change
hop
V
12,5/4.
2b/ea
Dataset Statisties Dyn-VQA is the first dataset specifically proposed for assessing the efficacy of mRAG systems. As shown in Table 2] it contains 1.55 questions in 9 domains, covering 3 types of question requiring complex dynamic retrieval: questions with rapidly changing answers, questions requiring multi-modal knowledge, multi-hop ques- tions.
Table 2: Statistics of VQA.
Stutistic
Number
Total Questions
1452
Domain
English questions
715 (18.2%%
Chinee questions
737 (50.8%)
Questions with fast updating
385 (26.5%)
&& requiring > 2-hop rensoning
(7.7%)
& requiring external visual knowledge
178 (12.3%%)
Questions with slow updating answers
494 (34.05))
Questions with never updaring answers
573 (39.5%
Comparative analysis between Dyn-VQA and other knowledge-seeking VQA datasets is also presented in Table [ It is evident that while other datasets also emphasize the necessity of external knowledge in the question solving, their knowledge typically pertains to only one category.
Questions requiring
reasening
387 (26.798
&& requiring external visual knowlodge Questions requiring S 2-hop reasoning
237
1065 (73.3%%
Questions requiring extemal visual knowledgee Questions requiring external visual knowledge
865 (59.6%)
587 (40.4%)
Average question longth
Max question length
contrast, each question in DQA originates from the real world, encom- passing a broader and more heterogeneous range of knowledge types, and featuring more open-ended answer styles. Reflecting in the length of ques- tions and answers, Dyn- VQA exhibits the longest entries compared to other datasets. Furthermore, Dyn-VQA employs more systematic question categorization schema, ensuring its challenging.
Average answer length
4.3
Max length
4

---

Published as a conference paper at ICLR 2025
Table 3: Human performances different VQA datasets.
Search Count
Dataset
Performance
Image Search with I. I.
Image Search with Q.
Web Search
VQAv2
0.05
74.31
A. O.OKVQA
0.18
0.06
60.19
InfoSeck
0.87
0.75
63.67
Dya-VQA
55.12
Unlike other datasets, which are primarily constructed through templated and automated processes, Dyn-VQA is meticulously curated by humans and requires ongoing human input to maintain the dataset with dynamically updated answers. Consequently, while Dyn- VQA may not match other datasets in scale, it far surpasses them in terms of quality, difficulty, and the cost of each instance. More details of Dyn-VQA can be found in Appendix.
Dataset Difficulty Questions in Dyn- VQA require more complex external knowledge, whose trieval process is not fixed. The inherent dynamism of Dyn-VQQ naturally ensures its difficulty. To illustrate more intuitively, we also conducted quantitative comparison of human performance on different datasets. As shown in Table 3] the questions in existing VQA datasets can typically be resolved within two reasoning steps, resulting in average search count of less than 2 per question. Besides, image search with textual query is not performed at all in other datasets, since the question therein do not require additional visual knowledge except the textual description of the object in image. In terms of overall accuracy, humans achieved the lowest performance on Dyn-VQA further demonstrating the challenges it presents.
4 RETRIEVAL BASELINES AND OMNISEARCH
In this section, establish several heuristic mRAG baselines and develop our OmniSearch to ad- dress dynamic questions that require complex retrieval. Retrieval tools in all methods are Google- based, including web search (textual web content retrieval with textual queries), image search with input images, and image search with textual query.
4.1 BASELINES
Single-hop heuristic mRAG baselines. The first heuristic baseline is to retrieve images with input images, which provides similar images alongside descriptive captions. This method augments MLLMs with visual knowledge about the objects depicted in the imput images. Similarly, the second heuristic baseline conducts web search with input textual questions and provides MLLMs with the top-k retrieved content as suppiementary knowledge. It is acknowledged that these two methods may not furnish precise supportive knowledge, since them only leverage partial modality of the input question as search query. Nonetheless, these baselines are still established to explore the benefits from wwwecrossmmoallii retrieval.
Two-hop heuristic mRAG baselines. Typically, existing mRAG methods can be generalized into two primary steps: first, converting the visual concepts in input images into text, and then combining the obtained text with input questions as search queries to retrieve related textual knowledge. For the first step, we employed the caption of the top-1 image retrieved by the input image and the output of the image caption model as textual representations of the input images, respectively. Subse- quently, the web search API is utilized to extract relevant knowledge from the Internet. Compared to single-step mRAG methods, two-step mRAG methods provide knowledge more precisely related to questions. However, they still encounter the limitation of a fixed retrieval process.
Estimated Upper Bound. We estimate the upper bound of mRAGs by the annotation of golden query, which simplifies the question to a last-hop question. Depending on whether the question requires external visual knowledge, either web search or image search API is called. We believe that in this simplified setting, the heuristic mRAG method can provide accurate information to MLLMs.
Descriptions of other baselines are available Appendix.
5

---

Published as a conference paper at ICLR 2025
Plansing
-I.LU>
Saaarh Hamiia ipprinngg
Artion hiloqunica -Sq-
(ydand FeeBbck +1
Cumfoors Whot
Bartiral
Quer <go.
Prvelook
Cearthc Text Karm SUT
Flxal Aucnee
port ane der Jhank
Mair inddal Hetriner
qul--quetion Salver
MILLM
Figure 3: Left: overall framework of OmniSearch. Right: running example of OmniSearch.
4.2 OMNISEARCH FRAMEWORK
Heuristic mRAG encounter the issues posed by rigid retrieval processes. A more flexible model that can plan the retrieval strategy tailored to the specific question and dynamically adjusts it throughout the process is desired. Therefore, we propose self-routing framework called OmniScarch for trieval planning. The underlying idea behind is to mimic the human of incrementally decomposing the complex question into sequence of solution actions. As illustrated in Figure 3l overall frame- work consists of three module. The planning agent is the core module that formulates sub-questions and plans the subsequent retrieval action based on real-world feedback (i.e., retrieved content or solver output). Actual retrieval action is execute by the retriever. Then, sub-question solver gener- ates feedback on sub-question based on the retrieval content and update it to the planner.
Planning Agent. Each planned action comprises four critical part: self-thought <ST> sub-question <SQ>, retrieval API <R>, and API query <Q> In each step, planning agent comprehends the given question and the real-world feedback in self-thought, then carefully determines the follow-up sub-question to tackle. Meanwhile, different retrieval APIs with query are invoked, depending on the knowledge type required for the sub-question. In a manner akin to human cognitive processes during question-solving, planning agent autonomously generates various potential actions, includ- ing: posing additional question to clarify ambiguous conflicting parts of the retrieved content; refining the retrieval query to acquire more supplementary knowledge: modifying the phrasing of sub-questions: verifying the response to the current sub-question; presenting the next sub sub-questions; concluding the final answer, etc.
Retriever executes actual retrieval operations. In our experiments, web search, image search with text and image retrieval with images are included. More retrieval tools can be considered in future.
Sub-question Solver summarizes the retrieved content and endeavors to address the sub-question accordingly. The feedback generated by the solver is then provided to the agent. Notably, the solver is compatible with arbitrary MILMS even the planning agent itself, i.e., directly returning the retrieved information to the planner. Depending on computational resources, MLLMs with larger or smaller sizes can be employed. Such pluggable and scalable feature is desired in industrial scenarios, which facilitates the flexible control of GPU cost.
The above process is fully automatic, with OmniSearch performing iterative actions until it believes it has retrieved sufficient knowledge to output final answer. We trained two versions of OmniSearch based on different MLLMs: proprietary GPT-4V and open-source Qwen-VL-Chan VL-Chat. For GPT-4V, we employed prompt engineering to stimulate its dynamic planning and decision-making capabilities. To facilitate Qwen-Vi--Cha's ability to plan and utilize retrieval APIs, we constructed a retrieval API training dataset, leveraging the GPT-4V synthetic data and the existing Infoseck dataset. We train the MLLM in a multi-round conversation mode. Additionally, general instruction data is also utilized to keep the general conversation capability of trained Agent.
The proposal of OmniSearch is inspired by from it essentially. The fundamental distinction between OmniSearch, multimodal agent, and CoT is its ability to utilize tools, interact with the environment, and response to the environment Zhang et al. (2023). In contrast, CoT methods primarily stimulate the model's inherent logical
(CoT) Wei et al. (022). but differs

---

Published as a conference paper at ICLR 2025
reasoning capabilities through prompts. The CoT approach is unable decouple intermediate pro- cesses, therefore can not be integrated with retrieval tools.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Backbone MILMs for heuristic mRAGs. We select several advanced MLLMs backbone model to show their effectiveness equipped with heuristic mRAGs. Qwen-VL-7B-Chan is large vi- sual language model with strong visual and text recognition abilities proposed by Bai al. (20236). GPT-4V and Dwen-VIL-Mayy are show how the larger MLLMs will affect the results. Additionally, Qwen-7B-Chat included in our experiments, which is a text-only LLM. We use this model to assess how multimodal RAG can solve the visual problem for textual LLM.
are also selected as representatives the closed-source models to
Evaluation Metric. The automated metric FI-Recall is utilized as the evaluation metric. We calculate the ratio common tokens between medee-generated responses and ground truth. Specif- ically, we first segment the generated text and golden text into token lists using word segmentation tools") then calculate the ratio of tokens generated by models belonging to the golden token list.
5.2
MAIN RESULTS
Table 4: Main results on Dyn.VQA.
Answer Update Frequency
Reasening Steps
Seeai-Seeking
Language
Model
all
fast
slow
never
S 2-hop 2-hop
no
yes
en
Original (M)LLMs
Owen VL- Chat
13.69
17.27
Qwen- B Chat
5.63
7,86
15.97
10.48
10.43
11.05
10.08
10.48
10:46
10.47
Qwen-VL-Man VL-Max
15.11
30.44
39.51
30.22
29.21
31.10
29.18
22.81
37.32
29.96
GPT-4V
12.63
27.80
40.82
28.74
26.44
34.18
30.25
Heuristic mRAG: Retrieving Imager with Input Images
Qwen. VL--hha
15.74
17.12
25.74
20.52
19.14
22.68
18.44
22.05
18.19
20.16
Qwen-7B- CB----
Qwen-
24.04
41.54
GPT-NV
20.18
33.21
50.00
35.94
35.65
42.63
31,33
30.90
40.32
33.87
Heuristic mRAG: Retrieving Web Pages with Input Questions
Qwen- ChaChat
20.78
17.94
22.59
Qwen-7B. BB-Chat
14.65
15.47
24.98
19.14
18.66
19.99
18.34
17.93
20.12
19.01
Qwen- VL-Max
26.71
27.37
35 84
30.65
30.22
31.14
30.13
30.27
30.82
30.54
GPT-IV
40.84
33.00
31.47
31:42
32.59
Houristle Two-Stop MRAG: Retrieving Image First, then Retrieving Wich Pages with Question appended In Retrieved Caption
Qwen- VL-Chat
19.17
20.02
28.54
23.35
22.66
23.84
22.69
24.11
22.17
23.16
Owen-7B-Chat
Owen- VL-Max
24.44
30.75
43.21
34.03
33.91
38.26
31.10
32.04
36.01
33.99
GPT-VV
20.37
33.98
48.46
36.12
36.04
40.19
33.32
32.99
39,30
36.10
Heuristic Two-Step Image Caption lhen Retrieving Web
Puges
Question
appended
Caption
Owen- VL-Chat
22.05
25,87
31.84
27.58
26.21
27.44
27.06
28.81
25.57
27.21
Qwen-7B Beea-BB-Chat PB-Chal
14.65
21.16
28.66
22.89
21.02
23.64
21.55
16.57
28.30
22.39
Owen VL-Max
24.27
32.93
44.03
35.04
34.97
36,16
34.92
31,10
39.05
35.02
GPT-4V
24.90
45.76
37.23
36.94
37.82
36.70
37.15
Generative Search Engine
Bing Chat
27.71
32.77
32.99
67.
30.80
35.44
28.64
29.62 62
32.74
31.44
Perplexity
34.88
33.51
Gemini
36.17
32,86
42.84
38.75
34.78
46.39
31.82
35.77
39.69
37.69
Ours
OmniSearch (Qwen-VL-CHH)
45.52
39.22
OmniSearch (-4)
44.04
49.58
$4.45
50.38
49.06
50.49
49.73
46.96
53.21
50.03
Estimated Upper Bound: Retrieving Web Pleger with Gloden Query
37.46
Owen 7B Chat
39.69
47.27
$7.76
49.53
49.02
50.94
43.36
46.02
52.88
49.40
Qwen-Vi. VL-Max
42.19
$3.01
56.60
$1.91
50.58
52.97
50.60
49.83
53.33
51.56
GPT-VV
52.81
Human Performance
51.63
60.02
53.19
54.12
58.31
$7.86
53.20
51.96
58.36
55.12
The performance various MLLMs with different mRAG methods are shown in Table from which we can find that:
2 Our evaluation was conducted up to July 1, 2024.
Jieba (hotps: // thub. com/ Exs jayjjiebb for Chinese, and NLTK (https: / www. m1tk. org/) for English.
7

---

Published as a conference paper at ICLR 2025
(1) Our OmniSearch (GPT-4V) significantly outperforms other models, encompassing both state-of- the-art MLLMs with heuristic mRAGs and commercial generative search engines. Even Qwen- VL- Chat-based OmniSearch surpasses the considerably larger GPT-4V equipped with two-step heuristic mRAG. We attribute this to two aspects: on the one hand, the OmniSearch decomposes a complex question into sequence of sub-questions, reducing the retrieval burden in single step. On the other hand, it rethinks the retrieved content and sub-questions to ensure the accuracy of the sub-answers, mitigating the risk of error propagation.
(2) Regarding overall performance, the OmniSearch closely parallel human and GPT-4V enhanced with content retrieved via gold query, highlighting its superior abilities. Nevertheless, significant gap remains between the OmniSearch and human performance on questions belong to the three most challenging subcategories (fast-changing. >2-hop, requiring external visual knowledge), which in- dicates substantial room for improvement in agent-based mRAG for real-world questions. How to generate more human-lit like search logic a promising direction for future research.
(3) Despite achieving more than 50% performance, VQA remains formidable challenge for both Al systems and humans. is observed that for questions necessitating multi-step retrieval or ad- ditional visual knowledge, all models consistently underperform compared to other questions within the same classification schema. Especially for questions with different answer update frequencies, the variance in model performance is high. We can conclude that questions requiring rapidly chang- ing knowledge pose the most intractable challenge, as such knowledge cannot be internalized by MLLMs.
(4) For two-step heuristic mRAGs, leveraging image caption model to transform visual concepts brings more gain the original MLLMs, which provides a more detailed image description for the next retrieval step. However, this advantage reverses for questions that do not require additional visual knowledge, primarily because the majority of them 2-hop (74%) and do not demand visual knowledge beyond the concepts presented in the image itself. Supplementary information from the image caption model does not substantially benefit the model.
(5) Commercial generative search engines generally perform poorly on Dyn-VQA. Even the best- performing engine, Gemini, only matching the performance of GPT-4V with two-step mRAG. Fur- ther case analysis reveals these generative search engines lack essential grounding capabilities: they fail to associate "it" the question with objects the image, nor can integrate multimodal infor- mation effectively. This suggests that questions in Dyn-VQA represent the real demand in industrial scenarios.
(6) Comparing -wen-7B-Chat and Qwen-V VL-Chat, observe that the performance gap between the models is reduced once equipped with mRAG. This phenomenon indicates that mRAG can assist pure text LLMs in addressing multi-modal issues.
Due to space constraints, more analysis experiments are placed in the appendices in the Supple- mentary Material, and they are highly recommended to the reader.
Table 5: Experiments on OmniSearch paired with different MLLMs as sub-question solvers. Om- niSearch (G) and OmniSearch (Q) refer to OmniSearch implementations based on GPT-4V and Qwen-VL-Chee VL-Chat, respectively.
Planning Model
Sub-question Solver
Answer Update Frequency
Reasoning Steps
Visunl-Secking
Language
all
fast
slow
never
S 22hoop
2-hop
zb
OnniSSeach (Q)
OrniiScarch (Q)
35.16
40.81
OunniSeurch (Q)
GPT-4Y
37.14
42.82
47.48
43.29
42.78
44.46
42.26
41.21
45.15
43.15
OmniSearch (Q)
GPT-IV + GPT-4
38.98
44.52
49.18
15.03
44,52
46.20
44.00
12.97
16.87
41.89
OmniSearch (Q)
Qwen VL-Chat
34.10
39.88
44.50
40.32
39.77
41.53
39.15
38.18
42.22
40.17
OrmniScarch (G)
OmmniSearc (G)
44.04
49.58
54.45
50.38
49.06
50.49
49.73
46.96
53.21
50.03
OrnniSearch (G)
Qwen-VLL-Chal VL-Chat
38.65
41.68
52.25
16.56
44.72
19.40
43.50
41.63
50.64
46.07
5.3 ANALYSIS EXPERIMENTS ON OMNISEARCH
In this section, we conduct extensive analysis experiments answer the following questions OmniSearch:
How different models sub-question solvers affect overall performance? As shown in Table 5) several observations can be made regarding the performance of OmmiSearch when paired with different MILMM as sub-question solvers:
8

---

Published as a conference paper at ICLR 2025
Table 6: Comparison of token costs and expenses for different models.
Planning Model
Sub-question Solver
Input Tokens
Output Tokens
Expensex (x 10-0)))
Performance
Two-Step mRAG
GFT-4V
1454.0 (G)
132.5 (G)
18.5
37.15
Two-Step mRAG
Qwen-VI-Che -L-Chat
749.9
(8)
0.2
27.21
OmniSearch (G)
Onniieearc (G)
3028,5 (G)
476.9 (G)
44.6
50.03
OmmiSearch (G)
Qwen "hal
1217.2 + (Q)
+ 124.8 (Q)
24.4
46.07
OmniSearch (Q)
OmniiSearch (Q)
9578.3 (Q)
572.5 (Q)
3.2
41.20
OmniSearch (Q)
GPT-4V
2371.5 (9)
281.0 (5)
32.8
43.15
(1) the case of the -hhChat Based OmniSearch, employing the larger model GPT-4V as the sub-question solver significantly enhances the performance of the OmniSearch, indicating the continued validity of scaling laws for sub-question solver. Meanwhile, substituting the sub-question solver of the GPT-4V Based OmniSearch with the smaller Owen-VL-Chatl leads to predictable decrease in model performance. Nonetheless, it still outperforms Qwen-VLe VL-Chat with two-step heuristics mRAG from Table 4
(2) we also explored a more complex invocation strategy for the sub-question solving model: lever- aging GPT-4V for sub-questions entailing multimodal contexts, and employing GPT-4 for those involving purely textual contexts, which is considered to be more capable on text-only questions compared to GPT-4V. This approach contributes further to performance enhancement. More refined invocation strategies is also worthy to be explored, such as having the uub-question solver output bounding boxes for certain objects in the image to guide more precise retrieval. We leave this topic in the future work.
(3) To assess whether the OmniSearch with retrieval path planning learning has been impaired in its question solving ability, we replaced the sub-question solver of the Qwen-VL-Chat Based Om- niSearch with the original Dw-n-VVL-Chat. Comparison of rows and 4 in Table 5 reveals that employing the OmniSearch as the sub-question solver instead improves question-solving ability. This enhancement demonstrate that the learning of retrieval path planning also involves the abil- ity to understand and reason about retrieval knowledge, potentially enhancing the model's problem solving ability and yielding cross-tast gains.
How different models sub- sub-question solvers affect token and expenses? In Table we further examine the token costs and actual expense: brought by different sub-question solvers. Although more costly, the enhancement provided by the OmniSearch relative the heuristic mRAG is con- siderably substantial. The correlation between the performance of the OmniSearch and the actual expenses is proportional, yet not linear. When substituting GPT-4V with Qwen-VL-Chat as the sub- question solver (rows 3 and 4), the absolute performance declines by under 4 points, approximately 7.9%, while the expenditure is nearly halved, demonstrating the excellent scalability of OmniSearch. The results also indicate that sub-question reasoning is not the bottleneck of current methods, rather the retrieval strategy of complex questions presents a more urgent challenge. This conclusion further supported by comparing the enhancements achieved by substituting the planning model (row to 5) and sub-question solver (row to 7) of the OmniSearch (Q) with GPT-4V. The benefits realized by the former are more pronounced. Consequently, when computational resources are con- strained, priority should be given to ensuring that retrieval planning model can utilize a larger model as backbone.
6 ANALYSIS EXPERIMENTS ON DYN- VQA DATASET
6.1 PERFORMANCE COMPARISON ON DIFFERENT DOMAINS
Figure 4 iillustrates the performance of methods across various domains. We can intuitively observe that each mRAG method enhances the efficacy of the original model. The coverage of both original models is notably broadened by the mRAG methods, particularly for the smaller Owen-VL-CChe VL-Chat. However, in several domains, such as transportation, the Qwen- Swwen-Vl-Chat-bbseed Swt--Chhh--aaeedmommSerrrrrr OmniSearch instead exhibits superior performance
and GPT-4V equipped with different mRAG
We refer wttee//ww......ooooooo.////////s for GPT-4V price.

---

Published as a conference paper at ICLR 2025
Sport and Bicreaton
19:00
15:22
00.11
24.10
14.13
11.11
00.11
+1.08
Cospaite,.sdPrddeet Prodact
4139
AMD
181
11.26
2044
3121
20.3
T2-44
6653
Tramspatato
Technology
2A0B
24
M1.11
Gengraphy
Eapguagsrand Celteres
48,97
54.77
26.91
51.00
3.67
12.00
Daily
22:4
22/1
06.00
11.3
11.0
1011
CIT LVV messtr slAG
Owe
Figure 4: Model performance of different do- Figure 5: Pairwise overlap between correctly mains.
answered questions of different models. 0-8: Owen-VLL-Chat, Qwen-7B-Chat, Deepseek-VI. 7B-Chat, VisualGI.M-6B, 7B, mPLUG-OWI2.1, InstrucoBLIP-vieuna-T Qwen-VL-Max, GPT-4V.
compared to the GPT-4V-based one. Further analysis reveals that this phenomenon is primarily attributed to the long-tail property of transportation domain, which contains only 10 VQA instances, with the majority comprising 2-hop questions or questions that do not involve changing knowledge. In these cases, GPT-4V-based OmniSearch tends to over-retrieve, e.g.. it has retrieved the necessary information but over-cautiously continues to gather additional information to validate the answer, resulting in the correct answer being obscured within large volume of retrieved information. This underscores the need for ongoing enhancements the robustness of our OmniSearch.
6.2 PREDICTION OVERLAP
In this subsection, we investigate the overlap of questions correctly answered by different models. Firstly, we observed that no questions the Dyn-VQA were correctly answered by all models, and 31% of the questions did not receive correct prediction from any model. Figure 5 illustrates the degree of pairwise overlap in correctly answering questions on Dyn-VQA. Each row indicates the proportion of questions correctly answered by the corresponding model that were also correctly answered by other models. Overall, the two
models, Qwen-Ven VL-Max and GPT- 4V, exhibited relatively high overlap, but still hover around 60%. Furthermore, looking at heat blocks (6, 8), we find that even for InstructBI.IP- Vicuna-nn- which demonstrated the weakest performance (12.33 overall Fl recall, as detailed Table 11 in the Appendix), 26.87% (100 73.13) of the questions it successfully answered could not be correctly answered by the best-performing GPT- 4V. This indicates substantial differences in model behavior and shows that although some models generally outperform others, their superiority is not attributable to correctly answering the "hard" questions while consistently getting the "easy" ones right. The varied challenges presented by Dyn- VQA affect models differently, highlighting ensemble-based and seff-consist-basee approaches as promising directions for future research.
7 CONCLUSION
In this paper, we study the multimodal retrieval augmented generation (mRAG). We argue that existing heuristic mRAGs typically predefined fixed retrieval processes, which causes two issues: (1) Non-adaptive Retrieval Queries. (2) Overloaded Retrieval Queries. However, these rigidity is- sues cannot be adequately reflected by current knowledge-seeking visual question answering (VQA) datasets. Therefore, we first construct Dyn-VQA dataset, consisting of three types of "dynamic" questions, which require complex knowledge retrieval strategies variable in query, tool, and time. Furthermore, OmniSearch is proposed as the first self-adape adlf-adaptive planning agent for multimodal re- trieval. Extensive experiments prove the effectiveness of our OmniSearch, also highlight the chal- lenges posed by Dyn-VQA.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This research project is supported by National Natural Science Foundation of China (Grant No.62276154), Research Center for ComputerNetwork (Shenzhen) Ministry of Educa- tion, the Natural Science Foundation of Guangdong Province (Grant No.2023A151501229 and 01050002113310000111 No.JCY/202103241220 2033,233, JCYJ202408131 12000000133 12009013
Basic Research Fund of Shenzhen City (Grant
and 01112202402181130000000 the Major Key Projectof PCL for Experiments and Applications (PCL2022A05 and PCL2023A09), NSF under Grants 111-1006558, and POSE-2346158.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308. 12966, 2023b.
Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint 2404 18930, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020
Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, et al. Mceval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.07436. 2024a.
Linzheng Chai, Jian Yang, Tao Sun, Hongcheng Guo, Jiaheng Liu, Bing Wang, Xinnian Liang, Jiaqi Bai, Tongliang Li, Qiyao Peng, and Zhoujun Li. xcot: Cross-lingual instruction tuning for cross- lingual chain-of-thought reasoning. arXiv preprint arXiv:2401.07003, abs/2401.070033 2024b. doi: 10.48550/ARXIV.20001 007037. Ut..hhppe://aoo..or00000055//5/3 07037
2401.
Meiqi Chen, Yixin Cao, Yan Zhang, and Chaochao Lu. Quantifying and mitigating unimodal biases in multimodal large language models: causal perspective. Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 16449-16469, Miami, Florida, USA, November 2024a. Association for Computational Linguisties. doi: 10. 002./6//////222...2/ 1/22024..nn.nn./s... mandings-emnl..960 URL https://cllathhllos orq/2024 findiiiiiiiiii senedding--mlllppp 960
Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wei Wei Chang. Can pre-trained vision and language models answer visual information-seekkng questions? arXiv preprint arXiv:230..11713 2023.
Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Measuring and im- proving chain-of of-t--utttttttttttt reasoning language models. In Proceedings of the 2024 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Pupers), pp. 192-210, 2024b.
Wenliang Dai. Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao Weisheng Wang, Boyang Li. Pascale Fung, and Steven Hoi. Instructblip: Towards general purpose vision-lande lssoonlanguaage models with instruction tuning, 2023.
general-purpose
11

---

Published as a conference paper at ICLR 2025
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320-33,, 2022.
Joseph L Fleiss. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):778, 1971.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan. Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. aetrieval-augmentee generation for large language models: A survey. arXiv preprint arX.v:2312.100 10997, 2023.
Yash Goyal. Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Liangke Gui, Borui Wang, Qiuyuan Huang, Alexander G Hauptmann, Yonatan Bisk, and Jianfeng Gao. Kat: A knowledge augmented transformer for
In Proceedings of the
2022 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, pp. 956-968, 2022.
Xia Hou, Qifeng Li, Jian Yang, Tongliang Li, Linzheng Chai, Xianjie Wu, Hangyuan Ji, Zhoujun Li, Jixuan Nie, Jingho Dun, al. Raw text all you need: Knowledge-intensive multi-turn instruction tuning for large language model. arXiv preprint arXiv:2407.030400 2024.
Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai- Wei Chang, Yizhou Sun, Cordelia Schmid, David Ross, and Alireza Fathi. Reveal: Retrieval-augmentt visual- Lasuul-anguage pre-training with multi-source multimodal knowledge memory. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 93369-23379, 2023.
Aman Jain, Mayank Kothyari, Vishwajeet Kumar, Preethi Jyothi, Ganesh Ramakrishnan, and Soumen Chakrabarti. Select, substitute, search: A new benchmark for knowledge-augment visual question answering. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2491-2498, 2021.
Jitesh Jain, Jianwei Yang, and Humphrey Shi. Vcoder: Versatile vision encoders for multimodal large language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 17992-28002, 2024.
Hangyuan Ji, Jian Yang, Linzheng Chai, Chaoren Wei, Liqun Yang, Yunlong Duan, Yunli Wang, Tianzhen Sun, Hongcheng Guo, Tongliang Li, et al. Sevenllm: Benchmarking, eliciting, and enhancing abilities of large language models in cyber threat intelligence. arXiv preprint wrXiv:2405.03446, 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Ann Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed. Mistral 7b, 2023.
Jihyung Kil, Farideh Tavazoee, Dongyeop Kang, and Joo-Kyung Kim. I1-MMR: Identifying and improving mudti-modal multi-hop reasoning in visual question answering. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics: ACL 2024, pp. 10698-107000 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 20.18663//1/2024,, Andings-acl.633 URL 2024. f wwwddiinnnnssann
org/
636
Paul Lerner, Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, José G Moreno, and Jesús Lovón Melgarejo. Viquae, dataset for knowledge-bast based visual question answering about named entities. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 3108-3120, 2022.
Weizhe Lin and Bill Byme Retrieval augmented visual question answering with outside knowledge. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11238-1 1254, 2022.
12

---

Published as a conference paper at ICLR 2025
Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byme. Fine- Fine-grained late- interaction multi-modal retrieval for retrieval augmented visual question answering. Advances in Neural Information Processing Systems, 36, 2024.
Yuanze Lin, Yujia Xie. Dongdong Chen, Yichong Xu, Chenguang Zhu, and Lu Yuan. Revive: Regional visual representation matters in knowledge-based visual question answering. Advances in Neural Information Processing Systems, 35: 35:10560-10571, 2022.
Hanchao Liu, Wenyuan Xue, Yifci Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint ar Xrivv:4402.0025. 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning, 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024c.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Representations, 2019.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, al. Deepseek-vl: towards real-world vision-language understanding. 2024a.
arXiv preprint ar
Haoyu Lu, Wen Liu, Bo Zhang. Bingxuan Wang. Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang. Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan, Deepseek-vl: Towards real-world vision-language understanding. 2024b.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cyf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pp. 146-162. Springer, 2022.
Zhiqing Sun, Sheng Shen. Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. /4525, 2023.
Tu Vu, Mohit lyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoe Le, et al. Freshlims: Refreshing large language models with search engine augmentation. arXiv preprint arXiv:2310.0321 10.0313, 2023.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvim: Visual expert for pretrained language models. arXiv preprint arXiv:2311.0300 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thouugh prompting elicits reasoning in large language models. Advances in neural information processing systems, 15:24824-24877, 2022.
Jayynn Wu, Wensheng Gan, Zefeng Chen. Shicheng Wan, and S Yu Philip. Multimodal large language models: survey. In 2023 IEEE International Conference on Big Data (BigData), pp- 2247-2256. IEEE, 2023.
13

---

Published as a conference paper at ICLR 2025
Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xinrun Du, Di Liang, Daixin Shu, Xianfu Cheng. Tianzhen Sun, et al. Tablebench: A comprehensive and complex benchmark for table question answering. arXiv preprint arXhv:2408.09174 2024.
Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, and Dongkuan Xu. Rewoo: Decoupling reasoning from observations for efficient augmented language models. arXiv preprint 2rX5v:2305.1183 2023.
Zhengyuan Yang. Zhe Gan, Jianfeng Wang. Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 3081-3089, 2022.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference Learning Representations (ICLR), 2023.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl: Modularization empowers large language models with multimodality, 2024.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306. /3549, 2023.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13807-13816 2024.
Bohan Zhai, Shijia Yang, Xiangchen Zhao, Chenfeng Xu, Sheng Shen, Dongdi Zhao, Kurt Keutzer, Manling Li, Tan Yan, and Xiangjun Fan. Halle-switch: Rethinking and controlling object existence hallucinations in large vision language models for detailed caption. arXiv preprint arXiv:2310.01779 2023.
Zhuosheng Zhang, Yao Yao, Aston Zhang, Xiangru Tang, Xinbei Ma, Zhiwei He, Yiming Wang, Mark Gerstein, Rui Wang, Gongshen Liu, et al. Igniting language intelligence: The hitchhiker's guide from chain-of-thought reasoning to language agents. arXiv preprint arXiv:2311.11997, 2023.
Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang. Wentao Zhang, and Bin Cui. Retrieval- -aummveel generation for ai-generated content: A survey, arXiv preprint arXiv:2402.1947. 2024.
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, Shafiq Joty, et al. Retrieving multimodal information for aug- mented generation: A survey. In The 2023 Conference on Empirical Methods in Natural Lan- guage Processing, 2023.
14

---

Published as a conference paper at ICLR 2025
A
MORE DETAILS ON DYN- VQA DATASET
A.1 STATISTICS
Figure illustrates the data distribution across various domains and the answer change frequencies in Dyn-VQA. Among the domains, Sports and Recreation, and Companies and Products consti- tute approximately 50% of the data. The distribution of questions with fast, slow, and never answer changes is relatively balanced within the classes and does not exhibit long tail, reflecting distri- bution that closely aligns with real-world scenarios.
Hustrry
woud Merrstttn
Langsage
REFIT
Seleece and
Harehowe
Jame
Prodlct
BETeR
Figure 6: Data distribution of different domains and answer change frequencies on the Dyn- VQA dataset.
A.2 DATASET QUALITY
To ensure dataset quality, following the initial annotation of Dyn-VQA, we employed two qual- ity control (QC) annotators to re-evaluate the dataset. This re-evaluation included verification of the answers, domains, answer update frequencies, reasoning steps, and whether require external vi- sual knowledge each data instance. Data deemed incorrect by both QC annotators was filtered out. The agreement between QC annotators #1 and #2 with the initial annotations, well as the agreement between the two QC annotators, are presented in Table 7 The inter-annotator agree- ment measured by Fleiss's Kappa (Fleiss 1971) all exceeded 0.8, demonstrating the reliability of the annotation results.
Table 7: The inter-annotator agreement of QC#1, QC#2 and Table 8: initial annotation with each other.
Question and answer diver-
sity. Mean pairwise cosine distances are used as metries.
Answer
Domain
Answer Update Freq.
Reasoning Step
External Visual-Seck
Dataset
Question
Answer
vs. QC#1
81.2
84.4
89.6
84.1
87.9
Init. vs. QCF2
84.2
85.6
85.4
82.3
84.5
VQAv2 0.8405
0.7606
QC#1 QC#2
A-OKVQA
0.8428
0.9078
Avg.
81.8
84.6
87.3
82.7
85.8
InfoSeck
0.7569
Dyn-VQA
0.8532
0.9135
A.3 DATASET DIVERSITY
To assess the diversity of Dyn compared with other datasets, we calculated pairwise cosine dis- tances for each dataset. Following A-OKVQA, we utilized centence-transformert as the encoder. As indicated in Table g Dyn-VQA exhibits more diverse data types, evidenced by larger cosine distances. Intuitively, questions in InfoSeek are primarily constructed through templates, resulting
1.M- 1.6 cos
15

---

Published as a conference paper at ICLR 2025
in more homogeneous data format. In contrast, Dyn-VQA is manually curated. Moreover, since the questions of Dyn-VQr VQA are all from open-ended domains, the answers of them feature longer response length compared with the same manually constructed VQAv2 and A-OKVQA, which can essentially be answered with a single word.
A.4 REGULAR UPDATES TO DYN-VQA
The biggest characteristic of DQn-VQA is that the knowledge required to answer the questions it contains is dynamically updated over time, that is, the answers the questions Dyn-VQA are constantly changing. Therefore, to ensure that Dyn-VQA can serve an effective research resource for the community the long term, is necessary for us to dynamically update and maintain Dyn- VQA's answer annotation information. In general, based on the analysis of the frequency of change of Dyn-VQQ answer, we commit to updating Dyn-VQA's answer annotation every three months to ensure its timeliness.
In particular, we plan to implement a humans work together. Specifically, For a certain sample in knowledge through the search engine, and then we use LLMs such as Qwen .5-72B to compare the latest knowledge retrieved with the original answer to determine whether the original answer for the sample needs to be updated. Note that here we only need LLMs to determine whether the answer needs to be updated, and do not need them accurately answer the latest answer, because we think this is simpler and more friendly for LLMs. After we have the prediction results of LLMs as a basis, we then carry out the manual update process. We require every human annotator involved in the data update process to accurately update each samplc's 's answer based on their common sense, the results of large model judgments, and the latest relevant knowledge retrieved by the search engines. We believe that the semi-automatic update mechanism, in which models and humans cooperate, will not only reduce the workload of manual annotators but also improve the accuracy data updates.
data update mechanism in which models and we first retrieve relevant text
B RELATED WORKS
B.1 MULTIMODAL LARGE LANGUAGE MODELS
In 2023, with the advent of GPT-4V
(Achiam et al.
2023 a series of MLLMS (Bai et al.
2023b
Lu al.) 2024a Dai al., 2023 Liu al. 2024c Wang al., 2023) have been proposed and
demonstrated superior results on a variety of vision- language tasks Yin et al. 2023 Wu et al. 2023). Despite promising results, MLLMs tend to haphazardly produce responses that appear plausible yet contain factual errors when faced with real-world questions. Many prior works explore mitigating this hallucination issue (Liu al., [024a]] Bai et al., (024)) by introducing additional knowledge- enhancing data tasks into the different training stages of MLLMs, including pre-training (Zhai) al. 2023), instruction fine-tuning (Chai et al., 2024b) Jain et al. 2024), and RLHF (Sun et al. 2023 Yu et al. 2024). However, the expensive training cost of MLLMs pose significant challenges to the scalability of these methods. Therefore, mRAG attracts growing interest as an effective and efficient alternative.
B.2 MULTIMODAL RETRIEVAL AUGMENTED GENERATION
Besides retrieval from knowledge bases as described in Related Works section in main content, some work explores retrieval from other knowledge sources. REVEAL (Hu et al. (023) integrates knowledge retrieved from multiple sources, including wikidata, wikipedia and other VQA datasets. PICa Yang et al. (2022) consider LLMSs as implicit knowledge bases and extract relevant information from GPT-3 [Brown al. 2020) with image description as prompt.
B.3 VQA DATASETS AS MRAG BENCHMARKS
Showledge-seekinng VQA datasets Marino et al. 2019 Jain et al., 2021: Schwenk et al. 2022
Kil et al. 2024 Chen et al. 2024b) are widely employed evaluate the performance of mRAGs, which rely on external information to address open-domain visual questions. For instance, the re- cently introduced Wikipedia-based VQA dataset, InfoSeek (Chen al. 2023), emphasizes fine-
16

---

Published as a conference paper at ICLR 2025
grained entity knowledge for open-domain questions. A-OKVQA (Schwenk et al. 2022 is new knowledge-barte baagge--beeddde VQA benchmark that necessitates a broad spectrum of commonsense and world knowledge. As illustrated in Table I several knowledgo-seeking VQA datasets have been proposed in recent years.
Nevertheless, the knowledge scope assessed by these datasets is constrained. More critically, the questions in these datasets often exhibit a fixed format, typically querying a specific property of the object in image, which is generally textual knowledge available on the internet. Such two-hop questions can be readily addressed by standard two-step retrieval. The static nature of these datasets Dyn-VQA, VQA, which requires the retrieval of dynamic knowledge. knowledge from a more diverse range of modalities, or more complex multi-hop knowledge.
prompts us to propose Dyn-
C EXPERIMENTS
C.I BASELINES
In the main experiment of Table 4 we also introduced generative search engine and human as baseline models. We describe them following:
Generative Search Engine Among the commercial Al products, LLM-powered generative search engines like Bing Chat, PerplexityAI, mita.ai, Tongyi, and GPT-4o stand out. For our experiment, select Bing Chat, pro-version of PerplexityAI, and Gemini-Advanee representatives of Al search engines. They have multimodal RAG ability for fair comparison.
Human Performance. We also investigated human performance Dyn-VQA, employing par- ticipants with at least a bachelor degree. These participants were not involved in the Dyn-VQA annotation process.
C.2 OTHER BACKBONE MLLMS FOR HEURISTIC MRAGS
In the Appendix, we supplement the main experiment with more MLLM as backbone for heuris- tie mRAGs. Deepseek-VL. L--BBChht is an open-source large visual language model introduced by Lu et (004b)). VisualGL.M66 is an open-source, bilingual, multi-modal large visual language model proposed by Du et al.) (2022). Hava-VL.6-Mistralll
is open-source large visual lan- guage model proposed by Liu et al. (2024b), which is trained by fine-tuning LLM on multimodal instruction-followiii data based on Jiang al. (2023). mPLUG-OWI2.1 (Ye et al. (024) is large visual language model trained with two-stage method for aligning image and text.
C.3 TRAINING DETAILS
OmniScarch (G) is constructed by prompt engineering for GPT-4V, whose prompt template can be found in the next subsection. OmniSearch (Q) is developed by instruction fine-tuning of Qwen- VL-Chat. we synthesize instruction data containing planning trajectories by using GPT-4V and raw InfoSeek data. ~40K data is synthesized, and then the data is filtered by sequence of predefined rules. 13K data is eventually obtained. Additionally, general instruction data from CLgVLMMSFF 3 is mixed in ratio of 1:2 with the planning instruction data. We use LoRA to parameter- efficient fine-tune DownnVVL-hhal with LoRA rank and alpha are 8 and 32. AdamW (Loshchilov & Hutter 2019) optimizer is employed for model training, with learning rate set of le-4 and weight decay of 0.1. We utilize a cosine learning rate schedule, warming up over 5% of the training steps. The model is fine-tuned with 1 epochs, with the batch size per device set 4 and the gradient accumulation step set to 8. The maximum sequence length is 8192. The training are run on 4 NVIDIA A100 SXM4 80GB GPUs
All data is in the form of multi-round conversations. During model training, we expect the model to learn to generate response given the instruction and input text, thus we compute the loss function by considering only the response tokens of each round and ignoring the input tokens.
MMSSFT3111K
17

---

Published as a conference paper at ICLR 2025
Table 9: Experiments on the impact of using different parts of retrieved content. performance of OmniSearch (G).
We report the
Retrieved Content
Using Part
Answer Update Frequency
Reasoning Steps
Visual-Srcking
Language
all
fast
slow
never
2-hopp
>> 2-hop
n0
yes
2h
en
Image Caption
44.01
49.58
54.45
50.38
19.06
50.19
19.73
46.96
53.21
50.03
Image
Caption
47.39
50.81
Image
42,58
48.14
52.97
48.95
47.60
48.98
48.30
45.50
51.74
48.57
Web Tile Description
44.04
49.58
54.45
50.38
19.06
50.49
49.73
46.96
53.21
50.03
Web Snippet
Knowledge
51.75
50.92
Web Tide
42,09
47.66
52.47
48.44
47.11
48.47
47.82
45.01
51.25
48.08
C.4 SUPPLEMENT TO MAIN EXPERIMENTS
Experiment results of using more MLLMs as backbones for heuristic mRAGs are supplemented in Table 10 from which can find:
(I) Among all open-sourced MLLMs,
Poopscck-VL-7B-CCE exhibits the best performance as backbone, achieving the highest overall performance. While heuristic mRAG methods show the most significant improvements with LnstructBLIP-vicuna-- where the average absolute gain across four heuristic mRAG approaches reaches 16.60, marking enhancement of 223.1% com- pared to the baseline performance of InstructBLIPP -ip--iiuna-7.... This substantial increase is likely attributed to the initially lower performance of the original DastructBLIP-iccnae
(2) The performance variance between MILMs that enhanced by the heuristic mRAG becomes smaller. For instance, with the incorporation of the image caption-based two-step mRAG method, the variance in performance among open- source MLLMs reduces from 4.92 to 2.42. Combined with the analysis in (1), this suggests that for MLLMs with suboptimal foundational capabilities, mRAG serves as optimal method to bolster model performance. It is not only less resource-dependen,, but also convenient to deploy.
(3) Although single-step heuristic mRAGs may not retrieve the most precise content, they still ben- efit the original model capabilities. Retrieving image information with input images enriches the MILMM with information about the objects in images, whereas web page retrieval with input ques- tions can also retrieve some relevant information due to certain keywords in the questions. Overall, web page retrieval yields greater improvement with an average of 6.78, which might be attributable to the inherent capacity of MLLMS to recognize some objects depicted the images. also in- dicates that unlike previous VQA datasets, the challenge of Dyn- VQA does not lie solely in the recognition of objects images.
D ANALYSIS EXPERIMENTS ON OMNISEARCH
D.1 ANALYSIS EXPERIMENTS ON RETRIEVED CONTENT
Is each part of the retrieved content useful? Table 9 presents the impact of utilizing different parts of the content returned by the retrieval APIs, from which we can find that each part of the retrieved content beneficial to overall performance. Utilizing different single part results in varying degrees of performance degradation compared to using the entire retrieved content. Notably, image captions contribute most to final performance. This phenomenon arises primarily because nearly all questions in Dyn-VQA necessitate object recognition in images, for which captions from retrieved similar images provide crucial additional information to the model. In contrast, the benefit derived from incorporating relevant knowledge provided by the search engine is relatively modest. This is predominantly due to such information typically constituting static background knowledge that lacks direct relevance to the specific problem at hand.
Is more retrieved content useful? Table 11 explores the impact of varying amounts of retrieved content model performance, from which we can find that:
(1) The model incorporating retrieved content consistently surpasses its counterpart without re- trieval, affirming the inherent advantage of mRAG.
18

---

Published as a conference paper at ICLR 2025
Table 10: Main results Dyn-VQA.
Answer Update Frequency
Reasoning Steps
Visual-Seeking
Language
Model
all
slow
never
2-hop
2-hop
no
yes
en
Original (MILLMs
Qwen-V1-Chat
13.69
14.20
17.27
15,56
14.49
15.50
15.13
16.92
13.58
15.28
Qwen-7B-Chat
7.86
15.97
10.48
10.43
11.05
10.08
10:47
BBhhha
11.66
18.57
30.16
22.12
19.08
23.26
19.99
19.94
22.73
21.31
VisualGI
Llava-VI 1116-Misttra--7
15.39
19.72
28.34
22.65
20.12
23.53
20.92
16.27
27.85
21.97
mPLUG-Ow12.1
11.25
18.44
13.81
12.44
14.48
12.74
19.01
13:44
InstructBLIP MnstretBLIPPViiunaaa/ Vicuna-7B
6.08
7.40
8.38
7.48
7.33
7.51
7.38
4.65
10.31
7.44
Qwen Max
15.11
30.44
39.51
30.22
29.21
31.10
18
22.81
37.32
29.96
GPT-4V
17.63
27.80
40.82
30.80
28.74
31.71
29.26
26.44
34.18
30.25
Heuristic mRAG: Retrieving Tmages with Input
Imagex
Owen-VLe Owen-VL-Chat
15.74
17.12
25.74
20.52
22.68
Qwen-7B Chat
10.97
15.04
25.91
18.76
16.86
24. 18
14.23
16.8
19.75
18.25
Deepsock-Vil VL--s-----BBB-hatt
15.95
36.80
26.49
26.35
32.27
25.97
26.94
26.45
VisuaIGLM-6B
15.51
16.69
29.08
21.73
19.99
25.75
18.22
20.84
21.71
21.27
Llava Blva--V16-Miide Mistral
17.38
19.74
33.48
25.42
22.10
32.00
19:47
17.01
32.29
24.54
mPL.UG-O 1w..--
13.49
16.52
28.15
21.42
17.20
84
16.54
13.45
27.36
20.30
InstmatBLIP--icmm
13.40
15.83
29. 29.02
20.19
19.73
25.
16.68
13.10
25.05
20.07
Owen-VL-Max
28.99
45.49
34.22
41.54
34.19
GPT-4V
20.18
33.21
50.00
35.94
35.65
42.63
31.33
30.90
40.32
35.87
Heuristic mRAG: Retrieving Web Pages with Input Qirestions
Owen-VL. VL-Chat
20,78
18.27
27.61
22.76
22.20
23.34
22.07
26.66
17.94
22.59
Qwen-7B Chat
14.65
15.47
24.98
19.14
18.66
19.99
18.34
17.93
20.12
19.01
Beepseck-VL-7B-..
20.34
23.44
32.02
26.13
25.65
27.48
25.03
27.01
26.00
VisualGI M-6B
20.71
18.02
28.56
23.13
22.23
23.47
22.50
22.53
23.26
22.89
Llava-VI 6-Mistral-7B
20.39
30.33
24.99
26.70
28.62
24.73
mPLUG-OW12.1
20.49
24.29
30.93
26.04
25.53
28.42
24.19
21.16
30.79
25.90
InstrucBLIIP Li..PP---uunaPPP
27.65
Qwen-Vl..he VL.-Max
26.71
27.37
35.84
30.65
30.22
31.14
30.13
30.27
30.82
30.54
GPT-4V
22.48
30.92
40.84
33.00
31.47
34.32
31.42
34.13
32.59
Heuristic Two-Step mRAG: Retrieving
Image
First, then Retrieving Web Pagex
with Question appended Retrieved Caption
Owen-VL-Chat
19.17
20.02
28.
23.33
22.68
23.84
22.69
24.11
22.17
23.16
Qwen-7B-Ce
15.27
28.53
26.65
Deepseck VL.-BB Chat
18.65
23.51
34.68
26.66
26.54
29.74
24.51
25.70
27.59
26.63
VisualGI
Llava-VI 6-Mistral-7B
18.41
20.81
34.70
26.27
23.97
30.49
22.37
19.10
32.41
25.65
mPLUG-OW12.1
14.10
19.28
20.24
22.77
20.74
26.96
19.02
16.24
28.41
22.23
InstructBLIP- BaaectBBII--iicnee
16.22
18.95
30.34
22,96
22.07
25.47
86
16.46
29.18
22.72
Owen-VL-Ma.
24.44
30.75
43.21
34.03
33.91
38.26
31.1
32.04
36.01
33.99
GPT-4V
20.37
33.98
48.46
36.12
36.04
40.19
33.32
32.99
39.30
36.10
Heuristic Two-Step mRtAG:
Image Caption First, then Retrieving Web Pages with Question appended to Caption
Qwen VL.-haa
25.87
31.84
28.81
25.57
Owen-7B-Choe
14,65
21.16
28 66
22.89
21.02
23 64
21.55
16.57
28.39
22.39
Deepsock-VLL VL--B-hat
21.12
27.65
36.27
29.41
29.08
26.84
31.88
29.32
VisuaIGLM-6B
19.60
21.75
33.23
25.88
25.23
27.27
24.65
22.95
28.56
25.71
Llava V1.6 Mistralt
21.09
26.41
33.87
28.20
27.25
29.76
26.71
21.66
34.42
27.94
mPL.UGOww/2.
20.46
26.67
34.91
28.77
26.90
28.31
28.25
20.33
36.47
28.27
InstmutBLIP-Vicunn
24.37
28.22
35.62
30.51
29.85
30.77
30.06
22.98
35.57
30.33
Owen-VL-Max
44.03
35.04
35.16
35.02
GPT-4V
24.90
36.74
45.76
37.23
36.94
37.82
36.70
31.65
42.81
37.15
Generative Search Engine
Bing Chat
27.71
32.77
32.99
31.67
30.80
35.44
28.64
29.62
32.74
31.44
Perplexity Al
29.62
34.69
34.88
33.67
32.81
37.46
30.67
31.59
34.80
33.51
Gemini
36.17
32.86
42.84
38.75
34.78
46.39
31.82
35.77
39.69
37.69
Ours
OmniSearch (Qwen-V V1..Chat)
45.52
41.34
Omnišearch (GPT-4V)
44.04
49,58
$4.45
50.38
49.06
50..49
49.77
46.96
53.21
$0.03
Estimated Upper Bound: + Retrieving Web Pages with Gloden
Query
Qwen-VL-Chat
37.45
46.52
52.18
46.73
45.28
47.94
43.88
48.90
46.35
Qwen-7B Chat
39.69
47.27
$7.76
49.53
49.02
50.94
48.36
46.02
52.88
49.40
Deepseek VL.-B Chat
$4.29
VisuaIGLM-6B
36.09
40.09
50.59
43.33
42.75
44.00
42.61
41.98
44.40
43.17
V1.6 Mistral-7B
49.67
mPL.GG-O A.P.UG-Ow12.
41.19
48.66
55.83
49.63
49.17
51.77
47.97
44.49
54.68
49.51
37.17
43,92
55.53
46.51
45.94
44.45
41.25
54.05
Owen-VL-Max
42.19
$3.01
56.6
$1.91
50.58
$2.97
50.6
49.83
$3.33
$1.56
GPT-4V
45.59
54.23
60.78
55.15
$2.81
$4.53
$4.51
$1.08
58.07
$4.52
Performanc
$3.19
54.12
19

---

Published as a conference paper at ICLR 2025
Table 11: Performances of GPT-4V and OmniSearch(G) with different top-k retrieved content.
Model
#.C.C
Answer Update Frequency
Reasoning Steps
Visual-Secking
Language
all
fast
slow
never
2-hop
2-hop
n10
zh
GPT-VV
None
17.63
27.80
40.82
30.80
28.74
31.71
29.26
26.44
34.18
30.25
GPT-4V
24.07
32.6
46.11
35.47
34.98
37.98
33.54
32.91
38.02
35.33
GPT-VV
26.66
37.7
47.65
39.15
38,02
40.01
37.94
37.37
40.21
38.78
GPT-4V
All
24.90
36.74
45.76
37.23
36.94
37.82
36.70
31.65
42.81
37.15
OmniSearch (G)
35.16
40.89
45.52
41.34
40.81
42.56
40.28
39.22
43.23
41.20
OmniSearch (G)
39.08
49.06
52.87
52.38
48.89
48.23
48.09
OmniSeurch (G)
All
44.04
49.58
54.45
50.38
49.06
50.49
49.73
46,96
53.21
50.03
Table 12: Consistency of Different Metries Consistency of Different Metrics Consistency of Differ- ent Metrics.
GPT-based Eval.
Human Eval.
Correlation
Model
Recall
#1 & #2
#1 & #3
#2 & #3
GPT-4V + mRAG
40.23
37.50
42.00
0.47
0.34
0.66
Qwen-VL--Chat VL--Chat mRAG
28.46
26.50
25.00
0.49
0.41
0.69
OmniSearch (G)
57.56
54.00
49,00
0.45
0.38
0.63
OmniSearch (Q)
41.39
40.50
39.50
0.43
0.37
0.61
(2) Unlike GPT-4V, which does not exhibit continual improvement with increased retrieval vol- ume, our OmniSearch demonstrates superior capacity for utilizing extensive retrieved content. This indicates that despite potential noise contained in the retrieved content that may be harmful to ques- tion solving, OmniSearch can effectively filters disregards such disturbances, thereby adequately leveraging the complex and voluminous retrieved content.
(3) For the English question, the performance of both models continues to grow with the increase of retrieved content. This is partly due to the fact that both models use GPT-4V as the backbone, which inherently more capable in English than in Chinese, and also arises from the fact that Google search is naturally more inclined towards English websites, leading to better support for English questions. This inspires us that the future, more search tools can be introduce based on language characteristics, such Bing, Baidu, etc., and even multiple search tools can be utilized to verify or vote for the final answers.
D.2 CONSISTENCY OF DIFFERENT EVALUATION METRICS
In the main experiment, we employed FI-Recall as an evaluation metric due to its convenience. To demonstrate that it reliably reflects the true capabilities of models, we introduced two supplemen- tary metrics: GPT-based Accuracy and Human- Accuracy. For these metrics, we presented questions with ground-truth answers to GPT-4V and human evaluators, respectively, asking them to assess the correctness of the model responses and then compute the percentage of correct an- swers. Table 12 delineates the scores of the different models across these three metrics, as well as their correlation, which is quantified by the Pearson correlation coefficient. The Pearson coefficient ranges from -1 to 1, with signifies a perfect positive correlation and -1 denotes a complete negative correlation. The trends of the different models across these three metrics entirely consistent, with all coefficients exceeding 0, affirming a positive correlation. This demonstrates that F1-Recall fully reflects model performance. While GPT-based Accuracy and Human-based Accuracy exhibit stronger consistency, which may prove that they are more reliable, FI-Recall remains ad advantagcous as an automated metric, offering significantly lower computational costs and better scalability.
D.3 SUPPLEMENTARY ANALYSIS COMPUTATIONAL COSTS
In Table 13 we further reported the computational cost OmniSearch on questions with different answer update frequencies (fast, slow, never). The results indicate that:
20

---

Published as a conference paper at ICLR 2025
Table 13:
The token cost of OmniSearch questions with different answer update frequencies.
Planning Model
Sub-qqeestin
Fast
Slow
Never
Solver
Input
Outpot T.
Perf.
Inpot T.
Output
Perf.
Input
Output T.
Porf.
OmmiSearch (G)
OmniSearch (G)
3098.5 (G)
166.3 (G)
44.04
3036,5 (G)
(G)
49.58
2974.6 (G)
483.9
54.45
OmmiSearch (G)
Owen Chat
1268.2 (G)
2165.4
403.1 (G)
148.4(Q)
38.65
1214.1 (G)
1925.7 (Q)
358.5 (G)
112.7 (Q) 44.68
1185.6 (G)
2138.8
398.2
119.3
52.25
OmuniSearch (Q)
OmmiSearch (Q)
10258.2 (Q)
638.1 (Q) 35.16
9874.8 (Q)
634.2
40.89
8866.9 (Q)
(
45.52
OmmiSeurch GPT-4V
2508.8 8 (G)
(Q)
269.3 (G)
(Q)
37.14
2452.5 (G)
10009 (Q) 490.
233.6
42.82
2209.6 (G)
1024 (Q)
329.7 (G)
(Q)
47.48
Table 14: The average model latency for question in Dyn-VQA.
Planning Model
Sub-question Solver
Search Time (s/%)
Inference Time (s/%)
Total Time (s)
OmniSeaach (G)
OmniSearch (G)
14.1 (44.8%)
18.4 (55.2%)
31.5
OmniSearch (G)
Qwen-VI. Vh.-
12.3 (44.6%)
15.3 (55.4%)
27.6
OmniSearch (Q)
OmniSearch (Q)
8.5 (38. (8.3%%
13.7 (61.7%)
22.2
OmniScarch (Q)
GPT-4V
11.8 (45.0%)
14.4 (55.0%)
26.2
(1) Overall, OmniSearch consumes more tokens for more complex questions, such as those with fast or slow answer updates, primarily because these questions inherently require more retrieval steps.
(2) The difference in resource consumption for questions of varying difficulty is more pronounced for the smaller model OmniSearch (Q). This is due to the behavioral differences between GPT-4V and Owen-VI.. Specifically, as described in the Section 6. GPT-4V tends to be more rigorous in proactively planning verification retrievals to validate final answers, leading to retrieval processes exceeding three steps even for some relatively easier questions.
(3) By comparing rows and 2 in the Table 13. we can observe that after replacing OmniSearch (G)'s Sub-quest Sub-question Solver with smaller Qwen- VL-Cha, the total token consumption of Qwen-VL-Chal VL-Chat and GPT-4V stil comparable to the original model. This suggests that OmniSearch (G) effectively offloads computational burden to smaller models without significant drop in overall performance.
Additionally, we also reported the average latency for each question in Table [4] The results show that substituting some modules in OmniSearch with smaller models effectively reduces latency. Moreover, the ratio of search time to model inference time is roughly 2:3, indicating significant potential for optimization in both aspects. It is important to note that latency is a complex system engineering issue that involves not only the model's complexity but also factors network config- uration of search APIs, caching strategies for retrieval content, inference model acceleration, and hardware FLOPS, etc.
E ANALYSIS EXPERIMENTS ON DYN- VQA DATASET
E.
MODEL PERFORMANCE ON OTHER VQA DATASETS
Table 15
presents the model performances on various VQA datasets, highlighting several observa-
tions:
(1) The original GPT-4V achieves an average performance exceeding on previous datasets, ap- proaching human-level proficiency in the Table 3 Conversely, its performance poorly on our Dyn- VQA with significantly lower Fi-reacll of 30.25, substantially lagging behind human capabilities. This discrepancy primarily arises because MILMS have internalized much of the knowledge per- tinent to traditional VQA datasets, where many questions rely on common-sense knowledge. For instance, VQAv2 frequently poses questions about object properties intentions behind actions, which are relatively specialized by GPT-4V.
(2) The performance of the heuristic mRAG method is unstable on different types of questions (datasets). It impairs the effectiveness of GPT-4V on all previous datasets, especially on the InfoSeek where the decline is over 10 points. Through case analysis, we found that the heuristic mRAG strug- gles predominantly with questions of images depicting buildings, specific flora and fauna, which the image caption model fails to describe accurately. Consequently, the search engine yields "shallow"
21

---

Published as a conference paper at ICLR 2025
Table 15: Model performance comparison on different VQA datasets. Heuristic mRAG in Table
represents image caption-basee based mARG.
Model
VQAv2
A-OKVQA
InfoSeek
Dyn-VQA
Original MILMS GPT-4V
68.00
83.63
70.44
30.25
Heuristic Two-Step mRAG
GPT-4V
65.36
81.00
58.64
37.15
Ours
OmniSearch (G)
70.34
84.12
71.48
50.03
knowledge, i.e., content that is relevant to the question topic but is actually irrelevant question, and instead misleads the original model. Moreover, given that InfoSeek is automatically generated from Wikipedia and characterized by homogenous question types, substantial proportion of these problematic questions magnifies the disability of heuristic mRAG. On the contrary, the questions of VQAv2 and A-OKVQA typically inquire about common-senn sense knowledge, which is quite different from the real-world knowledge on the Internet, therefore the retrieved content instead has a minor (but still present) negative impact on the model.
(3) Our OmniSearch method achieved steady gains on each dataset. Even for datasets such as VQAv2 and A-OKVQA, which demand less extensive real-world knowledge from the Internet, OmniSearch still achieved slight growth. Since OmniSearch potentially makes search determina- tions, allowing it to avoid unnecessary retrieval interference with the intrinsic model understanding of questions that actually do not benefit from external knowledge augmentation. OmniSearch proves more adaptable and robust to diverse question types compared to the heuristic mRAG.
Spects and Rececation 47.1%%
Compasio and Products 51.0% Sciecce and Tedunlogy 43.8%% Lenganggs and Cultures -olons--5524% Histeey 60,8%%
404444 eeeett aheetee 2042444 0844444 a s shesett a ahe4eth 2002
Figure 7: The performance of OmniSearch on different domains. Each category is represented by 7 bars of the same color, representing questions of different categories (in order fast, slow, never, <2-hop, hoh no, yes).
E.2 OMNISEARCH PERFORMANCE ON DIFFERENT DOMAINS
Figure displays the performance of OmniSearch across various domains, as an complement to Figure in the main content. We find that in most domains, the heights of the bars follow a sim- ilar pattern: 1st bar (fast) < 2nd bar (slow) < 3rd bar (never), 4th bar (less than 2-hop) > 5th bar (more than 2-hop), 6th bar (not require external visual knowledge) > 7th bar (require external visual knowledge). This indicates that OmniSearch generally underperforms on questions necessitating more complex retrieval processes. There exceptions in data-sparse domains such Transporta- tion, where OmniSearch also shows relatively poor performs in these domains.
E.3
CASE STUDY
presents several intriguing case studies. In the first case on the left, the model dynam- ically adjusted its retrieval strategy multiple times after an initial retrieval failure, demonstrating OmniSearch's 's ability to adapt complex retrieval environments and actively adjust its strategy based on environmental feedback. However, the subsequent two cases highlight different challenges faced by OmniSearch. In the middle example, after the model estimated the current GDP of South
Figure 8
22

---

Published as a conference paper at ICLR 2025
II
airendd cesarcae paouret imployeet
CONTERT
comean bsseaa Stte Bescing
Figure 8: Case study with successful case and two bad cases.
Korea by changing the direct search strategy to indirect one with the growth rate and the pre- vious year's GDP, it should have proceeded to retrieve Japan's current GDP. Instead, OmniSearch abruptly ceased its thought and retrieval. This indicates that OmniSearch struggles with questions requiring extended reasoning chains and tends to lose track of the original question and preceding information. In the right case, the initial planned retrieval strategy is incorrect from the start, as the main body of the image is a PowerPoint slice, and the person involved in the question only occupies small space the bottom right corner of the image. The OmniSearch focuses on the wrong visual evidence and gets caught in a "thinking trap". Ideally, the model should first search the image to find out that it screenshot of course video, then view the video to find out the name of the speaker, and further search the web page to find out information about his academic institution. Another alternative is to get the exact search region, i.c., the bottom-right corner of the image, through image object recognition and image cropping. Then the person information is obtained based the cap- tion of the retrieved image. However, both approaches cannot be perfectly achieved by the current OmniSearch, which does not support such complex and ime-grained retrieval. These failed cases bring us significant insights and inspirations: firstly, how to solve the question requiring long con- text knowledge is worth studying, analysed 100 error cases, and found that 73 of them encounter the issue of partially containing the correct answer, but the OmniSearch can't complete the full re- trieval process due to the long context. On the one hand, we need to improve the maximum length of the context window of MLLMs, on the other hand, how to denoise, compress, and summarize the context the direction that the sub- sub-problem solver of OmniSearch can be improved. Secondly, advancing more precise retrieval techniques and incorporating a broader range of retrieval tools is an urgent research to be carried out.
23


Reviews:
[{"review_id": "vy1QqVGINV", "reviewer": "Reviewer_6Ev1", "summary": "The paper introduces a new dataset: Dyn-VQA, that fills the gap in existing multi-modal RAG benchmarks by including questions that require varied and dynamic retrieval strategies, 1.5k samples for 9 domains, 2 languages and 3 categories of questions, including rapidly changing knowledge, multimodal knowledge required problem and multi-hop reasoning in mVQA, which is a very essential practical problem in VLM driven search engines.\nThe paper also presents: OmniSearch, a self-adaptive planning agent decomposes complex questions into sub-questions and dynamically plans retrieval actions to address these challenges. The authors conducted extensive experimental results show that OmniSearch improves performance over traditional heuristic mRAGs.", "questions": "- Can the authors elaborate more on the process of how the AI researchers are selected and trained to curate dataset, and how the distribution is defined?\n- Can the authors conduct experiments with GPT-4o and also include the latency evaluations? Would also love to see how the feedback generation are effective in other baselines.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Originality: The authors have an innovative and practical focus on the gap in existing benchmarks, especially the dynamic retrieval, and multi-modal multi-hop questions. This dataset is extensively curated manually and reflects some real-world complexities.\n\n- Clarity: The paper provides clear definitions, comprehensive descriptions of the dataset construction, and detailed experimental setups. Included some comparison with baselines and highlights the uniqueness and impact of OmniSearch.\n\n- Quality: The authors conducted extensive ablation experiments to compare the performance of OmniSearch.\n\n- Significance: This work provides valuable insights into dynamic planning agent for RAG in multimodal LLMs, focus on a crucial challenge for advancing AI's real-world applicability in mVQA tasks.", "weaknesses": "- The dataset's limited size of 1.5k samples, with only 178 questions covering all three challenging categories, raises questions about whether its complexity and diversity are sufficient to benefit the broader research community.\n\n- In data curation part, the dataset only includes English and Chinese, and the authors filtered intractable instances that doesn't translate well, this might limit the dataset's diversity or introduce bias. if more languages are included, and examples are elaborated on how human are correcting Google Translate API, the dataset would be more convincing to be used to evaluate model's generalizability and performance on culturally specific questions that might present real-world challenges.\n\n- For the omniSearch agentic flow, there is no discussion of the latency or computational overhead associated with the multi-step retrieval process.\n\n- The authors seem to not ablate the feedback generation effectiveness in the experiment sections.\n\n- In the experiment part, no reasoning-based commercial search engine such as GPT-4o are included as baselines.\n\n- Figure 3 seems to have a typo in the question, what's the price of this car rather than what's the price on this car?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u3xAYcr83v", "reviewer": "Reviewer_Z1ru", "summary": "This paper introduces Dyn-VQA a new dataset for benchmarking multimodal retrieval-augmented generation (mRAG) methods. The dataset is constructed specifically with tasks that require adaptive retrieval methods that can handle changing answers, multi-hop reasoning and multi-modal knowledge. The authors run several baselines/benchmarks on this dataset with different MLLM models, and mRAG methods. The authors also introduce OmniSearch, a self-adaptive planning agent designed to overcome the limitations of current mRAG methods (the limitations this dataset addresses), and show a performance gain from OmniSearch.", "questions": "1. Could you say more about the computational costs of OmniSearch specifically as it scales with multi-step, dynamic retrievals?\n\n2. It is interesting that no questions was correctly answered by all models - and in general Figure 5 is really interesting to see - do you have any intuition for this distribution? Could it be used to sort of profile what tasks a specific LLM is good at?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. They introduce a strong, novel dataset, Dyn-VQA, which offers a new, and uniquely hard multi-modal retrieval challenge for adaptable, multi-hop retrievals - which mimics real world settings well. \n\n2. Strong experimental section - they benchmark this dataset with several MLLMs, and several types of mRAG methods. \n\n3. They introduce the OmniSearch method which performs very well on this task. \n\n4. The paper is mostly clearly written and well motivated.", "weaknesses": "1. The dataset has a strong motivation, however, the abstract and introduction could more clearly address the concepts of (as you labeled them) (1) Non-adaptive Retrieval Queries and (2) Overloaded Retrieval Queries. Clarifying these issues up front would help position and motivate the work better - and I felt they weren't so clearly explained.\n\n2. There could be more discussion around the scalability and computational cost of OmniSearch to provide a better sense of its applicability in real-world/time settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wqxcrtr2ZT", "reviewer": "Reviewer_ia8N", "summary": "The authors focuses on a dynamic visual question answering (VQA) task, where the answers can change over time (e.g., \"What is his latest film?\"). The authors introduce a new dynamic VQA benchmark to investigate the limitation of current multimodal retrieval augmented generation (mRAG) methods. The authors propose a new retrieval method, OmniSearch, which is based on the self-adaptive planning agent.\n\nThe contributions of this paper include (i) a new benchmark for the dynamic VQA task and (ii) a new approach---a self-adaptive planning agent---to improve task peformance.", "questions": "Please see the weakness of the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- **Introduction of a new VQA benchmark**: The authors focus on \"dynamic\" visual questions, where answers can change over time. This type of question is frequently encountered in real-world scenarios but is underrepresented in existing VQA datasets. The authors propose a new benchmark featuring dynamic visual questions that reflect the complexity of real-world inquiries.\n\n- **Proposal of a new mRAG approach**: The author introduces a self-adaptive retrieval agent that plan seach retrieval actions in real time and demonstrate its effectivness on dymanic VQA tasks. This approach is designed as a plug-and-play module that can be incorporated into various MLLMs, showing its applicability.\n\n- **Presentation**: The paper is clearly written and easy to follow.", "weaknesses": "- **Sustainability of Benchmark Accuracy**: Since the answers to certain questions in this benchmark may change over time, there is a risk that the answers will become **outdated** after the benchmark is publicly released. This raises concerns about how to ensure accurate model evaluation in such cases (i.e., when the benchmark's ground-truth answers no longer reflect current information). How will the benchmark address this issue to continue providing reliable, up-to-date evaluations?\n\n- **Unclear Definition of \"Hops\" in Questions**: The benchmark offers \"multi-hop\" visual questions, which the authors claim are often missing in existing VQA benchmarks. However, the paper lacks clarity in defining a \"hop\". While the authors consider one reasoning step as one hop, they do not specify how they define a single reasoning step. Interpretations of reasoning steps can vary; for example, some may consider a particular reasoning process as requiring two steps, while others might view it as involving only one step. Setting clear criteria for defining a reasoning step is essential.\n\n- **Missing Related Work**: While the paper focuses on multi-hop visual questions, it does not address recent work in this area. For example, it omits the citation, \"Kil et al., II-MMR: Identifying and Improving Multi-modal Multi-hop Reasoning in Visual Question Answering, ACL'24.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NVStM1fT8Z", "reviewer": "Reviewer_t3io", "summary": "In this paper, the authors propose OmniSearch to address the limitations of existing Multimodal Retrieval Augmented Generation (mRAG) methods: (1) non-adaptive retrieval queries and (2) overloaded retrieval queries. Additionally, considering these limitations cannot be adequately reflected in current knowledge-seeking visual question answering datasets, the authors construct the Dyn-VQA dataset. It consists of three types of “dynamic” questions and requires complex knowledge retrieval strategies. Experiments demonstrate the effectiveness of the proposed OmniSearch.", "questions": "1.\tThe authors may consider expanding the scale of the Dyn-VQA dataset.\n2.\tThe Dyn-VQA dataset considers two languages (Chinese and English). I wonder if there are impacts on different LLM or MLLM models? For example, some models only support Chinese and English, but others support more languages.\n3.\tThe authors should explain how OmniSearch dynamically decomposes the sub-questions and how it determines whether a question has been resolved?\n4.\tThe authors should explain the difference between OmniSearch and the chain-of-thought method. \n5.\tIn experiments, the authors may consider evaluating the effectiveness of OmniSearch based on more LLMs or MLLMs.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe proposed Dyn-VQA dataset bridges the gap in the existing VQA-based mRAG benchmarks. It is oriented towards real-world problems and emphasizes the ability to retrieve dynamic knowledge during the question-answering process.\n2.\tThis paper provides the analysis of the statistical information, data quality and diversity of the proposed Dyn-VQA dataset.\n3.\tThe experiment results show the effectiveness of the proposed OmniSearch method.", "weaknesses": "1.\tThe scale of the proposed Dyn-VQA dataset is small, containing only 1,500 questions. It is significantly fewer than existing knowledge-seeking VQA datasets.\n2.\tAlthough the proposed OmniSearch shows the potential in application, the innovation is limited.\n3.\tAs mentioned in the case study, the OmniSearch method struggles with questions that require long-term reasoning chains and fine-grained questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)