Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering
Authors: Yuyang Hong
Jiaqi Gu
Qi Yang
Lubin Fan
Yue Wu
Ying Wang
Kun Ding
Shiming Xiang
Jieping Ye
Venue: NeurIPS 2025 poster ()
Keywords: Knowledge-based Visual Question Answering; Retrieval-Augmented Generation
Primary area: deep_learning
TL;DR: Give VLM the ability to call tools to retrieve and filter information

Abstract: The task of Knowlegde-Based Visual Question Answering (KB-VQA) requires the model to understand visual features and retrieve external knowledge. Retrieval-Augmented Generation (RAG) have been employed to address this problem through knowledge base querying. However, existing work demonstrate two limitations: insufficient interactivity during knowledge retrieval and ineffective organization of retrieved information for Visual-Language Model (VLM). To address these challenges, we propose a three-stage visual language model with Process, Retrieve and Filter (VLM-PRF) framework. For interactive retrieval, VLM-PRF uses reinforcement learning (RL) to guide the model to strategically process information via tool-driven operations. For knowledge filtering, our method trains the VLM to transform the raw retrieved information into into task-specific knowledge. With a dual reward as supervisory signals, VLM-PRF successfully enable model to optimize retrieval strategies and answer generation capabilities simultaneously. Experiments on two datasets demonstrate the effectiveness of our framework.

Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering
Yuyang Hong 12; Jiaqi Gu Qi Yang 1,, Lubin Fan
Yue Wu', Ying Wang", Kun Ding" Shiming Xiang , Jieping Ye
School of Artificial Intelligence, University of Chinese Academy of Sciences MAIS, Institute of Automation, Chinese Academy of Sciences Allibbba Cloud Computing
Abstract
Knowledge-baad based visual question answering (KB- VQA) requires visual language models (VLMs) to integrate visual understanding with external knowledge retrieval. Although retrieval-augmented generation (RAG) achieves significant advances in this task by combining knowledge-bass querying. it still struggles with the quality of multimodal queries and the relevance of retrieved results. To overcome these challenges, we propose a novel three-stage method, termed Wiki-PRF, including Processing. Retrieval and Filtering stages. The processing stage dynamically invokes visual tools to extract precise multimodal information for retricval. The retrieval stage integrates visual and text features to achieve multimodal knowledge retrieval. The filtering stage performs relevance filtering and concentration on retrieval results. To this end, introduce a visual language model trained with answer accuracy and format consistency as reward signals via a reinforcement learning manner. This enhances the model's reasoning. tool invocation for accurate queries, and filtering of irrelevant content. Experiments on benchmark datasets (E-VQA and InfoSeek) show significant improvements (36.0 and 42.8) in answer quality, achieving statc-of-the-as performance. Code is available at: https:
//github. www.comcccccc/o www.cqu- student/ /iikiiiiii-- PRE
Question
Retrieval
Reranking
Question: What that
Cate Alca
Manuel Capeda Perzaa
Man
Alci Catedral de San litefonso
Sony, don''t know
statue made out
Vanilla RAG Methods
Processing
Retrieval
Filtering
Knowledge Base
Manuel Capeda Peraza
Cemunal Marual Dappeta
This is monumentto
The casl iron.
decreed April
completed
1996.
A
VLM-PRF (Ours)
Figure 1: Illustration of vanilla RAG methods and our Wiki-PRE. Different from the traditional RAG methods (above), our method (below) employs multimodal tools processing stage and further filtering stage, enabling more effective retrieval and extraction of tesk-relevant information.
'Equal contribution: Vuyang Bong <hongruyagg2238ia.mm en>. Jingi Gu <radindzju. eetiejj..dd....m edu.cn> Correspunding authors:
-aa..bbb.bbbbaa--e....m.... Ran Ding dinn.inghha..ac. ac.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1 Introduction
2 3 have demonstrated remarkable capabilities in Visual 5 Despite their effectiveness. they still face challenges in addressing knowledge-based visual question answering (KB-VQA), as such questions require not only an understanding of visual content but also the integration of external knowledge. For instance, answering the question "What that statue made of?" in Figure | requires factual knowledge that goes beyond the visual content. To address this issue, retrieval-aue -aggenntd generation (RAG) methods enhance model performance by incorporating mechanisms to access and integrate external information. These methods typically involve three steps: first, retrieving knowledge based on the given visual or textual content; second, reranking the retrieved information for relevance; and third, generating answers using the most pertinent content.
Visual Language Models (VLMs)
Question Answering (VQA) tasks 4
often fail to retrieve the most relevant information when handling complex visual content and questions, leading to suboptimal answers. This issue primarily arises from two key challenges. (1) Fine-grained knowledge retrieval in complex visual scenes: When process information-rich images, existing retrieval methods that rely on full-image visual features are often insufficiently precise for effective knowledge retrieval. For example, when asking about statues near a bell tower in Figure the statue may occupy only small portion of the image, Consequently, the retrieval process can be heavily influenced by the more prominent bell tower, introducing excessive irrelevant information related to it. (2) Precise filtering of irrelevant information from large scale retrieved results: After retrieving contextual information, it is difficult to filter out irrelevant or low-quality content using paragraph reranking alone. The retrieved results typically contain significant amounts of extraneous information, which can affect the accuracy of the generated answers.
However, current methods 6
To address these challenges, we propose novel multimodal RAG method consisting of three stages: processing, retrieval, and filtering. The core idea is to obtain more relevant knowledge to generate accurate answers. For coarse retrieval limitations, we innovatively explored a tool-based fine-grained retrieval mechanism. For irrelevant information, we innovatively employ question-baig -aaed filtering stage. Specifically, in the processing stage, the visual language model autonomically invokes image- processing tools based on the input image and question. These tools perform operations such as image captioning, visual grounding, and image flipping to extract detailed information related to the question from the image, thereby generating high-quality multimodal retrieval inputs. In the retrieval stage, multimodal retrieval is conducted using both visual features and text descriptions to retrieve relevant knowledge. In the filtering stage, the retrieved contextual information is filtered and condensed to remove redundancy, extract the most relevant knowledge, and provide to the answer generator for generating accurate responses. To this end, introduce Wiki-PRF, RAG method that not only supports basic multimodal question-answering functionality but also enhances reasoning based on the input image and question. Wiki-PRI can flexibly invoke visual tools and demonstrates stronger capabilities in filtering and condensing retrieval results.
To enable the visual language model possess the aforementioned reasoning ability, we train VLM-PRF model using reinforcement learning (RL). This is because training data collected for complex visual question-answeriin tasks often lacks the intermediate reasoning steps, which necessary for effective supervised fine-tuning of VLMs. RL [22 13 14), as a paradigm for learning strategies achieve specific goals, has been widely adopted in recent years to enhance the reasoning
capabilities of VLMs [13] 16 [8] 19 (0) for specialized tasks. RL can utilize a small amount of
sample data, relying solely on answer accuracy as the reward signal, train the model generate high-quality retrieval content by accurately invoking task-specific tools. Additionally, it enables the model selectively retain and condense the most relevant retrieval results for the query. Specifically, we employ the LoRA [21] to train only a small number of additional parameters, enabling our Wiki- PRF to enhance its RAG capabilities without compromising its core question-ansee answering abilities. In summary, our main contributions are as follows:
A knowledge-based visual -anseerng method using a Processing-Retrivva-- Retrieval-FFittrrig framework is proposed, named Wiki-PRE. It effectively leverages external tools for information retrieval and systematically filters the retrieved knowledge to support the generation of precise answers.

---

We introduce VLM-PRF, a visual language model for multimodal RAG tasks, trained via rein- forcement learning to enhance reasoning. To our knowledge, this represents the first application of reinforcement learning to multimodal retrieval-augmente data while enabling flexible tool use and robust processing.
generation, requiring minimal training
Comprehensive experiments demonstrate that Wiki-PRF achieves state-of -f performance on E-VQA (36.0) and InfoSeek (42.8). Additional analyses further validate our method's 's effectiveness.
2 Related Work
2.1 Knowledge-based Visual Question Answering
Knowledge-Based Visual Question Answering (Knowledge-Basee VQA) [2] As a critical branch of Visual Question Answering (VQA), Knowledge-Basse VQA demands models to integrate the understanding of visual content and question with external knowledge bases for reasoning and answering. Based on knowledge base modalities, Knowledge-Baasee Based VQA frameworks can be categorized into unimodal [33 24 25 26 7 and multimodal [27 28
TO 11] paradigms. Uni- modal methods [26] 7 typically utilize text-only datasets such as Wiki-21MM29 and GS112K| (3) as external knowledge sources. For unimodal methods, TRiG facilitates knowledge passage retrieval and generative question answering by converting images into plain text, thus fully harnessing the power of large-scale knowledge bases and pre-trained language models.
8 9
For multimodal methods 88 [9 T0
II 31,, external knowledge bases typically incorporate datasets such as Encyclopedic VQA (Q-V 27 and InfoSeek (28). which include both Wikipedia images and corresponding textual information. MI represents multimodal knowledge through explicit triplets to capture the implicit relationships between visual objects and factual answers. EchoSight [9] first retrieves candidate Wikipedia articles using visual information, then re-ranks them based on text-image query relevance to improve retrieval performance. Unlike previous methods, our method enhances the utilization of external knowledge bases by enabling the model autonomously select and filter relevant information during the retrieval processing.
2.2 Reinforcement Learning for Visual Language Model
Reinforcement learning (RL) 12 13 14 a learning paradigm that improves model decision-making
through interaction with an environment and feedback in the form of rewards, has recently been
widely applied to vision-language models 32 33
34]
R5 (VLMs). Some works [15] 16. 17 [8]
19 20
focus on enhancing the reasoning capabilities of Vision-Language Models (VLMs) through reinforcement learning. R1-One Vision [20 framework innovatively bridges vision and language by encoding visual data into formalized textual representations, enabling robust and precise reasoning grounded in linguistic semantics. VisualThinker R1 -Zero I6] achieves the first successful realization of Ahha Moment' in multimodal reasoning using a 2B- 2B-parameter VLM. Other works 37 38
39 40] focus on leveraging reinforcement learning to improve the performance of Vision-Language Language Models in specific areas like mathematical reasoning and visual perception. Visual-RFF 39 leverages VLMMs to generate reasoning-enhanced responses and integrates task-specific verifiable rewards (e.g.. IoU for detection) via policy optimization methods like GRPO [41], improving model performance. In contrast to above work, our study uniquely introduces RAG capabilities into VLMMs via RI. As far as we are aware, this is the first exploration of RL-based method for RAG in VLM.
3 Method
3.1 Overview
Knowledge-based VQA requires answering question Q that is highly relevant to a given reference image I, with the assistance of a knowledge base KB. In our setup, KB E (1) (an, I,)) consists of a million-scale collection of entity articles {a:} along with their corresponding image set {4). Our goal is to improve multimodal retrieval quality by flexibly invoking visual tools and enhancing relevance through filtering and enrichment.
As illustrated in Figure 2 the overall architecture of Wiki-PRF consists of three key components:
an external knowledge base (KB), model (VLM-PRF) trained via reinforcement learning, and
3

---

Visual Question
Processing Stage
Retrieval Stage
Knowledge Base
VLM-PRF
Teol Application
Visual Retrieval
Toot
Tool Ressoning chite citinas
Tea Capalriing
Gaption....
What that statue made of
Format Reward
Answer Reward
Filtering Resut
Toxl Resooning
VLM-PRE
VLM-PRF
choob
Fliwiigg
Answer
Ground-iruth
Sa cenooro
Answer
Rewards for VLM-PRF Training
Filtering Stage
Iron.
Figure 2: Overview of Wiki-PRE. Wiki-PRF comprises three key stages: (1) Processing Stage:
VLM-PRF processes the input image and its corresponding question using external tools. (2) Retrieval Stage:
Relevant Wikipedia articles are retrieved, split into individual sections, and ranked based on their similarity the processed input. (3) Filtering Stage: The re-ranked article sections are further filtered by VLM-PRF to retain the most relevant content, which is then fed into the VLM for final answer generation. During training. VLM-PRF is supervised using types of reward signals: answer reward, which evaluates the correctness of the generated answer, and format reward. which ensures the output adheres the desired structure.
base model without extra trained parameters. The Wiki-PRF method comprises three main stages: (1) Processing Stage: The VLM-PRE model invokes external tools to process the reference image and question Q, generating precise retrieval queries Query. (2) Multimodal Retrieval Stage: The model performs multimodal information retrieval based the generated query Query and retrieves relevant information from the knowledge base. (3) Filtering Stage: The VLM-PRF model filters and extracts highly relevant information from the retrieval results and structures it into task-oriented knowledge, which is subsequently utilized to augment the answer.
3.2 Processing Stage
rely on raw input for retrieval, often missing key details due to lack of interactive processing. For instance, statue next church might be overlooked in favor of the church itself. To address this, our method introduces tool-based preprocessing before retrieval, enhancing results through secondary data refinement. We employ several representative tools: 1) A captioning tool that captures high-level semantic information from images. 2) A grounding tool that extracts regions of interest for precise, detailed retrieval. A flipping tool that adjusts the images orientation to mitigate the impact of direction on retrieval. Through these tools, Wiki-PRF achieves more comprehensive and accurate retrieval results. essence, VLM-PRF provides the strategy while 'LM-base delivers the core tool functionality, for captioning and grounding, which we define as VI.Meaptioningg and VLM VL.Mgrundingg respectively.
Previous methods 27 28 10
As in Figure 33, given an image and question Q, the VLM-PRF model reasons about which tools use and in what order within <tthink> tags, then outputs selected tools and their execution order in <tool> tags. After VLM-PRF plans the sequence of tool calls, the tasks are executed by VLM-base, foundational model (Qwen2.5-VL-7B), This model is invoked multiple times to power specific tools like captioning and grounding. For the captioning tool, VaMecptioning takes the init caption Cinit generated by VLM-PRF as input and produces the final caption C Cquery for retrieval:
Cguery VL.Mcspttioinng (Cinity Q),
Specifically, VLM-PRF first outputs the Censt to be processed by the captioning tool VLMeaptioning Then VLMeaptionin employs Cinst as input to generate the final query Cquery.
For grounding tool, VLMggunding takes the object output by VLM-PRF and returns the positional information. The image I is then corpped based on positional information and generate Igroundngg.
I Igrounding Crop(1, VL VLMgrounding (oMDondding(oobect)......
(2)

---

Processing Stage
within knouledge
unilize following tools query knowledge providing information need:
Query:
Precessing Annwer:
shows bee
does
Caption: Provide detailed description related ouestion.
Teal Reasoning
provide information the country where
the fourd///..ff </tlink>
2. Grounding: Identify the specific care subject related the question
VLM-PRF
1. EIR: flip the image left or right.
-Toal>
3 Flip: Flip the image left right.
2. groudding: the bee and the plant.
Enclose your reasering pracess within end without detriled illstratiins, and dy the tosls and contents you use within and odd querying the external knswledge base.
3. cantin: bee collecting nector from
purple flower. </hool>
Quadluu: whst country hox baz bon found??
Flipping
Grounding
Captioning
bes has been photographed collectity rec from purple flower in natural setting.
VIM--PK
Retrieval Stage
Retrieval
Retrieval
Top1
oforpe vchiiy scouan far
The dilie herre Min
has vile
Tap
Filtering Stage
Concat Top Tool Sections
Thai Saediar:: Booheer pay seploce iu speiles of bisdidee kaove by the senae conlarig baskiuhe in eatuars catters Nart Aravics wlee it ouc scou Canada and lnso esdure United State. gleen in 2. commentoo ling und intore comender wule the chomes, Come pebys. uncooess dircced mxcndrenio make op bne buteettouuil
F Fluciin Quary: Has pardin clarcousty
bus Beadf likes relooe infemmatio raturve
resioval Astiie Rature
Sectiss Tale Rattrs pacerorr
her koops migiess seecine mes entemoh... time the réo eleembrr has ga Uxpaueng ane Crundeg V... Linc homanal i audal fommm sem yoe rest oue noweee made frai evey penkk
Filtering
Flacring Roale: iittato sumweer The ienas. Barar capenier chekeas virgiamn, baxe Jund in the Liited Stapo and para of Chanata dountied in shrastioonl
itlermniie. a foaad Enougheci sach of North Amsriis cai of Rocky Mosilaae and la matt Adrasia authen Crasii and Mais. -aapwerr
VLM-PRF Answert The Eartern corpenter bex Olylocaps virginica) hot been found the United Shates and parts of Canada.
Ground Truth: ['Canoda']
Figure 3: Example of the tool calls and filtering. By analyzing the problem, VLM-PRF performs captioning, grounding and flipping operations on the images. Using the retrieved sections, VLM-PRF performs filtering and generates the task-oriented results.
The flipping tool applies a left-right inversion to reference image in order to mitigate the impact of angular variations on the retrieval performance. Finally, all Query results generated by the tools are aggregated to perform refined and accurate retrieval.
3.3 Multimodal Retrieval Stage
The goal of multimodal retrieval is to retrieve relevant articles a from knowledge database K B, based on the reference image I and the generated Query. Initially, retrieval is performed directly on I, and the most relevant article obtained from this process is selected as the base retrieval information, denoted as D. Leveraging the Query produced by the tools, additional information is retrieved from the knowledge base to enrich the search context.
Tool Search. The retrieval query Query is first embedded into a feature vector using EVA- CLIP [42). We then employ the Faiss library [3] with cosine similarity S to efficiently retrieve the top-k most relevant images and their corresponding documents from the knowledge base:
{
T
V.
max
(3)
|TI
|
(A, hool = {(A,, I)), i Spi).
(4)
T (QQurry) V. (T,),
(5)
where (A, denotes the top-kn retrieved articles and their associated images, t represents the feature extractor of EVA-CLIP, T is the embedding of the Query (Cinit, Igrunding ). and V; is the visual embedding of image T. The retrieved articles are split into sections S, and after removing titles, sections will be selected based on cosine similarity Stn. For captioning, we calculate the
5

---

cosine similarity between Query and sections. For grounding, we directly calculate the cosine similarity between the question embedding, T ( and sections to maximize the fusion of modal information. The top-kis most similar sections are then selected as the final retrieval results, where sf denotes the j-th section of the i-th article:
A {s] 22 on In
Kst
(6)
T"
((())
S
max
TT
[((())I|
By indexing S we obtain the sections Stoel returned by the corresponding tool and concatenate them to form the final search result Search. Subsequently, Search is fed into VLM-PRF along with the top-k article D, which is retrieved by directly searching the input image. Following the specification in Figure 3 Szearhh is filled in <<sarrch_ rericht_eeutt,,,. D is filled in <retrieved -nformatioon,,, where /!.../ represents long text.
3.4 Filtering Stage
During the retrieval processing, a large amount of redundant information is generated, with only a small fraction containing key details relevant answering the question. Previous approaches [1 attempt to mitigate this by reranking and selecting more relevant passages. However, article- or section-level reranking methods can only filter at the passage level, often retaining significant noise. To address this limitation, we propose training the model using reinforcement learning, guided by answer accuracy. This approach enables the model to filter retrieval results question-specificc manner, reducing the influence of irrelevant content. Specifically, Wiki-PRF guides VLM-PRF to process the directly retrieved information D, comprising both image-derived data from I and search results Swarch, and output its reasoning within <think> and </think> tags. The model then generates a compact, task-oriented knowledge representation F within <answer> and </answer> tags.
VI.M-PR((D,, Spench))),)
(9) where D denotes the retrieved information corresponding to 1, Ssenrch represents the external search results obtained via tool-based retrieval, and the filtered, task-oriented knowledge representation produced by the reinforcement learning module. After generating task-oriented knowledge F, Wiki-PRF uses context to generate the final answer A via the VLM.
A VLM Q),
3.5 Training via Reinforcement Learning
To improve the model's tool selection and information filtering strategies, we utilize GRPO 41 with removed KL divergence constraint for VLM-PRF training. The formula can be defined as:
G
[0) ((q-
1
min (ra,(0)Aru clip(r,,(0), e))
G
O
(10)
To oc R(q))
(11)
Te (01,144 (i, 01, R(g))*
where [ response. The term (o 0r,<r: R(q)) represents the conditional probability of token os, at decoding step f, conditioned on previous tokens 0j,et and retrieved information R(g))
i= denotes the G responses generated for question Q, and |o:| is the length of the i-th
Besides, we design reward function as the primary supervisory signal to guide the model in enhancing its tool invocation and its filtering of retrieved information. Specifically, we employ format-based reward encourage VLM-PRF to perform multi-step reasoning about tool usage within the <think> tags, make appropriate tool calls within the <tool> tags, and further process the retrieved information within the <think> tags. Finally, the refined and filtered results are output within the <answer> tags. Furthermore, we introduce an answer reward to supervise the content generated within the <answer> tags, ensuring that the model produces high-quality, relevant, and struuurree responses. The final reward function can be presented by:
To( y) aEM (.MM(pppn (apred: agt) + (M bool + 7M (aglters t(lter),
(12)

---

M y) (1
if match,
(13)
if unmatch.
where (z, y) represents the reward between input z and output y; EM denotes the evaluation function for answers, such as exact matching; M employs regular expression matching verify format compliance. a, f3 and are the weighting coefficients for these components, with values of 1, 0.3, and 0.7 respectively in our method. Moreover, Opred refers the model's output answer, Ogt represents the ground truth, while Grool and Grttr correspond to the model's outputs during the processing and filtering stages. Loot and liter represent the templates for tool usage and filtering.
4 Experiments
4.1 Experimental Setup
Datasets. We evaluated our experimental results two main datasets: InfoSeek [6] and Encyclopedic VQA (E-VQA) (1) [nfoSeek(6] contains 1.3M VQA pairs matched to 1IK images from OVEN 4 The training set (934K) and validation set (73K) are strictly divided by both entities and questions. The validation set is further categorized into two types: Unseen Entity and Unseen Question. Following the setup of [9]. we used knowledge base consisting of 100K Wikipedia entries and reported evaluation results on the entire validation split. (2) E- VQA (26] consists of over 221K unique question-answer pairs, each associated with up to five images sourced from iNaturalist [45] and the Google Landmarks Dataset v2 [46]. The dataset includes two types of questions: Single-hop and Two-hop. The samples are divided into training, validation, and test sets with 1M, 13.6K, and 5.8K items respectively. Like other methods [1]. we report our results on the 5,8K test set. We also evaluate OK-VQA
23 a 14K-question datasct spanning diverseknowledge domains.
Baselines. To validate the effectiveness of our Wiki-PRF, we establish two baselines: (1) Base Model: Directly answer questions without any RAG pipelines. (2) Wiki-PRF without RL: Answer questions with our Wiki-PRF method before RL fine- tne-tuning. These baselines serve to assess the contributions of our stree-stage Wiki-PRF design and the benefits of RL fine-tuning, respectively.
Evaluation Metrics. In the KB-VQA VBA task, we focus on evaluating both retrieval and QA metrics. For the retrieval metric, we use recall to determine whether the correct article appears among the top-k retrieved results. For the QA metric, following the original dataset settings, we apply VQA accuracy (5 23 for InfoSeek and BEM score/47
47 for E-VQA.
Implementation Details. Given the strong vision-language understanding capabilities of the Qwen2.5-VL series, we adopt Qwen2.5-VL-3B and Qwen2.5-VL.
as our base models. We
apply GRPO for reinforcement learning. Specifically, we set the number of generations to 8, the sampling temperature to 0.7, the number of training epochs to 2, and the learning rate to le-5. We utilize LoRA-based fine-tuning, with the LoRA rank to 64, the LoRA alpha to 128, and the dropout rate to 0.05. For the retriever, we use frozen EVA-CLIP 8B model for both retrieval and similarity computation. Image features are indexed and retrieved using cosine similarity with the Faiss-GPU library. By default, we use the Top-1 article retrieved from image retrieval alongside the Top-5 articles identified through tool calls. All sections from the image-retrieved article are retained, while the top-k sections from each of the tool-retrieved articles are kept. This combined information is then provided to the next stage for filtering. The total training process takes approximately 15 hours using A800 GPUs Our framework, Wiki-PRF, is implemented in two configurations: Wiki-PRF-33 and Wiki- PRF-7B. Models further fine-tuned using reinforcement learning are referred to as VLM-PRF-3B and VLMM-RR-77B
4.2 Main Results
VQA Results. We assess our method on the aforementioned VQA datasets, comparing against various MLLMs and aerieval-augmenterr approaches as shown in Table [] Among zero-shot MLLM approaches, we found that without RAG, the MLLM model achieves only relatively low accuracy in answering questions, highlighting the challenges presented by the KB-VQA task. Regarding retrieval- augmented methods, our Wiki-PRF before RL training, achieves an accuracy of 34.0 with the 3B model and 39.5 with the 7B model in InfoSeek dataset, surpassing several well-trained methods such as EchoSight (9] and Wiki-LLaVA (22. This finding further confirms the effectiveness of our proposed shree-stage approach. After RL training, our Wiki-PRF-7B establishes new state-of-the-an
7

---

E-VQA
InfoSeck
Method
Model
Retriever
Single-Hop
All
Unsccu-Q Umeaa-E
All
Zem- sbor MLLMs
BLIP-2
Flan-T5xL.
12.6
12.4
12.7
12.3
12.5
InstructHLiP IP
48
Plan-T5a
11.9
12.0
8.9
7.4
8.1
LLaVA-v1
Vicune-78
16.3
GPT-4V
1
26.9
28.1
15.0
14.3
14.6
Qwen2.5-VL..3 (Base)
2
17.9
19.6
20.4
21.9
21.4
Qwen2.5-V1-77
21.7
Werienal-Augmee Models
DPR 50
Multi pastii- BERT
CLIP ViT-B/32
29.1
12.4
RORA-VLM 51
Vicuna-7H
CLIP-Gooole Search
20.3
25.1
27.3
EchoSight
Miitaa-7BBLLLMM-33888
EVA-CLIP-SB
19.4
Wai-LLaaA
Vicuna-78 7B
CLIP WTT-L/14-Comrivver
17.7
20.3
30.1
27.8
28.9
ReflectiVA
[1
ILLMA-3,1-88
EVA-CLIP-S
28.0
29.2
40.4
39.8
40.1
MMKB-RAG
53
Qwen2-78
EVA-CLIP-S8
39.7
VLM-PRF (wro RL)
Qwen-2.55 Owen--25VVL-BB
EVA-CLIP-SB
26.5
25.6
34.2
33.7
34.0
VLM-PRI' (w/o RL.
Qwen Qwen-2.5VL. 2.5VL-7B
EVA-CLIP-SB
28.9
28.6
40.0
39.4
39.5
-Lagmented
Reinforcement Leaning
VLM-PRF (Owrs)
LL A-3.1888
EVA-CLIP-8H
36.3
35.5
41.3
40.6
40.8
VLM-PRE (Ours)
Owen-2 2. 2.5VL-3B
EVA-CLIP-SB
31.1
32.4
39.7
58.8
39.0
2.VL-BB5
EVA-CLIP-S0
VLM-PRF (Ours)
InternVI 3-88
EVA-CI JP-88
40.1
39.2
43.5
42.1
42.5
Table I: VQA accuracy on V-VQA and InfoSeek. The metries baselines and our methods are highlighted in light blue. indicates results that are not directly comparable due to different knowledge bases.
E-VQA
Hos
Aud and tha be bene for foo.. 30
Infoseek
Querree:: Which detetee trampiod smeccdes witt?
Todiy SMO
Teato GMA Callude
30
Groad (iottb Aid (0000000004)))
Figure 4: Qualitaive examples of Wiki-PRE.
accuracy of 36.0 on E-VQA and 42.8 on InfoSeck, outperforming all previous methods, including ReflectiVA B3I and BAMKB-RAG [53]. Moreover, method achieves consistent improvements across different base models, setting a new SOTA score of 39.2% on E- VQA with InternVL3-88 VL3-8B the base model. This underscores the impact of reinforcement learning in enhancing the model's RAG capabilities.
Table 2: Recall of retrieved articles.
Table 3: Tool usage statistics. Statistical analysis
Model
Retrioval Input
Retrivall Recall
of tool usage (mean and variance). Model
None
images
45.56
Combinations
Captioning
Grounding
Plipping
Qwen2.5. V.V-BB
images tools
48.32
Qwen2.5.VV
0.86/1.b0
Qwen2.5. VL--B
images tools
53.44
Owen2.5 VL.7B
2.43/ 1.03
0.85 0.228
0.22/0.45 0.42
VLM-PRE--B
images tools
50.16
VI MM-R--33 3B
53
1,52/ 1.22
0.54/0.00
80.5/
VLM-PRR-7B
images tools
54.89
VLM-PRF-7B
Effectiveness of the Processing Stage. Table presents the recall of articles retrieved from InfoSeek under various settings. The Top-1 retrieval performance using direct image retrieval is 45.56%. Combined with Top-5 article retrievals from our tools, this rate increases to 48.32% and 53.44%. Further improvement with reinforcement learning supervision raises it to 50.16% and 54.89%. Notably, we observe that models trained with RL supervision exhibit greater diversity and frequency in tool selection, as shown in Table 3 Specifically, the combinations of tool calls increase after RL training, demonstrating that the model can dynamically and flexibly construct tool invocation schemes. the same time, the captioning tool invoked most frequently, highlighting its role as the most common and direct tool for enhancing article recall. Overall, this demonstrates that RI can encourage the model leverage a broader array of tools by optimizing for the final answer reward, thereby showcasing its flexibility.
8

---

Table Performance on OK-VQA.
Table Filtering from oracles. VQA Accuracy
Method
Model
OK-VQA
in Oracle Setting with Ground-Truth Articles.
Qwen2.5- -B
62.1
Method
Model
VQA Accuracy
Qwen2.5-VL- -L-
72.4
KU-RAG 54
1L.VVA-Next-77 7B
73.1
Wiki-LLaVA 52
Vicuna-7B
$1.5
B-KKB-RAG 53
LLLMM-3.1--8
65.4
ReflectiVA [1
LLaMA-3.1-80
57.6
Wiki-PRE-BB
VLM-PRF-3B
68.6
Wiki-PRE-3B (Ours)
VLM-PRF-3B
64,4
Wiki-PRI-BB
VLM-PRE-7B
77.8
Wiki-PRF-7B (Ours)
VLM-PRF-7B
65.8
Table 6: Comparison of SFT and RL. We sampled 2,000 instances from InfoSeek and present a comparison between the results of SFT and RL.
Model
Unseen Question (UQ)
Unseen Entity (UE)
ALL
Owen2.5-VIL-77.
39.1
40.5
40.2
Wiki-PRE-7B (SFT)
41.5
41.9
41.8
Wiki-PRF-7B (RI.)
46.6
46.2
46.3
Effectiveness of the Filtering Stage. To evaluate whether our filtering stage can effectively extract accurate information from given sections, we conduct experiments similarly to other methods under an oracle setting, where the ground-trutt entity (i.e., the Wikipedia page associated with the query) provided. Compared with other methods, our final VQA accuracy is much higher (65.8%) as shown T the fine-tuned model can more efficiently locate the necessary information when given the oracle retrieval information.
Effectiveness of Reinforcement Learning. To investigate its effectiveness, we evaluated the use of supervised fine-tuning (SFT) for the filtering stage. Specifically, we trained a dedicated filtering model using SFT, keeping all other configurations identical reinforcement learning (RL). The results are presented in Table 6 The RL model significantly outperforms the SFT model on the test set. The reason that SFT tends imitate superficial patterns, which limits its generalization capability. In contrast, RL enables the model to understand the underlying principles of information filtering, leading to far more robust and generalizable performance.
Results on more benchmarks. As shown in Table 4 we evaluate our model on the widely used OK-VQA benchmark. We see that Wiki-PRE-7B achieves new state-of-the-art score of 77.8 on OK-VQA. The consistent performance improvement on multiple benchmarks confirms our method's strong generalization capability.
4.3 Ablation Studies
Modules and Tools. Table 3 presents the ablation study of our carefully designed stages and tools based on 10K samples from the InfoSeck validation set. In the module ablation, VLM-PRF-3B model improves the baseline by 2.54% and 2.02% when combining the processing stage and filtering stage, respectively, underscoring the effectiveness of these two modules. In the tool ablation, the caption and grounding tools increase performance by 1.94% and 0.98%, respectively. The combined use of all tools yields peak performance of 39. 99.48. This result demonstrates that Wiki-PRF significantly boost the final VQA accuracy by adeptly utilizing tools to provide more relevant information.
Table 7: Ablation studies on the scale of knowledge base on InfoSeek.
Method
Model
10k
50k
100k
Vanilla-MRAG
Owen2.5-VL-3B
49.7
32.1
21.4
Wiki-PRF-3B
VLM-PRR-3B
53.0
43.7
39.0
Vanilla-MRAG
Qwen2.5
56.3
39.6
23.7
Wiki-PRF-7B
VLM-PRF-7B
60.3
51.2
42.8
we verfied that the correct document for each evaluation question was included in knowledge bases of all scales, to assess the impact of knowledge base size. The result demonstrates that both our method and the baseline exhibit performance degradation as knowledge base size increases. This occurs because larger knowledge bases introduce additional noise, increasing retrieval difficulty, which is universal challenge for current RAG methods. Critically, Wiki-PRF demonstrates significantly slower rate of degradation for both 3B and 7B models.
Scale of Knowledge Base. In Table 7

---

Table 8: Modules and tools ablation. "Processing". "Retrieval" and "Filtering" denote the three distinct stages proposed in our approach.
Model
Processing
Retrieval
Filtering
Tools
VQA Accuracy
Modules Ablation
Qwen2.5-VL-3B
34,22
VLM-PRF-3B
36.24
VLM-PRF-3B
Multi-Tools
36.76
VLM-PRF-3B
Multi-Tools
39.48
Tools Ablation
VLM-PREF-B
Grounding
37.22
VLM-PRE-3B
Captioning Multi-Town Tooli-Tools
38.18
VLM-PRF-3B
39.48
Table 9: Ablation on training sample size.
Table 10: Ablation on retrieved article quantity.
VLM Model
2K
4K
6K
8K
Retrieved Article
Top-1
Top-3
Top-5
Top-7
VLM-PRF-3B
37.30%
39.48%
38. 38.92%
40.83%
VLM-PRE-7B 42.13%
43.10%
43.09%
43.80%
InfoSeek
38.85%
39.10%
39.48%
39.57%
Training Samples. Table 9 presents the accuracy achieved with different training sample sizes. As the number of training samples increases, we observe general upward trend in accuracy. Balancing accuracy with training time, we opt for 4K samples as our default experimental setting. Unlike other methods that necessitate training the complete training dataset (InfoSeek: 934K, E-VQA: 1M), our approach attains comparable or even superior results using only a small subset of samples. We attribute this efficiency reinforcement learning's 's ability to effectively stimulate the model to leverage tools for retrieving additional information and integrating existing knowledge to determine the correct answers, rather than relying on memorizing specific question-answer patterns.
Retrieved Articles. Table TO illustrates the ablation of retrieving top-k articles when utilizing tools. We evaluated the VQA accuracy of Wiki-PRF-3B on the InfoSeck dataset with varying numbers of retrieved articles: 3, 5, and The results indicate that as the number retrieved articles increases, VQA accuracy tends to improve gradually, albeit with diminishing returns. To balance inference time and accuracy, we set the K=5 as the optimal choice and utilize the Top-3 relevant sections from these K=5 articles to supplement knowledge. More experiments and details about the retrieval of tool calls are presented the supplementary materials.
Table 11: Inference time of each stage. We sampled 1,000 instances from InfoSeck and measured the average duration of each stage per sample.
Model
Processing & Retrieval
Filtering
Answering
Total
VLM-PRF-3B VIM-PRI-77
2.2s
3.4s
0.59s
6.23s
3.3s
4.6s
0.74s
8.77s
Inference Time. We evaluated the VLM-PRF model's stage-wise time cost per sample in Table 11 The results show that the Processing & Retrieval stages and Filtering stage consume more time than the Answering stage. This duration primarily stems from tool invocation and long-text processing, both which can be further optimized future improvements.
Conclusion
In this paper, we propose Wiki-PRF, a three- stage Siccess-Reerieval--Fieee framework that repre- sents first reinforcement learning method for multimodal retrieval- augmenttd generation. By guiding models to invoke tools for processing raw information during the processing stage and filtering retrieved knowledge during the filtering stage, the trained VLM-PRF model significantly enhances performance on Knowledge-Band -Base Visual Question Answering tasks. Extensive experiments demon- strate state-of-the-an results on EQV and InfoSeek benchmarks. While limited to three retrieval tools in this study, future work may explore expanded tool integration to further advance capabilities.
10

---

6 Acknowledgements
This work was supported by the Strategic Priority Research Program of Chinese Academy of Sciences
(Grant No. XDA0480200), the National Natural Science Foundations of China (Grant (0.62206310))
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 1xXi:23303.08774 2023.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint a3X1::2502.13923, 2025.
[3] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025.
[4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitmick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2425-2433, 2015.
[5] Yash Goyal, Tejas Khot, Douglas Summers- Stane, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pages 6904-6913, 2017.
[6] Weizhe Lin and Bill Byrne Retrieval augmented visual question answering with outside knowledge. arXiv preprint arXiv:2210.03809, 2022.
[7] Feng Gao, Qing Ping, Govind Thattai, Aishwarya Reganti, Ying Nian Wu, and Prem Natara- Natural language-centric outside-knowled visual question
jan. mennfform-etrreve-eeeeeee answering. In Proceedings of the IEEE/CVF conference Computer Vision and Pattern Recognition (CVPR), pages 5067-5077,200
[8] Yang Ding. Jing Yu, Bang Liu, Yue Hu, Mingxin Cui, and Qi Wu. Mukea: Multimodal knowledge extraction and accumulation for knowledge-basee visual question answering. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR), pages 5089-5098, 2022.
[9] Yibin Yan and Weidi Xie. Echosight: Advancing visual-language models with wiki knowledge. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1538-1551, 2024.
[10] Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozbeh Mottaghi. Multi-modal answer validation for knowledge-based vqa. In Proceedings of the AAAI conference on artificial intelligence (AAAI), volume 36, pages 2712-2721, 2022.
[11] Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William W Cohen. Murag: Multimodal petrieval-augmented generator for open question answering over images and text. arXiv preprint arXiv:2210.02928, 2022.
[12] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 112.5213, 2013.
[13] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence (AAAI), volume 30, 2016.
[14] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952 2015.

---

[15] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in IIms via reinforcement learning. arXiv preprint arXiv:2501.12945, 2025.
[16] Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng. Tianyi Zhou, and Cho-Jui Hsich. RI-zero's" aha moment" in visual reasoning on 2b non-sft model. arXiv preprint arXiv:2503.05 2025.
[17] Meng. L Du, Z Liu, Z Zhou, Lu, D Fu, B Shi, W Wang, J He, K Zhang, et al. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning. arXiv preprint arXiv:2503.07365, 2025.
[18] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics; Revisiting reinforce style optimization for learning from human feedback in llms. CoRR, 2024.
[19] Wouter Kool, Herke van Hoof, and Max Welling, Buy 4 reinforce samples, get a baseline for free! International Conference on Learning Representations Work Shop (ICLR work shop), 2019.
[20] Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang. Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503. 10615, 2025.
[21] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 1 (2):3, 2022.
[22] Jiaqi Deng, Zonghan Wu, Huan Huo, and Guandong Xu. comprehensive survey of knowledge- based vision question answering systems: The lifecycle of knowledge in visual reasoning task. arXiv preprint arXiv:2504. 17547, 2025.
[23] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on Computer Vision and Pattern Recognition (CVPR), pages 3195-3204, 2019.
[24] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision (ECCV), pages 146-162. Springer, 2022.
[25] Peng Wang, Qi Wu, Chunhua Shen, Authony Dick, and Anton Van Den Hengel. Fvqa: Fact-based visual question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 40110)2213224 2017.
[26] Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne Fine-grained late- interaction multi- retrieval for retrieval augmented visual question answering. Advances in Neural Information Processing Systems (Neurlps), 36:22820-291 2224000 2023.
[27] Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, André Araujo, and Vittorio Ferrari. Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3113-3124, 2023.
[28] Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming- Wei Chang. Can pre-trained vision and language models answer visual information-seeking questions? arXiv preprint arXiv:2302.11721 2023.
[29] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (I), pages 6769-6781, 2020.
12

---

[30] Man Luo, Yankai Zeng, Pratyay Banerjee, and Chitta Baral. Weakly-supervised vistal-retriever- reader for knowledge-beue question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (FMNLP 2021), 2021.
[31] Federico Cocchi, Nicholas Moratelli, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Augmenting multimodal Ilms with self-reflective tokens for knowledge-basee visual question answering. arXiv preprint arXiv:241 1616663, 2024.
[32] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems (Neurlps), 25:23716-237366 2022.
[33] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. In International Conference on Machine Learning (ICML), pages 17283-17300. PMLR, 2023.
[34] Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning (ICML), pages 19730-19742. PMLR, 2023.
[35] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint 24ki::2409.11991 2024.
[36] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-We Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[37] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems (Neurlps). 37:95095-95169, 2024.
[38] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-We Kei Chang, Yu Qiao, et al. Mathverse: Does your multi- Ilm truly the diagrams in visual math problems? In European Conference on Computer Vision (ECCV), pages 169-186. Springer, 2024.
[39] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning, arXiv preprint 25X1::2503.01785, 2025.
[40] Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-rl: A stable and generalizable r rl-style large lasion-language model. arXiv preprint ar MrXxv:2504.07611 2025.
[41] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03 2024.
[42] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang. and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint wxXv:2303.11388, 2023.
[43] Mathijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. CoRR, 2024.
[44] Hexiang Hu, Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. In Proceedings of the IEFF/CVE International Conference on Computer Vision (ICCV), pages 12065- 12075, 2023.
13

---

[45] Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In Proceedings of the IEEF/CVF conference on Computer Vision and Pattern Recognition (CVPR), pages 12884-12893, 2021.
[46] Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large- scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR), pages 2575-2584, 2020.
[47] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations (ICLR), 2019.
[48] Wenliang Dai, Jumnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023. URL htttt:t//axaiii org/ abs 06500.
[49] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern 2024.
Recognition (CVPR), pages
[50] Paul Lerner, Olivier Ferret, and Camille Guinaudeau. Cross-modal retrieval for knowledge- based visual question answering. In European Conference Information Retrieval (ECIR), pages 42 4-38. Springer, 2024.
[51] Jingyuan Qi, Zhiyang Xu, Rulin Shao, Yang Chen, dingnan jin, Yu Cheng, Qifan Wang, and Lifu Huang. Rora-vim: Robust retrieval-augmeettd vision language models. ArXiv, abs/2410.08876. 2024.
[52] Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Comia,, Lorenzo Baraldi, and Rita Cucchiara. Wiki-llava: Hierarchical retrieval-augmented generation for multimodal 1lms. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshop (CVPR workshop), pages 1818-1826, 2024.
[53] Zihan Ling, Zhiyao Guo, Yixuan Huang, Yi An, Shuai Xiao, Jinsong Lan, Xiaoyong Zhu, and Bo Zheng. Mmkb-rag: A multi-modal knowledge-basse based aenreeedl-angmented generation framework. arXiv preprint urXiv22504.10074 2025.
[54] Zhengxuan Zhang, Yin Wu, Yuyu Luo, and Nan Tang. Fine-grained cetrieval-augmente generation for visual question answering. CoRR, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We present the motivation of our method as clearly as possible in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We present the limitations of approach the end of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't i acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided the core of the paper should be complemented by formal proof' provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We present our method sufficient detail the Methods to allow readers to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibilityy can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: E-VQA and Infoseek are two public datasets we use. We will open source our code when appropriate.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
hhbllc//guddes///ddeetbmmmssss www.esuubbbssssooooooooooooo onPolicy for more details.
While encourage the release of code and data, understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central the contribution (e.g., for
open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips.cem wwwblicc/uiies/Coeeeeeee
serrissionii ssionPolccy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We explain our training and testing numbers detail in the Experimental
Details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We describe the metrics we use and their significance in section metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe the devices we used and the computation time in the implementa- tion details and further provide the inference time in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:p::://eerpppsscccpbb///////
Answer: [Yes]
Justification: Yes, our paper complies with NeurIPS ethical requirements.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We explain our motivation and the benefits the model in the abstract, which is consistent with the Broader impacts of L.LM research.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications. let alone deployments. However, if there is a direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition to attacks, mecha- nisms for monitoring misuse, mechanisms monitor how system learns from feedhack over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We use public data sets and do not involve risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We mainly base it on VLM-RI, whose code is licensed under CC-BY 4.0.
http://gtttbb..ommmmmaa-bb//L//u
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We use open source code on github and manage our code using github's open source license.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Studies of retrieval enhancement generation do not involve subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

for Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification:
We finetune Qwen2.5-VL. 3B/7B in our method.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.M policy (https: what should should not be described.
This appendix presents additional materials and results. First, describe the complete workflow of our method in Sec. to enhance comprehension. Then, we give further descriptions of our prompts in experiments Sec. B Next, we provide more ablation studies for Wiki-PRF in Sec. Finally, series of visual results are presented in Sec. D and the broader impacts are discussed in Sec.
A
Workflow of Wiki-PRF
We detail the complete workflow of Wiki-PRF below.
Processing Stage
Query Anysis: Given the reference image and question Q. Wiki-PRF begins by analyzing the
key information needed to solve the problem <think> and </think> and subsequently specifies the required tools using the <tool> Tool: Content </tool> format.
Tool Calling: Upon capturing tool request, Wiki-PRF parses tools enclosed in <tool> and </tool> tags and sequentially executes the corresponding functions.
For Captioning, Wiki-PRF feeds the content following the caption to VLM-PRF to generate
the retrieval query Quer Ycaptioning-
Querycaptistigg
For Grounding. Wiki-PRF first obtains the object coordinates from VLM-PRF, followed by performing the image cropping operation based on the coordinates. The resulting cropped image is then returned as the retrieval query Querygrouding-
For Flipping, VIM-PRF directly returns the flipped image Tnip-
Retrieval Stage: the retrieval stage, Wiki-PRF follows a two-step process: it first retrieves the top-k articles D) based on the reference image and then conducts further searches using the queries returned by the tools.
Captioning Search: Given QuerVcaptionin- Wiki-PRF initially retrieves the top ls most similar images and their associated documents from the knowledge base. These documents are then segmented into sections denoted as Seaptioning Subsequently, Wiki-PRE computes the similarity between Q Puerrycaptionigg and each section Scaptioning, and selects the top-k;, most relevant sections as the final retrieval results.
Grounding Search: Given Querygrounings same as Captioning Search, Wiki-PRF follows procedure similar to that of captioning search by first retrieving the sections Sgrounding. The key difference lies in the subsequent step, where the Wiki-PRE computes the similarity between the question Q and each section in Sgrounding. Finally, top-k, sections are selected as the retrieval results.
Constructing Search Result: Wiki-PRI takes the union of all retrieval results, and then concate- nates the sections in the union Seearch-
Filtering Stage: Given the documents D and the sections Search: Wiki-PRF leverages VLM-PRF to filter relevant information guided by the reference image and question Q. The reasoning process of VLM-PRE is presented within <think> and </think>, while the resulting task-oriented knowledge F is output within <answer> and </answer>.
Answering: With the task-oriented knowledge F, Wiki-PRF generates the final answer A.
21

---

B Prompts Details in Wiki-PRF
B.1 Processing Stage
Prompt for Tool Calling:
Given questoo whose auswer is within knowleige base. you need utilize one more folluwing sools the knowledge base by providing information you need: \ (cappion) i Provide detailed deseripiion related the question and the information will used (he external knowledge relevast knowledge pouns. ''goonnding'' V Ideaudy specitic ure. /Plip) ": Flip the image left or right. Enclose your reasoning process within <think> and <otinii without detailed illustatons, and specity the tools and contents you use within <tuols and shool to sid in querying the external knowlegge base. Example: <thinkorussuinng procee//hinkk> <tuol>
USER:
subject
question
retum
Plpp: Hipp left.
grounding:
panda
caption: A panda is climbing the tree with bird beside it.
</ool> Here as the user question, [Question..
Prompt for Captioning:
USER: Here the question. (Question) Liere the caption, (Caption). describe the image the cuntext of the question and the caption."
Prompt for Grounding:
USER: "Locose (object), oupput its coordinates using ISUN
B.2 Filtering Stage
"Here is user question, equestione> (Questicn) eiquestions. Here relevant infonmation retrieedd through inage retrivvl, exetrieved innforms- /Seacch)ctoabs, icttobs, <search_ressto resulh resure To cheain nstful information must conduct remoning inside <tiink <ttiikks fins time you new retrievod inforration. After reasoning, should the fittored information inside canswers clanswers, withost detailed illusiations."
USER:
booz [Uocement] cleenieved jnformatooo. Here as the relevast information through <tnab> Search
B.3 Prompt for Answer
USER: "Hore question, [Question). Hose retrieval information, (Senchh, results). shoet inswer:"
C Additional Experiments
C.1 Training Loss
In this section, we present the training curve of VLM-PRF-7B under reinforcement learning in E- E-Q.A. Figure displays three key metrics: answer reward, format reward, and task-oriented knowledge tokens. As shown in F both the answer reward and format reward exhibit a consistent upward trend, indicating that the model learns invoke tools and filter relevant information, its accuracy in answering knowledge-base VQA questions gradually improves. This clearly demonstrates the effectiveness of GRPO in enhancing the model's RAG capabilities.
Moreover, the tokens of the task-oriented knowledge decreases progressively with the number of training steps. This phenomenon suggests that the model becomes increasingly adept identifying and retaining only the most relevant knowledge during the learning process.
Answer Reward
Format Reward
Task oriented Knewledge To telens
EVVA
Step
Step
Step
Figure 5: The training curve of VLM-PRF-7B in E-VQA.
22

---

C.2 Tool Inference Time
In Table 12 we analyze the execution time individual tools under varying numbers of recalled articles (i.e., 3, 5, and 7). The results reveal that grounding requires more time than captioning. This can be attributed the image processing operations involved in grounding, which result in increased computational demands during execution.
Table 12:
Tool Calling Time Per Sample.
Model
Recall Numbers
Captioning
Grounding
3
0.99s
1.64s
VLM-PRF-31
5
1.07s
1.69s
7
1.11s
1.73s
C.3 Weights of Rewards
As shown in Table 13 we investigate the influence of varying the weight of answer reward (i.e., a) and the weight of format reward (i.e., ,3 + 7) in the overall objective function on the InfoSeek dataset. We fix the values of / and to 1.0, and continuously adjust the ratio of and (8 + 77). By gradually decreasing a (3 + 7) from 3 to 1 we observe that the optimal performance is achieved when equally weighted. Consequently, in our experiments, adopt equal ratio, both components where a 2.0, B 1.0, and 1.0.
Table 13: Ratios of Answer Reward and Format Reward Weights.
Model
3:1
2:1
1:1
1:2
1:3
VI.M-PRF-BB
38.80
38.20
39.48
38.89
38.53
C.4 The Number of Selected Sections
present ablation studies conducted on VLM-PRF-3B to evaluate the impact of varying the number of retrieved articles and sections during tool-based retrieval on the InfoSeek and E-VQA datasets. The tables report the final accuracy of Wiki-PRF-3B when top-1 and top-3 retrieved articles or sections are used during training.
In Table 14 and Table 15
The results indicate that the model performance generally improves as the number of selected sections increases. However, when only a single article is considered, the overall relevance of the article becomes the primary determinant of accuracy. The inclusion of redundant sections introduces noise and may lead decline performance.
Table 14: Retrieved Settings Ablation on Infoseek.
Table 15; Retrieved Settings Ablation on E-VQA.
Retrieved Settings
Top-1 Section
Top-3 Sections
Retrieved Settings
Top-1 Section
Top-3 Sections
Top-1 Article
38.96%
38.85%
Top-1 Article
24.28%
24.31%
Top-3 Articles
39.03%
39.10%
Top-3 Articles
28.15%
28.94%
Top-5 Articles
39.39%
39.48%
Top-5 Articles
32.10%
32.38%
D Qualitative Results
D.1
Comparison of Wiki-PRE
We conduct a comparison between our method and two baselines: Vanilla RAG and Wiki-PRF without the reinforcement learning fine-tuning (Wiki-PRF w/o RL). As shown in Figure we present comparison across various scenes, including plants, buildings, and animals. Examples 2 and 3 in
23

---

Figure and example in Figure 7 demonstrate the accuracy of our method in answering number- related questions. Examples 3 and 4 in Figure show that our method can still accurately answer questions when the target subject is far away. The comparison results fully illustrate the effectiveness of our method.
D.2 Illustration of Wiki-PRF on Variours Questions
In this section, we mainly show the case examples of Wiki-PRE. Figure and Figure show examples of visualizations of different tools working individually. Figure and Figure 10 show scenarios where the two tools work together, but are called in different orders. Figure 12 demonstrates that through tool calls, Wiki-PRF extends information retrieval retrieve information that contributes to the answer. Figure 13 shows an example of calling the captioning combination, proving that Wiki-PRF will make specific tool combinations according to the question. The filtered results across all samples are shorter than the original messages while retaining the correct answer, illustrating the effectiveness of the filtering stage.
Broader Impacts of Wiki-PRF
In this section, focus the broader impacts of our method. Our method facilitates assisting VLM to achieve better answers through knowledge retrieval. Importantly, any potentially harmful information encountered during the retrieval stage be mitigated by appropriately restricting the scope of the knowledge base.
24

---

E-VQA
Question: How many rooms does this hotel have?
Vanilla RAG The Izmailovo Hotel has 5,000 rooms.
VLM-PRF (w/o RL) The hotel in the image has 5,000 rooms. X
VLM-PRF The hotel has 1.77Z rooms. An Question: How many rooms does this hotel have? \n Short answer:
Ground Truth: [' 1,777 rooms']
Question:
What kind of perianth radial symmetry do the flowers of this plant have? Vanilla RAG The flowers of Schefflera arboricola are radially X symmetrical in their perianth.
VLM-PRF (w/o RL): The image provided does not show flowers, //nce//iiice the image only shows the foliage, which characteristic feature Schefflera arboricola, the discussion about perianth symmetry would be X inappropriate.
VLM-PRF flowers of this plant double perianth redial symmetry
Ground Truth: ['double']
Question: Who designed this palace?
Vanilla The Palacio de Marquesa de Sonora was designed by the architect Juan de Villanueva the early 19th century. //..///
X
VLM-PRF (w/o RL) The palace in the image//...// by Juan Gómez de Mora Juan de Hernera. Therefore, it not possible to determine the designer of the polace from the given information or the image.
VLM-PRF Ricardo Velázquez Bosco designed the Palace of Agriculture.
Ground Truth: ('Ricardo Velázquez Bosco')
Figure 6: Comparison on E-VQA
25

---

Infoseek
Question:
What is the weight of male of this bird gram?
Vanilla RAG A male least grebe typically weighs between 129-161 X grams.
VLM-PRF (w/o RL) The weight a male least grebe ranges from 112 to 180 grams. The bird in the picture appears to be male least grebe, given its size and characteristics. Therefore, the weight of this bird could be X anywhere between 112 to 180 grams.
VLM-PRF weight of the bird in the picture approximately
129 grams.
Ground Truth: [[ ('wikidata': 129,0, 'range': [1161000000000000000,,,,,,,,,,,, 141,9]]
Question: What is the sea level in metre of this mountain?
Vanilla RAG The provided image and details do not contain information about the sea// such the elevation the mountain the village.
VLM-PRF (w/o RL) The image does not provide specific data about the sea level or //.../ elevation of the mountain.
VLM-PRF The sea level in meters of this mountain 1228 meters.
Ground Truth: [[ ('wiiidata': 1228.0, 'range": [1105.2, 33508800000000000211
Question: What the immediately prior item that this bridge follows in series?
Vanilla RAG The Charles Bridge. X
VLM-PRF (w/o RL) The image does not provide enough context to determine what the immediately prior item in series. //..///Chrrles Bridge in Praque, Czech Republic, and does not indicate any preceding item.
VLM-PRF
The immediately prior item that this bridge follows in series is the Judith Bridge.
Ground Truth: ('Judith bridge")
Figure 7: Comparison on InfoSeek
26

---

Queshion: In which part(s) of the world does this fish lie? Visual Question
Processing Stage
you need to uhiize one or more following too is query the knowledge bose by providing information metion you seed:
Processing Queryi Given question whose ordwer withis Incwrledge base.
Tool Reasoning
Processing Anseer:
Ta deternine the partts) of the world where the fish lives, need to dentifyy the species and then the typical habitot. 11,1)
<think
Caption Provide a detailed desonption related the question
drgunddng Identiffy the specific core subject re teleted to the question. Flip: flip the image left right.
VLM-PRF
1. Cactianing The shows yellow and black striped fish coral reef structires.
Enclese your ressaning proceai within and without detailed illattratiend, querying
specify knowledge
contents you
&ecandiy The fish is a danselfiah species, likell of the genus Chromis. </hoal>
Captioning
Grounding
The image shows e fish with yellow and block stripes omong caral reef structures, poasibly found tropical around the
Retrieval Stage
Retrieval
Retrieval
Top
Top
Cord clyse inverteerates make up Uned butterflyfish' This tensitorialia contected their
Section gruet sllvery crean with stripes bedy. //
Sectior
fomily Chastodontidon thought Corel butterllyfish J1..!!
Section Title: Tspegraphy bounded the by Cucharas the by Purto !!..
spowning sutterfiyfish however believed they spaer/i...!
Very
Section Title Oceonogrsphy Bahia located Cubs in!
asdl Section Titlee Butherflyfidh When properly cored for, lutterflyfish can make booutiful and distinctive odditions fish only morine squariues. !!.!!!
#M Section Tinle: Froemation pimperii it species of rop-firned fien the family Hacmulidee native to the westem Atlontic Ocean (1.3)
Filtering Stage
Concot Top Tool Sections
Tool Sections: The white grunt is silvery crean color with rumercus yellow and blue horizontdl stripes// airipel/..//Maemraaa plunieni, the white grunt commen grunt, i species noy-firmed fish fomily iommulidor notive the western Atlantic Ocean/). //Cord palype. enall antmorax, alyse iveeteorttes make the lined bytterflyfish diet.
Filtering Query: Here the user question, In which part(s) the world does this Fish live? Here the relevant information retmoved through image retrieval, Thus fish lilely lives in The waters of Florida, United States/.... // Here is the relevent information in thrzugh Caphoning and Grounding TT. Tool Sections)../ ln To obitin aseful information. you must conduct reasoning inside first every time you get new retrieved information. After reasoting, you shou provide the fittered information inside and, witheat detailed ilsttrations.
Filtering
VLM-PRF
Filtering Result: <thisk ////thinic conpaer> The fish in the picture is likeyy the queen tri triggerfish rish (Boleense vetulo), as motchee the choroctensstos described in the retricved information, This netive western Atlantic Oceon and commanty fourd
with neets The ihhabit the
further
being quaes tnggerfihh supports liklitood mentioned that the species be of Fioridd" erticle babitut of the queen trogerfish, which includes canal reets and diverse morine life. Thorefore, the fish the image likely the queen triggerfish and can be found the waters Fondda United States, or the Bay of Pigs in Cuba. +/amwerr
trigerfias
included the of Cuba. The manne environment depicted is image aligni with
the of Pigi, which
VLM-Base
WI6-FRFF Answer: The fish in the picture lillyy the quees trigeefish (Balistes vetuis), which lives the western Atlantic Ocean, including aress with coral It be waters Flarids, States, Boy Pigs. lecoted Therefore, lives waters the westere Atlantic Oceas, apecifically in the regiond of Flarids and the Boy Figs.
Truth: ('western Ocson']
Figure 8: Illustration of Wiki-PRE on Question E-VQA_114 from E-VQA.
27

---

Questions In what country has the bes been feund?
Visual Question
Processing Stage
Processing Query: Siven quest or whose onzwer is within knowledge base, you need to utlize one or more following tools to query the knowlelge base by providing information you nued.
Pracessing Arower:
<think image shows plant, provide any about
Tool Reasoning
Coption frovide detailed description reloted the question.
</think
country
Grounding: Identify specific subject related the question. Fll: Flip image right.
VLM-PRF
-teel-
1. Elp: flpp the image left right.
your procees and specify knowledge
illustrations, quarying
2. goourding: the bee and the plant.
3. coption: bee collecting nechar from a purple flower.
Grounding
Captioning
A bes has been photographed collecting nector from purple flower natural setting
Retrieval Stage
Retrieval
Retrieval
Top
Tog
perplesue perplesue
Section Thtle
important
for much the commeecia velue. 2005, the //. //
#8 dissected
Canada,
Narth
Section Title Vicla
(tufted setch, bird vetch.//.. Canada
Section Title Ecology Erology (from Ancionn olwo (dikee) 'houne', stu
populacions incressing interests into other pollinstor like the /l..!f
Title
survival
logio)
onships ergenssns, Wi.!!
Top
The mand bles have reddish tipe. It has white hars on the hend and legs and yellow. This species occurs the norltine regicra of Canada, !/..!/
## Section Title The western honey bee impor polinntor crops, this
service accounts for much of the species commertal value //_
Concist Top Tool SecTions
Filtering Stage
Tesl Sections: Bombus perplexus species of bumblebee known the comman confusing bumolebee: Et is native sorthem North Americs, where it occurs across Conada and ints the eastem united States. The queen 17 2.1 centireter tosete nake up the lnsd butteriy Sa's
under centineter wice of the abdomes, //.. pohps, small chemones, and
Fiitering Query: Hore is the user qusstion, In what country hos this bee beon found? Ve Hene the relevant information retriesed through image netraval Wiki Artids Enstemn corperter boe Rit Section Timle Eastern corperter bee Xylocopo wingnice, conetines rel refereef/..l Here the relevent information on through Captioning and Grounding /T....Iee bachinal // obton useful information, you must conduct renseiing made every time you get netrieved information. reasoring, you should provide
Filtering
VLM-PRF
Filtering Realt: <think/// //f/thinkk vanavery The bes in the image, Eartemn carpenter bes (Xy loyooops vingin ca), has been found the Unined Shates and parts Canada, as delombed the retrieved information It found throughout much of North Americs the Rocky Mountais and for wo Nebreska, southern Ontene, and
</onpeer>
ILM-Base
PRE
vinginica) found
Canada
Ground Truth: ("Canadik)
Figure Illustration of Wiki-PRE on Question E-VQA 1182 from E-VQA.
28

---

this cartle?
Visual Question
Processing Stage
Processing Atswer
mee to utilize nore following tools the knowledge bose by informot
shos buildig with architecurd fechres that aococitted with histoni institution, porentidly 2t.col//..1f
nead!
Reasoning
Captton: Frevide detailed description ralated the ouestion.
</hiank
Grounding Identify the spacific core subject related to the question. 3. Fip: Flip the mmgg left or right.
VLM-PRF
-teal>
1. Caprion: image depicts a plaque informatio about
Enclese your reasoning pracess withis and whout de toiled illustattion, and specify the tools contents you use within to queryinn the external knowledge bose.
complex, rndisating
siafficance anchieechudd features,
2. Greunding: The main sinject of the image the ploque proveingg detall about Zoorash complax.
Captioning
Grounding
mor kink the historical and architscturd significance of the Zborazh Castle complex, copocicting it with XV- XVII century structures and protected by state low.
The image
Retrieval Stage
Retrieval
Retrieval
de Section Title repostiion Exposttin hells the cadle has been houning signifiant collection of ari and wiilects informing //_!f
Fightficant collection and artifcets informing !!.!! Top
Top
## d Zberezh
busts
Besides these
woul find
Besides Zbereth countryside three expesttions //./
Section Titll: National Bank of Ukraine Souiding) The Nationall Bank Ukraine building (Uleneniar bypoese Hinichatwerr
Volpograd Volgograd (Russian Boarorpad, (negg orati (isterj). foomenly Thurityyn
Tvarisyet
Top
ee Section Titler Church of the Holy Apost es Peter and Paul, Ras The Church of the Holy Aposties Peter and Faul (Serbian:
be Section Titllr History Section Title: Trantyn Although the
orfanased 1555, documented evidence Traritynn the confluence of the Tearitsa
Filtering Stage
Concat Top Sections
Tool Sechions: naterials art collection foundetion compled expssthions the Zoorosh castle poloce, There archeologica and etnrogyaphi sxtibits, ochiesemente !!.
hall secred arts Hieterit Architectural Preservation expanded and anriched the trad tion and
relerget
Zbenazh Cestic zu Section Title Zberesh Castle Zberazh Castle (Jiranizs: 36ppazusi savan, Puish: Zunck Zbersau) is fortifies es deferee strenghald in Zbarashv/ 2/ Here the relewsee information through Captiorin and Grounding // [Top Teal Sectione] // le To obton usefil information, you must contuct reosoning insid first every time get new retievedd information After reasoning you shoul provide the filtered infomationn inside and without deteiled illuttetiors.
Filtering
VLM-PRF
Result: <thinke//... belanging histeric
Zberazh, "Hall Socced
houses collection significant toarspoople significancs
century
cestury, highighhtnng
VLM-Base
Wii-PPR Answer: name the hall ar this castle the Sacred
Truth: ["Sacred arts']
Figure 10: Illustratoon of Wiki-PRF on Question E-VQA 1747 1747 from E-VOA
29

---

longth building
Visual Question
Processing Stage
Precessing
need Query:
following
the Inowledge
shows lorge hitterical buildigg the background, anth the centrol structure resericting St. Peter's Bosilico in Vutticn City </think-
providing information
Reasoning
Captian: Provics detaited desioipt iption related the quest
2 Grounding: Identify the specific core subject related the question. 5 Flp: Fiip the mmge left or right
VLM-PRF
right. the man alements the the hatarical building and the
Enclede your reasoning process within and without defoiled illustrations, and
specify the toels and contents you use within and to querying the externa knowledge base.
street the foregreand
caption: detalled deser prian the imgge neaded question
length
Grounding
Captioning
How the building image meters?
Retrieval Stage
Retrieval
Retrieval
Top
Tap
Chelyshingk siperbelice
Section Titl
atrropphere
on//_.//
Ferestnch Pasanged Hotel, 210 flcorsff...)
Chriation
mile: Atmorphaetc entry meteorod through atrogghree
passage
## St. James's Bruges Sr. Janen's (int-Joksbukker) Brages.
Design
mezuarig
the deck//_..
Hit Section Mortie, Madrid Church San Martin (Sparipi: Tglesia San Martin) church locahed Madrid, Spain //.2/
mee Section Thtler The buildin was completed 1939 and opened carty 1940 xits great /1..7, and South Afrizon architects and digeihanes unthused about it. IV.
Filtering Stage
Concat Top Tool Sections
Tool dections: Under conctruction Atideeter Adrviistrativs Tower: 143 m (469 ft) risors. On hold Faredhtsh Roserged Honel, 210 (690 fr), 46 floors Expinos Peicce Hotel, 116 m (357 fr), Flors Tohron Third millemun Tower, Floos, Didor Complex, 26 floors Alborz Tower, 26 floors Narges Honel, /f..!
Fittering Query: Hers is user quection, What is the length this buildng netro? Here the relevest information retrieved through image netrievel The Popal Beallice Verioan simply Peter's Bustica is church Castioning should provide the pitered information insids atthout detalled llustration.
City /f Rercissance style ing every you get information
Filtering
VLM-PRF
Filtering Reault: cthikk///.////hiet <anpwer> The length of St. Peter's Besilice apprexinately 210 meters. //onswer>
VLM-Base
length building image
Boslio builing
resemb is medern office buildng, not the iconic besilca If you cre aaking about St. Peter's Senlice its length is ipprasinately 2.0 maters."
Truth: ( f''ikidata':
o, 'range": (198 o, 00000000000003))
Figure 11: Illustration of Wiki-PRF on Question Infoseck (atbeckk00011222 00012299 from Infoseek.
30

---

Question: What the conservetion stotus of this figh?
Visual Question
Processing Stage
Precessing ARSWEr:
Processing Query: siven o question whose orswer within a knowledge bass. you need to utiice one or more following took query the knowleige bose by providing informet mction you need:
ethint.
The iaag shous longe listarical buildin, bacigreund, with the central resembiing
Tool Reasoning
/tty//..//
Coption: frovide dehoiled description reloted the question question.
</think-
Grounding Identify specific care subject
VLM-PRF
stool>
fllp image right.
Coptiee: The indivduell is helding lorge fish, pessibly a Geloth Graper. fisting best. The fish hos greensst-breen coloration, the person dressed froling The
reasoning process and you
ilustronions, querying
specify leswledge
Gelorth Groper (Ppinepholks ihajore).
</toal
Captioning
Grounding
The individual halding lange fish, patantially a Goliath Grouper, while fisting on bast. It has greanish brown coloration on Detailed conservation on stotus infernation not provided in the original capriar.
Retrieval Stage
Retnieval
Retrieval
Top
## Section Title Conservation Myctenoparca banao TUCh Red near
speciis
threatened spacies
threatenndd
Tup?
back grouper.
#2 Section Conservation Myctersperzs bonaci on Red List near threstened species, vulnerbble incressse because is relatively slow breadar.
black reckfish rockfish. species roy-finned
##
predators Mycteropercs
#s Section Title: Pootfic halibut Hippalossss sterelapi, the Poofic halibut, is species of righteye Founderr This very lang species /i.!f Top
benaci include sondber shark (Corcherhiuu plumbeus) homeeli
NW Section Title Uniltsstion Mychhrrperrs benoci quite todty and important Feed fish 21t fished for eals and for sport //_1/
jett Myctensperse Myctersperce bonaci, grouper black Fish/7..1
reckfish mombled
Filtering Stage
Concat Top Tool Sections
Tool Sections: They considened apecies of lesst concern the Usited States and The Bchamas, but considered be near threatsned the westtem At Atlantic
Ocean, becouse their vilserable status in South America and reported threats throughout many oreas of Gestrol Amenica and the Cariebeon /1_//
Filtering Query: Hore the user quastion, What is the conservation status of this Fish? in Hers the relevent informat on retrevved through
ratrieval Mycherspercs
bonon, The block grouper rode rodefalh or marbled redifish. i speoes of morine ray-fimed fish. grouper from the aubfumily // /// Hene is the nelevent information through Captianing Ground //.. TTp Toel becheral. J/ sbroin useful information, you must condact reasoning de first every time you get new netrieved information, Affer information
reasoning,
illstratione.
Filtering
VLM-PRF
Fitteing Rendt: 11
The consoevation shotus the black Near Threstened, c/onpeer>
PLM-Base
picturn
Ground Truth: ('NT". "Near Threstuned', 'LA/nt')
Figure 12: Illustration of Wiki-PRF on Question Infoseek 00033513 from Infoseek.
31

---

Questions Which strect is this building located
Visual Question
Processing Stage
Precesting Antwer:
Processing Query: Given question whose arower within, a knowledge base. you need to utilize one or more followin tools query the knowledge base by providing information or you nead
asks for the
where builing
Tool Reasoning
//./
Coption: Provide
desoription
the question. question
</think
Identify speoific subyect
VLM-PRF
-tool-
Flip the imas process within and thoho detailed illustrations, within to querying the right.
Coption: Provide detailed desoription related to the question.
Enclose
specify the tools
Captisn: image omottel designed entrance bookstore satting, porably within European //hoai>
called
Mederme" The suggests
external knowledig bose.
Captioning
Coptioning
The image the entrance Nistorical buildin, possibly brokettone
The individst is helding lorge fish, potentially Golloth Grouper, while fahing on boat: It has a preenihh brean coloration. betailed conservation status information not provided the original caption.
suggested by 'Liesriie Anciume Medorna." // Ol and Modem Booketore." The architecture featunes intricate, 11, // and clack adding hamericc
Retrieval Stage
Retrieval
Retrieval
Top
Tap
#F Section Title: Gallery ee Section Title: Location The gallery has entrances from the tue des Peeto-Champs. rue de Banqqe and //_//
Kll Section Title Location The gallery has entrances from the rue des Putis- Champs, rue de la Banque and rue Vivienne.
Tog
Iog
Har Section Trtle La Posta Monaco the company responsible for postal service in Monaco. It branch Groupe La Poste (Francel.
xtt Section Tide: Gallery es Section Trile Location The gallery has entrances from The rue dos Petits Champs, rue de la Banque and rue Vivienne.
itW Section Title: Myctersperce banaci Mycterwgerca benaci, the bleck grouper, black reckfish mecbiled rockfiih. is species morie roy fiened Fish //
e* Section Titl:: Aspodiated notable people Artist Pablo Picasso married Oiga Kho here on 12 July 1918: the withesses were Jean Coctoau
Top
Tap
Hist Section Thil: Despite several alteration to the post office on George Street, by [351, 1, special Boand of Erequry wiieblished by the colomial government 11..1
Section Thtle The monumenta staircase No. led to the former home of Eugine François Vidacq ahter his disgracel)
Filtering Stage
Concat Top Tool Sections
Tool Sections: The gellery has entrences from the rue des Pehtte- rue la Banque rue Vivienne The gollery was sult 1823 by Manchous, President of the de Seront hotel Pehits Fores pazsage. It based plars drown up by the anchitest France's is Joan /T...
Natorica, location
Filhering Query: Here the user question, Which street this ba bolding located at? Here the relexant information retrieved through mage retripee reeie The Galene Vivienne information passages of Paris Grounding netrieved informet lan After rossening should provide the fiitered information insids and without detailed illutrations.
one
(n) long and (r) gellery registered N/ reasoning iraide every
//
Filtering
VLM-PRF
Filterig Result: chisso ////thinko vanswer> The Galeric Viserve lecoted the 2nd orrendissement Paris, France, It shuated betteen the Polye- Ports Sourse, and the Grands Boulewards. The estronces to the Golerie Vivienne ore from the rue des Pefits-Champe rue de la. Banque, and rue Vivienne. </eraerr>
TLM-Base
WIM-FRF Anower: This building is located at the Galene Vivienne The spec entrance shown in the image from the rue de la Banque.
Ground ("rux Vivieme'. Sanque')
Figure 13: Illustration of Wiki-PRE on Question Infoseek 00005094 from Infoseek.
32


Reviews:
[{"review_id": "RTlizns6UV", "reviewer": "Reviewer_aWe7", "summary": "This paper presents a retrieval-augmented generation (RAG) method named Wiki-PRF to address knowledge-based visual question answering (KB-VQA). To improve multimodal query quality and the relevance of retrieved results, Wiki-PRF utilizes a three-stage method consisting of Processing, Retrieval, and Filtering stages.  \n\nIn the Processing stage, a model, named VLM-PRF, invokes visual tools (captioning, grounding, flipping) to convert original images and questions into retrieval queries. The Retrieval stage uses EVA-CLIP and FAISS to perform multimodal knowledge retrieval on Wikipedia articles. The Filtering stage, VLM-PRF is used to filter irrelevant information and retain the most relevant content\n\nThe training of VLM-PRF is basically borrowed from the DeepSeek-R1 and its GRPO algorithm.\nExperiments on E-VQA and InfoSeek datasets show significant improvements, achieving state-of-the-art performance. Ablation studies validate the effectiveness of each stage and tool, while analyses demonstrate the efficiency of RL training with limited samples.", "questions": "Major concerns are raised in the Weaknesses section. Besides, I would like to ask the following questions:\n\n1. Why trains a single model to perform two tasks (query generation based on tool-use and retrieved results filtering) via RL? What if we train two separate models, one for each task?\n2. What is the scale of the knowledge base used for retrieval? How does the performance change with larger scale knowledge bases?\n3. In Table 4, what does \"combinations\" mean? why the sum of mean usage of all three tools is not approximately equal to the combination value?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**:  \n- Wiki-PRF’s three-stage pipeline, Processing-Retrieval-Filtering, is succinct and effective to address the challenges of multimodal RAG in KB-VQA. Leveraging visual cues to boosts retrieval quality, through tool-based query generation (e.g., grounding to focus on relevant regions) and VLM-powered result filtering. \n- As far as I can tell, in the field of KB-VQA, this is the first attempt to build a multimodal RAG workflow that is enhanced by reinforcement learning and tool-use techniques.  \n- Experiments on benchmark datasets (E-VQA, InfoSeek) demonstrate consistent SOTA results. Ablation studies confirm the contributions of each stage and tools.\n\n**Weaknesses**:  \n- **Limited novelty**: Despite the decent performance, the approach lacks novelty. It mainly combines existing techniques (RAG, RL, tool-use) without introducing significant new concepts or methodologies, thus it is valuable in terms of engineering practice but lacks academic contribution.\n- **Unclear description for Processing stage**: Section 3.2 does not clearly explain how the Processing stage works. For example, it is confusing how VLM is involved multiple times (the invocation order and input-output relationships). $Caption_{init}$ is also not defined.\n- **Missing details of training data**: The paper does not provide sufficient details on the training data used for VLM-PRF, which are crucial since RL training requires labeled data. It is unclear how the training data is constructed to contain the ground-truth of tool-use and filtering.\n- **Lack of Generalization Analysis**: While effective on E-VQA and InfoSeek, the paper does not thoroughly explore performance on other KB-VQA datasets or more diverse visual domains (e.g., medical images, satellite imagery).", "was_revised": true, "final_justification": "Thank you to the Authors for your patient rebuttal; my concerns have now largely been addressed. The experimental results and conclusions (1. MM RAG can be improved by RLVR; 2. MM RAG is helpful for knowledge-based VQA) of the paper are convincing. Thus, I believe the paper is worthy of acceptance.", "rebuttal": "Thank you for your constructive reviews and we will address your concerns as follows.\n\n**Q1**: Efficient engineering practice with limited academic contribution.\\\n**A1**: In this work, we recognize two fundamental challenges in KB-VQA problem: (1) fine-grained knowledge retrieval in complex visual scenes and (2) precise filtering of irrelevant information from large scale retrieved results. Based on the challenges above, our framework features three core innovations:\n\n1.  For coarse retrieval limitations, we innovatively explored a tool-based fine-grained retrieval mechanism.\n    \n2.  For irrelevant information, we innovatively employ a question-based filtering stage, rather than simple reranking\\[5\\].\n    \n3.  For the lack of extensive labeled data, we innovatively employ RL to train the above stages.\n    \n\nThe Processing-Retrieval-Filtering (PRF) framework resolves critical KB-VQA limitations in RAG, not only through architectural design but also via component-level technical innovations, enhances multi-step reasoning capability without extensive labeled data (Reviewer bmZr), and provides foundational insights for multimodal RAG design (Reviewer VEoU).\n\n**Q2**: More detailed description for Processing stage.\\\n**A2**:  We appreciate the reviewer’s feedback. The function of the VLM-PRF in the Processing stage is to generate an execution tool-usage plan based on the visual and textual inputs. As referenced in L131-133, given the image and question, the VLM-PRF outputs selected tools and execution order in `<tool>` tags.\n\n---\n\n*   Input: {Image, Question}\n    \n*   Output: {`<think>`Thought Process`</think>` `<tool>` 1. ToolA: ContextA; 2. ToolB: ContextB; ...  `</tool>` }\n    \n\n---\n\nComprehensive case study of InfoSeek and E-VQA illustrating this mechanism is available in the page 7-12 of supplementary material. We will add more detailed description in L131-133 to help understand the work of the processing stage.\n\n**Q3**: How VLM is involved multiple times.\\\n**A3**: After VLM-PRF plans the sequence of tool calls, the tasks are executed by VLM-base, a foundational model (Qwen2.5-VL-7B). This model is invoked multiple times to power specific tools like captioning and grounding. In essence, VLM-PRF provides the strategy while VLM-base delivers the core tool functionality, which explains its multiple involvements in the processing stage. We will further clarify the distinct roles of VLM-PRF and VLM-base in Section 3.2 in the revision.\n\n**Q4**: The definition of $Caption\\_{init}$.\\\n**A4**: Thank you for pointing this out. $Caption\\_{init}$ is the specific instruction for the caption tool when VLM-PRF calls it. For example, in a complex photo of flowers, it might be 'Please describe the small insect on the purple flower'. The VLM-Base then executes the caption tool, taking this instruction as input to produce the final caption. This allows our system to focus on relevant details. We will add this definition in L135 and annotate it detailed in Fig. 3.\n\n**Q5**: Missing details of training data.\\\n**A5**: Thank you for pointing this out. The training data follows the basic VQA format in EVQ-A \\[1\\] and Infoseek \\[2\\].\n\n*   Input: {Image, Question}\n    \n*   Label: {Answer}\n    \n\nWe clarify that our Wiki-PRF model requires no ground-truth labels for tool use or information filtering, as it learns these skills via RL. The model is guided by a reward signal derived solely from the final answer's correctness, which incentivizes it to autonomously develop effective strategies for tool invocation and content filtering. We will add details about the training data at L184.\n\n**Q6**: Generalization Analysis on other KB-VQA datasets.\\\n**A6**: To address the concern about generalization, we evaluate our model on the widely-used OK-VQA benchmark \\[3\\], following the standard setup from M2KR \\[4\\] for a fair comparison.\n\nWe can see that our Wiki-PRF-7B achieves a new state-of-the-art score of 77.8 on OK-VQA. The consistent performance improvement on multiple benchmarks confirms our method's strong generalization ability.\n\nDue to limited time and lack of suitable readily available benchmarks, we leave the evaluation of KB-VQA on medical/satellite images to future work. However, based on our current tests, we believe our approach is still applicable.\n\n| Method | Model | OK-VQA | E-VQA | InfoSeek |\n| --- | --- | --- | --- | --- |\n| Qwen2.5-VL-3B | - | 62.1 | 19.6 | 21.4 |\n| Qwen2.5-VL-7B | - | 72.4 | 20.3 | 23.7 |\n| KU-RAG | LLaVA-Next-7B | 73.1 | 11.0 | 9.1 |\n| MMKB-RAG | LLaMA-3.1-8B | 65.4 | 35.9 | 36.4 |\n| **Wiki-PRF-3B** | VLM-PRF-3B | **68.6** | **32.4** | **39.0** |\n| **Wiki-PRF-7B** | VLM-PRF-7B | **77.8** | **36.0** | **42.8** |\n\n**Table R1:** Performance on OK-VQA, E-VQA, and InfoSeek.\n\n**Q7**: Why trains a single model to perform two tasks via RL? What if we train two separate models, one for each task?\\\n**A7**: Our primary goal is to develop a single model for RAG based multimodal QA. This model must not only achieves state-of-the-art accuracy but also considers resource consumption and performance in applications.\n\n1.  Through task analysis, we implemented joint training via RL to unify image tool selection and retrieved information filtering. Joint training enables the model to generate inputs explicitly optimized for downstream task performance.\n    \n2.  Training two separate models requires deploying two 7B models (14B in total) and doubles training time to ~30 hours. It is a computationally inefficient approach compared to our unified framework.\n    \n\n**Q8**: What is the scale of the knowledge base used for retrieval?\\\n**A8**: We adopt the identical knowledge base configuration established in \\[5\\], utilizing 100K Wikipedia articles for InfoSeek and 2M Wikipedia articles for E-VQA.\n\n**Q9**: How does the performance change with larger scale knowledge bases?\\\n**A9**: We evaluated our Wiki-PRF on the largest open-source benchmarks, InfoSeek (100k) and E-VQA (2M). Table 2 demonstrates its SOTA performance. To evaluate scalability with limited rebuttal time, we sample varying knowledge base proportions from InfoSeek (10K–100K articles) and evaluate the scalability of 3B and 7B model.\n\nTable R2 below shows that both our method and the baseline exhibit performance degradation as knowledge base size increases. This occurs because larger knowledge bases introduce additional noise, increasing retrieval difficulty, which is a universal challenge for current RAG methods. Critically, Wiki-PRF demonstrates significantly slower degradation for both 3B and 7B models compared to the baseline. Specifically, when scaling from 10K to 100K articles:\n\n*   Wiki-PRF-7B declines by 17.5 percentage points (60.3% -> 42.8%)\n    \n*   Vanilla-MRAG (7B) declines by 32.6 percentage points (56.3% -> 23.7%)\n    \n*   At 100K articles, Wiki-PRF-7B achieves 42.8% accuracy(1.8× the baseline's 23.7%), demonstrating superior scalability.\n    \n| Method | Model | 10K | 50K | 100K |\n| --- | --- | --- | --- | --- |\n| Vanilla-MRAG [6] | Qwen2.5-VL-3B | 49.7% | 32.1% | 21.4% |\n| Wiki-PRF-3B | VLM-PRF-3B | 53.0% | 43.7% | 39.0% |\n| Vanilla-MRAG | Qwen2.5-VL-7B | 56.3% | 39.6% | 23.7% |\n| Wiki-PRF-7B | VLM-PRF-7B | 60.3% | 51.2% | 42.8% |\n\n**Table R2:** Ablation studies on the scale of knowledge base on InfoSeek.\n\n**Q10**: What does \"combinations\" mean?\\\n**A10**: A \"combination\" refers to aa ordered set of called tools, for example, {grounding, flip, captioning}, {captioning, grounding}, and {grounding, captioning} are three distinct combinations.  The number of tools being combined indicates whether the model has further explored more ways to use the tools through training. As shown in Table 5 in original paper, the model does use a wider variety of ordered tool combinations. We will add the above explanation  to L256.\n\n**Q11**: Why the sum of mean usage of all three tools is not approximately equal to the combination value?\\\n**A11**: Because \"mean\" (per-tool frequency) and \"combinations\" (distinct sequence count) measure fundamentally different dimensions. The \"mean\" value per tool represents the average number of times it is invoked per problem instance. For example, a value exceeding 1.0 indicates the model can complete complex tasks through repeated tool invocation. We will add the description of \"mean\" in L258.\n\n**References**\n\n\\[1\\] Mensink, T., Uijlings, J., Castrejon, L., Goel, A., Cadar, F., Zhou, H., ... & Ferrari, V. (2023). Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories. (ICCV2023)\n\n\\[2\\] Chen, Y., Hu, H., Luan, Y., Sun, H., Changpinyo, S., Ritter, A., & Chang, M. W. (2023, January). Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?. (ACL2023)\n\n\\[3\\] Marino, K., Rastegari, M., Farhadi, A., & Mottaghi, R. (2019). Ok-vqa: A visual question answering benchmark requiring external knowledge. （CVPR2019）.\n\n\\[4\\] Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Byrne. 2024. PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers. (ACL2024)\n\n\\[5\\] Yan, Yibin, and Weidi Xie. \"EchoSight: Advancing visual-language models with Wiki knowledge.\" (EMNLP 2024).\n\n\n\\[6\\] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., ... & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. (Neurlps 2020)"}, {"review_id": "TjkIdGZhom", "reviewer": "Reviewer_bmZr", "summary": "This paper proposes Wiki-PRF, a three-stage framework (Processing, Retrieval, Filtering) for knowledge-based visual question answering (KB-VQA). By dynamically invoking visual tools (e.g., grounding, captioning) to extract precise multimodal features, integrating visual-textual retrieval, and applying reinforcement learning (RL) for relevance filtering, the method addresses challenges in retrieval-augmented generation (RAG), such as imprecise queries and noisy results. Experiments on E-VQA and InfoSeek datasets show that Wiki-PRF achieves state-of-the-art accuracy (36.0 and 42.8, respectively), outperforming prior RAG and vision-language models. The RL-trained VLM-PRF enhances reasoning and tool adaptation, demonstrating robust performance with minimal training data.", "questions": "Besides the weaknesses mentioned above, there are still some questions that are expected to be addressed in the rebuttal.\n1. In Figure 2, what is the specific VLM model used in the Filtering Stage? It seems that its inputs are pure text. Why not use an LLM instead?\n2. Could you provide comparisons with recent multimodal LLMs (e.g., GPT-4O, InternVL3) and discuss compatibility with the latest commercial VLM?\n3. Prior works like MMKB-RAG have conducted experiments on the OKVQA and M2KR datasets. I suggest adding results on these two commonly used datasets.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The three-stage design (Processing-Retrieval-Filtering) innovatively combines multimodal tool processing with RL-based filtering, addressing key limitations in traditional RAG for KB-VQA.  \n2. The RL-based training improves the model's reasoning ability without extensive labeled data.  \n3. Comprehensive experiments and ablation studies validate the effectiveness of each stage and tool, with clear performance gains over state-of-the-art baselines (e.g., ReflectiVA, MMKB-RAG).\n\nWeaknesses:\n1. The framework relies on a limited set of tools (captioning, grounding, flipping), which may not cover all KB-VQA scenarios.  \n2. The comparisons in Table 2 are not fair, as the base LLMs are different. For rigor, I suggest using the same LLaMA-3.1-8B model to compare with ReflectiVA and MMKB-RAG. \n3. The terms Wiki-PRF and VLM-PRF are confusing. I don't think it is necessary to use two different names.", "was_revised": true, "final_justification": "The meticulous response from the authors has addressed my concerns, and I believe the proposed multi-stage framework is reasonable. On the other hand, the relatively complex pipeline hinders the practicality and reproducibility. Therefore, I would maintain my initial rating as borderline accept.", "rebuttal": "Thank you for your constructive reviews and we will address your concerns as follows.\n\n**Q1**: The toolset used within the framework is limited.\\\n**A1**: In this study, we have chosen  three highly relevant tools (flipping, captioning, and grounding) for KB-VQA. However, our framework is flexible, not confined to a specific toolset, but rather tailored to the specific needs of the task. We will explore more tools for other tasks in future work.\n\n**Q2**: Use the same base LLMs (i.e., LLaMA-3.1-8B) for comparision. Provide comparisons with recent multimodal LLMs (e.g., GPT-4O, InternVL3) and discuss compatibility with the latest commercial VLMs.\\\n**A2**：Thank you for pointing this out.\n\n1.  We supplement a new set of experiments using LLaMA-3.1-8B as a unified backbone to fairly compare our Wiki-PRF framework with other methods.\n    \n2.  We extend this analysis to an additional powerful base model (i.e., InternVL3-8B) and compare against the latest commercial VLMs (i.e., GPT-4o) to demonstrate our method's general applicability.\n\nThe table  R1 demonstrates that under identical LLaMA-3.1-8B base model, Wiki-PRF achieves superior performance with consistent gains over ReflectiVA (+8.3 on EVQA, +0.7 on InfoSeek) and MMKB-RAG (+4.4 on InfoSeek), validating its efficacy.\n\nMoreover, Wiki-PRF adapts to different base models and can achieve consistent improvements. With a stronger base model (InternVL3-8B), our framework achieves a new SOTA score of 39.2% on E-VQA, surpassing all competitors. We will incorporate these comparisons in the Table 2.\n\nThe updated results are presented below:\n\n| Method | Model | E-VQA (Single) | E-VQA (All) | InfoSeek (UQ) | InfoSeek (UE) | InfoSeek (All) |\n| --- | --- | --- | --- | --- | --- | --- |\n| GPT-4o | \\- | \\- | 15.2 | \\- | \\- | 36.1 |\n| LLaMA-3.1-8B | \\- | 20.3 | 20.4 | 8.8 | 7.6 | 7.9 |\n| Qwen2.5-VL-7B | \\- | 21.7 | 20.3 | 22.8 | 24.1 | 23.7 |\n| InternVL3-8B | \\- | 24.3 | 23.7 | 24.2 | 21.4 | 22.0 |\n| ReflectiVA | LLaMA-3.1-8B | 28.0 | 29.2 | 40.4 | 39.8 | 40.1 |\n| MMKB-RAG | LLaMA-3.1-8B | 39.7 | 35.9 | 36.4 | 36.3 | 36.4 |\n| Wiki-PRF | LLaMA-3.1-8B | 36.3 | 35.5 | 41.3 | 40.6 | 40.8 |\n| Wiki-PRF | Qwen2.5-VL-7B | 37.1 | 36.0 | 43.3 | **42.7** | **42.8** |\n| Wiki-PRF | InternVL3-8B | **40.1** | **39.2** | **43.5** | 42.1 | 42.5 |\n\n**Table R1:** Performance on different base models.\n\n**Q3**: The terms Wiki-PRF and VLM-PRF are confusing.\\\n**A3**: We use Wiki-PRF and VLM-PRF to distinguish the method from the model.\n\n*   Wiki-PRF is our proposed framework (the full Processing-Retrieval-Filtering pipeline), as in L47.\n    \n*   VLM-PRF is the specific model that is trained via RL for tool-use and information-filtering within this framework, as in L52.\n    \n\n**Q4**: What is the specific VLM model used in the Filtering Stage? It seems that its inputs are pure text. Why not use an LLM instead?\\\n**A4**:\n\n1.  The VLM model used in the Filtering Stage is our VLM-PRF trained on Qwen2.5-VL-7B, same as the model in the Processing Stage.\n    \n2.  The filtering stage requires the image as a critical verification source to align visual content with retrieved information. For example, when answering \"What species is the green bird in the image?\" The visual grounding constitutes a fundamental requirement that pure LLMs cannot fulfill. Therefore, the VLM cannot be replaced with LLM. We will highlight the input image in Fig. 4.\n    \n\n**Q5**: Suggesting more experiments on the OKVQA and M2KR datasets.\\\n**A5**: Thank you for the suggestion. Table R2 presents the evaluation of Wiki-PRF on OK-VQA benchmark. It is worth noting that M2KR\\[2\\] is an evaluation framework for OK-VQA.\n\nOur method demonstrates consistent generalization, with our Wiki-PRF trained on Qwen2.5-VL-7B achieving a new state-of-the-art result of 77.8% (+4.7% and +12.4% over KU-RAG and MMKB-RAG, respectively).\n\n| Method | Model | OK-VQA | E-VQA | InfoSeek |\n| --- | --- | --- | --- | --- |\n| Qwen2.5-VL-3B |  | 62.1 | 19.6 | 21.4 |\n| Qwen2.5-VL-7B |  | 72.4 | 20.3 | 23.7 |\n| KU-RAG | LLaVA-Next-7B | 73.1 | 11.0 | 9.1 |\n| MMKB-RAG | LLaMA-3.1-8B | 65.4 | 35.9 | 36.4 |\n| **Wiki-PRF-3B** | **VLM-PRF-3B** | **68.6** | **32.4** | **39.0** |\n| **Wiki-PRF-7B** | **VLM-PRF-7B** | **77.8** | **36.0** | **42.8** |\n\n**Table R2:** Performance on OK-VQA, E-VQA, and InfoSeek.\n\n\n**References**\n\n\\[1\\] Marino, K., Rastegari, M., Farhadi, A., & Mottaghi, R. (2019). Ok-vqa: A visual question answering benchmark requiring external knowledge. （CVPR2019）\n\n\\[2\\] Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Byrne. 2024. PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers. (ACL2024)"}, {"review_id": "apytqLzVlB", "reviewer": "Reviewer_VEoU", "summary": "This paper introduces a three-stage framework for knowledge-based visual question answering. The first stage processes the raw query with several tools, including captioning, grounding, and flipping to obtain revised queries. Then it uses multi-modal retrieval to obtain an initial set of retrieved documents with the revised queries. A VLM-based filtering is used to summarize and generate informative knowledge representation from the initial document set, and the selected documents are used for final answer generation. The filtering is trained from reinforcement learning using the reward designed from the final answer generation. The authors conduct experiments on E-VQA and InfoSeek to illustrate the effectiveness of the proposed method.", "questions": "1. Is the VLM used for final answer generation a fixed VLM model?\n\n2. From the inference time, it seems that the filtering stage is the most time-consuming part. Is it because of the long input or the reasoning part?\n\n3. Can you provide the training curve so that we can have an idea regarding the stability of the RL training?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. It is an interesting idea to use different tools for query expansion and use RL to guide the selection of useful knowledge representation. Those directions may provide new insights into the multi-modal retrieval-augmented generation framework design.\n\n2. The authors conduct extensive experiments on E-VQA and InfoSeek to compare with some strong models. They also provide a detailed analysis of the retrieval performance, filtering performance, and ablation study for different tools.\n\n**Weaknesses**\n\n1. One weakness is the verification of the effects of RL. I appreciate that the authors provide Table 5 to verify the effectiveness of the filtering. But since you have a set of \"ground-truth\" entities, it seems to me that a more straightforward way would be to train the answer filtering to provide context regarding the ground-truth entities, rather than performing complex RL-based training. I would expect the authors to provide the comparison or provide additional explanations that it is not feasible. It would help us better understand the rationale for using the RL-based training.\n\n2. There are a couple of narratives that require additional explanations:\n- The authors choose GRPO with the removed KL divergence. Why do you remove the KL penalty?\n- The design of the reward also includes multiple terms. It would be beneficial if the authors could conduct an ablation study on the reward design.\n- The authors use another VLM for final answer generation. Why do you use two passes rather than complete the knowledge filtering and answer generation in one LLM?", "was_revised": true, "final_justification": "I decided to keep my positive score after reading the authors' response and other reviewers' comments. The author have clarified my concerns with additional experiments. I also think most other reviewers' comments have been addressed.", "rebuttal": "Thank you for your constructive reviews and we will address your concerns as follows.\n\n**Q1**: Verification of the effects of RL\\\n**A1**: Thank you for pointing this out. To investigate its effectiveness, we evaluated using SFT for the filtering stage. We trained a dedicated filtering model with SFT, maintaining the same configuration as our RL setup. The results are as in Table R1.\n\nThe RL model significantly outperforms the SFT model on the test set. We believe this is because SFT tends to imitate surface-level patterns, limiting its generalization capabilities. In contrast, RL enables the model to grasp the underlying principles of information filtering, leading to a far more robust and generalizable performance.\n\n\n| Model | Unseen Question (UQ) | Unseen Entity (UE) | ALL |\n| --- | --- | --- | --- |\n| Qwen2.5-VL-7B | 39.1 | 40.5 | 40.2 |\n| Wiki-PRF-7B (SFT) | 41.5 | 41.9 | 41.8 |\n| **Wiki-PRF-7B (RL)** | **46.6** | **46.2** | **46.3** |\n\n**Table R1**: Performance on the Test set.\n\n**Q2**: Why do you remove the KL penalty?\\\n**A2**: \n1.  To maximize exploration of tool usage, we remove the KL divergence term, which constraints the output distribution of the trained model to remain close to the base model while inherently limiting exploration. We explore tool-usage combinations while supervising result correctness via answer rewards during training.\n    \n2.  Following DAPO \\[1\\], we adopt KL divergence removal for unconstrained exploration, which is a strategy validated by recent work \\[2–4\\] that enhances exploration through this approach.\n    \n\n**Q3**: The design of the reward also includes multiple terms. It would be beneficial if the authors could conduct an ablation study on the reward design.\\\n**A3**: The reward weight ablation study is presented in Appendix C.3 (L68-72). We show the ablation result in the table below. Results demonstrate that a 1:1 weighting of reward terms yields optimal performance. This confirms both format and answer rewards are essential.\n\n| Method|3:1 | 2:1 | 1:1 | 1:2 | 1:3 |\n| --- | --- | --- | --- | --- | --- |\n| Wiki-PRF-3B | 38.80 | 38.20 | 39.48 | 38.89 | 38.53 |\n\n**Table R2:** Performance on different reward designs. X:Y means the weight ratio of answer reward and format reward.\n\n**Q4**: Why do you use two passes rather than complete the knowledge filtering and answer generation in one LLM?\\\n**A4**: After retrieval, our method retrieves substantial information, including both relevant content and semantically similar but irrelevant noise. Inspired by long-text processing techniques, we propose inserting a VLM-based filtering stage before answer generation. This filter selects relevant passages, removes noise, and then generates answers using the filtered information. Experiment results in Table R3 demonstrates that our method significantly improves question-answering accuracy.\n\n| Method | Processing | Filtering | VQA Accuracy |\n| --- | --- | --- | --- |\n| Wiki-PRF-3B | ✔ | \\- | 36.76 |\n| Wiki-PRF-3B | ✔ | ✔ | 39.48 |\n\n**Table R3** Ablation of Filtering\n\n**Q5**: Is the VLM used for final answer generation a fixed VLM model?\\\n**A5**: The VLM used for final answer generation is flexible. But our framework employs a single vision-language backbone across all stages, enabling parameter sharing that reduces total parameters by half while maintaining inference efficiency through weight inheritance. This design eliminates redundant model instantiation, directly addressing scalability constraints in multimodal RAG systems.\n\n**Q6**: Why is the filtering stage is the most time-consuming part?\\\n**A6**: As in Table 9 (L293-L294), compared to the reasoning part, processing a large amount of retrieval information (i.e., long input) takes more time. This makes the filtering stage the most time-consuming part.\n\n**Q7**: Can you provide the training curve so that we can have an idea regarding the stability of the RL training?\\\n**A7**: We provide the training curves in Appendix C.1 (L53-60) demonstraing the convergence of our RL training process.  As shown in the figure, the answer reward and format reward exhibit stable improvement, while the number of tokens decreases steadily over training. It indicates that the RL successfully optimizes both answer quality and conciseness.\n\n**References**\n\n\\[1\\] Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., ... & Wang, M. (2025). Dapo: An open-source llm reinforcement learning system at scale. _arXiv preprint arXiv:2503.14476_. (2025).\n\n\\[2\\] Zhang, X., Sun, H., Zhang, Y., Feng, K., Lu, C., Yang, C., & Meng, H. (2025). Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback. _arXiv preprint arXiv:2506.03106_.\n\n\\[3\\] Chu, X., Huang, H., Zhang, X., Wei, F., & Wang, Y. (2025). Gpg: A simple and strong reinforcement learning baseline for model reasoning. _arXiv preprint arXiv:2504.02546_.\n\n\\[4\\] Liu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., ... & Lin, M. (2025). Understanding r1-zero-like training: A critical perspective. _arXiv preprint arXiv:2503.20783_."}, {"review_id": "V3nFflys8J", "reviewer": "Reviewer_CtMV", "summary": "This paper focuses on the task of knowledge-based visual question answering (KB-VQA) and proposes a new three-stage RAG paradigm, which includes multimodal processing, multimodal retrieval, and filtering.", "questions": "See the weakness.", "limitations": "See the weakness.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n\nThe motivation of the paper is clear. \nThe paper is well-written and easy to follow.\n\n\nWeaknesses:\n\n\nThe authors claim that KB-VQA involves two distinct challenges. However, based on the discussion in the introduction, these two challenges appear to essentially describe the same underlying issue.\n\nIt is unclear why widely-used benchmark datasets for KB-VQA were not selected for evaluation. This omission weakens the experimental validation.\n\nWhile the paper emphasizes its novel RAG paradigm, the reported improvements seem to primarily stem from reinforcement learning (RL). The proposed RAG approach differs from traditional RAG mainly in the multimodal processing stage (e.g., through image captioning and object detection). It would be helpful for the authors to clarify whether the RL component could be replaced by existing large language models (LLMs). If so, the novelty of this work might be limited.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive reviews and we will address your concerns as follows.\n\n**Q1:** Differences between the two challenges mentioned in introduction. \\\n**A1:** In the introduction, we outlined two current challenges in the Visual RAG problem \\[1\\]:\n\n1.  In the retrieval stage, reliance solely on global image-text features prevents fine-grained analysis of the image, resulting in ineffective recall of high quality information.\n    \n2.  In the filtering stage, the retrieved information inevitably contains noise, which current ranking methods cannot fundamentally address, degrading answer quality.\n    \n\nThese two challenges are addressed by improving the granularity/accuracy of retrieved content and the relevance of retrieval results to the question.\n\nTo this end, we designed the Processing and Filtering stages:\n\n1.  By invoking image processing tools, we enhance fine-grained image understanding for the question and improve accuracy of retrieved content .\n    \n2.  By filtering recalled content using a visual language model, we remove irrelevant/redundant information, thereby improving accuracy of answer .\n    \n\n**Q2**: Why widely-used benchmark datasets for KB-VQA were not selected for evaluation?\\\n**A2**: In the paper, we evaluated our method on E-VQA and InfoSeek, which are two widely-used benchmarks in KU-RAG, MMKB-RAG, etc. To further validate our method, we experiment on OK-VQA benchmark \\[6\\] with the standard setup from M2KR \\[7\\] for a fair comparison.\n\nThe results indicate our Wiki-PRF-7B achieves a new state-of-the-art score of 77.8 on OK-VQA.\n\n| Method | Model | OK-VQA | E-VQA | InfoSeek |\n| --- | --- | --- | --- | --- |\n| Qwen2.5-VL-3B |  | 62.1 | 19.6 | 21.4 |\n| Qwen2.5-VL-7B |  | 72.4 | 20.3 | 23.7 |\n| KU-RAG | LLaVA-Next-7B | 73.1 | 11.0 | 9.1 |\n| MMKB-RAG | LLaMA-3.1-8B | 65.4 | 35.9 | 36.4 |\n| **Wiki-PRF-3B** | VLM-PRF-3B | **68.6** | **32.4** | **39.0** |\n| **Wiki-PRF-7B** | VLM-PRF-7B | **77.8** | **36.0** | **42.8** |\n\n**Table R1**: Performance on OK-VQA, E-VQA, and InfoSeek.\n\n**Q3**: Concern about the improvements seem to primarily stem from RL.\\\n**A3**: Actually, the performance improvement stems from the combined effects of the novel RAG framework and RL training. To reveal this, we implement a standard RAG based on Qwen2.5-VL as a baseline for evaluation, termed Vanilla-MRAG. As shown in Table R2, when both use the 3B model, our Wiki-PRF-3B (w/o RL) outperforms Vanilla-MRAG-3B by 2.54 points (36.76% vs. 34.22%). After using RL training, an additional 2.72 points of improvement are achieved (39.48% vs. 36.76%). The two improvements have an overall comparable impact on RAG capabilities. Similar results are observed for the 7B model. This demonstrates that our PRF framework plays an important role in improving the overall performance of RAG. We conduct our experiments on a 2K-subset of the InfoSeek test split.\n\n| Method | Model | InfoSeek |\n| --- | --- | --- |\n| Vanilla-MRAG-3B | Qwen2.5-VL-3B | 34.22 |\n| **Wiki-PRF-3B (w/o RL)** | Qwen2.5-VL-3B | **36.76** |\n| **Wiki-PRF-3B (with RL)** | VLM-PRF-3B | **39.48** |\n|Vanilla-MRAG-7B | Qwen2.5-VL-7B | 37.53 |\n| **Wiki-PRF-7B (w/o RL)** | Qwen2.5-VL-7B | **39.71** |\n| **Wiki-PRF-7B (with RL)** | VLM-PRF-7B | **43.12** |\n\n**Table R2:** Ablation of our framework on a 2K-subset of the InfoSeek test split. (w/o: without)\n\n**Q4**: The proposed RAG approach differs from traditional RAG mainly in the multimodal processing stage.\\\n**A4**: Actually, the filtering stage is also one of the main differences. We use the filtering stage to replace the reranker module in traditional RAG. As in Tab. 6 and (Ln268-Ln275), our processing stage and filtering stage contribute equally to overall performance. Combining the analysis in A1, results in Table R3 show that the processing stage improves performance by 2.02% over the base model by enhancing recall of highly relevant documents, while the filtering stage achieves another 3.24% gain by filtering the most relevant information. Therefore, in Wiki-PRF, the processing stage and the filtering stage are equally important.\n\n| Processing | Retrieval | Filtering | VQA Accuracy |\n| --- | --- | --- | --- |\n| \\- | ✔ | \\- | 34.22 |\n| ✔ | ✔ | \\- | 36.24 |\n| \\- | ✔ | ✔ | 36.76 |\n| ✔ | ✔ | ✔ | **39.48** |\n\n**Table R3:** Ablation of Processing stage and Filtering stage on Wiki-PRF-3B.\n\n**Q5**: Whether the RL component could be replaced by existing LLMs.\\\n**A5**: More powerful VLMs will, to some extent, replace the gains from our RL component. However, the RL component is more suitable for our Wiki-PRF framework.  This is because reinforcement learning training can improve model performance without intermediate annotations. Coincidentally, our training data lacks labeled tool combinations (intermediate annotations) due to the high cost of annotation. This creates an implicit and irreplaceable advantage.\n\nBesides, larger VLMs bring greater deployment overhead and API costs in applications. Our RL component, however, can achieve better performance at a lower cost. As shown in Table R4, even Vanilla-MRAG-32B, a model four times larger, cannot achieve the same performance as our Wiki-PRF-7B on InfoSeek.\n\n| Method | Model | InfoSeek (UQ) | InfoSeek (UE) | InfoSeek (All) |\n| --- | --- | --- | --- | --- |\n| Qwen2.5-VL-32B | \\- | 33.9 | 32.7 | 32.9 |\n| Vanilla-MRAG-32B [1] | Qwen2.5-VL-32B | 41.2 | 40.3 | 40.4 |\n| Wki-PRF-7B (with RL) | VLM-PRF-7B | **43.3** | **42.7** | **42.8** |\n\n**Table R4**: Performance on larger models.\n\n**References**\n\n\\[1\\]Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., ... & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks.  (Neurlps 2020)\n\n\\[2\\] Faysse, M., Sibille, H., Wu, T., Omrani, B., Viaud, G., Hudelot, C., & Colombo, P. (2024). Colpali: Efficient document retrieval with vision language models.  （ICLR2025).\n\n\\[3\\] Lin, W., Chen, J., Mei, J., Coca, A., & Byrne, B. (2023). Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering.  (Neurlps 2023)\n\n\\[4\\] Abdallah, A., Piryani, B., Mozafari, J., Ali, M., & Jatowt, A. (2025). Rankify: A comprehensive python toolkit for retrieval, re-ranking, and retrieval-augmented generation. _arXiv preprint arXiv:2502.02464_. \n\n\\[5\\] Li, N., Liu, J., Shan, Y., Huang, M., & Li, T. (2025). Graph-based RAG Enhancement via Global Query Disambiguation and Dependency-Aware Reranking. _arXiv preprint arXiv:2506.11106_.\n\n\\[6\\] Marino, K., Rastegari, M., Farhadi, A., & Mottaghi, R. (2019). Ok-vqa: A visual question answering benchmark requiring external knowledge. （CVPR2019）\n\n\\[7\\] Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Byrne. 2024. PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers. (ACL2024)"}]

Author rebuttal:

Meta-review: Existing knowledge base visual question answering (KB-VQA) systems suffer from insufficient interactivity during knowledge retrieval and ineffective organization of retrieved information for Visual-Language Model (VLM). To address these issues, this paper proposes a three-stage visual language model based on the Process, Retrieve, Filter (VLM-PRF) framework.


This paper proposes Wiki-PRF, a three-stage (processing–search–filtering) pipeline that is highly effective at addressing the challenges of multimodal RAG in KB-VQA. The proposed method combines reinforcement learning and tool utilization. It is the first attempt in this field and provides new insights into query expansion and knowledge representation selection. Comprehensive experiments on E-VQA and InfoSeek, along with ablation studies, demonstrate the effectiveness of each stage and tool by consistently outperforming strong baselines such as ReflectiVA and MMKB-RAG.


The author's detailed response and new experiments have addressed many of the concerns that were raised. The effectiveness of the proposed method and its conclusions (that MM RAG can improve RLVR and is useful for knowledge-based VQA) are considered convincing. As a result, multiple reviewers have maintained a positive stance, deeming the paper worthy of acceptance. However, one reviewer issued a borderline rejection. This reviewer's main concern pertains to the evaluation setup of the VQA dataset. The AC has thoroughly reviewed this discussion and believes that the authors have appropriately addressed the reviewer's concerns.


Based on the author's careful rebuttal and additional experiments, the concerns have been resolved, and the proposed method and conclusions are considered convincing. Therefore, the AC has decided to accept the paper, and it is strongly recommended that the reviewers' comments be reflected in the camera-ready version.
Decision: Accept (poster)