Title: Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering
Venue: NeurIPS 2025
Accepted: yes
Keywords: Knowledge-based Visual Question Answering; Retrieval-Augmented Generation
Abstract: The task of Knowlegde-Based Visual Question Answering (KB-VQA) requires the model to understand visual features and retrieve external knowledge. Retrieval-Augmented Generation (RAG) have been employed to address this problem through knowledge base querying. However, existing work demonstrate two limitations: insufficient interactivity during knowledge retrieval and ineffective organization of retrieved information for Visual-Language Model (VLM). To address these challenges, we propose a three-stage visual language model with Process, Retrieve and Filter (VLM-PRF) framework. For interactive retrieval, VLM-PRF uses reinforcement learning (RL) to guide the model to strategically process information via tool-driven operations. For knowledge filtering, our method trains the VLM to transform the raw retrieved information into into task-specific knowledge. With a dual reward as supervisory signals, VLM-PRF successfully enable model to optimize retrieval strategies and answer generation capabilities simultaneously. Experiments on two datasets demonstrate the effectiveness of our framework.


Peer reviews:
comments: SUMMARY:
This paper focuses on the task of knowledge-based visual question answering (KB-VQA) and proposes a new three-stage RAG paradigm, which includes multimodal processing, multimodal retrieval, and filtering.

STRENGTHS AND WEAKNESSES:
Strengths:


The motivation of the paper is clear. 
The paper is well-written and easy to follow.


Weaknesses:


The authors claim that KB-VQA involves two distinct challenges. However, based on the discussion in the introduction, these two challenges appear to essentially describe the same underlying issue.

It is unclear why widely-used benchmark datasets for KB-VQA were not selected for evaluation. This omission weakens the experimental validation.

While the paper emphasizes its novel RAG paradigm, the reported improvements seem to primarily stem from reinforcement learning (RL). The proposed RAG approach differs from traditional RAG mainly in the multimodal processing stage (e.g., through image captioning and object detection). It would be helpful for the authors to clarify whether the RL component could be replaced by existing large language models (LLMs). If so, the novelty of this work might be limited.

QUESTIONS:
See the weakness.

LIMITATIONS:
See the weakness.
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-17

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a three-stage framework for knowledge-based visual question answering. The first stage processes the raw query with several tools, including captioning, grounding, and flipping to obtain revised queries. Then it uses multi-modal retrieval to obtain an initial set of retrieved documents with the revised queries. A VLM-based filtering is used to summarize and generate informative knowledge representation from the initial document set, and the selected documents are used for final answer generation. The filtering is trained from reinforcement learning using the reward designed from the final answer generation. The authors conduct experiments on E-VQA and InfoSeek to illustrate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. It is an interesting idea to use different tools for query expansion and use RL to guide the selection of useful knowledge representation. Those directions may provide new insights into the multi-modal retrieval-augmented generation framework design.\n\n2. The authors conduct extensive experiments on E-VQA and InfoSeek to compare with some strong models. They also provide a detailed analysis of the retrieval performance, filtering performance, and ablation study for different tools.\n\n**Weaknesses**\n\n1. One weakness is the verification of the effects of RL. I appreciate that the authors provide Table 5 to verify the effectiveness of the filtering. But since you have a set of \"ground-truth\" entities, it seems to me that a more straightforward way would be to train the answer filtering to provide context regarding the ground-truth entities, rather than performing complex RL-based training. I would expect the authors to provide the comparison or provide additional explanations that it is not feasible. It would help us better understand the rationale for using the RL-based training.\n\n2. There are a couple of narratives that require additional explanations:\n- The authors choose GRPO with the removed KL divergence. Why do you remove the KL penalty?\n- The design of the reward also includes multiple terms. It would be beneficial if the authors could conduct an ablation study on the reward design.\n- The authors use another VLM for final answer generation. Why do you use two passes rather than complete the knowledge filtering and answer generation in one LLM?\n\nQUESTIONS:\n1. Is the VLM used for final answer generation a fixed VLM model?\n\n2. From the inference time, it seems that the filtering stage is the most time-consuming part. Is it because of the long input or the reasoning part?\n\n3. Can you provide the training curve so that we can have an idea regarding the stability of the RL training?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Wiki-PRF, a three-stage framework (Processing, Retrieval, Filtering) for knowledge-based visual question answering (KB-VQA). By dynamically invoking visual tools (e.g., grounding, captioning) to extract precise multimodal features, integrating visual-textual retrieval, and applying reinforcement learning (RL) for relevance filtering, the method addresses challenges in retrieval-augmented generation (RAG), such as imprecise queries and noisy results. Experiments on E-VQA and InfoSeek datasets show that Wiki-PRF achieves state-of-the-art accuracy (36.0 and 42.8, respectively), outperforming prior RAG and vision-language models. The RL-trained VLM-PRF enhances reasoning and tool adaptation, demonstrating robust performance with minimal training data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The three-stage design (Processing-Retrieval-Filtering) innovatively combines multimodal tool processing with RL-based filtering, addressing key limitations in traditional RAG for KB-VQA.  \n2. The RL-based training improves the model's reasoning ability without extensive labeled data.  \n3. Comprehensive experiments and ablation studies validate the effectiveness of each stage and tool, with clear performance gains over state-of-the-art baselines (e.g., ReflectiVA, MMKB-RAG).\n\nWeaknesses:\n1. The framework relies on a limited set of tools (captioning, grounding, flipping), which may not cover all KB-VQA scenarios.  \n2. The comparisons in Table 2 are not fair, as the base LLMs are different. For rigor, I suggest using the same LLaMA-3.1-8B model to compare with ReflectiVA and MMKB-RAG. \n3. The terms Wiki-PRF and VLM-PRF are confusing. I don't think it is necessary to use two different names.\n\nQUESTIONS:\nBesides the weaknesses mentioned above, there are still some questions that are expected to be addressed in the rebuttal.\n1. In Figure 2, what is the specific VLM model used in the Filtering Stage? It seems that its inputs are pure text. Why not use an LLM instead?\n2. Could you provide comparisons with recent multimodal LLMs (e.g., GPT-4O, InternVL3) and discuss compatibility with the latest commercial VLM?\n3. Prior works like MMKB-RAG have conducted experiments on the OKVQA and M2KR datasets. I suggest adding results on these two commonly used datasets.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a retrieval-augmented generation (RAG) method named Wiki-PRF to address knowledge-based visual question answering (KB-VQA). To improve multimodal query quality and the relevance of retrieved results, Wiki-PRF utilizes a three-stage method consisting of Processing, Retrieval, and Filtering stages.  \n\nIn the Processing stage, a model, named VLM-PRF, invokes visual tools (captioning, grounding, flipping) to convert original images and questions into retrieval queries. The Retrieval stage uses EVA-CLIP and FAISS to perform multimodal knowledge retrieval on Wikipedia articles. The Filtering stage, VLM-PRF is used to filter irrelevant information and retain the most relevant content\n\nThe training of VLM-PRF is basically borrowed from the DeepSeek-R1 and its GRPO algorithm.\nExperiments on E-VQA and InfoSeek datasets show significant improvements, achieving state-of-the-art performance. Ablation studies validate the effectiveness of each stage and tool, while analyses demonstrate the efficiency of RL training with limited samples.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:  \n- Wiki-PRF’s three-stage pipeline, Processing-Retrieval-Filtering, is succinct and effective to address the challenges of multimodal RAG in KB-VQA. Leveraging visual cues to boosts retrieval quality, through tool-based query generation (e.g., grounding to focus on relevant regions) and VLM-powered result filtering. \n- As far as I can tell, in the field of KB-VQA, this is the first attempt to build a multimodal RAG workflow that is enhanced by reinforcement learning and tool-use techniques.  \n- Experiments on benchmark datasets (E-VQA, InfoSeek) demonstrate consistent SOTA results. Ablation studies confirm the contributions of each stage and tools.\n\n**Weaknesses**:  \n- **Limited novelty**: Despite the decent performance, the approach lacks novelty. It mainly combines existing techniques (RAG, RL, tool-use) without introducing significant new concepts or methodologies, thus it is valuable in terms of engineering practice but lacks academic contribution.\n- **Unclear description for Processing stage**: Section 3.2 does not clearly explain how the Processing stage works. For example, it is confusing how VLM is involved multiple times (the invocation order and input-output relationships). $Caption_{init}$ is also not defined.\n- **Missing details of training data**: The paper does not provide sufficient details on the training data used for VLM-PRF, which are crucial since RL training requires labeled data. It is unclear how the training data is constructed to contain the ground-truth of tool-use and filtering.\n- **Lack of Generalization Analysis**: While effective on E-VQA and InfoSeek, the paper does not thoroughly explore performance on other KB-VQA datasets or more diverse visual domains (e.g., medical images, satellite imagery).\n\nQUESTIONS:\nMajor concerns are raised in the Weaknesses section. Besides, I would like to ask the following questions:\n\n1. Why trains a single model to perform two tasks (query generation based on tool-use and retrieved results filtering) via RL? What if we train two separate models, one for each task?\n2. What is the scale of the knowledge base used for retrieval? How does the performance change with larger scale knowledge bases?\n3. In Table 4, what does \"combinations\" mean? why the sum of mean usage of all three tools is not approximately equal to the combination value?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}