SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge
Authors: Chuanhao Li
Zhen Li
Chenchen Jing
Shuo Liu
Wenqi Shao
Yuwei Wu
Ping Luo
Yu Qiao
Kaipeng Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Internet-Augmented Generation
Large Vision-Language Models
Primary area: natural_language_processing

Abstract: Large vision-language models (LVLMs) are ignorant of the up-to-date knowledge, such as LLaVA series, because they cannot be updated frequently due to the large amount of resources required, and therefore fail in many cases. For example, if a LVLM was released on January 2024, and it wouldn't know the singer of the theme song for the new Detective Conan movie, which wasn't released until April 2024. To solve the problem, a promising solution motivated by retrieval-augmented generation (RAG) is to provide LVLMs with up-to-date knowledge via internet search during inference, i.e., internet-augmented generation (IAG), which is already integrated in some closed-source commercial LVLMs such as GPT-4V. However, the specific mechanics underpinning them remain a mystery. In this paper, we propose a plug-and-play framework, for augmenting existing LVLMs in handling visual question answering (VQA) about up-to-date knowledge, dubbed SearchLVLMs. A hierarchical filtering model is trained to effectively and efficiently find the most helpful content from the websites returned by a search engine to prompt LVLMs with up-to-date knowledge. To train the model and evaluate our framework's performance, we propose a pipeline to automatically generate news-related VQA samples to construct a dataset, dubbed UDK-VQA. A multi-model voting mechanism is introduced to label the usefulness of website/content for VQA samples to construct the training set. Experimental results demonstrate the effectiveness of our framework, outperforming GPT-4o by $\sim$30\% in accuracy.

SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge
Chuanhao LP.133 Zhen Li?, Chenchen Jing', Shuo Liu', Wengi Shao
Yuwei 2 Ping Luo Yu Qiao', Kaipeng Zhang 163
'OpenGVLab, Shanghai Al Laboratory Beijing Institute of Technology Shenzhen MSU-BIT University "Zhejiang University 'The University of Hong Kong
Abstract
Large vision-language models (LVLMs) are ignorant of the up-to-date knowledge, such as LLaVA series, because they cannot be updated frequently due the large amount of resources required, and therefore fail in many cases. For example, if LVLM was released on January 2024, and it wouldn't know the singer the theme song for the new Detective Conan movie, which wasn't 't released until April 2024. To solve the problem, promising solution motivated by retrieval- augmented generation (RAG) is to provide LVLMs with up-to-date knowledge via internet search during inference, i.e., internet- augmmet- generation (IAG), which is already integrated in some closed-soured commercial LVLMs such as GPT-4V. However, the specific mechanies underpinning them remain a mystery. this paper, we propose plug-and-play framework, for augmenting existing L.V.Mss in handling visual question answering (VQA) about up-to-date knowledge, dubbed SearchLVLMs A hierarchical filtering model is trained to effectively and efficiently find the most helpful content from the websites returned by a search engine to prompt LVLMs with up-to-date knowledge. To train the model and evaluate our framework's 's performance, we propose pipeline to automatically generate news- related VQA samples to construct dataset, dubbed UDK-VQA A multi-model voting mechanism introduced to label the usefulness of website/content for VQA samples construct the training set. Experimental results demonstrate the effectiveness of our framework, outperforming GPT-4o by ~30% in accuracy.
1 Introduction
Large vision-language models (LVLMss e.g. GPT-4V [9), Gemini Series 20 and Grok 21 have received much attention for their impressive generative capabilities. They require large resource for data collection, cleaning, and training, restricting them from frequently updating models. However, new information and knowledge are created every time, making LVLMs ineffective many scenarios. For example, if we talk with LLaVA-1.6 23 (relcased on January 30, 2024) about the new Detective Conan movie (realeased on April, 2024), such as "the singer of the theme song", performs very badly. It is promising to augment LVLM VLMs by retrieving up-to-date knowledge via internet search during inference, i.e., internet-augmenter generation (IAG). Although commercial LVLMs such as
This work was done during the internship at Shanghai Al Laboratory. Corresponding Authors: wuyuwei bit. eit.edu.cn; kp_zhang@ fom
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

B
Query Generator
Search Engine
OO
Tille Tattetiv Corth
Delediive Conse, The Millio-
Google
Parser
Tille special
Llama
GPT
dollaz Ferbarant * Theme strg.
colaborntive
Content:
Visual Search
Segrest by The mosic
Segaretd 2: The nargle...
Q: W Woo sings this theme song?
Bing
websitn
A Aummente Generation
Hierarchical Filtering Model
Gemini
Diversity Selection
Content Filter
Website Filter
GPT
LLaVA
Qwen
Delective
Somment 3
Sognent
Millios-dolar Pentngram's theme song is
Segmert 2, clo...
Sugnent 1 Sagniett 2g
main
Answer: Aiko /
"Mutwal Love" by Alko.
segmentt
Figure 1: The proposed SearchLVLMs, a framework for LVLMs to access up-to-date knowledge.
GPT-4V [9] and Claude3 22 have the ability of IAG, the specific mechanics underpinning them remain undisclosed. This paper proposes Alug-and-play framework to augment different LVLMs in handling visual question answering (VQA) about up-to-date knowledge, named ScarchLVLMs.
We first introduce our overall framework applicable to different LVLMs for equipping them with up-to-date knowledge during inference. consists of four components: query generator, search engine, hierarchical filtering model, and augmented generation, as shown in Figure || Specifically, we begin by extracting queries via Bing Visual Search and LLMs for an image-related question. Then, we acquire helpful websites through search engines and extract their contents by web scraping. However, is impractical to augment LVLMs directly with the entire content of all websites, because: (1) Most LVLMs are poor at handling such long contexts. (2) Handling such long contexts is computationally intensive and consucnn.. To this end, a hierarchical filtering model trained to find the most helpful content for answering the question, which first efficiently sifts the websites based on each website's ' title and snippet, and then identifies the most helpful content from the filtered websites. Finally, the filtered content is fed to LVLMs to assist them in answering the question.
We construct a dataset called UDK- UDK-VQA based on up-to-date news, used to train the hierarchical filtering model and evaluate our framework's 's performance. We propose pipeline to automatically scrape the up-to-date news and generate news-related VQA samples. Specifically, we use search terms from Google Daily Search Trends and manually collected popular search terms queries to search for hot news. For each news piece, we segment its content and ask GPT-3.5 to generate question-anae annwerr pairs from each segment. Then, extract an entity for each question and replace with its hypernym. To compose a VQA sample, we use Bing to search images of the replaced entity and cluster them to reduce the outliers among them. In doing answering the generated VQA samples requires models to consider both visual and textual information. We use queries from different time periods to scrape news from different time periods to generate samples for the training and test sets, avoiding test data exposure in the training data. In the training set, we further use multi-model voting mechanism to label website's usefulness and content's usefulness for VQA samples, and combine the samples with websites and their content based on the label for training the hierarchical filtering model. In the testing set, we conduct manual screening to ensure its correctness.
To validate the effectiveness and generalizability of the proposed framework, we incorporate 15 ofate-of-the-art LVLMs into the framework, such GPT-4V [19] and LLaVA-1.6 (2). Notably, once the hicrarchical filtering model trained, our framework can adapt different LVLMs and improve their performance without any fine-tuning, Extensive experimental results demonstrate that our framework can significantly improve LVLM's ability to answer questions about up-to- date knowledge. Incorporating the LLaVA-1.6 model of our framework even outperforms the self-contained IAG-capable GPT-4o by ~30% in accuracy on UDK- VQA test set.
Our contributions are summarized as follows. (1) We propose the first open-source framework seamlessly incorporating existing LVL.Ms with up-to-date knowledge during inference. (2) We propose a pipeline that automatically generates VQA samples related to up-to-date news and construct

---

the first test set for evaluating LVLMs' ability to handle VQA on up-to-date knowledge. (3) Extensive experimental results on 15 satte-of-the-at LVLMs demonstrate the effectiveness of our framework.
2
Related Work
2.1 Retrieval-Augmented Generation
Recently retrieval-augmented generation (RAG) attracted increasing attention of both the natural language processing [ 9 10
18 27 28). REALM [ uses the and lision-ad--aanuuage [3] 4
query to retrieve the top k most relevant article snippets, and uses large language models (LLMs) to generate & responses, which are then combined to obtain a final output for question answering. Recently, [41 10 43 explores the internet-augmented generation (IAG) of LLMs to enable language models to access up-to-date information via search engines. Komeili et.al. [41] show that LLMs enhanced via search engines can generate less factually incorrect information during dialogue with humans. Lazaridou et.al. [10] uses few-shot prompting to enable LLMs to exploit knowledge returned from Google search to answer questions about factual and up-to-date information. In vision-aue and language, REVEAL (3] builds a memory by encoding open-world knowledge including image-text pairs, etc., and uses retriever find the most relevant knowledge entries in the memory. The memory, encoder, retriever, and generator are pre-trained in end-to-end manner. Re-ViLA ReLM 4 augments Flamingo [5], by retrieving relevant image-text pairs from the external image-text datasets 6 7 & for zero and in-context few-shot image-to-text generations. RA-CM3 [42] performs retrieval from an external memory for generating images and text. Differently, we focus on enabling LVLMs to retrieve up-to-date knowledge via Internet search during inference.
pairs, question-answernigg
2.2
Large Models with Search Engine
Recent years have witnessed a growing interest in exploring external tools for LLMs M 15 12
16 13 52.
Among them, some methods [13 17 can use search engines access up-to-date knowledge. Nonetheless, these methods usually focus on how to appropriately use different tools to enhance LLMs, such as using Python interpreter to generate complex programs [15), incorporating more external tools [13], or updating tools by acquiring new knowledge [17]. Although they can access up-to-date knowledge, they usually directly use the website snippets for augmenting generation. By contrast, this work focuses on interet-auugmett
generation and explores how to obtain more relevant up-to-date knowledge and effectively use retrieved knowledge augment LVLMs.
3 SearchLVLMs Framework
In this section, we introduce SearchLVLMs, a framework that seamlessly incorporate existing LVLMs, allowing these LVLMs access up-to-date knowledge without fine-tuning. The whole framework is illustrated in Figure| [ For a natural language question @ about an image V, we first extract queries for both @ and V via the query generator. Then we enter the queries into search engines, and the search engine would return related websites, each of which consist of a title and a snippet. To identify the most helpful content within the websites, website filter is used to filter the websites based on their titles and snippets, and content filter is further used to filter the content of the websites filtered by the website filter. Finally, we stitch the filtered content together to prompt existing LVLMs.
3.1 Query Generator
Question Query Generator. To get queries that make search engines return websites containing helpful content, we leverage large language models (LLMs) to extract queries for Q. Thanks to the language understanding capability of LLMs, the role played by each word can be well inferred from the grammatical information of Q, even if certain words are unknown for the LLMs. We use "Do not try to answer the question, just print the most informative more than three entities in the question. Put them on one line and separate them with comm." " to prompt LLMs to generate queries.
Image Query Generator. For an image V, we leverage Bing Visual Search to analyze the image entities of as queries. The reason for using Bing Visual Search rather than a LVLM to extract queries for V is that current LVLMs are inadequate in extracting image entities especially for emerging
3

---

entities. Notably, Bing Visual Search is a tool different from commonly used search engines, returning image-related attributes, including image entity names, image-related search terms and image-related websites. However, entity names are missing in most cases. To address this problem, we extract the longest public ancestor of related search terms and related website titles as the queries for V.
3.2 Search Engine
The extracted queries are fed into a search engine, and the search engine returns relevant websites with their titles and snippets. However, the returned titles and snippets often contain limited and incomplete information. For example, for website with title "Pororo Dragon Castle Adventure", the entire snippet returned by Bing is "Pororo and his friends were having fun when a little red dragon named Arthur appears above them Arthur who claims to be the king of dragons commands Pororo and his friends to search for his Dragon... ...". obviously there is more about "Pororo Dragon Castle Adventure" contained in the website. Thus we parse the textual content of all websites. For a website, not all of its content contributes to answering questions, we empirically divide the website content into segments every third sentence for a more granular selection of content.
3.3 Hierarchical Filtering Model
Since most of the existing LVLMs cannot receive long context as inputs, and long contexts can be computationally intensive and time consuming for them, it's necessary to filter the website content after obtaining the websites via the search engine. Towards this goal, we train a hierarchical filtering model, which consists of a website filter and a content filter to perform a two-step filtering.
The aim of the website filter is perform the filtering of websites based on their titles and snippets. Specially, a website scoring model is trained via instruction tuning, to predict how helpful a website will be answering a question, and the N websites with higher scores would be kept. The training samples are in the format (T. S, Q, V, Rw). where Rw is a quantitative usefulness in the interval [0, 1] representing how helpful a website with title and snippet S will be in answering Q related to V. Based on the samples, we construct instructions like "How helpful is an article with such a title and snippet in answering the question based on the image? Choose the best option. Titl: <T> Snippet: <S> Question: <Q> Options: A. 1.0 B. 0.8 C. 0.6 D. 0.4 E. 0.2 F. 0.0". In doing so, the score regression problem is converted into a classification problem, which is easier to learn.
Website Filter.
Content Filter. The content filter is used to select the most helpful content segments from the websites filtered by the website filter. For each content segment, we predict how helpful is it for answering Q by a content scoring model. The content scoring model is trained by samples in the format (C. Q. V. Re), where C is a content segment, and Re is the quantitative usefulness of C in answering Q. The instructions for training the content scoring model are in the format: "How helpful this context in answering the question based on the image? Choose the best option. Context: <C> Question: <Q> Options: A. 1.0 B. 0.8 C. 0.6 D. 0.4 E. 0.2 F. 0.0". We use the model to sort all content segments and select the M highest scoring ones as the obtained segments.
Diversity Selection. To avoid LVLMs answer questions using bias from repetitive contexts, we performed quadratic selection on the obtained segments based on diversity. Specially, we extract CLIP features (37] for all the segments and cluster them using k-means [44). The segments closest to the center of each cluster are stitched together as the final obtained content for prompting the LVLMs.
3.4 Augmented Generation
We augment existing LVLMs by prompting them with the final obtained content. Taking answer the multiple choice questions as an example, for question Q with candidate answers AI, A2. A and
A4. we use the prompt "Given context: <X> Question: <Q> Answers: A. A.<Ai> B. B.< C. C,< D. D.< Answer with the option's 's letter from the given choices directly based on the context and the image." where X denotes the final content obtained by the hierarchical filtering model.
4 UDK-VQA Dataset
To evaluate the effectiveness of our framework, we propose a pipeline to automatically scrape the up-to-date news and generate news-related VQA. The whole pipeline is demonstrated in Figure 2

---

Query Collection
Question G Generation
Image A ssignment
Google Daily Seurch Treads
G Google
GPT-3.5
Entity Replace
Image Search
Image Clusting
Bing
Human Collection
Parser
yamike::
Vin Co dodea
Brood XI11, Chiikawa, Opeenhemmer ...
Iile Dietis on sqpeatt chicg oat metir lie
viteg menenae
Apa
webaite Indocmattio
peir
pair
Images entity
cluster
Pseudo-Score Generation
Manual Screening
code Sais
Plan donde
GARPIE and
Biie sugoist Coiag
LLaVA
Qwen
Monkey
mPLUG-Owl2
and for data from different time pertods
Figure 2: Overall pipeline of the sample generation for the UDK-VQA dataset. For brevity, we only show one output item at several steps, such as the content segment returned by the Parser. Notably, we use queries from different time periods scrape news from different time periods to generate training samples and test samples, which is not reflected in this figure for brevity.
The pipeline is also used collect training samples for the hierarchical filtering model. We first collect hot search terms as queries to scrape relevant news returned by search engines. For each piece of news, every third sentence divided into segment. We then employ GPT 3.5 to generate question-answer pair for each segment, and extract entity in the question, replacing it with its hypernym. Bing Image Search used to find images for the replaced entity, and after removing outliers from the images using clustering, the images and the deeestion-nsweer pair are composed into VQA samples. We combine the VQA samples and website information (e.g., title, snippet and content), and introduce multi-mode voting mechanism to generate pseudo-score, constituting the training set. For the test set, manual screening is conducted to ensure the correctness test samples.
4.1 Query Collection
Google daily search trends is an available data source that reflects what's hot in real time, and is well suited as the query used to construct our dataset. However, we observe that most search terms of the Google daily search trends are related to politics and sports, which poses great limitation. Therefore, we further manually collect popular search terms to improve the query diversity. The popular search terms are collected from many other domains including films, technological products, anime characters, places of interest, and so on. These human-collecte queries were mixed with queries from Google daily search trends to be used for subsequent sample generation.
4.2 Question Generation
For each query, we use Bing search for relevant and up-to-date news. We divide every third sentence int the scraped news content into a segment, and prompt GPT-3.5 to generate a question-answe pair and several confused answers for each segment by: "Given context: <Con> Filling the blanks to generate a question about the most informative event of the context, generate an correct answer to th question in no more than three words based on context, and generate three incorrectly confused answers of no more than three words based on context. Question: where <Con> denotes segment. We design simple but effective rule to ensure the correctness of the generated pairs, which requires a model can answer a question Q with A based on a segment C, if the model can generate a question-ansee answer pair (Q. A) based on C.
Correct answer: Incorrect
answers: A. B. C.
4.3 Image Assignment
To generate VQA samples and avoid the model's reliance on language priors for answering, we create samples that necessitate an understanding of the image for correct answers. Firstly, we extract an entity for each question via named an entity recognition (NER) model [38]. Images of the entity
5

---

tht Wiber del Autirn heet tot
sggeess The Tipc vill op to 2027 cansiaa; When dd thi Collag battnaal Coa falloria dhe 30l--4 uuse??
qeedies Whkt mos cace of
(t5)
pdl.
Dhagg Waae
50c
wigges Stuph wikki
Lindd of cull sscle
Come las Gost pobl
dical Palt" perothaeve:: K2-
eoung senmere parile
nus Rirey alauee
(37%)
(6%L
(4%)
(b)
(c)
Figure 3: (a) Training samples. (b) Test samples. (c) Category statistics for the test set of UDK-VQA
are then obtained by Bing Image Search. Since the images returned by the search engine are noisy (outlier images), we cluster them based on the CLIP feature [37] of the images and keep only the images in the cluster with the highest number of images. Finally, the kept images are assigned the new question-answe -nnwerr pairs where the entity is replaced by its hypernym, to compose VQA samples. An obtained VQA sample can denoted as (V. Agt, { where Q is the generated question, V the entity image, A represents the ground-truth answer, and AG, is i-th confused wrong answer.
4.4 Pseudo-Score Generation
For a VQA sample generated from the content segment C, which we denote as the ground-truth segment for the sample, it is certain that C is most helpful in answering this sample. Inevitably, we must consider to what extent do the other content segments contribute to answering the sample? We propose a pseudo-score generation method that uses five LVLMs for voting to quantify how helpful content segment to a VQA sample into six values: 1.0, 0.8, 0.6, 0.4, 0.2 and 0.0. Specially, for a VQA sample with the ground-truth segment C from news fetched for query, we first sample four content segments from the news for the query beyond its ground truth segment. Then we use each sampled segment to prompt each of the five LVLMs to answer the VQA sample and count the rate of LVLMs that answer correctly as the pscudo-score for the segment.
In doing so, we obtain training samples for the content filter, in the format (C, Q, V. RR), where C' is a content segment, Q denotes a question related the image V. and R is the pseudo-score of how helpful C is to answer Q. Moreover, we count the maximum pseudo-score of all content segments in a news for a VQA sample as the pseudo-score for the news website, dubbed Rw, to build training samples for the website filter. The training sample format for the website filter is (T. Q. V, Rw), where T is the website title, S is the website snippet. By merging these training samples into the training instructions mentioned in Section 3.3 the hierarchical filtering model can be implemented.
4.5 Manual Screening
For constructing the test set, we do not use the pseudo-score generation method. A test sample (C, V. Q, Agr. ( can be seen as VQA sample with its ground-truth content segment C. is worth noting that C is only provided when testing the upper bound of performance. For each test sample, we randomly mix Agı and {A then assign them the options (i.e. A, B, and D), and add complementary option E. No Correct Answers, to evaluate LVLMs in multiple choice format. Moreover, we manually review all test samples to ensure that they are correct.
4.6 Dataset Analysis
To prevent test data exposure in the training set, we use queries and news from different time periods to construct the training and test sets. For the training set, we use queries from February 17 to March 31, 2024, to scrape news before April 10, 2024. The training sample counts for the website and content filters are 599,700 and 850,267. For the test set, queries from April I to April 31, 2024, are used to scrape news after April 10, yielding 1,000 test samples. We divide the test sample into seven categories: politics, entertainment, announcement, sports, economic, technology and society, based on their required knowledge. We visualize some samples and the statistics for test samples in Figure 3 A new version, UDK- 200-249-0024 is constructed, with details in the Appendix.

---

5 Experiments
5.1 Settings
Training. We implement two versions of the hierarchical filtering model, one using VLaVA-1.5- vicuna-7b [39] and the other using Qwen-V -L-Cht [2]]. In each version, we use same hyper- parameters to fine-tune two same LVLMs with LoRA 24 as the website filter and the content 7b or VL-hh,,, the entire training
filter, respectively. Whether finettnnig process is facilitated on two Nvidia A100 GPUs, using batch size of 128 over 3 epochs.
Baselines. We incorporate 15 representative LVLMs into the proposed framework including Gemini 1.5 Pro 20). GPT-4V [9] GPT-4o, InternVI.-11. 45 LLaVA-1.6 23 LLaVA-115 [39] XCom-
poser2 26 Monkey 25), CogVIM B0), MiniCPM- V2 [5), mPLJUG-O12 [1], Qwen- VL. (2).
MMAlaya [34, Xtuner (32) and VisualGLM [3). We implement Gemini 1.5 Pro. GPT-4V and
GPT-4o via their official webs and APIs. We implement other LVLMs based on VLMEvalKit (36.
Evaluation. We evaluate LVLMs on four datasets including GQA 46. InfoSeek 477, A-OKVQA
48 and the proposed UDK-VQA. The reason behind selecting GQA, InfoSeek, and A-OKVQA is to evaluate the generalization capability of our framework across datasets that do not necessitate up-to-date knowledge. In addition evaluating LVLMs via VLMEvalKit, we design additional matching patterns for each LVLM with respect to its answer format. For example, we additionally use the pattern "The answer XXX." for XComposer2 as often answers in this format. All evaluations are conducted with single Nvidia A A100 GPU.
5.2 Quantitative Comparison with SOTA LVLMs
We compare with state-of-the-art LVLMS on the UDK-VQA test set, including Gemini 1.5 Pro 20] GPT-4V [19], GPT-4o,, LLa 23 and Intern Intern 45] For Gemini 1.5 Pro, GPT-4V and GPT-4o, we implement their Raw version via official APIs, which do not have the ability of IAG.
Table I: Comparision with SOTA LVLMs UDK-VQA, where "Raw" represents the model without IAG ability (e.g., official API version), "IAG" represents the model with self-contained IAG-capable ability (official web version). "LC" represents the model with long context input. "Gen.", "Cham." " and "CLIP (C-F)" denote the method from [51), [13] and 47]. respectively. *** indicates that the method leverages our framework access up-to-date knowledge. "Ours" stands for incorporating the Raw baseline into our framework. The value outside/in () indicates the accuracy over samples that do not violate the content management policy of current/all model(s).
Model Variant
pol.
ent.
sp.
tech.
Soc.
overall
Gemini Raw
6.2 (5.7) 15.8 (16.3) 10.2 (11.9) 7.4 (8.1) 2.3 (2.5) 8.0 (6.2) 3.0 9.1 (9.5)
LC
61.7 (65.7) 71.5 (77.3) 73.5 (76.2) 77.2 (79.2) 72.7 (72.5) 81.3 (83.1) 62.1 (66.7) 76.4 (76.
1.5 Pro
Ours
82.8 (82.9) 79.6 (79.0) 91.8 (92.9) 81.5 (80.1) 97.7 (97.5) 84.0 (83.1) 90.9 (88.9) 83.3 (82.3)
Raw
21.1 (23.8) 31.5 (30.9) 16.3 (19.0) 16.7 (17.5) 15.9 (17.5) 41.3 (38.5) 21.2 (22.2) 24.2 (23.8)
GPT
IAG
62.5 (68.0 0) 61.9 (63.6) 63.3 (66.7) 62.4 (63.3) 70.5 (67.5) 80.0 (78.5) 69.7 (70.4) 64.5 (65.9)
4V
Ours
76.6 (83.8) 85.8 (85.8) 89.8 (90.5) 86.2 (86.4) 97.7 (97.5) 92.0 (90.8) 87.9 (92.6) 87.2 (87.4)
Raw
36.7 (40.0) 34.2 (36.1) 42.9 (47.6) 28.6 (30.4) 40.9 (42.5) 65.3 (67.7) 36.4 (38.9) 37.2 (37.8)
GPT
IAG
61.7 16 57.3 (58.4) 69.4 (64.3) 48.4 (47.9) 70.5 (75.0) 81.3 (81.5) 62.1 (61.1) 57.8 (57.9)
4o
Ours
86.7 (92.4) 89.6 (91.4) 98.0 (100) 83.9 (88.0) 97.7 (100) 90.7 (96.9) 89.4 (94.4) 91.8 (91.6)
Raw
43.8 (45.7) 32.3 (31.8 22.4 (23.8) 24.9 (23.2) 25.0 (25.0) 53.3 (55.4) 33.3 (31.5) 31.8 (31.2)
Gen.
39.1 (40.0) 31.5 (28.8) 18.4 (19.0) 25.7 (25.3) 36.4 (37.5) 44.0 (44.6) 39.4 (37.0) 31,3 (30.4)
LLaVA
Cham.
58.6 (58.1) I) 57.3 (57.5) 53.1 (57.1) 65.3 (67.2) 52.3 (52.5) 72.0 (67.7) 74.2 (72.2) 62.3 (62.7)
1.6
C-
55.5 (56.2) 56.5 (57.9) 34.7 (35.7) 54.0 54.5 (52.5) 62.7 (64.6) 56.1 (53.7) 54.7 (55.3)
Ours
86.7 (87.6) 91.9 (918) 93.9 (97.6) 88.1 (88.0) 90.9 (90.0) 93.3 (9388 95.5 (100) 90.2 (90.7)
Raw
43.8 (43.8) 53.1 (52.4) 49.0 (47.6) 29.9 (30.7) 34.1 (35.0) 73.3 (76.9) 37.9 (35.2) 42.6 (42.8)
Gen.
29.7 (30.5) 28.1 (26.2) 28.6 (26.2) 22.8 (23.5) 31.8 (32.5) 42.7 (46.2) 28.8 (27.8) 27.6 (27.6)
Intern
Cham.
59.4 (5..1) 61.9 (618) 55.1 (57.1) 55.6 (55.4) 52.3 (52.5) 65.3 (67.7) 71.2 (70.4) 59.3 (59.2)
VL 1.5
C-
59.4 (58.1) I) 65.0 (64.8) 44.9 (42.9) 54.2 (55.7) 47.7 (50.0) 65.3 (66.2) 53.0 (50.0) 57.7 (58.0)
Ours
90.6 (89.5) 95.4 (95.3) 98.0 (97.6) 88.9 (88.0) 100 (100) 96.0 (95.4) 98.5 (98.1) 92.9 (92.3)
7

---

Table 2: Experiments on GQA 46 InfoSeek 47
A-OKVQA 48
where GQA does Not Rely on
external Knowledge (NRK), InfoSeek and A-OKVQA Rely on Commonsense Knowledge (RCK).
Retrieval Sources
Datasets
Model
Variant
Local Data (clear GT)
Internet Data
GQA (NRK)
InfoSeek (RCK)
A-OKVQA (RCK)
(unclear GT)
CFR [50
72.10
Dracle-FII 47
45.60
Omni-SMol.A
49
84.10
Raw
61.66
37.86
75.53
LLaVA-1.6
Ours
62.33
41.25
76.22
Raw
74.03
51.13
84,53
Intem VL.115
/
Ours
74.41
53,10
84,59
Since Gemini 1.5 Pro is famous for receiving long contexts, we use all website content returned by the search engine of our framework to prompt directly, dubbed LC. For GPT-4V and GPT-4o, we test their self-contained IAG-capable ability via prompting their official web versions with "Retrieve relevant news and answer the question directly from the given options using the option letters based on the image.", "ogge. dubbed IAG. We incorporate each Raw baseline into framework Ours.
The experimental results on UDK- VQA are listed in Table we can observe that: (1) Intern with our framework achieves the best performance on almost categories of UDK-VQN VVA. (2) For all four baselines, our framework consistently improves their accuracy (e.g.. 22.7% and 34.0% absolute performance gains in overall accuracy for GPT-4V and GPT-4o, respectively).
(3) Our framework uses shorter contexts but has higher accuracy (e.g., 76.4% vs 83.3% in accuracy for LC and Ours variants of Gemini, respectively). The observations suggest that our framework is generalizable and effective enhancing the ability of LVLMs to answer questions about up-to-date knowledge.
In addition, the experimental results on GQA, InfoSeek and A-OKVQA are listed in Table 2 Since these datasets do not rely on the up-to-date knowledge, we use a simple strategy avoid misleading LVLMs with the up-to-date knowledge by invoking our framework when they respond with "E" (as mentioned Section 4.55) without retrieval. From the table, we can observe that our framework improves the performance of different LVLMs across various datasets. The improvements on these three datasets are not as significant as on our UDK-VOL dataset for the following reasons: (1) The GQA dataset does not rely on external knowledge and is used to evaluate the reasoning ability of LVLMs, which is beyond the scope of our framework. (2) Our framework focuses on retrieving the up-to-date knowledge, whereas the InfoSeek dataset and the A-OKVQA dataset rely on commonsense knowledge, much which has already been used the training data LVLMs.
5.3 Ablation Studies
The experimental results of ablation studies on the proposed UDK-VQL dataset are shown in Table 3 where we use LLa.6- (3 as the baseline. Firstly, we investigate simple IAG methods, including using the similarity between questions and segments to select segments, i.e., IAG (SIM Q), using the similarity between images and segments to select segments i.e., IAG (SIM V), using the averaged similarity of the the above two similarities to select segments, i.e., IAG (SIM QV). These methods show limited improvements and achieve unsatisfactory accuracy.
Then, we study the influences of different components of our framework on the performance. For the hierarchical filtering model, we study two popular LVLMs, LLaVA-1.5 For the query generator, we conduct experiments with NER (8). LLaMA3 40), GPT-3.5 and Bing Visual Search. We observe that: (1) Using different backbone for the hierarchical filtering model has little effect on performance. (2) Using multiple question query generators at the same time can result in better performance than using only one. (3) Using both the question query generator and the image query generator gives the best performance. These observations suggest that all components of our framework are effective in improving the baseline, and components complementary to each other.
[39 and Qwen VL [29].

---

Table 3; Ablation studies of our framework on UDK-VQA
Hierarchical Filtering Model
Query Generator (Q)
Query Generator (V)
Acc.
Model
Variant
1.vVA-155
QWen-VI.
NER ILaMA3 GPT-3.5
Bing Visual Search
(%)
Raw
31.8
IAG (SIM @)
46.1
IAG (SIM V)
47.1
LLaVA-1.6
IAG (SIM QV)
47.7
49.3
65,9
7
81.4
Ours
7
86.6
/
87.6
/
90.2
/
89.6
5.4 Analysis of Pseudo-Score Generation
We analyze the influences of using different LVLMs to gen- erate pseudo-scores on the performance. We categorize 10 LVLMs into two groups based their released date, the first group contains LLaVA-1.6, XComposer2, Monkey, CogVLM VLM and MiniCPM-V2, the second group contains mPL.UGOO112 Qwen-VL, MMAlaya, Xtuner and VisualGL.. Using these two groups generate pseudo-scores and PSG with G2. Experimental results are shown in Figure 4 which reveal that: (1) The proposed framework can be directly used to boost LVLMs that are not used for generating pseudo- scores, which show the transferability of our framework. (2) The use of more recent LVLMs for generating pseudo-scores al- lows for greater improvements in general. (3) Different LVLMs have different performance upper bound, some of them achieve limited accuracy (e.g., 70% accuracy for VisualGLM) even are augmented with ground-trute segments (GT Segment).
Copstat stersoyy
KCnmparerz
dubbed PSG with GI
Figure 4: Accuracy using different LVLMs generate pseudo-scores,
5.5 Analysis Training Strategy
Would jointly training the hierarchical filtering model with LVLMs result in greater improvements? We use LLa VA-1.5 391 as the backbone for the hierarchical filtering model and
Table 4: Experiments of different
training strategies on UDK-VQA
conduct experiments with Qwen- VL (29 and 1L. as the LVLMs. As shown Table 4 separate training, where the LVLMs are frozen during the training of the hierarchi- cal filtering model, leads more significant improvement in performance. The main reasons are: (1) Our training data uses pseudo-labeling instead of high-quality human annotation. Training based on such data may cause LVLMs to lose their original semantic understanding capabilities. (2) Our training and testing sets are generated from news from different time periods, involving different entities and having different dis- tributions. Training LVLMs on our training set easily leads to overfitting, resulting lower generalization the test set.
Training Strategy
LVLM
Variant
(%)
Raw
35.2
QWen-Vi WW
Ours
Joint
68.5
Ours
Separate
84.8
Raw
41.2
LLaVA 1.5
Ours
Joint
68.0
Ours
Separate
88.9
5.6 Analysis of Diversity Selection
In this section, we investigate the necessity of diversity selection. We compare our diversity selection (Div-K) with Top-K selection, and the experimental results of 10 LVLMs are shown in Figure

---

LLavA-1.6
XCamposer2
Monkey
CogVLM
Top-K
60
50
40
72
40
20
68
15
20
10
20
5
15
20
10
15
20
10
15
20
MMAlaya
MPLUG
Qwen-VL
Xtunor
VisualGLM
80
10
15
10 20 15
20 10
15
20
5
10
15
20
Figure 5: Comparison between Top-K selection and diversity selection (Div-K), where K denotes the number of stitched content segments for prompting LVLMs. For each sub-figure, the horizontal coordinate is K and the vertical coordinate is the accuracy. Note that an accuracy of means that the model fails the context length under the current setting of K, and is labeled as a triangle.
The Top-K selection means stitching content segments with the highest scores together to prompt the LVLMs. For Div-K, K denotes the number of clusters. Experimental results demonstrate that: (1) Our diversity selection outperforms the Top-K selection regardless of the setting of K for most LVLMs. (2) As K increases, the performance using the Top-K selection plummets. This is because content with high scores is similar, and if LVLM receives too many duplicate content as inputs, will misinterpret the instruction and thus repeat the inputs instead of answering the question. These experimental results prove the necessity and effectiveness of the diversity selection.
5.7 Analysis of Website Filter
Acc
An important capability of the website filter is the trade-off between the content filter efficiency and the LVLMs' accuracy. Adjusting the filtered website number N can control the token number that the content filter needs to process as a percent- age of the total token number returned by the search engine, dubbed 0. The variation in accuracy LVLMs as increases is shown in Figure g we can observe that: (1) The accuracy of LVLMs increases with 0, especially when s 40%. (2) The 40%.
increase in accuracy of LVLMs slows down after Therefore, setting = 40% achieves better trade-off, because the accuracy obtained by processing 40% tokens is close to 98% the accuracy obtained when processing 100% tokens.
10%
25%
49
65%
Figure 6: Accuracy under the con- tent filter processing different per- centages of website content.
5.8 Analysis Snippet Completeness
We consider whether providing complete snippets to the website filter could result in further improve- ments. The experimental results of snippet completeness are presented in the Appendix.
Conclusion
In this work, we have presented SearchLVLMs, a plug-and-play framework to augment LVLMS in handling visual question answering about up-to- knowledge. By introducing hierarchical filtering model, the framework enables LVLMs to access up-to-date knowledge. A UDK-VQA dataset further curated by scraping up-to-date news and generating news-related VQA samples. The dataset enables quantitatively evaluate the ability of LVLMs respond to questions about up-to-date knowledge. Experimental results on UDK-VQA demonstrate that our framework can significantly boost the performance of LVLMs for answering questions requiring up-to-date knowledge.
10

---

Acknowledgments This work was supported by the Natural Science Foundation of China (NSFC) under Grants No. 62172041 and No. 62176021, Natural Science Foundation of Shenzhen under Grant Key Research Platforms and Projects of the Guangdong Provincial Department of Education under Grant No. 20.2003/ 1034, and the China Postdoctoral Science Foundation (No. 2023M743003). This work was partially supported by the National Key R&D Program of China (No.2022Z0016000, No. No.20222D01601 No.2022ZD0161002 and the General Research Fund of Hong Kong 17.1720622 and 17209324.
No. NYY12023080714227000000
References
[1] K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang, "Retrieval augmented language model pre- pre-training," in International conference on machine learning. PMLR, 2020, pp. 3929-3938.
[2] A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi, "Self-rag: Learning to retrieve, generate, and critique through self-reflection," arXiv preprint arXiv:2310.
2023.
[3] Z. Hu, A. Iscen, C. Sun, Z. Wang. K.-W. Chang, Y. Sun, C. Schmid, D. A. Ross, and A. Fathi, "Reveal: Retrieval-augmentte visual-language pre-training with multi-source multimodal knowl- edge memory." in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 23 369-2333 379..
[4] Z. Yang, W. Ping, Z. Liu, V. Korthikanti, W. Nie, D.-A. Huang, L. Fan, Z. Yu, S. Lan, B. et al., "Re-vilm: Retrieval-augmenete visual language model for zero and few-shot image captioning." arXiv preprint arXiv:2302.0488
2023.
[5] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al.. "Flamingo: a visual language model for few-shot learning," Advances in neural information processing systems, vol. 35, pp. 23 716-23 736, 2022.
[6] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. "Microsoft coco: Common objects in context," in Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-1 2014, Proceedings, Part V 13. Springer, 2014, pp. 740-755.
[7] P. Sharma, N. Ding. S. Goodman, and R. Soricut, "Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning." in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 2556-2565.
[8] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut, "Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 3558-3568
[9] J. Sun, C. Xu, L. Tang, S. Wang, C. Lin, Y. Gong, H.-Y. Shum, and J. Guo. "Think-on on-graph: Deep and responsible reasoning of large language model with knowledge graph," arXiv preprint arXiv:2307.07697, 2023.
[10] A. Lazaridou, F. Gribovskaya, W. Stokowiec, and N. Grigorev, "Internet-augmented lan- guage models through few-shot prompting for open-domain question answering," arXiv preprint arXiv:2203.05 15, 2022.
[11] T. Gupta and A. Kembhavi, "Visual programming: Compositional visual reasoning without training." in CVPR, 2023, pp. 14 14953-1496 962.
[12] D. Gao, L. Ji, L. Zhou, K. Q Lin, J. Chen, Z. Fan, and M. Shou, "Assistgpt: general multi-modal assistant that can plan, execute, inspect, and learn," arXiv preprint arXiv:2306.088 08640, 2023.
[13] Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y. N. Wu, S.-C. Zhu, and J. Gao, "Chameleon: Plug-and-play compositional reasoning with large language models," in NeurIPS, 2023, pp. 43 447-43477
[14] Z. Yang, L. Li. J. Wang, K. Lin, E. Azarnasab, F. Ahmed, Z. Liu, C. Liu, M. Zeng, and
I. Wang, "Mm-react: Prompting chatgpt for multimodal reasoning and action," arXiv preprint arXiv:2303.1 1181, 2023.
[15] D. Surís, S. Menon, and C. Vondrick, "Vipergpt: Visual inference via python execution for reasoning," in ICCV, 2023, pp. 11 888-11 898.

---

[16] C. Wu, S. Yin, W. Qi, X. Wang, Z. Tang, and N. Duan, "Visual chatgpt: Talking, drawing and editing with visual foundation models," " arXiv preprint arXiv:2303. 04671, 2023.
[17) Z. Gao, Y. Du, X. Zhang, X. Ma, W. Han, S.-C. Zhu, and Q. Li, "Clova: A closed-loop visual assistant with tool usage and update." in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR). 2024.
[18] C. Jing, Y. Li, H. Chen, and C. Shen, "Retrieval-angmentee augmented primitive representations for com- positional zero-shot learning," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 3, 2024, pp. 2652-2666 2660..
[19] S. Bubeck, V. Chandrasckaran. R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg et al., "Sparks of artificial general intelligence: Early experiments with gpt-4," arXiv preprint arXiv:2303. 12712, 2023.
[20] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth et al., "Gemini: a family of highly capable multimodal models," arXiv preprint
arXiv:23/2. /805, 2023.
[21] G. Contributors, "Grok," htttp::/ggitubb.comm
grok- 2024.
[22] C. Contributors, "Claude," https://clauddee ai/ 2024.
[23] H. Liu, C. Li, Y. Li, B. Li, Zhang, Shen, and Y. J. Lee, "Llava-next: Improved reasoning. ocr, and world knowledge," January 2024. [Online]. Available: https://1lava-vl.guti github
1o/blog/ 202//2024-000 101-30-1110 10ava-llaaaaaa 1aa--aaaavaeeaa////aea
[24] E. J. Hu, Y. Shen, P. Wallis, 7. Allen-Zhu, Y. Li, S. Wang. L. Wang, and W. Chen, "LoRA: Low-rank adaptation of large language models," in International Conference on Learinng Repre- sentations, 2022. [Online]. Available: https: //s::://oppe /pppreevepeeeeeeee//eeeeee meet/forum?ddnnZeWkeete
[25] Z. Li, B. Yang, Q. Liu, Z. Ma, S. Zhang. J. Yang, Y. Sun, Y. Liu, and X. Bai, "Monkey: Image resolution and text label are important things for large multi-modal models," arXiv preprint arXiv:231 1.06607, 2023.
[26] X. Dong, P. Zhang, Y. Zang, Y. Cao, B. Wang, L. Ouyang, X. Wei, S. Zhang, H. Duan, M. Cao, W. Zhang, Y. Li, H. Yan, Y. Gao, X. Zhang, W. J. Li, K. Chen, C. He, X. Zhang, Y. Qiao, D. Lin, and Wang, Iiteenimmxccompseee2 Mastering free-form text-image composition and comprehension in vision-language large model," arXiv preprint
2024.
[27] C. Li, C. Jing, Z. Li, M. Zhai, Y. Wu, and Y. Jia, "In-context compositional generalization for large vision-language models," in Proceedings the the Conference Empirical Methods in Natural Language Processing (EMNLP), 2024.
[28] C. Li, Z. Li, C. Jing. Y. Wu, M. Zhai, and Y. Jia, "Compositional substitutivity of visual reasoning for visual question answering," in Proceedings of the European Conference Computer Vision (ECCV), 2024.
[29] Bai, S. Bai, S. Yang. S. Wang, S. Tan, P. Wang. J. Lin, C. Zhou, and J. Zhou, "Qwen-vl: A frontier large vision-language model with versatile abilities," arXiv preprint arXiv: /2966. 2023.
[30] W. Wang, Q. Lv, W. Yu. W. Hong. J. Qi, Y. Wang. J. Ji, Z. Yang, L. Zhao. X. Song et al., "Cogvlm: Visual expert for pretrained language models," arXiv preprint arXiv:2311.03079 2023.
[31] Q. Ye, H. Xu, J. Ye, M. Yan, H. Liu, Q. Qian, J. Zhang, F. Huang, and J. Zhou, "mplug-ow Revolutionizing multi-modal large language model with modality collaboration," arXiv preprint arX0v2231.00425,, 2023.
[32] X. Contributors, "Xtuner: A toolkit for efficiently fine-tuning Ilm," httts:///githui: com/ InternLM/xtuner 2023,
[33] 7. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang, "Glm: General language model pretraining with autoregressive blank infilling." in Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2022, pp. 320-335.
[34] D. Ltd., "mmalaya," http:////ggiaabbccca/aaaaaaaaaaa
2024.
[35] OpenBMB, "Miniipm-""lttttss//ggitte com Saamm/0peeBMBB//mnnn LLM, 2024.
[36] O. Contributors, "Opencompass: A universal evaluation platform for foundation models,"
https://github.. http:///pp....ooopppp/appppaaaaa 2023.
12

---

[37] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., "Learning transferable visual models from natural language supervi- sion," in Proceedings of the International Conference on Machine Learning (ICML), 2021, pp. 7748-8763.
[38] Devlin, M. Chang, K. Lee, and K. Toutanova, "BERT: pre-training of deep bidirectional transformers for language understanding." CoRR, vol. abs/18 2018. [Online]. Available:
http://arxij hopp:///rxiv..o orgooogg///./ooo aww/aa/0.....///.../ 04805
[39] H. Liu, C. Li, Y. Li, and Y. J. Lee, "Improved basclines with visual instruction tuning," 2023.
[40] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, Azhar et al., "Ilaam: Open and efficient foundation language models," arXiv preprint arXiv:2302. 13971, 2023.
[41] M. Komeili, K. Shuster, and J. Weston, "Internet-augmented dialogue generation," " arXiv preprint wrxiv:21070076666 2021.
[42] M. Yasunaga, A. Aghajanyan, W. Shi, R. James, J. Leskovec, P. Liang, M. Lewis, L. Zettle- moyer, and W.-T. Yih, "Rerrievalauugmented multimodal language modeling," in International Conference on Machine Learning. PMLR, 2023, pp. 39 755-3957777 769.
[43] J. Tian, H. Chen, G. Xu, M. Yan, X. Gao, J. Zhang, C. J. Liu, W. Xu, H. Xu et al., "Chatplug: Open-domain generative dialogue system with internet-augmented instruction tuning for digital human," arXiv preprint arXiv:2304.0777
2023.
[44] A. K. Jain and R. C. Dubes, Algorithms for clustering data. Prentice- Hall, Inc., 1988.
[45] Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, F. Cui, W. Tong, K. Hu, Luo, Z. Ma et al., "How far are
gpt-4v? closing the gap commercial multimodal models with open-source suites," arXiv preprint arXiv:2404. 1682 1, 2024.
[46] D. A. Hudson and C. D. Manning. "Gqa: A new dataset for real-world visual reasoning and compositional question answering." in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2019, pp. 6700-6709.
[47] Y. Chen, H. Hu, Y. Luan, Sun, Changpinyo, A. Ritter, and M.-W. Chang, "Can pre-trained vision and language models answer visual information-seeking questions?" in Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, 2023, pp. 14 948--19968
[48] D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi, "A-okvqa: A benchmark for visual question answering using world knowledge," in European conference on computer vision. Springer, 2022, pp. 162.
[49] J. Wu, X. Hu, Y. Wang, B. Pang, and R. Soricut, "Omni-smola: Boosting generalist multimodal models with soft mixture of low-rank experts." in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024, pp. 14 205-14215.
[50] B. X. Nguyen, T. Do, H. Tran, E. Tjiputra, Q. D. Tran, and A. Nguyen, "Coarse-to-fine reasoning for visual question answering." in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2022, pp. 4558 4566.
[51] W. Yu, D. Iter, Wang, Y. Xu, M. Ju, S. Sanyal, C. Zhu, M. Zeng, and M. Jiang, "Generate rather than retrieve: Large language models are strong context generators," in International Conference on Learning Representations, 2023.
[52] Li, Z. Gao, B. Zhang, T. Yuan, Y. Wu, M. Harandi, Y. Jia, S.-C. Zhu, and Q Li, "Fire: A dataset for feedback integration and refinement evaluation of multimodal models," Advances in Neural Information Processing Systems, 2024.
13

---

A
Appendix
A.1 Analysis of Snippet Completeness UDK-VQA
The incomplete snippets returned by search engines lead us to consider whether providing complete snippets the website filter could result in further improvements. We present the experimental results of snippet completeness on our UDK-VE VDK dataset in the Table [ where 0 represents the percentage as mentioned Section 5.7 For each website snippet, attempt locate the full sentence corresponding to the snippet by crawling the website's s content. However, the content of many websites could not be crawled. For such websites, we experiment with two strategies: (1) Discarding these websites during training and testing. (2) Using the incomplete snippets. The experimental results are shown in the table below, where "Raw" represents all snippets without completion, "Discard" represents strategy (I), and "Mixture" represents strategy (2). From the experimental results we can observe that directly discarding the websites leads a significant performance loss, as discarding reduces the number of usable wehsites by approximately half, thereby limiting the performance of the website filter. Furthermore, as increases, the performance of strategy (2) becomes increasingly close to that of all snippets without completion (i.e., Raw), which validates that the completeness of the snippets has little impact on accuracy.
Table 5: Experiments of snippet completeness.
0
Baseline
Variant
25%
40%
55%
70%
85%
100%
Raw
80.8
86.1
88.4
89.4
89.7
90.0
90.2
LL.
Discard
76.6
80.2
80.7
82.1
81.9
81.3
81.6
(Ours)
Mixture
83.6
87.9
89.4
89.6
89.8
90.4
90.2
Raw
84.6
89.2
91.1
92.9
92.4
92.7
92,9
Intern VL-1.5
Discard
79.7
82.2
82.9
83.8
84.0
82.8
83.2
(Ours)
Mixture
86.0
89.5
92.2
92.3
92.2
92.5
92.9
A.2 UDK-VQA-20240906 Dataset
To test the capabilities of LVLMs handle visual question answering on the latest up-to-date knowledge, we construct new UDK VQA--2220900000 test set based on the proposed data generation pipeline. For the UDK- VQA-20240905 test set, we use the queries of Google Daily Search Trends from September 1, 2024 to September 5, 2024, to scrape news after September 5, 2024. In addition, test set covers a
we manually construct some test samples to ensure that the ADK-VQA-20220 wider range of categories of knowledge. The total number of test samples is 50, and the samples can be divided into five categories, including game (5), sports (24), society (8), entertainment (9), and economic (4), based their required knowledge. The number in parentheses after each category indicates the number of samples in that category. As shown Figure answering the samples of HDK--QA-2024000 requires up-to-date knowledge. For example, the first sample with the question "Where is the Meditation Spot located at this Temple?" about the game "Black Myth: Wukong," which released on August 20, 2024, after the release date of existing LVLMs.
Quetion: Whem is the Meditatio Spot lecatod et thi Temple?
Question: la vutic county was the
peetur smmtee?
MultsChaaise
Madti-Chaioe
(A) Temple Gardes
(1) Swedon
Nenvix
Tozple
Ficland
Figure 7: Test samples of the UDK- NII-VQQ--2000 -00240009 test set.
14

---

Table 6: Comparison with state-of-the-art LVLMS on UDK-VQA-20240905
Model
Variant
game
sp.
SOC.
ent.
cco.
overall
Raw Ours
0.0
0.0
0.0
11.1
0.0
2.0
Gemini 1.5
20.0
58.3
75.0
55.6
100
60.0
Raw
20.0
25.0
12.5
22.2
50.0
24.0
GPT-4o
40.0
70.8
62.5
55.6
Ours
80.0
70.8
87.5
77.8
100
78.0
Raw
20.0
20.8
0.0
11.1
50.0
18.0
LLaVA-1.6
Ours
80.0
75.0
87.5
66.7
100
78.0
Raw
60.0
33.3
37.5
55.6
25.0
40.0
Inter VL-1.5
Ours
80.0
83.3
87.5
77.8
100
84.0
A.3 Experiments on UDK- UDK-VQA-20240905
We compare with state-of-the-an LVLMs on the UDK-VQA LOK-VQA-20240900 test set, including Gemini 1.5 Pro (20). GPT-4o, 1L. 23 and Inter1 45 Similar the principle for testing on the UDK-VQA dataset, we implement the Raw version of Gemini 1.5 Pro and GPT-4o via their official APIs, which do not have the ability of internet-augmenteed generation (IAG). For GPT-4o, we prompted its official web versions with "Retrieve relevant news and answer the question directly from the given options using the option letters based on the image." to test its self-contained IAG-capable ability, dubbed IAG. For LLaVA-1.6 and Intern InternVL-1.3 VL-1.5, we implement them via the VLMEvalKit toolkit [36.. Each Raw baseline is incorporated into our framework, dubbed as Ours.
The experimental results on the UDK- VQQ-22220902 test set are listed in Table 0 from which we can observe that: (1) Intern with our framework performs the best, suppressing closed-source business models including Gemini 1.5 Pro and GPT-4o. (2) All baseline models (Raw version) perform worse on the UDK 20AK-VQA-0220000 test set compared to the UDK-VQA test set, indicating that the newer the knowledge required by the test samples, the more challenging these samples are for LVLMs. (3) Compared to the experimental results on the UDK- VQA dataset, the improvement of our framework over the baseline has decreased. This is because there is greater discrepancy in entity distribution between the UDK-VQA-09D dataset and the training set of SearchLVLMS. (4) The accuracy of the Raw version of Gemini 1.5 Pro is lower, because this model tends to choose "E. Correct Answers" while other models are more inclined to select from (A, B, C, D), even though they do not possess the knowledge required answer such questions correctly.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract contains our main claims including motivation, the up-to-date knowledge retrieval-auge -augmeeedd framework, the pipeline for generating news-realated VQA samples and the curated dataset UDK-VQA.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations in a separare section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[NA]
Justification:
Our work is not related to theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide the implementation details including hyperparameter settings, baseline selection and evaluation details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
17

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: We do not provide open access to the data and code at this time, but can publish part of them at the rebuttal stage if the reviewers need it. The complete data and code will be published after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//abps.cc/pullic/uuiiee htapollii/gguides//ooeiieeeeeeee
ssssssssicyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the implementation details including hyperparameter settings, baseline selection and evaluation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow existing work in the areas work in and do not provide statistical significance for fair comparisons.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computer resources for reproducing the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our work conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of our work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We use search engines to access Internet data, and search engines have their own methods to avoid security safety risks. Moreover, samples in the test set we curated have been reviewed case by case.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We've cited the original paper of the code and model we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We will provide open access to part of the new assets the rebuttal stage if the reviewers need it. The complete assets will be published after the paper is accepted.
Guidelines:
The answer NA means that the paper does not release assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The test samples our curated UDK-VQA dataset are checked by co-authors.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The test samples of our curated UDK-VQA dataset are checked by co-authors.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "MG2f9eazJQ", "reviewer": "Reviewer_qhJo", "summary": "The paper proposes a framework that enhances large vision-language models (LVLMs) to handle visual question answering (VQA) tasks involving up-to-date knowledge. The system utilizes a search engine to retrieve relevant websites and parses their content for more comprehensive information. To manage the large volume of data, a hierarchical filtering model is introduced, consisting of a website filter and a content filter. These filters score and select the most relevant information, which is then used to prompt LVLMs for improved performance on VQA tasks. The framework also includes a diversity selection mechanism to ensure varied and unbiased content selection.", "questions": "Please see above.", "limitations": "The authors have made a commendable effort to address the limitations of their work, including 1) Separate Training for Filters and 2) LVLMs and   Content Snippet Limitations.\n\nSuggestions for Improvement:\n    1) Detailed Impact of Separate Training: The authors could provide more insights into how the separate training of filters and LVLMs specifically impacts performance and whether integrating these components could lead to substantial improvements.\n    2) Addressing Incomplete Snippets: While the paper acknowledges the limitation of incomplete snippets, it would be beneficial to discuss potential strategies to enhance snippet completeness or alternative methods to initially extract more comprehensive information.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Clear Motivation: The paper is well-motivated and addresses a very practical problem.\n2. Hierarchical Filtering Model: The two-step filtering process effectively manages large data volumes, enhancing the performance of LVLMs by ensuring that only the most relevant content is used.\n3. Comprehensive Evaluation: The introduction of the UDK-VQA dataset and the experimental comparisons with LVLMs seem to to provide evidence of the framework’s effectiveness.", "weaknesses": "1. \"First open-source framework which seamlessly incorporates existing LVLMs to equip them with up-to-date knowledge during inference.\" I am concerned about this statement. There have been many frameworks utilizing external knowledge. Many of them can seamlessly apply to LVLMs. I am wondering what makes the proposed method different from them. \n\n2. Lack of Comparison against Existing SOTA Methods Utilizing External Knowledge: I encourage the author to conduct experiments to include more SOTA methods directly utilizing retrieved external knowledge for clear comparison.\n\n2. The author mentions the construction of the dataset as one of the contributions. However, I am concerned there is limited novelty in the construction pipeline of UDK-VQA as the steps are very engineering and practical.\n\n3. The paper claims UDK-VQA to be the first test set for evaluating the ability of LVLMs in handling VQA about up-to-date knowledge. I have concerns about this. There have been plenty of VQA benchmarks requiring external knowledge in the field. I am wondering what really makes UDK-VQA different from the existing ones. Are the up-to-date knowledge? However, this terminology is very blurry. What is the definition of up-to-date knowledge? How do you define the timestamp threshold? \n\n4. Broader Evaluation on Other Benchmarks Requiring External Knowledge: Extend the evaluation to include more diverse datasets and LVLMs to verify the generalizability of the framework across different benchmarks requiring external knowledge.\n\n5. Performance Dependency: The framework’s performance is heavily dependent on the quality of the hierarchical filtering model, which may vary across different LVLMs and datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper.\nWe value your suggestions.\nHere, we would like to mention that our work focuses on multimodal Internet-Augmented Generation (IAG), a type of method to enhance LVLMs in handling multimodal prompts based on Internet knowledge.\nIAG is an emerging topic in the field of Retrieval-Augmented Generation (RAG), offering significant potential and application value.\nFor example, OpenAI's recently revealed SearchGPT employs IAG as one of its main technologies.\nWe have carefully addressed your comments, but due to the length, **some responses have been placed in the comments section**.\n\n---\n> **Weakness1:\nDifferences from existing frameworks.**\n\n**Response:**\nOur proposed framework significantly differs from existing works. Existing frameworks for LVLMs utilizing external knowledge can be categorized into two types: (1) Frameworks that retrieve from local knowledge bases. (2) Frameworks based on external tools. The first type of framework [2][6] often retrieves external commonsense knowledge, such as \"the Eiffel Tower is in Paris\" and \"the important function of a folding chair is portability\". This type of knowledge is not frequently updated and can be easily embedded in LVLMs during training. Differently, we focus on the up-to-date knowledge, which is frequently updated over time, such as \"how many gold medals did the USA win in the Olympics on August 6, 2024.\". LVLMs cannot acquire this type of knowledge during the training phase. The second type of framework [7][8] uses search engines to obtain the up-to-date knowledge knowledge, but they focus on the orchestration of different tools. Differently, we focus on how to retrieve and use the obtained knowledge to augment LVLMs, which is orthogonal to the second type of framework and our method can be embedded into their framework.\n\n---\n> **Weakness2:\nLack of comparison against existing SOTA methods utilizing external knowledge.**\n\n**Response:**\nThanks for your suggestions.\nWe immediately conducted experiments using more methods: GENREAD [6], Chameleon [7], and CLIP→FID [2].\nGENREAD uses LLMs for knowledge retrieval and cannot access the up-to-date knowledge. Chameleon is an external tool-based framework that uses snippets returned by search engines for augmentation. CLIP→FID is a framework based on local database retrieval, and we integrated it with some components of our framework to allow it to access the up-to-date knowledge.\nThe experimental results are shown in the table below, where **QG** denotes Query Generator and **HFM** denotes Hierarchical Filtering Model.\nWe can observe that our framework achieves the best performance, proving the effectiveness of our approach.\nIf you have any suggestions for frameworks to compare, please let us know. We look forward to discussing them with you.\n\n| Baseline   | Variant       | QG (*Q*) | QG (*V*) | HFM   | UDKAG  |\n|------------|---------------|----------|----------|-------|--------|\n| LLaVA 1.6 | Raw           | -        | -        | -     | 31.8   |\n|            | GENREAD [6]   | -        | -        | -     | 31.3   |\n|            | Chameleon [7] | -        | -        | -     | 62.3   |\n|            | CLIP→FID [2]  | ✓        | ✓        | -     | 54.7   |\n|            | **Ours**      | **✓**        | **✓**        | **✓**     | **90.2** |\n| InternVL 1.5 | Raw           | -        | -        | -     | 42.6   |\n|            | GENREAD [6]  | -        | -        | -     | 27.6   |\n|            | Chameleon [7] | -        | -        | -     | 59.3   |\n|            | CLIP→FID [2]  | ✓        | ✓        | -     | 57.7   |\n|            | **Ours**      | **✓**        | **✓**        | **✓**     | **92.9** |\n| Qwen-VL    | Raw           | -        | -        | -     | 35.2   |\n|            | GENREAD [6]   | -        | -        | -     | 23.5   |\n|            | Chameleon [7] | -        | -        | -     | 43.8   |\n|            | CLIP→FID [2]  | ✓        | ✓        | -     | 36.7   |\n|            | **Ours**      | **✓**        | **✓**        | **✓**     | **84.8** |\n| LLaVA 1.5  | Raw           | -        | -        | -     | 41.2   |\n|            | GENREAD [6]   | -        | -        | -     | 31.9   |\n|            | Chameleon [7] | -        | -        | -     | 60.7   |\n|            | CLIP→FID [2]  | ✓        | ✓        | -     | 58.5   |\n|            | **Ours**      | **✓**        | **✓**        | **✓**     | **88.9** |\n\n---\n> **Weakness3:\nLimited novelty in the construction pipeline of UDK-VQA.**\n\n**Response:**\n(1) Automatically generating high-quality internet VQA samples is challenging.\nWe employed several strategies to improve the quality of the generated data, resulting in VQA samples that can be used for training. The strategies include using an entity replacement strategy to avoid generating samples with meaningless images, and using consistency answering (as described in Section 4.2, second paragraph) and image clustering to automatically filter out incorrect samples.\n(2) With our pipeline, it is easy to collect a large number of VQA samples for training and regularly generate test data that is free from data contamination issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RRLKU3zUxC", "reviewer": "Reviewer_jo4p", "summary": "This paper investigates a novel approach to augment large vision-language models with up-to-date knowledge from the internet. The author proposes to extensively leverage existing search engines (e.g., Bing and Google) and foundation models like ChatGPT for web information searching and parsing. A hierarchical filtering model has been proposed to filter the irrelevant information from the retrieved websites. The method can augment existing LVLM for augmented generations and achieve the state-of-the-art (SOTA) performance on a newly proposed VQA dataset, i.e., UDK-VQA, where the proposed plug-and-play method can improve SOTA LLM/VLMs like Gemini 1.5 Pro, GPT-4v, GPT-4o and LLaVA 1.6. Extensive ablation studies have been conducted to analyze the effectiveness of each component of the proposed framework.", "questions": "Please refer to the Weaknesses section for more details.", "limitations": "N/A.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is very well written and strongly motivated, given the necessity of augmenting the LVLM with up-to-date knowledge. To the best of my knowledge, this should be the first paper on augmenting the LVLM with internet information in a comprehensive manner.\n\n2. The proposed method is novel, given its design choice and intelligent combination of search engine and foundation model to enable precise information retrieval from the internet. The reviewer believes that such a new systematic system will inspire future exploration of IAG for multi-modality foundation models.\n\n3. The proposed method can be plug-and-play into many existing foundation models without trial and error, demonstrating its universality and significance. Moreover, the foundation models augmented by the proposed method can be improved significantly, highlighting the effectiveness of the proposed method.\n\n4. Extensive analysis has been conducted on the proposed method to verify further the effectiveness of the proposed framework.", "weaknesses": "Overall, this is a strong paper. The primary concern is the cost of leveraging the search engine and LLM like ChatGPT. The author should elaborate more on this aspect to show whether such an IAG is cost-efficient. Moreover, the reviewer is also curious about whether the code of the proposed framework will be open-source to serve as the cornerstone for future research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the significance of my work and for your constructive comments, which will help enhance the quality of my paper.\n\n---\n\n> **Weakness1: The primary concern is the cost of leveraging the search engine and LLM like ChatGPT. The author should elaborate more on this aspect to show whether such an IAG is cost-efficient.**\n\n**Response:** Thanks for your suggestions.\nHere is a cost calculation for using our framework for inference. The main costs involved are generating queries with ChatGPT, calling Bing Text Search, and calling Bing Visual Search.\nThe cost of ChatGPT is `$1.50` per 1M input tokens and  `$3.00` per 1M output tokens.\nThe cost of Bing Text Search and Bing Visual Search are `$0.025` and `$0.015` per search, respectively.\nFor our framework, testing a VQA sample involves approximately 50/20 input/output tokens for ChatGPT, 3 Bing Text Searches and 1 Bing Visual Search.\nTherefore, the total cost for testing a VQA sample using our framework is approximately `$0.09`.\n\n---\n\n> **Weakness2: Moreover, the reviewer is also curious about whether the code of the proposed framework will be open-source to serve as the cornerstone for future research.**\n\n**Response:** We will release the dataset and the code once the paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r2bkMRjXnO", "reviewer": "Reviewer_PaHW", "summary": "The paper presents UDKAG, a novel framework designed to enhance the capabilities of Large Vision-Language Models (LVLMs) by integrating up-to-date knowledge retrieved from the internet during the inference phase. The authors have developed a hierarchical filtering model to identify pertinent content from search engine results and a UDK-VQA dataset to evaluate the model's performance on VQA tasks.", "questions": "I'm curious about the generalization ability after integrating up-to-date knowledge. Will the model forget some important information and perform worse on other data?", "limitations": "Yes", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces an novel internet-augmented generation framework to update newest knowledge.\n2. The creation of the UDK-VQA dataset is a valuable contribution.\n3. Significant performance improvement.", "weaknesses": "I'm not familiar with this field and from my perspective this paper may not have obvious weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in reviewing our paper and for your recommended acceptance.\nWe sincerely appreciate your valuable comments and feedback.\n\n---\n\n> **Question: I'm curious about the generalization ability after integrating up-to-date knowledge. Will the model forget some important information and perform worse on other data?**\n\n**Answer:** The performance of large vision-language models (LVLMs) with our framework remains robust even on datasets that do not require up-to-date knowledge. We have conducted experiments on more existing datasets, including GQA [1], INFOSEEK [2] and A-OKVQA [3]. GQA is designed to evaluate the reasoning abilities of LVLMs and does Not Rely on external Knowledge **(NRK)**. INFOSEEK and A-OKVQA Require external Commonsense Knowledge **(RCK)**, such as \"the Eiffel Tower is in Paris\" and \"the important function of a folding chair is portability\", rather than up-to-date knowledge. In contrast, our UDKAG dataset Require Up-to-Date Knowledge **(RUDK)**. Our experimental results, presented in the table below, demonstrate that our framework consistently improves various baselines across these three datasets, underscoring the generalizability of our approach.\n\n---\n\n| Baseline      | Variant   | Local Data (clear GT) | Internet Data (unclear GT) | GQA [1] (NRK) | INFOSEEK [2] (RCK) | A-OKVQA [3] (RCK) | UDKAG (RUDK) |\n|---------------|-----------|-----------------------|----------------------------|---------------|--------------------|-------------------|--------------|\n| CFR [5]       | -         | -                     | -                          | 72.10         | -                  | -                 | -            |\n| Oracle→FID [2]| -         | ✓                     | -                          | -             | 45.60              | -                 | -            |\n| Omni-SMoLA [4]| -         | ✓                     | -                          | -             | -                  | 84.10             | -            |\n| LLaVA-1.6     | Raw       | -                     | -                          | 61.66         | 37.86              | 75.53             | 31.8         |\n|               | **Ours**  | **-**                     | **✓**                          | **62.33**     | **41.25**          | **76.22**         | **90.2**     |\n| InternVL-1.5  | Raw       | -                     | -                          | 74.03         | 51.13              | 84.53             | 42.6         |\n|               | **Ours**  | **-**                    | **✓**                          | **74.41**     | **53.10**          | **84.59**         | **92.9**     |\n\n---\n[1] Hudson et al. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR 2019.\n\n[2] Chen et al. Can pre-trained vision and language models answer visual information-seeking questions? In EMNLP 2023.\n\n[3] Schwenk et al. A-okvqa: A benchmark for visual question answering using world knowledge. In ECCV 2022.\n\n[4] Wu et al. Omni-SMoLA: Boosting generalist multimodal models with soft mixture of low-rank experts. In CVPR 2024.\n\n[5] Nguyen et al. Coarse-to-fine reasoning for visual question answering. In CVPRW 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### **General response**

We would like to thank the Area Chairs and the Reviewers for carefully reading our paper and providing valuable comments.

We are pleased to hear that the majority of the reviewers found our paper "well-written " (jo4p) and "strongly motivated" (qhJo, jo4p).
We also appreciate the reviewers' recognition of the "novelty" (jo4p, PaHW) and the "universality and significance" (jo4p) of our work.
Additionally, we are glad to receive positive feedback on our experiments, which "provide evidence of the framework's effectiveness" (qhJo), "verify further the effectiveness of the proposed framework" (jo4p), and show "significant performance improvement" (PaHW).

We look forward to the upcoming discussion sessions and to making further contributions to the vision-and-language community.

Meta-review: This paper receives mostly positive rates (1 accept, 1 weak accept, and 1 borderline reject) thanks to a novel internet-augmented generation framework with convincing experimental results. Although overall reviews are positive, reviews still have some concerns such as computational costs and incomplete snippets, which should be addressed in the final version.
Decision: Accept (poster)