Title: SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge
Venue: NeurIPS 2024
Accepted: yes
Keywords: Internet-Augmented Generation

Large Vision-Language Models
Abstract: Large vision-language models (LVLMs) are ignorant of the up-to-date knowledge, such as LLaVA series, because they cannot be updated frequently due to the large amount of resources required, and therefore fail in many cases. For example, if a LVLM was released on January 2024, and it wouldn't know the singer of the theme song for the new Detective Conan movie, which wasn't released until April 2024. To solve the problem, a promising solution motivated by retrieval-augmented generation (RAG) is to provide LVLMs with up-to-date knowledge via internet search during inference, i.e., internet-augmented generation (IAG), which is already integrated in some closed-source commercial LVLMs such as GPT-4V. However, the specific mechanics underpinning them remain a mystery. In this paper, we propose a plug-and-play framework, for augmenting existing LVLMs in handling visual question answering (VQA) about up-to-date knowledge, dubbed SearchLVLMs. A hierarchical filtering model is trained to effectively and efficiently find the most helpful content from the websites returned by a search engine to prompt LVLMs with up-to-date knowledge. To train the model and evaluate our framework's performance, we propose a pipeline to automatically generate news-related VQA samples to construct a dataset, dubbed UDK-VQA. A multi-model voting mechanism is introduced to label the usefulness of website/content for VQA samples to construct the training set. Experimental results demonstrate the effectiveness of our framework, outperforming GPT-4o by $\sim$30\% in accuracy.

SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge

Abstract

Large vision-language models (LVLMs) are ignorant of the up-to-date knowledge, such as LLaVA series, because they cannot be updated frequently due to the large amount of resources required, and therefore fail in many cases. For example, if a LVLM was released on January 2024, and it wouldn't know the singer of the theme song for the new Detective Conan movie, which wasn't released until April 2024. To solve the problem, a promising solution motivated by retrievalaugmented generation (RAG) is to provide LVLMs with up-to-date knowledge via internet search during inference, i.e. , internet-augmented generation (IAG), which is already integrated in some closed-source commercial LVLMs such as GPT-4V. However, the specific mechanics underpinning them remain a mystery. In this paper, we propose a plug-and-play framework, for augmenting existing LVLMs in handling visual question answering (VQA) about up-to-date knowledge, dubbed SearchLVLMs. A hierarchical filtering model is trained to effectively and efficiently find the most helpful content from the websites returned by a search engine to prompt LVLMs with up-to-date knowledge. To train the model and evaluate our framework's performance, we propose a pipeline to automatically generate newsrelated VQA samples to construct a dataset, dubbed UDK-VQA. A multi-model voting mechanism is introduced to label the usefulness of website/content for VQA samples to construct the training set. Experimental results demonstrate the effectiveness of our framework, outperforming GPT-4o by ∼ 30% in accuracy.

1 Introduction

Large vision-language models (LVLMs, e.g. , GPT-4V [19], Gemini Series [20], and Grok [21]) have received much attention for their impressive generative capabilities. They require a large resource for data collection, cleaning, and training, restricting them from frequently updating models. However, new information and knowledge are created every time, making LVLMs ineffective in many scenarios. For example, if we talk with LLaVA-1.6 [23] (released on January 30, 2024) about the new Detective Conan movie (realeased on April, 2024), such as 'the singer of the theme song', it performs very badly. It is promising to augment LVLMs by retrieving up-to-date knowledge via internet search during inference, i.e. , internet-augmented generation (IAG). Although commercial LVLMs such as

Q:

Who sings this theme song?

Llama   GPT

Visual

Search

Detective  Conan,

The

Million-

dollar Pentagram,

Theme song,

…

Search Engine

Google

website 1

website T

Title:

…

Snippet:

From what is the

holy war to where does

…

Content:

Segment 1 T

: It's been a

…

Segment 2 T

: Thankfully

…

Bing

Parser

Augmented Generation

Answer:

Aiko

…

GPT    LLaVA   Qwen

Hierarchical Filtering Model

Website Filter

Diversity Selection

Content Filter

Detective

Conan:

The

Million-dollar

Pentagram's

main

theme

song

is

'Mutual Love' by Aiko

.

segment 1

segment M

Segment 3

5

:

the holy war in

motion  …

website 1

website N

Segment 1 N

:

…

Segment 2 N

:

…

…

Title:

Snippet:

A special

collaborative music video

…

Content:

Segment 1 1

: The music

…

Segment 2 1

: The manga

…

Segment 1 1

:

…

Segment 2 1

:

…

…

Segment 2 1

:

… 'mutual love' by

aiko, …

名辣顶

8

100片2五楼星

C

女

GPT-4V [19] and Claude3 [22] have the ability of IAG, the specific mechanics underpinning them remain undisclosed. This paper proposes a plug-and-play framework to augment different LVLMs in handling visual question answering (VQA) about up-to-date knowledge, named SearchLVLMs.

We first introduce our overall framework applicable to different LVLMs for equipping them with up-to-date knowledge during inference. It consists of four components: query generator, search engine, hierarchical filtering model, and augmented generation, as shown in Figure 1. Specifically, we begin by extracting queries via Bing Visual Search and LLMs for an image-related question. Then, we acquire helpful websites through search engines and extract their contents by web scraping. However, it is impractical to augment LVLMs directly with the entire content of all websites, because: (1) Most LVLMs are poor at handling such long contexts. (2) Handling such long contexts is computationally intensive and time-consuming. To this end, a hierarchical filtering model is trained to find the most helpful content for answering the question, which first efficiently sifts the websites based on each website's title and snippet, and then identifies the most helpful content from the filtered websites. Finally, the filtered content is fed to LVLMs to assist them in answering the question.

We construct a dataset called UDK-VQA based on up-to-date news, used to train the hierarchical filtering model and evaluate our framework's performance. We propose a pipeline to automatically scrape the up-to-date news and generate news-related VQA samples. Specifically, we use search terms from Google Daily Search Trends and manually collected popular search terms as queries to search for hot news. For each news piece, we segment its content and ask GPT-3.5 to generate question-answer pairs from each segment. Then, we extract an entity for each question and replace it with its hypernym. To compose a VQA sample, we use Bing to search images of the replaced entity and cluster them to reduce the outliers among them. In doing so, answering the generated VQA samples requires models to consider both visual and textual information. We use queries from different time periods to scrape news from different time periods to generate samples for the training and test sets, avoiding test data exposure in the training data. In the training set, we further use a multi-model voting mechanism to label website's usefulness and content's usefulness for VQA samples, and combine the samples with websites and their content based on the label for training the hierarchical filtering model. In the testing set, we conduct manual screening to ensure its correctness.

To validate the effectiveness and generalizability of the proposed framework, we incorporate 15 state-of-the-art LVLMs into the framework, such as GPT-4V [19] and LLaVA-1.6 [23]. Notably, once the hierarchical filtering model is trained, our framework can adapt different LVLMs and improve their performance without any fine-tuning. Extensive experimental results demonstrate that our framework can significantly improve LVLM's ability to answer questions about up-todate knowledge. Incorporating the LLaVA-1.6 model of our framework even outperforms the self-contained IAG-capable GPT-4o by ∼ 30% in accuracy on UDK-VQA test set.

Our contributions are summarized as follows. (1) We propose the first open-source framework seamlessly incorporating existing LVLMs with up-to-date knowledge during inference. (2) We propose a pipeline that automatically generates VQA samples related to up-to-date news and construct

the first test set for evaluating LVLMs' ability to handle VQA on up-to-date knowledge. (3) Extensive experimental results on 15 state-of-the-art LVLMs demonstrate the effectiveness of our framework.

2 Related Work

2.1 Retrieval-Augmented Generation

Recently retrieval-augmented generation (RAG) attracted increasing attention of both the natural language processing [1, 9, 10, 2] and vision-and-language [3, 4, 18, 27, 28]. REALM [1] uses the query to retrieve the top 𝑘 most relevant article snippets, and uses large language models (LLMs) to generate 𝑘 responses, which are then combined to obtain a final output for question answering. Recently, [41, 10, 43] explores the internet-augmented generation (IAG) of LLMs to enable language models to access up-to-date information via search engines. Komeili et.al. [41] show that LLMs enhanced via search engines can generate less factually incorrect information during dialogue with humans. Lazaridou et.al. [10] uses few-shot prompting to enable LLMs to exploit knowledge returned from Google search to answer questions about factual and up-to-date information. In vision-andlanguage, REVEAL [3] builds a memory by encoding open-world knowledge including image-text pairs, question-answering pairs, etc., and uses a retriever to find the most relevant knowledge entries in the memory. The memory, encoder, retriever, and generator are pre-trained in an end-to-end manner. Re-ViLM [4] augments Flamingo [5], by retrieving relevant image-text pairs from the external image-text datasets [6, 7, 8] for zero and in-context few-shot image-to-text generations. RA-CM3 [42] performs retrieval from an external memory for generating images and text. Differently, we focus on enabling LVLMs to retrieve up-to-date knowledge via Internet search during inference.

2.2 Large Models with Search Engine

Recent years have witnessed a growing interest in exploring external tools for LLMs [11, 15, 12, 16, 13, 52]. Among them, some methods [13, 14, 17] can use search engines to access up-to-date knowledge. Nonetheless, these methods usually focus on how to appropriately use different tools to enhance LLMs, such as using Python interpreter to generate complex programs [15], incorporating more external tools [13], or updating tools by acquiring new knowledge [17]. Although they can access up-to-date knowledge, they usually directly use the website snippets for augmenting generation. By contrast, this work focuses on internet-augmented generation and explores how to obtain more relevant up-to-date knowledge and effectively use retrieved knowledge to augment LVLMs.

3 SearchLVLMs Framework

In this section, we introduce SearchLVLMs, a framework that seamlessly incorporate existing LVLMs, allowing these LVLMs to access up-to-date knowledge without fine-tuning. The whole framework is illustrated in Figure 1. For a natural language question 𝑄 about an image 𝑉 , we first extract queries for both 𝑄 and 𝑉 via the query generator. Then we enter the queries into search engines, and the search engine would return related websites, each of which consist of a title and a snippet. To identify the most helpful content within the websites, a website filter is used to filter the websites based on their titles and snippets, and a content filter is further used to filter the content of the websites filtered by the website filter. Finally, we stitch the filtered content together to prompt existing LVLMs.

3.1 Query Generator

Question Query Generator. To get queries that make search engines return websites containing helpful content, we leverage large language models (LLMs) to extract queries for 𝑄 . Thanks to the language understanding capability of LLMs, the role played by each word can be well inferred from the grammatical information of 𝑄 , even if certain words are unknown for the LLMs. We use ' Do not try to answer the question, just print the most informative no more than three entities in the question. Put them on one line and separate them with comm. ' to prompt LLMs to generate queries.

Image Query Generator. For an image 𝑉 , we leverage Bing Visual Search to analyze the image entities of 𝑉 as queries. The reason for using Bing Visual Search rather than a LVLM to extract queries for 𝑉 is that current LVLMs are inadequate in extracting image entities especially for emerging

entities. Notably, Bing Visual Search is a tool different from commonly used search engines, returning image-related attributes, including image entity names, image-related search terms and image-related websites. However, entity names are missing in most cases. To address this problem, we extract the longest public ancestor of related search terms and related website titles as the queries for 𝑉 .

3.2 Search Engine

The extracted queries are fed into a search engine, and the search engine returns relevant websites with their titles and snippets. However, the returned titles and snippets often contain limited and incomplete information. For example, for a website with title ' Pororo Dragon Castle Adventure ', the entire snippet returned by Bing is ' Pororo and his friends were having fun when a little red dragon named Arthur appears above them Arthur who claims to be the king of dragons commands Pororo and his friends to search for his Dragon ... ', obviously there is more about ' Pororo Dragon Castle Adventure ' contained in the website. Thus we parse the textual content of all websites. For a website, not all of its content contributes to answering questions, we empirically divide the website content into segments every third sentence for a more granular selection of content.

3.3 Hierarchical Filtering Model

Since most of the existing LVLMs cannot receive long context as inputs, and long contexts can be computationally intensive and time consuming for them, it's necessary to filter the website content after obtaining the websites via the search engine. Towards this goal, we train a hierarchical filtering model, which consists of a website filter and a content filter to perform a two-step filtering.

Website Filter. The aim of the website filter is to perform the filtering of websites based on their titles and snippets. Specially, a website scoring model is trained via instruction tuning, to predict how helpful a website will be in answering a question, and the 𝑁 websites with higher scores would be kept. The training samples are in the format ( 𝑇, 𝑆, 𝑄, 𝑉, 𝑅 𝑤 ) , where 𝑅 𝑤 is a quantitative usefulness in the interval [ 0 , 1 ] representing how helpful a website with title 𝑇 and snippet 𝑆 will be in answering 𝑄 related to 𝑉 . Based on the samples, we construct instructions like ' How helpful is an article with such a title and snippet in answering the question based on the image? Choose the best option. Title: < 𝑇 > Snippet: < 𝑆 > Question: < 𝑄 > Options: A. 1.0 B. 0.8 C. 0.6 D. 0.4 E. 0.2 F . 0.0 '. In doing so, the score regression problem is converted into a classification problem, which is easier to learn.

Content Filter. The content filter is used to select the most helpful content segments from the websites filtered by the website filter. For each content segment, we predict how helpful is it for answering 𝑄 by a content scoring model. The content scoring model is trained by samples in the format ( 𝐶, 𝑄,𝑉, 𝑅 𝑐 ) , where 𝐶 is a content segment, and 𝑅 𝑐 is the quantitative usefulness of 𝐶 in answering 𝑄 . The instructions for training the content scoring model are in the format: ' How helpful is this context in answering the question based on the image? Choose the best option. Context: < 𝐶 > Question: < 𝑄 > Options: A. 1.0 B. 0.8 C. 0.6 D. 0.4 E. 0.2 F. 0.0 '. We use the model to sort all content segments and select the 𝑀 highest scoring ones as the obtained segments.

Diversity Selection. To avoid LVLMs answer questions using bias from repetitive contexts, we performed a quadratic selection on the obtained segments based on diversity. Specially, we extract CLIP features [37] for all the segments and cluster them using k-means [44]. The segments closest to the center of each cluster are stitched together as the final obtained content for prompting the LVLMs.

3.4 Augmented Generation

We augment existing LVLMs by prompting them with the final obtained content. Taking answer the multiple choice questions as an example, for a question 𝑄 with candidate answers 𝐴 1 , 𝐴 2 , 𝐴 3 and 𝐴 4 , we use the prompt ' Given context: < 𝑋 > Question: < 𝑄 > Answers: A.< 𝐴 1 > B.< 𝐴 2 > C.< 𝐴 3 > D.< 𝐴 4 > Answer with the option's letter from the given choices directly based on the context and the image. ', where 𝑋 denotes the final content obtained by the hierarchical filtering model.

4 UDK-VQA Dataset

To evaluate the effectiveness of our framework, we propose a pipeline to automatically scrape the up-to-date news and generate news-related VQA. The whole pipeline is demonstrated in Figure 2.

Pseudo-Score Generation

Query Collection

Brood XIII, Chiikawa,

Oppenheimer,

…

Question Generation

Parser

Manual

Screening

Google Daily

Search Trends

Human

Collection

Google    Bing

GPT-3.5

LLaVA      Qwen      Monkey    mPLUG-Owl2

…

website information

Image Assignment

Entity Replace

Image Search

Image Clusting

and       for data from different time periods

QA pair

images about entity

images in Max cluster

question:

When  do  cicadas  from

Brood

XIII

typically

emerge in Chicago?

GT answer:

mid-May

wrong answers:

1. early June

2. late June,

3. late April

QA pair

question:

When

do

cicadas

from

Brood

XIII

typically

emerge in this city?

entity:

Chicago

GT answer:

mid-May

wrong answers:

1. early June

2. ate June

3. late April.

question:

When  do  cicadas  from

Brood

XIII

typically

emerge in this city?

GT answer:

mid-May

wrong answers:

early June, late June, late

April.

GT segment:

In  Chicago,  said  typically,

periodical cicadas emerge in

mid-May,

…

title:

Cicadas emerge in Chicago,

…

snippet:

Some  cicadas  from  Brood,

…

segment:

In

Chicago,

said

typically,

periodical cicadas emerge in mid-May,

…

question:

When do

cicadas  from  Brood

XIII typically  emerge

in this city?

pseudo-score:

1.0

title:

…

snippet:

Some cicadas

…

segment:

In Chicago, said

typically, periodical cicadas

emerge in mid-May,

…

C

G

女

The pipeline is also used to collect training samples for the hierarchical filtering model. We first collect hot search terms as queries to scrape relevant news returned by search engines. For each piece of news, every third sentence is divided into a segment. We then employ GPT 3.5 to generate a question-answer pair for each segment, and extract an entity in the question, replacing it with its hypernym. Bing Image Search is used to find images for the replaced entity, and after removing outliers from the images using clustering, the images and the question-answer pair are composed into VQA samples. We combine the VQA samples and website information ( e.g. , title, snippet and content), and introduce a multi-model voting mechanism to generate pseudo-score, constituting the training set. For the test set, manual screening is conducted to ensure the correctness of test samples.

4.1 Query Collection

Google daily search trends is an available data source that reflects what's hot in real time, and is well suited as the query used to construct our dataset. However, we observe that most search terms of the Google daily search trends are related to politics and sports, which poses a great limitation. Therefore, we further manually collect popular search terms to improve the query diversity. The popular search terms are collected from many other domains including films, technological products, anime characters, places of interest, and so on. These human-collected queries were mixed with queries from Google daily search trends to be used for subsequent sample generation.

4.2 Question Generation

For each query, we use Bing to search for relevant and up-to-date news. We divide every third sentence int the scraped news content into a segment, and prompt GPT-3.5 to generate a question-answer pair and several confused answers for each segment by: ' Given context: < 𝐶𝑜𝑛 > Filling the blanks to generate a question about the most informative event of the context, generate an correct answer to the question in no more than three words based on context, and generate three incorrectly confused answers of no more than three words based on context. Question: Correct answer: Incorrect answers: A. B. C. ', where < 𝐶𝑜𝑛 > denotes a segment. We design a simple but effective rule to ensure the correctness of the generated pairs, which requires a model can answer a question 𝑄 with 𝐴 based on a segment 𝐶 , if the model can generate a question-answer pair ( 𝑄, 𝐴 ) based on 𝐶 .

4.3 Image Assignment

To generate VQA samples and avoid the model's reliance on language priors for answering, we create samples that necessitate an understanding of the image for correct answers. Firstly, we extract an entity for each question via named an entity recognition (NER) model [38]. Images of the entity

segment: The Tigers will cap the 2023-24

season by being ranked No. 17, ..

team finish in the final USA TODAY Sports

title: Where did Auburn finish in the ..

snippet: Auburn basketball ended its SEC, .

question: Where did this College basketball

Coaches Poll for the 2023-24 season?

pseudo-score: R=1.0, R=1.0

wrong answers: For Broken Ears, Essence, If

album Born in the Wild was done, ..

question: What was the name of Tems’ debut

Orange Was a Place.

GT segment: Tems announced that her debut

album announced at this music festival?

GT answer: Born in the Wild

(5%)

ann.

ent.(26%)

pol.

(13%)

title: Everything about Banff National Park

snippet: Simplicity marks the origin ...

GT segment: ..., acts as a barrier for over-

land runoff of mutrients and eroded soil, ...

(7%)

SOC.

question: What caused the stranded passengers

segment: Simplicity marks the origin of Banff

Canada’s first national park, ...

in this National Park?

question: What is the purpose of planting new

woodland in this River's SAC catchment?

wrong answers: provide shade, reduce river

GT answer: a barrier for over-land runoff

(37%)

sp.

eco.

tech.

(8%)

pseudo-score: R=0.4, R=0.6

temperature, adapt to climate change.

(4%)

(a)

(b)

(c)

are then obtained by Bing Image Search. Since the images returned by the search engine are noisy (outlier images), we cluster them based on the CLIP feature [37] of the images and keep only the images in the cluster with the highest number of images. Finally, the kept images are assigned to the new question-answer pairs where the entity is replaced by its hypernym, to compose VQA samples. An obtained VQA sample can be denoted as ( 𝑉, 𝑄, 𝐴 𝑔𝑡 , { 𝐴 𝑖 𝑤 } 3 𝑖 = 1 ) , where 𝑄 is the generated question, 𝑉 is the entity image, 𝐴 represents the ground-truth answer, and 𝐴 𝑖 𝑤 is 𝑖 -th confused wrong answer.

4.4 Pseudo-Score Generation

For a VQA sample generated from the content segment 𝐶 , which we denote as the ground-truth segment for the sample, it is certain that 𝐶 is most helpful in answering this sample. Inevitably, we must consider to what extent do the other content segments contribute to answering the sample? We propose a pseudo-score generation method that uses five LVLMs for voting to quantify how helpful a content segment is to a VQA sample into six values: 1.0, 0.8, 0.6, 0.4, 0.2 and 0.0. Specially, for a VQA sample with the ground-truth segment 𝐶 from a news fetched for a query, we first sample four content segments from the news for the query beyond its ground-truth segment. Then we use each sampled segment to prompt each of the five LVLMs to answer the VQA sample and count the rate of LVLMs that answer correctly as the pseudo-score for the segment.

In doing so, we obtain training samples for the content filter, in the format ( 𝐶, 𝑄,𝑉, 𝑅 𝑐 ) , where 𝐶 is a content segment, 𝑄 denotes a question related to the image 𝑉 , and 𝑅 𝑐 is the pseudo-score of how helpful 𝐶 is to answer 𝑄 . Moreover, we count the maximum pseudo-score of all content segments in a news for a VQA sample as the pseudo-score for the news website, dubbed 𝑅 𝑤 , to build training samples for the website filter. The training sample format for the website filter is ( 𝑇, 𝑆, 𝑄, 𝑉, 𝑅 𝑤 ) , where 𝑇 is the website title, 𝑆 is the website snippet. By merging these training samples into the training instructions mentioned in Section 3.3, the hierarchical filtering model can be implemented.

4.5 Manual Screening

For constructing the test set, we do not use the pseudo-score generation method. A test sample ( 𝐶,𝑉,𝑄, 𝐴 𝑔𝑡 , { 𝐴 𝑖 𝑤 } 3 𝑖 = 1 ) can be seen as a VQA sample with its ground-truth content segment 𝐶 . It is worth noting that 𝐶 is only provided when testing the upper bound of performance. For each test sample, we randomly mix 𝐴 𝑔𝑡 and { 𝐴 𝑖 𝑤 } 3 𝑖 = 1 , then assign them the options ( i.e. A, B, C and D), and add a complementary option E. No Correct Answers , to evaluate LVLMs in a multiple choice format. Moreover, we manually review all test samples to ensure that they are correct.

4.6 Dataset Analysis

To prevent test data exposure in the training set, we use queries and news from different time periods to construct the training and test sets. For the training set, we use queries from February 17 to March 31, 2024, to scrape news before April 10, 2024. The training sample counts for the website and content filters are 599,700 and 850,267. For the test set, queries from April 1 to April 31, 2024, are used to scrape news after April 10, yielding 1,000 test samples. We divide the test sample into seven categories: politics, entertainment, announcement, sports, economic, technology and society, based on their required knowledge. We visualize some samples and the statistics for test samples in Figure 3. A new version, UDK-VQA-20240925, is constructed, with details in the Appendix .

5 Experiments

5.1 Settings

Training. We implement two versions of the hierarchical filtering model, one using LLaVA-1.5vicuna-7b [39] and the other using Qwen-VL-Chat [29]. In each version, we use same hyperparameters to fine-tune two same LVLMs with LoRA [24] as the website filter and the content filter, respectively. Whether fine-tuning LLaV A-1.5-vicuna-7b or Qwen-VL-Chat, the entire training process is facilitated on two Nvidia A100 GPUs, using a batch size of 128 over 3 epochs.

Baselines. We incorporate 15 representative LVLMs into the proposed framework including Gemini 1.5 Pro [20], GPT-4V [19], GPT-4o, InternVL-1.5 [45], LLaVA-1.6 [23], LLaVA-1.5 [39] XComposer2 [26], Monkey [25], CogVLM [30], MiniCPM-V2 [35], mPLUG-Owl2 [31], Qwen-VL [29], MMAlaya [34], Xtuner [32] and VisualGLM [33]. We implement Gemini 1.5 Pro, GPT-4V and GPT-4o via their official webs and APIs. We implement other LVLMs based on VLMEvalKit [36].

Evaluation. We evaluate LVLMs on four datasets including GQA [46], InfoSeek [47], A-OKVQA [48] and the proposed UDK-VQA. The reason behind selecting GQA, InfoSeek, and A-OKVQA is to evaluate the generalization capability of our framework across datasets that do not necessitate up-to-date knowledge. In addition to evaluating LVLMs via VLMEvalKit, we design additional matching patterns for each LVLM with respect to its answer format. For example, we additionally use the pattern 'The answer is XXX.' for XComposer2 as it often answers in this format. All evaluations are conducted with a single Nvidia A100 GPU.

5.2 Quantitative Comparison with SOTA LVLMs

We compare with state-of-the-art LVLMs on the UDK-VQA test set, including Gemini 1.5 Pro [20], GPT-4V [19], GPT-4o, LLaVA-1.6 [23] and InternVL-1.5 [45]. For Gemini 1.5 Pro, GPT-4V and GPT-4o, we implement their Raw version via official APIs, which do not have the ability of IAG.

Since Gemini 1.5 Pro is famous for receiving long contexts, we use all website content returned by the search engine of our framework to prompt it directly, dubbed LC . For GPT-4V and GPT-4o, we test their self-contained IAG-capable ability via prompting their official web versions with ' Retrieve relevant news and answer the question directly from the given options using the option letters based on the image. ', dubbed IAG . We incorporate each Raw baseline into our framework as Ours .

The experimental results on UDK-VQA are listed in Table 6, we can observe that: (1) InternVL-1.5 with our framework achieves the best performance on almost categories of UDK-VQA. (2) For all four baselines, our framework consistently improves their accuracy (e.g., 22.7% and 34.0% absolute performance gains in overall accuracy for GPT-4V and GPT-4o, respectively). (3) Our framework uses shorter contexts but has higher accuracy (e.g., 76.4% vs 83.3% in accuracy for LC and Ours variants of Gemini, respectively). The observations suggest that our framework is generalizable and effective in enhancing the ability of LVLMs to answer questions about up-to-date knowledge.

In addition, the experimental results on GQA, InfoSeek and A-OKVQA are listed in Table 2. Since these datasets do not rely on the up-to-date knowledge, we use a simple strategy to avoid misleading LVLMs with the up-to-date knowledge by invoking our framework when they respond with 'E' (as mentioned in Section 4.5) without retrieval. From the table, we can observe that our framework improves the performance of different LVLMs across various datasets. The improvements on these three datasets are not as significant as on our UDK-VQA dataset for the following reasons: (1) The GQA dataset does not rely on external knowledge and is used to evaluate the reasoning ability of LVLMs, which is beyond the scope of our framework. (2) Our framework focuses on retrieving the up-to-date knowledge, whereas the InfoSeek dataset and the A-OKVQA dataset rely on commonsense knowledge, much of which has already been used in the training data of LVLMs.

5.3 Ablation Studies

The experimental results of ablation studies on the proposed UDK-VQA dataset are shown in Table 3, where we use LLaVA-1.6 [23] as the baseline. Firstly, we investigate simple IAG methods, including using the similarity between questions and segments to select segments, i.e. , IAG (SIM 𝑄 ) , using the similarity between images and segments to select segments i.e. , IAG (SIM 𝑉 ) , using the averaged similarity of the the above two similarities to select segments, i.e. , IAG (SIM 𝑄𝑉 ) . These methods show limited improvements and achieve unsatisfactory accuracy.

Then, we study the influences of different components of our framework on the performance. For the hierarchical filtering model, we study two popular LVLMs, LLaVA-1.5 [39] and Qwen-VL [29]. For the query generator, we conduct experiments with NER [38], LLaMA3 [40], GPT-3.5 and Bing Visual Search. We observe that: (1) Using different backbone for the hierarchical filtering model has little effect on performance. (2) Using multiple question query generators at the same time can result in better performance than using only one. (3) Using both the question query generator and the image query generator gives the best performance. These observations suggest that all components of our framework are effective in improving the baseline, and components are complementary to each other.

5.4 Analysis of Pseudo-Score Generation

We analyze the influences of using different LVLMs to generate pseudo-scores on the performance. We categorize 10 LVLMs into two groups based on their released date, the first group contains LLaVA-1.6, XComposer2, Monkey, CogVLM and MiniCPM-V2, the second group contains mPLUG-Owl2, Qwen-VL, MMAlaya, Xtuner and VisualGLM. Using these two groups to generate pseudo-scores are dubbed PSG with G1 and PSG with G2 . Experimental results are shown in Figure 4, which reveal that: (1) The proposed framework can be directly used to boost LVLMs that are not used for generating pseudoscores, which show the transferability of our framework. (2) The use of more recent LVLMs for generating pseudo-scores allows for greater improvements in general. (3) Different LVLMs have different performance upper bound, some of them achieve limited accuracy ( e.g. , ∼ 70% in accuracy for VisualGLM) even are augmented with ground-truth segments ( GT Segment ).

5.5 Analysis of Training Strategy

Would jointly training the hierarchical filtering model with LVLMs result in greater improvements? We use LLaVA-1.5 [39] as the backbone for the hierarchical filtering model and conduct experiments with Qwen-VL [29] and LLaVA-1.5 as the LVLMs. As shown Table 4, separate training, where the LVLMs are frozen during the training of the hierarchical filtering model, leads to a more significant improvement in performance. The main reasons are: (1) Our training data uses pseudo-labeling instead of high-quality human annotation. Training based on such data may cause LVLMs to lose their original semantic understanding capabilities. (2) Our training and testing sets are generated from news from different time periods, involving different entities and having different distributions. Training LVLMs on our training set easily leads to overfitting, resulting in lower generalization on the test set.

5.6 Analysis of Diversity Selection

In this section, we investigate the necessity of diversity selection. We compare our diversity selection (Div𝐾 ) with Top𝐾 selection, and the experimental results of 10 LVLMs are shown in Figure 5.

25

60

80

90

MiniCPM-V2

LLaVA-1.6

XComposer2

Monkey

CogVLM

MMAlaya

mPLUG-Owl2

Qwen-VL

Xtuner

VisualGLM

MiniCPM-V2

Raw

GT Segment

PSG with G1

PSG with G2

5

10

15

20

20

40

60

MiniCPM-V2

5

10

15

20

88

89

90

91

LLaVA-1.6

5

10

15

20

68

70

72

74

XComposer2

5

10

15

20

70

75

80

Monkey

5

10

15

20

0

20

40

60

CogVLM

Top-

K

K

5

10

15

20

0

20

40

60

80

MMAlaya

5

10

15

20

50

60

70

80

mPLUG-Owl2

5

10

15

20

60

70

80

Qwen-VL

5

10

15

20

40

60

80

Xtuner

5

10

15

20

50

60

70

VisualGLM

The Top𝐾 selection means stitching 𝐾 content segments with the highest scores together to prompt the LVLMs. For Div𝐾 , 𝐾 denotes the number of clusters. Experimental results demonstrate that: (1) Our diversity selection outperforms the Top𝐾 selection regardless of the setting of 𝐾 for most LVLMs. (2) As 𝐾 increases, the performance using the Top𝐾 selection plummets. This is because content with high scores is similar, and if a LVLM receives too many duplicate content as inputs, it will misinterpret the instruction and thus repeat the inputs instead of answering the question. These experimental results prove the necessity and effectiveness of the diversity selection.

5.7 Analysis of Website Filter

An important capability of the website filter is the trade-off between the content filter efficiency and the LVLMs' accuracy. Adjusting the filtered website number 𝑁 can control the token number that the content filter needs to process as a percentage of the total token number returned by the search engine, dubbed 𝜃 . The variation in accuracy of LVLMs as 𝜃 increases is shown in Figure 6, we can observe that: (1) The accuracy of LVLMs increases with 𝜃 , especially when 𝜃 ≤ 40% . (2) The increase in accuracy of LVLMs slows down after 𝜃 ≥ 40% . Therefore, setting 𝜃 = 40% achieves a better trade-off, because the accuracy obtained by processing 40% tokens is close to 98% of the accuracy obtained when processing 100% tokens.

5.8 Analysis of Snippet Completeness

We consider whether providing complete snippets to the website filter could result in further improvements. The experimental results of snippet completeness are presented in the Appendix .

6 Conclusion

In this work, we have presented SearchLVLMs, a plug-and-play framework to augment LVLMs in handling visual question answering about up-to-date knowledge. By introducing a hierarchical filtering model, the framework enables LVLMs to access up-to-date knowledge. A UDK-VQA dataset is further curated by scraping up-to-date news and generating news-related VQA samples. The dataset enables quantitatively evaluate the ability of LVLMs to respond to questions about up-to-date knowledge. Experimental results on UDK-VQA demonstrate that our framework can significantly boost the performance of LVLMs for answering questions requiring up-to-date knowledge.

10%

25%

40%

55%

70%

85%

100%

50

55

60

65

70

75

80

85

90

Acc

MiniCPM-V2

LLaVA-1.6

XComposer2

Monkey

CogVLM

mPLUG-Owl2

Qwen-VL

Xtuner

VisualGLM

Acknowledgments This work was supported by the Natural Science Foundation of China (NSFC) under Grants No. 62172041 and No. 62176021, Natural Science Foundation of Shenzhen under Grant No. JCYJ20230807142703006, Key Research Platforms and Projects of the Guangdong Provincial Department of Education under Grant No.2023ZDZX1034, and the China Postdoctoral Science Foundation (No. 2023M743003). This work was partially supported by the National Key R&D Program of China (No.2022ZD0161000, No.2022ZD0160101, No.2022ZD0160102) and the General Research Fund of Hong Kong No.17200622 and 17209324.

References

K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang, 'Retrieval augmented language model pre-training,' in International conference on machine learning . PMLR, 2020, pp. 3929-3938.

A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi, 'Self-rag: Learning to retrieve, generate, and critique through self-reflection,' arXiv preprint arXiv:2310.11511 , 2023.

Z. Hu, A. Iscen, C. Sun, Z. Wang, K.-W. Chang, Y. Sun, C. Schmid, D. A. Ross, and A. Fathi, 'Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge memory,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2023, pp. 23 369-23 379.

Z. Yang, W. Ping, Z. Liu, V. Korthikanti, W. Nie, D.-A. Huang, L. Fan, Z. Yu, S. Lan, B. Li et al. , 'Re-vilm: Retrieval-augmented visual language model for zero and few-shot image captioning,' arXiv preprint arXiv:2302.04858 , 2023.

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al. , 'Flamingo: a visual language model for few-shot learning,' Advances in neural information processing systems , vol. 35, pp. 23 716-23 736, 2022.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, 'Microsoft coco: Common objects in context,' in Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 . Springer, 2014, pp. 740-755.

P. Sharma, N. Ding, S. Goodman, and R. Soricut, 'Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,' in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2018, pp. 2556-2565.

S. Changpinyo, P. Sharma, N. Ding, and R. Soricut, 'Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 3558-3568.

J. Sun, C. Xu, L. Tang, S. Wang, C. Lin, Y. Gong, H.-Y. Shum, and J. Guo, 'Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph,' arXiv preprint arXiv:2307.07697 , 2023.

A. Lazaridou, E. Gribovskaya, W. Stokowiec, and N. Grigorev, 'Internet-augmented language models through few-shot prompting for open-domain question answering,' arXiv preprint arXiv:2203.05115 , 2022.

T. Gupta and A. Kembhavi, 'Visual programming: Compositional visual reasoning without training,' in CVPR , 2023, pp. 14 953-14 962.

D. Gao, L. Ji, L. Zhou, K. Q. Lin, J. Chen, Z. Fan, and M. Z. Shou, 'Assistgpt: A general multi-modal assistant that can plan, execute, inspect, and learn,' arXiv preprint arXiv:2306.08640 , 2023.

P. Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y. N. Wu, S.-C. Zhu, and J. Gao, 'Chameleon: Plug-and-play compositional reasoning with large language models,' in NeurIPS , 2023, pp. 43 447-43 478.

Z. Yang, L. Li, J. Wang, K. Lin, E. Azarnasab, F. Ahmed, Z. Liu, C. Liu, M. Zeng, and L. Wang, 'Mm-react: Prompting chatgpt for multimodal reasoning and action,' arXiv preprint arXiv:2303.11381 , 2023.

D. Surís, S. Menon, and C. Vondrick, 'Vipergpt: Visual inference via python execution for reasoning,' in ICCV , 2023, pp. 11 888-11 898.

C. Wu, S. Yin, W. Qi, X. Wang, Z. Tang, and N. Duan, 'Visual chatgpt: Talking, drawing and editing with visual foundation models,' arXiv preprint arXiv:2303.04671 , 2023.

Z. Gao, Y. Du, X. Zhang, X. Ma, W. Han, S.-C. Zhu, and Q. Li, 'Clova: A closed-loop visual assistant with tool usage and update,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , 2024.

C. Jing, Y. Li, H. Chen, and C. Shen, 'Retrieval-augmented primitive representations for compositional zero-shot learning,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 3, 2024, pp. 2652-2660.

S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg et al. , 'Sparks of artificial general intelligence: Early experiments with gpt-4,' arXiv preprint arXiv:2303.12712 , 2023.

G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth et al. , 'Gemini: a family of highly capable multimodal models,' arXiv preprint arXiv:2312.11805 , 2023.

G. Contributors, 'Grok,' https://github.com/xai-org/grok-1 , 2024.

C. Contributors, 'Claude,' https://claude.ai/ , 2024.

H. Liu, C. Li, Y. Li, B. Li, Y . Zhang, S. Shen, and Y . J. Lee, 'Llava-next: Improved reasoning, ocr, and world knowledge,' January 2024. [Online]. Available: https://llava-vl.github. io/blog/2024-01-30-llava-next/

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, 'LoRA: Low-rank adaptation of large language models,' in International Conference on Learning Representations , 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9

Z. Li, B. Yang, Q. Liu, Z. Ma, S. Zhang, J. Yang, Y. Sun, Y. Liu, and X. Bai, 'Monkey: Image resolution and text label are important things for large multi-modal models,' arXiv preprint arXiv:2311.06607 , 2023.

X. Dong, P. Zhang, Y. Zang, Y. Cao, B. Wang, L. Ouyang, X. Wei, S. Zhang, H. Duan, M. Cao, W. Zhang, Y. Li, H. Yan, Y. Gao, X. Zhang, W. Li, J. Li, K. Chen, C. He, X. Zhang, Y. Qiao, D. Lin, and J. Wang, 'Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model,' arXiv preprint arXiv:2401.16420 , 2024.

C. Li, C. Jing, Z. Li, M. Zhai, Y. Wu, and Y. Jia, 'In-context compositional generalization for large vision-language models,' in Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2024.

C. Li, Z. Li, C. Jing, Y . Wu, M. Zhai, and Y . Jia, 'Compositional substitutivity of visual reasoning for visual question answering,' in Proceedings of the European Conference on Computer Vision (ECCV) , 2024.

J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, 'Qwen-vl: A frontier large vision-language model with versatile abilities,' arXiv preprint arXiv:2308.12966 , 2023.

W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y. Wang, J. Ji, Z. Yang, L. Zhao, X. Song et al. , 'Cogvlm: Visual expert for pretrained language models,' arXiv preprint arXiv:2311.03079 , 2023.

Q. Ye, H. Xu, J. Ye, M. Yan, H. Liu, Q. Qian, J. Zhang, F. Huang, and J. Zhou, 'mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration,' arXiv preprint arXiv:2311.04257 , 2023.

X. Contributors, 'Xtuner: A toolkit for efficiently fine-tuning llm,' https://github.com/ InternLM/xtuner , 2023.

Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang, 'Glm: General language model pretraining with autoregressive blank infilling,' in Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2022, pp. 320-335.

D. Ltd., 'mmalaya,' https://github.com/DataCanvasIO/MMAlaya , 2024.

OpenBMB, 'Minicpm-v,' https://github.com/OpenBMB/OmniLMM , 2024.

O. Contributors, 'Opencompass: A universal evaluation platform for foundation models,' https://github.com/open-compass/opencompass , 2023.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , 'Learning transferable visual models from natural language supervision,' in Proceedings of the International Conference on Machine Learning (ICML) , 2021, pp. 8748-8763.

J. Devlin, M. Chang, K. Lee, and K. Toutanova, 'BERT: pre-training of deep bidirectional transformers for language understanding,' CoRR , vol. abs/1810.04805, 2018. [Online]. Available: http://arxiv.org/abs/1810.04805

H. Liu, C. Li, Y. Li, and Y. J. Lee, 'Improved baselines with visual instruction tuning,' 2023.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al. , 'Llama: Open and efficient foundation language models,' arXiv preprint arXiv:2302.13971 , 2023.

M. Komeili, K. Shuster, and J. Weston, 'Internet-augmented dialogue generation,' arXiv preprint arXiv:2107.07566 , 2021.

M. Yasunaga, A. Aghajanyan, W. Shi, R. James, J. Leskovec, P. Liang, M. Lewis, L. Zettlemoyer, and W.-T. Yih, 'Retrieval-augmented multimodal language modeling,' in International Conference on Machine Learning . PMLR, 2023, pp. 39 755-39 769.

J. Tian, H. Chen, G. Xu, M. Yan, X. Gao, J. Zhang, C. Li, J. Liu, W. Xu, H. Xu et al. , 'Chatplug: Open-domain generative dialogue system with internet-augmented instruction tuning for digital human,' arXiv preprint arXiv:2304.07849 , 2023.

A. K. Jain and R. C. Dubes, Algorithms for clustering data . Prentice-Hall, Inc., 1988.

Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma et al. , 'How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites,' arXiv preprint arXiv:2404.16821 , 2024.

D. A. Hudson and C. D. Manning, 'Gqa: A new dataset for real-world visual reasoning and compositional question answering,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 6700-6709.

Y. Chen, H. Hu, Y. Luan, H. Sun, S. Changpinyo, A. Ritter, and M.-W. Chang, 'Can pre-trained vision and language models answer visual information-seeking questions?' in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , 2023, pp. 14 948-14 968.

D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi, 'A-okvqa: A benchmark for visual question answering using world knowledge,' in European conference on computer vision . Springer, 2022, pp. 146-162.

J. Wu, X. Hu, Y. Wang, B. Pang, and R. Soricut, 'Omni-smola: Boosting generalist multimodal models with soft mixture of low-rank experts,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 14 205-14 215.

B. X. Nguyen, T. Do, H. Tran, E. Tjiputra, Q. D. Tran, and A. Nguyen, 'Coarse-to-fine reasoning for visual question answering,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 4558-4566.

W. Yu, D. Iter, S. Wang, Y. Xu, M. Ju, S. Sanyal, C. Zhu, M. Zeng, and M. Jiang, 'Generate rather than retrieve: Large language models are strong context generators,' in International Conference on Learning Representations , 2023.

P. Li, Z. Gao, B. Zhang, T. Yuan, Y. Wu, M. Harandi, Y. Jia, S.-C. Zhu, and Q. Li, 'Fire: A dataset for feedback integration and refinement evaluation of multimodal models,' Advances in Neural Information Processing Systems , 2024.

A Appendix

A.1 Analysis of Snippet Completeness on UDK-VQA

The incomplete snippets returned by search engines lead us to consider whether providing complete snippets to the website filter could result in further improvements. We present the experimental results of snippet completeness on our UDK-VQA dataset in the Table 5, where 𝜃 represents the percentage as mentioned in Section 5.7. For each website snippet, we attempt to locate the full sentence corresponding to the snippet by crawling the website's content. However, the content of many websites could not be crawled. For such websites, we experiment with two strategies: (1) Discarding these websites during training and testing. (2) Using the incomplete snippets. The experimental results are shown in the table below, where ' Raw ' represents all snippets without completion, ' Discard ' represents strategy (1), and ' Mixture ' represents strategy (2). From the experimental results we can observe that directly discarding the websites leads to a significant performance loss, as discarding reduces the number of usable websites by approximately half, thereby limiting the performance of the website filter. Furthermore, as 𝜃 increases, the performance of strategy (2) becomes increasingly close to that of all snippets without completion ( i.e. , Raw), which validates that the completeness of the snippets has little impact on accuracy.

A.2 UDK-VQA-20240905 Dataset

To test the capabilities of LVLMs to handle visual question answering on the latest up-to-date knowledge, we construct a new UDK-VQA-20240905 test set based on the proposed data generation pipeline. For the UDK-VQA-20240905 test set, we use the queries of Google Daily Search Trends from September 1, 2024 to September 5, 2024, to scrape news after September 5, 2024. In addition, we manually construct some test samples to ensure that the UDK-VQA-20240905 test set covers a wider range of categories of knowledge. The total number of test samples is 50, and the samples can be divided into five categories, including game (5), sports (24), society (8), entertainment (9), and economic (4), based on their required knowledge. The number in parentheses after each category indicates the number of samples in that category. As shown in Figure 7, answering the samples of UDK-VQA-20240905 requires up-to-date knowledge. For example, the first sample with the question 'Where is the Meditation Spot located at this Temple?' is about the game 'Black Myth: Wukong,' which was released on August 20, 2024, after the release date of existing LVLMs.

Question:

Where

is

the

Meditation

Spot

located at this Temple?

Multi-Choices:

(A) Temple Garden

(B) Temple Pavilion

(C) Temple Entrance

(D) Temple Sanctuary

Question:

In which country was the girl in the

picture arrested?

Multi-Choices:

(A) Sweden

(B) Norway

(C) Denmark

(D) Finland

A.3 Experiments on UDK-VQA-20240905

We compare with state-of-the-art LVLMs on the UDK-VQA-20240905 test set, including Gemini 1.5 Pro [20], GPT-4o, LLaVA-1.6 [23] and InternVL-1.5 [45]. Similar to the principle for testing on the UDK-VQA dataset, we implement the Raw version of Gemini 1.5 Pro and GPT-4o via their official APIs, which do not have the ability of internet-augmented generation (IAG). For GPT-4o, we prompted its official web versions with ' Retrieve relevant news and answer the question directly from the given options using the option letters based on the image. ' to test its self-contained IAG-capable ability, dubbed IAG . For LLaVA-1.6 and InternVL-1.5, we implement them via the VLMEvalKit toolkit [36]. Each Raw baseline is incorporated into our framework, dubbed as Ours .

The experimental results on the UDK-VQA-20240905 test set are listed in Table 6, from which we can observe that: (1) InternVL-1.5 with our framework performs the best, suppressing closed-source business models including Gemini 1.5 Pro and GPT-4o. (2) All baseline models (Raw version) perform worse on the UDK-VQA-20240905 test set compared to the UDK-VQA test set, indicating that the newer the knowledge required by the test samples, the more challenging these samples are for LVLMs. (3) Compared to the experimental results on the UDK-VQA dataset, the improvement of our framework over the baseline has decreased. This is because there is a greater discrepancy in entity distribution between the UDK-VQA-0905 dataset and the training set of SearchLVLMs. (4) The accuracy of the Raw version of Gemini 1.5 Pro is lower, because this model tends to choose 'E. No Correct Answers' while other models are more inclined to select from { 𝐴, 𝐵, 𝐶, 𝐷 } , even though they do not possess the knowledge required to answer such questions correctly.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract contains our main claims including motivation, the up-to-date knowledge retrieval-augmented framework, the pipeline for generating news-realated VQA samples and the curated dataset UDK-VQA.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations in a separare section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our work is not related to theorems.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the implementation details including hyperparameter settings, baseline selection and evaluation details.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We do not provide open access to the data and code at this time, but can publish part of them at the rebuttal stage if the reviewers need it. The complete data and code will be published after the paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the implementation details including hyperparameter settings, baseline selection and evaluation details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We follow existing work in the areas we work in and do not provide statistical significance for fair comparisons.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the computer resources for reproducing the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our work conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of our work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: We use search engines to access Internet data, and search engines have their own methods to avoid security safety risks. Moreover, samples in the test set we curated have been reviewed case by case.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We've cited the original paper of the code and model we used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: We will provide open access to part of the new assets at the rebuttal stage if the reviewers need it. The complete assets will be published after the paper is accepted.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The test samples of our curated UDK-VQA dataset are checked by co-authors. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The test samples of our curated UDK-VQA dataset are checked by co-authors.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper presents UDKAG, a novel framework designed to enhance the capabilities of Large Vision-Language Models (LVLMs) by integrating up-to-date knowledge retrieved from the internet during the inference phase. The authors have developed a hierarchical filtering model to identify pertinent content from search engine results and a UDK-VQA dataset to evaluate the model's performance on VQA tasks.

STRENGTHS:
1. The paper introduces an novel internet-augmented generation framework to update newest knowledge.
2. The creation of the UDK-VQA dataset is a valuable contribution.
3. Significant performance improvement.

WEAKNESSES:
I'm not familiar with this field and from my perspective this paper may not have obvious weakness.

QUESTIONS:
I'm curious about the generalization ability after integrating up-to-date knowledge. Will the model forget some important information and perform worse on other data?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a novel approach to augment large vision-language models with up-to-date knowledge from the internet. The author proposes to extensively leverage existing search engines (e.g., Bing and Google) and foundation models like ChatGPT for web information searching and parsing. A hierarchical filtering model has been proposed to filter the irrelevant information from the retrieved websites. The method can augment existing LVLM for augmented generations and achieve the state-of-the-art (SOTA) performance on a newly proposed VQA dataset, i.e., UDK-VQA, where the proposed plug-and-play method can improve SOTA LLM/VLMs like Gemini 1.5 Pro, GPT-4v, GPT-4o and LLaVA 1.6. Extensive ablation studies have been conducted to analyze the effectiveness of each component of the proposed framework.\n\nSTRENGTHS:\n1. The paper is very well written and strongly motivated, given the necessity of augmenting the LVLM with up-to-date knowledge. To the best of my knowledge, this should be the first paper on augmenting the LVLM with internet information in a comprehensive manner.\n\n2. The proposed method is novel, given its design choice and intelligent combination of search engine and foundation model to enable precise information retrieval from the internet. The reviewer believes that such a new systematic system will inspire future exploration of IAG for multi-modality foundation models.\n\n3. The proposed method can be plug-and-play into many existing foundation models without trial and error, demonstrating its universality and significance. Moreover, the foundation models augmented by the proposed method can be improved significantly, highlighting the effectiveness of the proposed method.\n\n4. Extensive analysis has been conducted on the proposed method to verify further the effectiveness of the proposed framework.\n\nWEAKNESSES:\nOverall, this is a strong paper. The primary concern is the cost of leveraging the search engine and LLM like ChatGPT. The author should elaborate more on this aspect to show whether such an IAG is cost-efficient. Moreover, the reviewer is also curious about whether the code of the proposed framework will be open-source to serve as the cornerstone for future research.\n\nQUESTIONS:\nPlease refer to the Weaknesses section for more details.\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a framework that enhances large vision-language models (LVLMs) to handle visual question answering (VQA) tasks involving up-to-date knowledge. The system utilizes a search engine to retrieve relevant websites and parses their content for more comprehensive information. To manage the large volume of data, a hierarchical filtering model is introduced, consisting of a website filter and a content filter. These filters score and select the most relevant information, which is then used to prompt LVLMs for improved performance on VQA tasks. The framework also includes a diversity selection mechanism to ensure varied and unbiased content selection.\n\nSTRENGTHS:\n1. Clear Motivation: The paper is well-motivated and addresses a very practical problem.\n2. Hierarchical Filtering Model: The two-step filtering process effectively manages large data volumes, enhancing the performance of LVLMs by ensuring that only the most relevant content is used.\n3. Comprehensive Evaluation: The introduction of the UDK-VQA dataset and the experimental comparisons with LVLMs seem to to provide evidence of the framework’s effectiveness.\n\nWEAKNESSES:\n1. \"First open-source framework which seamlessly incorporates existing LVLMs to equip them with up-to-date knowledge during inference.\" I am concerned about this statement. There have been many frameworks utilizing external knowledge. Many of them can seamlessly apply to LVLMs. I am wondering what makes the proposed method different from them. \n\n2. Lack of Comparison against Existing SOTA Methods Utilizing External Knowledge: I encourage the author to conduct experiments to include more SOTA methods directly utilizing retrieved external knowledge for clear comparison.\n\n2. The author mentions the construction of the dataset as one of the contributions. However, I am concerned there is limited novelty in the construction pipeline of UDK-VQA as the steps are very engineering and practical.\n\n3. The paper claims UDK-VQA to be the first test set for evaluating the ability of LVLMs in handling VQA about up-to-date knowledge. I have concerns about this. There have been plenty of VQA benchmarks requiring external knowledge in the field. I am wondering what really makes UDK-VQA different from the existing ones. Are the up-to-date knowledge? However, this terminology is very blurry. What is the definition of up-to-date knowledge? How do you define the timestamp threshold? \n\n4. Broader Evaluation on Other Benchmarks Requiring External Knowledge: Extend the evaluation to include more diverse datasets and LVLMs to verify the generalizability of the framework across different benchmarks requiring external knowledge.\n\n5. Performance Dependency: The framework’s performance is heavily dependent on the quality of the hierarchical filtering model, which may vary across different LVLMs and datasets.\n\nQUESTIONS:\nPlease see above.\n\nLIMITATIONS:\nThe authors have made a commendable effort to address the limitations of their work, including 1) Separate Training for Filters and 2) LVLMs and   Content Snippet Limitations.\n\nSuggestions for Improvement:\n    1) Detailed Impact of Separate Training: The authors could provide more insights into how the separate training of filters and LVLMs specifically impacts performance and whether integrating these components could lead to substantial improvements.\n    2) Addressing Incomplete Snippets: While the paper acknowledges the limitation of incomplete snippets, it would be beneficial to discuss potential strategies to enhance snippet completeness or alternative methods to initially extract more comprehensive information.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-19"}