Vision Search Assistant: Empower Vision-Language Models as Multimodal Search Engines

## Abstract

Abstract Search engines enable the retrieval of unknown information with texts. However, traditional methods fall short when it comes to understanding unfamiliar visual content, such as identifying an object that the model has never seen before. This challenge is particularly pronounced for large vision-language models (VLMs): if the model has not been exposed to the object depicted in an image, it struggles to generate reliable answers to the user’s question regarding that image. Moreover, as new objects and events continuously emerge, frequently updating VLMs is impractical due to heavy computational burdens. To address this limitation, we propose Vision Search Assistant, a novel framework that facilitates collaboration between VLMs and web agents. This approach leverages VLMs’ visual understanding capabilities and web agents’ real-time information access to perform open-world Retrieval-Augmented Generation via the web. By integrating visual and textual representations through this collaboration, the model can provide informed responses even when the image is novel to the system. Extensive experiments conducted on both open-set and closed-set QA benchmarks demonstrate that the Vision Search Assistant significantly outperforms the other models and can be widely applied to existing VLMs.

## 1 Introduction

The advent of Large Language Models (LLMs) [, , , , , ] has significantly enhanced the human capacity to acquire unfamiliar knowledge through powerful zero-shot Question-Answering (QA) capabilities. Building upon these advancements, techniques such as Retrieval-Augmented Generation (RAG) [, , ] have further reinforced LLMs in knowledge-intensive, open-domain QA tasks. Concurrently, recent progress in visual instruction tuning [, , ] has led to the development of large Vision-Language Models (VLMs) that aim to equip LLMs with visual understanding capabilities. By scaling model parameters and training on extensive text-image datasets, VLMs such as LLaVA-1.6-34B [], Qwen2-VL-72B [], and InternVL2-76B [] have achieved state-of-the-art performance on the OpenVLM leaderboard(^1^11[https://huggingface.co/spaces/opencompass/open_vlm_leaderboard](https://huggingface.co/spaces/opencompass/open_vlm_leaderboard)). However, LLMs and VLMs are subject to the limitations imposed by their knowledge cutoff dates. They may provide incorrect answers when asked about events or concepts that occurred after their knowledge cutoff dates (Figure 1) To overcome this limitation for LLMs, they are often connected to web agents [, , , , ], which enable internet access and information retrieval, allowing them to obtain the most up-to-date data and improve the accuracy of their responses. Such agents are designed to interpret natural language instructions, navigate complex web environments, and extract relevant textual information from HTML documents, thereby enhancing the accessibility and utility of vast amounts of web-based textual data for a wide range of applications.

However, for VLMs facing unseen images and novel concepts, their ability to learn and use up-to-date multimodal knowledge from the internet remains a pressing challenge. As the existing web agents predominantly rely on searching the user’s question and summarizing the returned HTML content, they present a notable limitation when handling tasks involving images or other visual content: the visual information is often overlooked or inadequately processed.

In this work, we enable the VLM to answer a question regarding an unseen image or novel concept, which behaves like a human searching the Internet. It 1) understands the query, 2) decides which objects in the image it should look at and infers the correlations among the objects, 3) respectively generates the texts to search, 4) analyzes the contents returned from the search engine based on the query and inferred correlations, and 5) judges if the obtained visual and textual information is sufficient for generating the answer or it should iterate and refine the above process. Regarding the concrete designs of such a framework, we make contributions by answering the following three questions that remained unanswered in the literature

- •
What to search: shall we search the descriptions of the whole image or some critical objects?
- •
How to search: shall we search once and summarize a huge amount of returned content or search progressively to obtain more related content?
- •
By what to conclude: shall the final answer be generated with the eventually summarized web knowledge or all the knowledge acquired through the entire searching process?

Figure: Figure 2: Comparsion with Closed-Source Models including GPT-4o [], Gemini [], Claude 3.5 Sonnet [] with Vision Search Assistant shows that Vision Search Assistant satisfies users’ needs better even if the image is novel.
Refer to caption: x2.png

By exploring such three aspects, we propose Vision Search Assistant, a framework based on VLM-agent collaboration, which empowers an arbitrary VLM to become a multimodal automatic search engine. We integrate VLMs into web agents to understand what the user wants, where to look, what to search, how to learn from the returned multimodal information, and whether to conclude or search another time. More specifically, Vision Search Assistant conducts three steps (Figure 3):

- •
Visual Content Formulation (§ 3.1) is proposed to represent the visual content with VLM-generated textural descriptions of critical visual objects and their underlying correlations. Through this step, we obtain a correlated formulation for each critical object, which is a textual representation that considers its correlations with other objects.
- •
Web Knowledge Search (§ 3.2) is a novel algorithm that drives the search process. It generates multiple sub-questions with the web agent regarding the user prompt and the correlated formulation of each critical object. Each of such sub-questions can be viewed as a node in a directed graph. For each correlated formulation and each sub-question, we construct the search query by combining the correlated formulation and sub-question and use the LLM to analyze and select useful contents returned from the search engine, then summarize the web knowledge from the answers obtained with all such sub-questions. After that, we iterate the above step by proposing more sub-questions based on the previous sub-questions and known web knowledge, which can be seen as expanding the directed graph. We use the LLM to judge if the latest iteration has obtained sufficient web knowledge to answer the user’s question and terminate the process if so.
- •
Collaborative Generation (§ 3.3)) is proposed to use the VLM to generate the eventual answer with all the critical objects in the image, the initial question, all of their correlated formulations, and the web knowledge obtained in every iteration.

As shown in Figure 2, Vision Search Assistant can generate more precise answers than powerful closed-source models such as GPT-4o [], Gemini [], and Claude 3.5 Sonnet [], which further validates the necessity and promising improvement of VLM-Agents collaboration in tackling the growing complexity of multimodal web data and the rapid influx of novel visual content.

## 2 Related Work

Vision-Language Models.
Pioneering models such as Flamingo [], BLIP-2 [], LLaVA [], and MiniGPT-4 [] have been instrumental in training vision-language models for the tasks such as image captioning and visual question answering. Recent works focus on higher-quality datasets [] and developing lightweight, trainable models [] to enhance efficiency and accessibility. Further progress includes extending large language models (LLMs) to additional modalities and domains, such as audio processing [, ], and more modalities [, , ]. Additionally, KOSMOS-2 [], InternVL2 [], MiniGPT-2 [], and LLaVA-1.5 [] incorporate region-level information by encoding visual regions to embeddings of language models. However, despite scaling model parameters and training data, VLMs’ ability to handle unseen images remains limited, as they heavily rely on previously seen text-image pairs. To overcome this, we propose to enhance VLMs’ performance on novel data by improving generalization without relying solely on extensive training pairs.

Figure: Figure 3: Overview of Vision Search Assistant. We first identify the critical objects and generate their descriptions considering their correlations, named Correlated Formulation, using the Vision Language Model (VLM). We then use the LLM to generate sub-questions that leads to the final answer, which is referred to as the Planning Agent. The web pages returned from the search engine are analyzed, selected, and summarized by the same LLM, which is referred to as the Searching Agent. We use the original image, the user’s prompt, the Correlated Formulation together with the obtained web knowledge to generate the final answer. Vision Search Assistant produces reliable answers, even for novel images, by leveraging the collaboration between VLM and web agents to gather visual information from the web effectively.
Refer to caption: x3.png

Web Search Agents.
The development of web search agents has progressed through integrating advanced learning techniques, enhancing autonomy, and optimizing efficiency in web automation. Early models like WebGPT [] and WebGLM [] primarily focused on retrieving information for question-answering tasks, while newer models, such as AutoWebGLM [], address deployment challenges with compact designs. Despite their strong web navigation skills, larger models such as WebAgent [] are constrained by size. Incorporating reinforcement learning [] and behavior cloning [] has further boosted the efficiency of web agents, as demonstrated by MindAct [], which integrates cognitive functionalities for complex task execution. While these advances are leading to more scalable and versatile solutions for real-world use, current web agents still struggle with processing visual content directly from the web. We introduce Vision-Language Models to enable web agents to effectively interpret and interact with visual data, significantly expanding their capabilities in handling complex, multimodal tasks. We hope it can make web agents more powerful and adaptable in real-world applications.

Retrieval-Augmented Generation.
Integrating retrieval from large corpora into language models has become essential for knowledge-intensive tasks like open-domain question answering. Instead of relying solely on pre-trained data, the retriever-reader architecture [, ] enables models to fetch relevant information based on an input query, which the language model then uses to generate accurate predictions. Recent research has enhanced retrievers [, , , , ], improved readers [, , , ], jointly fine-tuned both components [, , , ], and integrated retrieval directly within language models [, , ].

Therefore, we propose the Vision Search Assistant framework, which introduces an open-world retrieval-augmented generation framework that extends beyond text-based retrieval to operate across both vision and language modalities on the web. It enables VLMs to access real-time, dynamic information, improving their ability to handle novel, cross-modal queries. By pushing the boundaries of retrieval beyond static knowledge sources, we address the challenge of incorporating web-based, multimodal data into generative tasks, offering a more adaptable and scalable solution for RAG.

## 3 Vision Search Assistant

### 3.1 Visual Content Formulation

The Visual Content Formulation is proposed to extract the object-level descriptions and correlations among objects in an image. Given the input image ${\bm{X}}_{I}$, we first use the open-vocab detector $\mathcal{F}_{\mathrm{det}}(\cdot)$ [] to obtain $N$ regions of interests in the original image,

$$ $\{{\bm{X}}_{I}^{(i)}\}_{i=1}^{N}=\mathcal{F}_{\mathrm{det}}({\bm{X}}_{I}),$ (1) $$

where $i$ indicates the $i$-th region ${\bm{X}}_{I}^{(i)}$ in the image ${\bm{X}}_{I}$. Then we employ the pretrained VLM (^2^22Our experiments are conducted with LLaVA-1.6-Vicuna-7B model, which is publicly available at [https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b](https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b).) $\mathcal{F}_{\mathrm{vlm}}(\cdot,\cdot)$ to caption these regions $\{{\bm{X}}_{I}^{(i)}\}_{i=1}^{N}$ conditioned on the tokenized user’s textual prompt ${\bm{X}}_{T}$, and obtain the visual caption ${\bm{X}}_{r}^{(i)}$ for the $i$-th region:

$$ ${\bm{X}}_{r}^{(i)}=\mathcal{F}_{\mathrm{vlm}}({\bm{X}}_{I}^{(i)},{\bm{X}}_{T}).$ (2) $$

In this way, we make the regional captions $\{{\bm{X}}_{r}^{(i)}\}_{i=1}^{N}$ contain specific visual information obtained based on the user’s interests. To formulate the visual content more comprehensively, we further correlate these visual regions to obtain precise descriptions of the whole image. More specifically, for each region, we concatenate its corresponding caption and the captions of all the other regions. The resultant text, denoted by $[{\bm{X}}_{r}^{(i)},\{{\bm{X}}_{r}^{(j)}\}_{j\neq i}]$, encodes the underlying correlations. It is fed into the VLM together with the image region ${\bm{X}}_{I}^{(i)}$. The output is referred to as the correlated formulation of each region $\{{\bm{X}}_{c}^{(i)}\}_{i=1}^{N}$.

$$ ${\bm{X}}_{c}^{(i)}=\mathcal{F}_{\mathrm{vlm}}({\bm{X}}_{I}^{(i)},[{\bm{X}}_{r} ^{(i)},\{{\bm{X}}_{r}^{(j)}\}_{j\neq i}])).$ (3) $$

We will use the correlated formulations of such regions to perform the following web search.

### 3.2 Web Knowledge Search: The Chain of Search

The core of Web Knowledge Search is an iterative algorithm named Chain of Search, which is designed to obtain the comprehensive web knowledge of the correlated formulations $\{{\bm{X}}_{c}^{(i)}\}_{i=1}^{N}$. We take an arbitrary $i$-th region ${\bm{X}}_{c}^{(i)}$ to elaborate on the Chain of Search algorithm and drop the superscript $(i)$ for convenience.

We use the LLM in our VLM to generate sub-questions that lead to the final answer, which is referred to as the Planning Agent. The web pages returned from the search engine are analyzed, selected, and summarized by the same LLM, which is referred to as the Searching Agent. In this way, we can obtain web knowledge regarding the visual content. Then, based on each of such sub-questions, the Planning Agent generates more sub-questions, and the Searching Agent obtains web knowledge for the next iteration. Formally, we define a directed graph to represent this process, which is ${\mathcal{G}}=\left\langle V,E\right\rangle$, where $V=\{V_{0}\}$ is the set of nodes, $V_{0}$ is the initial node, and $E=\varnothing$ is the set of edges. A node represents a set of known information so that $V_{0}$ should represent what we know about the region before any web search, i.e., the correlated formulation ${\bm{X}}_{c}$. This is formulated as $V_{0}\leftarrow{\bm{X}}_{c}$. When we search with a sub-question, we will update the graph with a new node representing the web knowledge gained through the sub-question.

For the $1$-st update, we generate sub-questions based on $V_{0}$ and denote the generated sub-questions by $({\bm{X}}_{s}^{1})=\{({\bm{X}}_{s}^{1})_{i}\}_{i=1}^{N_{v}^{1}}$, where $N_{v}^{1}$ is the number of sub-questions, i.e., the number of new nodes.

Figure: Figure 4: The Chain of Search algorithm (§ 3.2). We deduce the update of the directed graph when $k=1,2,\cdots$, and the web knowledge is progressively extracted from each update.
Refer to caption: x4.png

Let $j$ be the index of the sub-question, the new node $\Delta V_{j}^{(1)}$ is a child of $V_{0}$, which corresponds to a search with sub-question $({\bm{X}}_{s}^{1})_{j}$. The returned set of web pages are formatted as HTML documents. The Searching Agent uses the LLM in our VLM, which is denoted by $\mathcal{F}_{llm}(\cdot)$, to judge their relevance to the parent node $V_{0}$ and the corresponding sub-question $({\bm{X}}_{s}^{1})_{j}$ and select those of the highest relevance. The selected web page index $\tau_{j}^{1}$ can be formulated

$$ $\tau_{j}^{1}=\mathcal{F}_{\mathrm{llm}}([V_{0},({\bm{X}}_{s}^{1})_{j}])\,.$ (4) $$

We use $\tau_{j}^{1}$ to select a subset of the HTML documents at the $1$-st update, and those selected for sub-question $j$ are denoted by $\{P_{j}^{1}\}$. We derive the search response $R_{j}^{1}$ for sub-question $j$ at the $1$-st update by summarizing the selected pages with the LLM, which is $R_{j}^{1}=\mathcal{F}_{\mathrm{llm}}(\{P_{j}^{1}\})$. By the definition of the directed graph, the new node $\Delta V_{j}^{(1)}$ should represent $R_{j}^{1}$, that is, $\Delta V_{j}^{(1)}\leftarrow R_{j}^{1}$. We add $\Delta V_{j}^{(1)}$ into the node set and $(V_{0},V_{j}^{(1)})$ into the edge set. In this paper, $\Delta V_{j}^{(1)}$ is synonymous with “the search response $R_{j}^{1}$ obtained with sub-question $({\bm{X}}_{s}^{1})_{j}$”.

Figure: Figure 5: Open-Set Evaluation: We conduct a human expert evaluation on open-set QA tasks. Vision Search Assistant significantly outperformed Perplexity.ai Pro and GPT-4o-Web across three key objectives: factuality, relevance, and supportiveness.
Refer to caption: x5.png

Then, we summarize the search responses of all the $N_{v}^{1}$ nodes at the $1$-st update and obtain the comprehensive web knowledge ${\bm{X}}_{w}^{(1)}$, which is denoted by

$$ ${\bm{X}}_{w}^{(1)}=\mathcal{F}_{\mathrm{llm}}([R_{1}^{1},R_{2}^{1},\cdots,R_{N _{v}^{1}}^{1}])\,.$ (5) $$

**Table 1: Closed-Set Evaluation on the LLaVA-W benchmark. We use GPT-4o (0806) for evaluation. Naive search here denotes the VLM with Google image search.**
| Model | Conversation (%) | Detail (%) | Reasoning (%) | Overall (%) |
| --- | --- | --- | --- | --- |
| LLava-1.6-7B (Baseline) | 72.9 | 76.5 | 84.2 | 78.5 |
| LLava-1.6-7B (naive search) | 70.3 | 76.7 | 85.8 | 78.9 |
| LLava-1.6-7B (w/ § 3.2) | 72.6 | 78.9 | 89.8 | 82.7 |
| Vision Search Assistant | 73.3 (+0.4) | 79.3 (+2.8) | 95.0 (+10.8) | 84.9 (+6.4) |

For the following updates with $k>1$, we expand the graph similarly but with minor differences:

- •
For each node at update $k-1$, we use the LLM to generate further sub-questions, just like how we expand $V_{0}$ at the $1$-st update.
- •
When we select the most relevant web pages for a node $\Delta V_{j}^{(k)}$, we analyze their relevance to not only $V_{0}$ and the corresponding sub-question $({\bm{X}}_{s}^{k})_{j}$ (just like the $1$-st update), but also the search response of its parent node.
- •
When we summarize the comprehensive web knowledge ${\bm{X}}_{w}^{(k)}$, except for the search responses of all the nodes at the current update, we also use all the known comprehensive web knowledge $\{{\bm{X}}_{w}^{(i)}\}_{i=1}^{k-1}$ and the search responses of all the previous nodes $\{R_{m}^{n}\}_{\{m=1,n=1\}}^{\{m=N_{v}^{n},n=k-1\}}$.

Formally, when $k>1$:

$$ $\displaystyle\small\tau_{j}^{k}=$ $\displaystyle\mathcal{F}_{\mathrm{llm}}([V_{0},({\bm{X}}_{s}^{k})_{j},R_{i}^{k -1}]),$ (6) $\displaystyle{\bm{X}}_{w}^{(k)}=$ $\displaystyle\mathcal{F}_{\mathrm{llm}}(\{{\bm{X}}_{w}^{(i)}\}_{i=1}^{k-1},\{R _{m}^{n}\}_{\{m=1,n=1\}}^{\{m=N_{v}^{n},n=k-1\}},\{R_{i}^{k}\}_{i=1}^{N_{v}^{k }}).$ $$

At each update, the search agent uses the LLM to judge if the knowledge currently obtained is sufficient to answer the initial question. If so, we terminate the process.

### 3.3 Collaborative Generation

We use the original image ${\bm{X}}_{I}$, the user’s initial prompt ${\bm{X}}_{T}$, and the Correlated Formulations $\{{\bm{X}}_{C}^{(i)}\}_{i=1}^{N}$ together with the obtained web knowledge $\{{\bm{X}}_{W}^{(i)}\}_{i=1}^{N}$ to collaboratively generate the final answer ${\bm{Y}}$ with the VLM:

$$ ${\bm{Y}}=\mathcal{F}_{\mathrm{vlm}}({\bm{X}}_{I},\{{\bm{X}}_{c}^{(i)}\}_{i=1}^ {N},\{{\bm{X}}_{w}^{(i)}\}_{i=1}^{N},{\bm{X}}_{T})\,.$ (7) $$

## 4 Experiments

### 4.1 Open-Set Evaluation

Figure: Figure 6: Comparisons among Qwen2-VL-72B, InternVL2-76B, and Vision Search Assistant. We compare the open-set QA results on both novel events (the first two rows) and images (the last two rows). Vision Search Assistant excels in generating accurate and detailed results.
Refer to caption: x6.png

Setup. In the Open-Set Evaluation, we performed a comparative assessment by 10 human experts evaluation, which involved questions of 100 image-text pairs collected from the news from July 15th to September 25th covering all fields on both novel images and events. Human experts conducted the evaluations across three critical dimensions: factuality, relevance, and supportiveness.

Figure: Figure 7: Ablation Study on What to Search. We use the object description to avoid the visual redundancy of the image.
Refer to caption: x7.png

Figure: Figure 8: Ablation Study on How to search. We propose the Chain of Search (§ 3.2) to obtain related web knowledge for VLMs progressively.
Refer to caption: x8.png

Figure: Figure 9: Ablation Study on Complex Scenarios. We use visual correlation to improve the ability in multiple-object scenarios.
Refer to caption: x9.png

Figure: Figure 10: A series of demos of Vision Search Assistant on novel images, events, and in-the-wild scenarios. Vision Search Assistant delivers promising potential as a powerful multimodal engine.
Refer to caption: x10.png

Results and Analysis. As illustrated in Figure 5, Vision Search Assistant demonstrated superior performance across all three dimensions compared to Perplexity.ai Pro and GPT-4-Web: 1) Factuality: Vision Search Assistant scored 68%, outperforming Perplexity.ai Pro (14%) and GPT-4-Web (18%). This significant lead indicates that Vision Search Assistant consistently provided more accurate and fact-based answers. 2) Relevance: With a relevance score of 80%, Vision Search Assistant demonstrated a substantial advantage in providing highly pertinent answers. In comparison, Perplexity.ai Pro and GPT-4-Web achieved 11% and 9%, respectively, showing a significant gap in their ability to maintain topicality with the web search. 3) Supportiveness: Vision Search Assistant also outperformed the other models in providing sufficient evidence and justifications for its responses, scoring 63% in supportiveness. Perplexity.ai Pro and GPT-4-Web trailed with scores of 19% and 24%, respectively. These results underscore the superior performance of Vision Search Assistant in open-set tasks, particularly in delivering comprehensive, relevant, and well-supported answers, positioning it as an effective method for handling novel images and events.

### 4.2 Closed-Set Evaluation

Setup. We conduct the closed-set evaluation on the LLaVA-W [] benchmark, which contains 60 questions regarding the Conversation, Detail, and Reasoning abilities of VLMs in the wild. We use the GPT-4o(0806) model for evaluation. We use LLaVA-1.6-7B as our baseline model, that has been evaluated in two modes: the standard mode and a “naive search” mode that utilizes a simple Google Image search component. Additionally, an enhanced version of LLaVA-1.6-7B, equipped with improvements outlined in section § 3.2, is also evaluated.

Results and Analysis. As shown in Table 1, the Vision Search Assistant demonstrates the strongest performance across all categories. Specifically, it achieves a 73.3% score in the conversation category, representing a modest gain of +0.4% compared to the LLaVA models. In the detail category, the Vision Search Assistant stands out with a score of 79.3%, outperforming the highest-performing LLaVA variation by +2.8%.
When it comes to reasoning, our method brings out +10.8% above the best-performing LLaVA model. This suggests that the Vision Search Assistant’s advanced integration of visual and textual search greatly enhances its reasoning capabilities. The overall performance of the Vision Search Assistant is 84.9%, marking an improvement of +6.4% over the baseline model. This shows that the Vision Search Assistant excels in both conversational and reasoning tasks, giving it a clear advantage for in-the-wild abilities.

### 4.3 Ablation Study

What to search: Object-Level Descriptions. As illustrated in Figure 7, if we use the image-based caption, the search agent can not precisely focus on the key information (the handbag in this figure), meanwhile, the image contains visual redundancy, which obstacles the textual description to drive web agent and retrieve the most relevant web pages, therefore, we use the object-level description in the following ablation study.

Complex Scenarios of Search: Visual Correlation. We find that the caption can not fully support the search ability in multiple-object scenarios. As shown in Figure 9, the caption of Biden can not answer the questions on the group-wise debate, the visual correlation (“debate” in this demo) between Trump can effectively improve the answer quality.

How to search: Chain of Search (§ 3.2). The trivial idea to incorporate web search with VLMs is to introduce a Google search engine and re-rank the large-scale related pages. As shown in Figure 8, we found it difficult to directly obtain the required knowledge since the page-rank method prefers more hyper-link pages instead of exact relevance. The VLM is also limited to its context length to summarize a large number of pages. Therefore, we propose the chain of search and enable the progressive summary of web knowledge aiming to answer the user’s questions.

## 5 Conclusion and Discussion

In this paper, we seek to improve the generalization ability of VLMs of novel images and extend the capacity of web agents to solve visual content tasks. Through the synergistic collaboration between VLMs and web agents, we find that VLMs can generate more reliable answers regarding novel images with the help of real-time web knowledge retrieval, and web agents can solve more challenging tasks than HTML documents only. Meanwhile, there are also some limitations inside the Vision Search Assistant framework such as the exact inference speed of VLMs, the web condition of web agents, and the retrieval efficiency. We hope this paper can inspire more research to address the challenges of VLMs in user experience and improve the automation abilities of web agents across diverse modalities.

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. Could you provide more detailed explanations of the terms "actuality", "relevance", and "supportiveness" in the context of the Open-Set Evaluation? This might help clarify their specific meanings for the readers.
2. Would it be possible to include quantitative results in the ablation study, perhaps in tabular form, to enhance the clarity and impact of your findings?
3. Could you clarify the specific type of score being referenced in your experiments? This additional detail would help readers better understand the results.
soundness: 2
strengths: 1. This paper effectively addresses the challenges posed by traditional methods in comprehending unfamiliar visual content.
2. The experiment results achieved good performance.
summary: This paper proposes the Vision Search Assistant to address the issue that traditional methods struggle with understanding unfamiliar visual content. A workflow comprising three parts is proposed: Visual Content Formulation, Web Knowledge Search, and Collaborative Generation. Experiments on both open-set and closed-set QA tasks demonstrate that the model proposed in this paper outperforms other models significantly and can be widely applied to existing VLMs. The ablation studies validate each component of the model effectively.
weaknesses: 1. The framework appears to be a typical application framework for LLMs that combines captioning, searching, and RAG. This seems to be a combination of A + B + C. The author needs to highlight their contributions.  
2. Which search engine is being used? It doesn't seem to be mentioned in the paper.
3. The discussion of factuality, relevance, and supportiveness in the Open-Set Evaluation could be more detailed, as the current descriptions may not fully convey their specific meanings.
4. In the ablation study, providing quantitative results, such as in tabular form, would enhance the clarity and impact of the findings.
5. The experiments could benefit from a clearer explanation of the specific type of score being referenced, as this would help readers better understand the results.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Is Vision Search Assistant allowed to access the internet during the Closed-set Evaluation? If so, does its search scope differ from that of other comparison models?
2. Why does the paper use textual descriptions of objects (line 424) instead of visual object representations for search?
3. How is the performance of the web knowledge search measured, and why is this aspect not explained in the paper?
4. How is the performance of the web knowledge search measured? This paper does not explain how the performance of the Web Knowledge Search component is measured.
5. If the web knowledge search is not accurate, how does this affect the final results returned by the system? What measures are in place to handle such inaccuracies?
soundness: 2
strengths: 1. This paper tackles the challenge of handling unseen images and novel concepts, which traditional VLMs struggle with due to their reliance on previously seen data.
2. This paper introduces an algorithm that allows for progressive summarization of web knowledge, enhancing the system's ability to handle novel and unseen data.
3. This paper tries to answer the remained questions and makes some ablation studies, such as what to search, how to search, and by what to conclude.
summary: To tackle the challenge of handling unseen images and novel concepts, this paper proposes a novel framework called Vision Search Assistant designed to empower vision-language models (VLMs) to function as multimodal search engines. This framework facilitates collaboration between VLMs and web agents, enabling the system to perform open-world RAG by leveraging the visual understanding capabilities of VLMs and the real-time information access of web agents.
weaknesses: 1. The introduction of the experimental section is somewhat careless. For instance, Figure 5 is provided in this paper, but there is no corresponding explanation or discussion of its content within the main text.
2. This paper lacks a detailed description of the open-set dataset used in the evaluation.
3. In Figure 6, the proposed method significantly outperforms GPT-4 web, but there is no clear explanation in the method section that justifies this improvement. Additionally, the paper does not analyze or discuss this result.
4. The experiments conducted are not sufficient, and this paper lacks comparisons with more open-world methods.
5. This paper does not provide information on the computational efficiency or time consumption of the proposed framework, particularly the Chain of Search algorithm, which is noted for its complexity.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: Please see weaknesses.
soundness: 2
strengths: The paper tackles the challenge of enabling vision-language models (VLMs) to understand and respond to unfamiliar visual content by integrating real-time web search capabilities. To this end, the authors introduce the **Vision Search Assistant**, a framework that combines VLMs with web agents to perform open-world Retrieval-Augmented Generation, enhancing the model's ability to handle novel images and events.

Here are Key Contributions:
  - **Visual Content Formulation**: Extracts and correlates object-level descriptions from images.
  - **Web Knowledge Search**: Uses an iterative algorithm to progressively gather and summarize relevant web knowledge.
  - **Collaborative Generation**: Integrates visual and textual information to generate accurate responses.

Extensive experiments show that the Vision Search Assistant improves the performance of the base model in both open-set and closed-set QA benchmarks.
summary: The paper tackles the challenge of enabling vision-language models (VLMs) to understand and respond to unfamiliar visual content by integrating real-time web search capabilities.

The authors introduce the **Vision Search Assistant**, a framework that combines VLMs with web agents to perform open-world Retrieval-Augmented Generation, enhancing the model's ability to handle novel images and events.

Extensive experiments show that the Vision Search Assistant achieves good performance in both open-set and closed-set QA benchmarks.
weaknesses: ### Method
1. **Complexity**: The approach is overly complicated, essentially trading time for performance.
2. **Correlated Formulations**: How are these extracted? The function \( F_{VLM} \) takes an image and two captions, so how can it output correlated formulations? Is \( F_{VLM} \) specifically trained for this task?

### Experiments
1. **Insufficient Experiments**: The proposed method is essentially retrieval-augmented generation (RAG), yet the authors barely discuss its relevance to existing RAG works. Furthermore, there are no experimental comparisons between the proposed method and state-of-the-art works, which is unacceptable in research. Given that New Bing (Copilot) is known as an excellent RAG system, it would be more convincing to compare with it.
2. **Impact of Open-Set Detectors**: How do open-set detectors affect the performance of the proposed method?
3. **Generalization Across VLLMs**: The proposed method appears to be training-free. Why were experiments only conducted on LLaVA 1.6? It is crucial to validate the generalization of this method across different VLLMs.
4. **Open-Set Benchmark**: What does the open-set benchmark look like? How is fairness ensured during subjective evaluation?
5. **Overclaim**: Why is the Chain of Search algorithm considered the answer to “How to search”? There are no quantitative results to support this claim. Additionally, the explanation for “By what to conclude” should be “How to conclude”. “What to conclude” focuses more on content selection.
6. **Efficacy of the Proposed Method**: The authors use many qualitative results to prove the efficacy of the method, which is not convincing. Sometimes, if VLLMs have never seen the objects before, the captions extracted from VLLMs make it difficult to retrieve results from the web.

### Additional Comments
- **Political Examples**: Using examples related to politics is not advisable. Many VLLMs avoid discussing such topics, which may lead to unfair implications about model performance.
- **Overall Assessment**: In my view, this paper is not ready for publication due to numerous weaknesses. I vote to reject this paper and hope the authors can further improve their work.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. What is the motivation for transforming visual regions into text descriptions for search, instead of directly using images? In my view, some critical information may be lost in the descriptive process, and many models struggle with entity recognition (such as the athlete’s name in Fig. 3), potentially affecting search accuracy. How do the authors think of this issue?
2. In Table 1, what does "LLava-1.6-7B (Baseline)" represent? Does it indicate zero-shot LLava? Why does the performance on Conversation decline when image search is introduced (naive search)? The way Google Image Search is utilized requires further explanation.
3. There is a lack of detailed qualitative analysis. The authors could provide a case study showing descriptions of each region, the results of each search chain step, and the final model-generated response to illustrate the benefits of the proposed method.
4. In Fig. 6, the authors conduct human evaluations of open-domain QA. What criteria are used to assess Factuality, Relevance, and Supportiveness? If these definitions follow prior work, citations should be provided; otherwise, detailed explanations are needed. Furthermore, are the human evaluation results statistically significant and consistent?
soundness: 2
strengths: 1. Open-domain visual search is valuable in practice, and combining multimodal large language models with search represents an intriguing research direction.
2. The paper identifies three critical questions in visual search — what to search, how to search, and what to conclude, which are interesting research points.
summary: This paper introduces a new framework called Vision Search Assistant, which aims to enhance the capabilities of mutimodal large language models and web agents through collaboration, targeting multimodal search tasks involving unfamiliar images and novel concepts. The authors identify three key aspects of visual search: what to search, how to search, and what to conclude. To address these, the proposed framework utilizes three steps: Visual Content Formulation, Chain of Search, and Collaborative Generation. The Vision Search Assistant is evaluated on the LLaVA-Wild dataset, and further assessed through open-set evaluations by human experts. Results indicate that Vision Search Assistant achieved the best outcomes across multiple metrics.
weaknesses: 1. There is a lack of efficiency analysis, especially concerning the average latency per query, number of LMM calls, or token consumption. While efficiency is often an issue with large models, an essential analysis should not be overlooked.
2. The evaluation has some limitations. The paper only tests on the LLaVA-Wild dataset, which contains only 60 queries. In practice, many VQA datasets involving external knowledge, such as A-OKVQA, could serve as more comprehensive evaluation benchmarks.
3. There is insufficient information regarding reproducibility. Sampling parameters, such as temperature and top-p during large model inference, are not provided. Additionally, prompts used with large models (e.g., in Eq. 4 and Eq. 7) are not specified.