MMSearch: Unveiling the Potential of Large Models as Multi-modal Search Engines

Authors: Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, jiayi lei, Pengshuo Qiu, Pan Lu, Zehui Chen, Guanglu Song, Peng Gao, Yu Liu, Chunyuan Li, Hongsheng Li
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Large Multimodal Model, AI Search Engine, Benchmark

Abstract:
The advent of Large Language Models (LLMs) has paved the way for AI search engines, e.g., SearchGPT, showcasing a new paradigm in human-internet interaction. However, most current AI search engines are limited to text-only settings, neglecting the multimodal user queries and the text-image interleaved nature of website information. Recently, Large Multimodal Models (LMMs) have made impressive strides. Yet, whether they can function as AI search engines remains under-explored, leaving the potential of LMMs in multimodal search an open question. To this end, we first design a delicate pipeline, MMSearch-Engine, to empower any LMMs with multimodal search capabilities. On top of this, we introduce MMSearch, a comprehensive evaluation benchmark to assess the multimodal search performance of LMMs. The curated dataset contains 300 manually collected instances spanning 14 subfields, which involves no overlap with the current LMMs' training data, ensuring the correct answer can only be obtained within searching. By using MMSearch-Engine, the LMMs are evaluated by performing three individual tasks (requery, rerank, and summarization), and one challenging end-to-end task with a complete searching process. We conduct extensive experiments on closed-source and open-source LMMs. Among all tested models, GPT-4o with MMSearch-Engine achieves the best results, which surpasses the commercial product, Perplexity Pro, in the end-to-end task, demonstrating the effectiveness of our proposed pipeline. We further present error analysis to unveil current LMMs still struggle to fully grasp the multimodal search tasks, and conduct ablation study to indicate the potential of scaling test-time computation for AI search engine. We hope MMSearch may provide unique insights to guide the future development of multimodal AI search engine.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730282080176

thread_type:
official_review

<thread_object>
creator: reviewer_gura

<content>
summary:
The authors present MMSEARCH-ENGINE, a multimodal AI search engine pipeline, and MMSEARCH, a comprehensive evaluation benchmark to assess the multimodal search performance of LMMs.

strengths:
1. This paper introduces the first mm RAG pipeline (at least based on the related work introduced) and the first mm search evaluation.
2. The methods for building the MM RAG pipeline and evaluation are solid and with comprehensive statistics.
3. The methodology statement is easy to follow.
4. Comprehensive experiments.
5. Step-wise evaluation of the RAG pipeline is cool.

weaknesses:
1. The evaluation data creation pipeline is not efficient, which may result in difficulties to keep the evaluation dynamic.
2. Evaluation data only comprises 300 entries, which is far from comprehensive. Generally, a benchmark should have 1k+ samples to be robust.
3. The authors didn't exhibit the step-wise scores for the proposed RAG pipeline, which harms the result soundness.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730512348119

thread_type:
official_review

<thread_object>
creator: reviewer_mwvo

<content>
summary:
This paper introduces MMSearch, a benchmark that evaluates the capacity of large multimodal models (LMMs) as AI-driven search engines for handling complex multimodal queries (text and image). The study proposes MMSearch-Engine, a tailored pipeline enabling LMMs to address multimodal search tasks, divided into stages: requery, rerank, and summarization. Experimental results show that models like GPT-4o, integrated into this pipeline, perform better than current commercial systems, such as Perplexity Pro, in end-to-end multimodal search tasks. The paper presents detailed error analyses, offering insights into the limitations of LMMs in search-oriented subtasks, especially in requery and rerank.

strengths:
(1) This paper constructs a novel benchmark MMSearch. It fills an important gap in multimodal AI evaluation by focusing on search capabilities and interaction with multimodal data. It offers a unique benchmark that pushes beyond standard image-text alignment tasks.

(2) This paper proposes an effective multimodal retrieval pipeline MMSearch-Engine, which consists of requery, rerank, and summarization. Experiments have shown that this pipeline can effectively improve the performance of the model.

(3) The experimental analysis is comprehensive and provides good reference conclusions. The error analysis and comparison between commercial and open-source models are valuable for understanding current model limitations, providing useful insights for improving multimodal search models.

(4) This paper uses a weighted score of four scores to evaluate the effect of the model, which not only focuses on the correctness of the model's results, but also focuses on the correctness of its process, which can provide a more comprehensive and effective understanding of the model's performance.

weaknesses:
(1) Although MMSearch covers a wide range of news and knowledge domains, the total number is only 300 instances. Such a number and scale are not enough to fully reflect the generality of the model, etc. The author may need to further expand the scale of the dataset in the future.

(2) Lack of task complexity hierarchy. The complexity of current MMSearch tasks is relatively consistent, lacking a hierarchy of tasks from simple to complex. In the future, tasks of different difficulty levels can be designed to better measure the performance of the model when dealing with tasks of increasing complexity.

(3) There are still limitations in the validation of the model's adaptability. The adaptability of this method in different fields and application scenarios is still unclear, especially on data from special or professional fields (such as medicine or law). Introducing datasets from these fields can more comprehensively evaluate the versatility and adaptability of the model.

questions:
(1) In future versions, could the pipeline include user feedback to improve task accuracy iteratively?

(2) How general and adaptable is the approach to data from specific or specialized fields, such as medicine or law?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730635531219

thread_type:
official_review

<thread_object>
creator: reviewer_3mnq

<content>
summary:
This paper is about benchmarking LLMs as multimodal search engines. The authors propose:
1) A pipeline that enables this type of multimodal search, where images are translated into textual queries before querying a search engine and summarizing the results.
2) A small, but high-quality dataset as benchmark. They do their best to ensure that there is no information leakage between the LLM's training data and the benchmark, by collecting long-tail knowledge data, and very recent news.
They present results using commercial search engines, closed-source, and open-source LLMS, and show that there is much room for improvement, analysing the sources of error along the way.

strengths:
1) The paper is very well-written, with excellent presentation and clarity. In my view, the scope is very well-defined, and there is no redundant information.
2) I particularly like that the authors try to create a dataset that LLMs have not seen before.
3) There is a very large appendix with interesting additional experiments and qualitative analysis.

weaknesses:
1) The way the authors construct the dataset is not future-proof. We have to assume that every piece of news that comes out is immediately ingested, at the very least by large corporations with commercial search engines. In that sense, the benchmark will be obsolete in a couple of months. I would love to see a piece of discussion that proposes a way to make the data collection and annotation pipeline future-proof.
2) The benchmark dataset, even though high quality, is rather small. Maybe I am missing something, but are there any indications that its coverage over the problem space is sufficient?

questions:
Please, refer to the "Weaknesses" section.

I have another question for the authors that I do not consider a weakness, just interesting: In line 409 you state "Any-resolution input only provides slight or no improvement.". Have you tried to corrupt a clean image with differend kinds of corruptions (e.g., occlusions, blur, gaussian noise, etc.), and see how it impacts the search results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721432629

thread_type:
official_review

<thread_object>
creator: reviewer_7dle

<content>
summary:
This work describes a new pipeline to empower LMMs with multimodal search capabilities called MMSearch-Engine, and introduce MMSearch, a comprehensive evaluation benchmark to assess the multimodal search performance of LMMs which includes 300 high-quality instances across 14 subfields. The study also points out LMMs' challenges in multimodal search and suggests scaling test-time computation could improve AI search engines.

strengths:
* MMSearch-Engine is a well-designed three steps pipeline that empower LLMs with multimodal search capabilities. 
* MMSearch is a fair evaluation benchmark because its data is all dated later than the model's knowledge base update time.
* Case analysis is comprehensive.

weaknesses:
* The small amount of data may lead to a certain degree of randomness.
* When constructing the data pipeline, human annotators perform multiple requery operations in cases where no website is classified as valid. Only one requery operation may not fully demonstrate the LMMs' capabilities.

questions:
* Due to the sequential nature of requery, rerank, and summarization, errors in each step can lead to subsequent errors. How to ensure the reasonableness of the weight of each score in formula 1?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732285637827

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response To All Reviewers (1/2)

comment:
We thank all reviewers for their insightful comments and great questions. We greatly appreciate all the reviewers' **acknowledgment of our paper's motivation, design details, and experimental analysis**. Here, we are sharing a summary response covering the common concerns of reviewers.

**Benchmark Size and Future Expansion**

Reviewers commented about the relatively small benchmark size, which is 300 questions in total. We present several observations that may support the appropriateness of the benchmark scale. Then, we propose an automated/semi-automated data curation pipeline to expand the benchmark efficiently. The introduction of the pipeline could not only address the concern of limited data scale but also enable high efficiency for benchmark evolution.

Here are **a few reasons why the current scale is enough**:

1. The news domain in MMSearch covers all hot topics available on mainstream news websites like CNN or The Guardian. It includes most daily search scenarios with comprehensive categories of news.
2. Both the two domain data in MMSearch aim to be beyond the internal knowledge of LMMs. Different from other benchmarks, all the data in MMSearch truly examines the searching capability without any leakage.
3. Each instance includes four tasks: end-to-end, requery, rerank, and summarization. A model is required to be evaluated on each task with the task-specific annotated data. Therefore, there are 1,200 questions in total in MMSearch.

The current scale also allows for efficient evaluation. In the end-to-end setting, we need to interact with the search engine and also record the search results and screenshots. The evaluation of 300 questions now takes 5 to 8 hours for the end-to-end task to complete. A larger dataset may harm the development efficiency.

Now we introduce our **automated/semi-automated data curation pipeline**. The figure is shown in **Figure 8 in Appendix B**. We focus on the news domain here. We first define a website pool and a model pool. The website pool contains general news websites like CNN and expertise websites like Arxiv. The model pool contains state-of-the-art models for the data curation pipeline to guarantee diversity and fairness.

1. End-to-end data curation.  We employ a web crawler to obtain all the subsites published later than a specific date. However, not all the websites are suitable for raising a question to test the LMMs' searching capability. For example, some websites do not contain any recent news, while some websites' contents are difficult to convert into a question with a definite answer. Therefore, we randomly choose a model from the model pool to serve as a news filter, prompting it to filter valid websites by providing some few-shot examples. Next, we provide the text contents and screenshots of the valid websites to a model from the model pool. The model is asked to raise several questions based on the website content. It is encouraged to raise questions that cannot be answered only by text. As for the question with an image, the model is asked to describe the image content briefly, and we will later use the description to search in Bing and obtain the first result image. Finally, we apply the quality check of the generated questions and their corresponding images either by human or a model from the model pool. 
2. Requery data generation. We provide the question, the question image content, and the answer to a model. The model is prompted to come up with a suitable requery. 
3. Rerank data generation. We provide the generated requery to the search engine and retrieve K websites for rerank. Again, we provide the question, the question image content, and the answer to a model and ask to categorize each website into valid, unsure, or invalid. 
4. Summarization data generation. We randomly choose one website from the websites marked as valid in the last step and obtain its full content.

Notably, human check is a must for the requery data generation process. There should be at least one valid website to guarantee the effectiveness of the generated requery. Only after human verification of this step, the quality of rerank and summarization data generation is assured.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary:**

This paper proposes a straightforward approach to developing a multi-modal search engine by integrating LLMs, Google Lens, and other tools. The core components of the system are re-querying, reranking, and summarization. The scope of the paper is well-defined and offers valuable insights into the proposed direction. Notably, the benchmark datasets provided are expected to be highly beneficial and could stimulate further research in this area, despite their relatively small size. Furthermore, the paper includes extensive experiments and detailed explanations in the Appendix, thoroughly showcasing the authors' ideas, methodology, and evaluation process.

**Strength:**
- A well-designed three-step pipeline (re-query, rerank, summarization) that enhances LLMs with multimodal search capabilities.
- Fills a critical gap in multimodal AI evaluation by emphasizing search and interaction with multimodal data, beyond standard image-text alignment tasks.
- Authors created a dataset that LLMs have not encountered before, making it a unique and valuable resource for research.

**Weakness:**
- The dataset contains only 300 instances, which is insufficient to fully reflect the generality of the model's performance.
- Human annotators perform multiple requery operations when no website is classified as valid, but a single requery may not fully demonstrate the capabilities of LLMs.
- No clear indication that the dataset sufficiently covers the problem space, raising concerns about the robustness of the benchmark.