MMSearch: Benchmarking the Potential of Large Models as Multi-modal
  Search Engines
Authors: name: Dongzhi Jiang; name: Renrui Zhang; name: Ziyu Guo; name: Yanmin Wu; name: Jiayi Lei; name: Pengshuo Qiu; name: Pan Lu; name: Zehui Chen; name: Guanglu Song; name: Peng Gao; name: Yu Liu; name: Chunyuan Li; name: Hongsheng Li
Published: 2024-09-19 17:59:45

Abstract: The advent of Large Language Models (LLMs) has paved the way for AI search
engines, e.g., SearchGPT, showcasing a new paradigm in human-internet
interaction. However, most current AI search engines are limited to text-only
settings, neglecting the multimodal user queries and the text-image interleaved
nature of website information. Recently, Large Multimodal Models (LMMs) have
made impressive strides. Yet, whether they can function as AI search engines
remains under-explored, leaving the potential of LMMs in multimodal search an
open question. To this end, we first design a delicate pipeline,
MMSearch-Engine, to empower any LMMs with multimodal search capabilities. On
top of this, we introduce MMSearch, a comprehensive evaluation benchmark to
assess the multimodal search performance of LMMs. The curated dataset contains
300 manually collected instances spanning 14 subfields, which involves no
overlap with the current LMMs' training data, ensuring the correct answer can
only be obtained within searching. By using MMSearch-Engine, the LMMs are
evaluated by performing three individual tasks (requery, rerank, and
summarization), and one challenging end-to-end task with a complete searching
process. We conduct extensive experiments on closed-source and open-source
LMMs. Among all tested models, GPT-4o with MMSearch-Engine achieves the best
results, which surpasses the commercial product, Perplexity Pro, in the
end-to-end task, demonstrating the effectiveness of our proposed pipeline. We
further present error analysis to unveil current LMMs still struggle to fully
grasp the multimodal search tasks, and conduct ablation study to indicate the
potential of scaling test-time computation for AI search engine. We hope
MMSearch may provide unique insights to guide the future development of
multimodal AI search engine. Project Page: https://mmsearch.github.io