MindSearch: Mimicking Human Minds Elicits Deep AI Searcher
Authors: name: Zehui Chen; name: Kuikun Liu; name: Qiuchen Wang; name: Jiangning Liu; name: Wenwei Zhang; name: Kai Chen; name: Feng Zhao
Published: 2024-07-29 17:12:40

Abstract: Information seeking and integration is a complex cognitive task that consumes
enormous time and effort. Inspired by the remarkable progress of Large Language
Models, recent works attempt to solve this task by combining LLMs and search
engines. However, these methods still obtain unsatisfying performance due to
three challenges: (1) complex requests often cannot be accurately and
completely retrieved by the search engine once (2) corresponding information to
be integrated is spread over multiple web pages along with massive noise, and
(3) a large number of web pages with long contents may quickly exceed the
maximum context length of LLMs. Inspired by the cognitive process when humans
solve these problems, we introduce MindSearch to mimic the human minds in web
information seeking and integration, which can be instantiated by a simple yet
effective LLM-based multi-agent framework. The WebPlanner models the human mind
of multi-step information seeking as a dynamic graph construction process: it
decomposes the user query into atomic sub-questions as nodes in the graph and
progressively extends the graph based on the search result from WebSearcher.
Tasked with each sub-question, WebSearcher performs hierarchical information
retrieval with search engines and collects valuable information for WebPlanner.
The multi-agent design of MindSearch enables the whole framework to seek and
integrate information parallelly from larger-scale (e.g., more than 300) web
pages in 3 minutes, which is worth 3 hours of human effort. MindSearch
demonstrates significant improvement in the response quality in terms of depth
and breadth, on both close-set and open-set QA problems. Besides, responses
from MindSearch based on InternLM2.5-7B are preferable by humans to ChatGPT-Web
and Perplexity.ai applications, which implies that MindSearch can already
deliver a competitive solution to the proprietary AI search engine.