MindSearch: Mimicking Human Minds Elicits Deep AI Searcher

Authors: Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, Feng Zhao
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: language model, search engine, multi-agent system

Abstract:
Information seeking and integration is a complex cognitive task that consumes enormous time and effort. Inspired by the remarkable progress of Large Language Models, recent works attempt to solve this task by combining LLMs and search engines. However, these methods still obtain unsatisfying performance due to three challenges: (1) complex requests often cannot be accurately and completely retrieved by the search engine once (2) corresponding information to be integrated is spread over multiple web pages along with massive noise, and (3) a large number of web pages with long contents may quickly exceed the maximum context length of LLMs. Inspired by the cognitive process when humans solve these problems, we introduce MindSearch to mimic the human minds in web information seeking and integration, which can be instantiated by a simple yet effective LLM-based multi-agent framework. The WebPlanner models the human mind of multi-step information seeking as a dynamic graph construction process: it decomposes the user query into atomic sub-questions as nodes in the graph and progressively extends the graph based on the search result from WebSearcher. Tasked with each sub-question, WebSearcher performs hierarchical information retrieval with search engines and collects valuable information for WebPlanner. The multi-agent design of MindSearch enables the whole framework to seek and integrate information parallelly from larger-scale (e.g., more than 300) web pages in 3 minutes, which is worth 3 hours of human effort. MindSearch demonstrates significant improvement in the response quality in terms of depth and breadth, on both close-set and open-set QA problems. Besides, responses from MindSearch based on InternLM2.5-7B are preferable by humans to ChatGPT-Web and Perplexity.ai applications, which implies that MindSearch can already deliver a competitive solution to the proprietary AI search engine.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729865899515

thread_type:
official_review

<thread_object>
creator: reviewer_ic9f

<content>
summary:
This paper introduces a system called MindSearch, designed to emulate human cognitive processes to enhance web information retrieval and integration tasks. By combining large language models (LLMs) with search engines, the system addresses limitations in handling complex queries, fragmented information, and lengthy content through an LLM-based multi-agent framework.
* WebPlanner: Simulates the cognitive process of multi-step information seeking by breaking down user queries into atomic subproblems, represented as nodes in a graph. The graph is then progressively expanded based on search results from WebSearcher.
* WebSearcher: Conducts hierarchical information retrieval for each subproblem, using a search engine to gather valuable information for WebPlanner.

This multi-agent design enables MindSearch to search and integrate information from vast web sources within three minutes, equivalent to saving three hours of manual effort. MindSearch demonstrates significant response quality improvements in both closed-set and open-set QA tasks.

strengths:
- The paper presents a clear and logical approach to the problem, with a well-organized visual format that is easy to understand and read. 
- This method provides a novel question-answering retrieval method based on directed acyclic graphs, which makes the RAG more reasonable.

weaknesses:
- The method part is not detailed enough to show the technical details. For instance, the design of DAG and the use of DAG is not clear.
- Few baseline methods from the same category are included, and many RAG-based question-answering approaches are left unexamined, such as ChatKBQA, AutoReAct, etc. 
- The backbone was only tested on GPT-4 (close sourced) and InternLM2.5 (open sourced). Under this setting, it is hard to tell if the MindSearch will work for all (at least most) LLMs.

questions:
- When constructing the DAG, how does MindSearch automatically create graph nodes? What are some tips for structuring question as graph nodes?
- When large amounts of content are retrieved, how does WebSearcher reduce noise? And, as the rapid growth of web content can easily exceed the maximum context length of the LLM, how does WebSearcher effectively limit content length?
- Additionally, could the experiment include more closed-source and open-source LLMs to further validate the effectiveness of the method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730126464377

thread_type:
official_review

<thread_object>
creator: reviewer_xr8b

<content>
summary:
This paper proposes a novel tool agent, MindSearch, which decomposes user queries into atomic sub-questions represented as graph nodes and progressively extends the graph based on the search results from WebSearcher. For each sub-question, WebSearcher performs hierarchical information retrieval using search engines to gather relevant information for WebPlanner. Extensive experiments are conducted, including both open-set and closed-set datasets, and using open-source models alongside close-sourced LLMs, demonstrating the its effectiveness.

strengths:
S1: The writing and framework of this paper are clear and easy to follow.

S2: The method is novel, utilizing the agents WebPlanner and WebSearcher to perform web search tasks.

S3: Extensive experiments are conducted, demonstrating both the effectiveness and efficiency of this approach.

weaknesses:
W1: In Figure 5, the words should also be accompanied by English translations.

W2: For WebSearcher, how does the LLM select the most valuable pages from all the retrieved web content? More details should be provided. Additionally, regarding answer generation, the statement, "After reading these results, the LLM generates a response to answer the original question based on the search results," requires further elaboration, such as information on input design or specific prompt construction.

W3: For the open-set evaluation, five experts are chosen. The author should provide more details, including whether these experts had prior exposure to the answers generated by MindSearch. Furthermore, examples should be included to intuitively demonstrate the differences between the responses generated by MindSearch, ChatGPT-Web, and Perplexity.ai.

W4: The author could provide information on token consumption to help the community manage the budget when using MindSearch in their projects.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730977322756

thread_type:
official_review

<thread_object>
creator: reviewer_8bqd

<content>
summary:
The paper presents MindSearch, a multi-agent system for complex tasks that uses large language models (LLMs) and search engines for complex web information-seeking tasks. MindSearch addresses complex queries by decomposing them and retrieving information hierarchically, modeling the process as an iterative graph construction to enhance precision and recall. By distributing tasks across specialized agents, the framework manages complex and extended contexts effectively. The authors show that experimental results using GPT-4o and InternLM2.5-7B MindSearch outperforms benchmarks like ChatGPT-Web and Perplexity.ai, with human evaluators preferring its responses.

strengths:
1. The problem is both interesting and important. Multi-agent systems for complex QA tasks that are robust and effective 

2. Easy to follow and the methods are simple and well explained.

3. Experiments that include inference cost analysis is well considered.

weaknesses:
1) The work fails to cite and compare to other relevant baselines. For complex QA tasks like HotpotQA or MusiqueQA self-ask[1] with search is a relevant baseline. Similarly Searchain[2] is particularly relevant as it also forms a global reasoning chain or graph where the query is decomposed into subquestions that comprise the nodes of the chain and this planning is similar in philosophy to Mindsearch. I think Assistantbench[3] released in July 2024 is also very relevant and useful to evaluate on. The method SeeplanAct proposed in the paper would serve as a strong baseline. SeeAct[4] is also a relevant baseline. While the authors have cited the same they have not compared to this approach. Other RAG baselines in AssistantBench are also relevant.

2) Some claims are unsupported. For instance the claim made in abstract and section 2.3 regarding the utility of Mindsearch : “Mindsearch performs in 3 minutes tasks worth 3 hours of human effort” has no related evidence cited in the paper. Was there any qualitative evaluation on the benchmark where several human subjects were involved in performing the task with corresponding measurement of time taken ? to compare to mindsearch ?.

3) The work also misses on some important ablations. What happens when Webplanner and code style interaction is not employed ? Is query decomposition required for all queries in web-searcher ? There is also a lack of qualitative analysis of failure scenarios. What happens when response at one node of the chain is wrong ? Does it result in cascading failures. Is there ayn mechanism for the Webplanner to detect such mistakes with feedback from websearcher ? The current approach is a simple tool use based approach which has been well explored in existing WebAgent based works. The additional analysis and error handling mentioned above may help strengthen and understand the core contributions of MindSearch

[1] Measuring and Narrowing the Compositionality Gap in Language Models, Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, Mike Lewis

[2] Search-in-the-Chain: Interactively Enhancing Large Language Models with Search for Knowledge-intensive Tasks, Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, Tat-Seng Chua [3] AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?, Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, Jonathan Berant [4] GPT-4V(ision) is a Generalist Web Agent, if Grounded, Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, Yu Su

questions:
1. What happens when Webplanner and code style interaction is not employed ? Is query decomposition required for all queries in web-searcher ? There is also a lack of qualitative analysis of failure scenarios. What happens when response at one node of the chain is wrong ? Does it result in cascading failures. Is there ayn mechanism for the Webplanner to detect such mistakes with feedback from websearcher ? 

2. Was there any qualitative evaluation on the benchmark where several human subjects were involved in performing the task with corresponding measurement of time taken ? to compare to mindsearch ?


3. how do you respond to the first point in the weakness 1.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731290531068

thread_type:
official_review

<thread_object>
creator: reviewer_bgrw

<content>
summary:
The paper proposes MindSearch, an LLM-based multi-agent information-seeking framework for complex multi-step information-seeking questions. MindSearch includes a Web Planner which decomposes the user query into atomic sub-questions as nodes in a dynamic graph and progressively extends the graph based on results from the WebSearcher. MindSearch considerably improves in response quality in terms of depth and breadth and also improves over the baseline react-based iterative search system.

strengths:
1)	The paper demonstrates considerably better output responses for MindSearch, compared to proprietary AI-Search engines like Perplexity Pro and ChatGPT-Web.

2)	MindSearch also works considerably better than the closed-book and ReACT baselines on a variety of multi-hop question-answering datasets. 

3)	Extensive analysis and evaluation provided in terms of the prompting strategy for WebPlanner along with using a graph-based methodology vs JSON-based and code-based.

weaknesses:
1)	While the paper only evaluates for final response quality, it does not consider the attribution quality of the generated response. Popular AI search engines like Perplexity.AI and ChatGPT-web also provide citations as part of the generated output. The authors do not discuss whether MindSearch provides any kind of attribution, and if yes, what does the citation quality look like (based on automatic evaluations like ALCE [1])

2)	No analysis was provided with regard to the dynamic graph constructed by the WebPlanner. Does the number of hops in the question match the depth of the tree? How often is an incomplete graph created? Also, it would be interesting to see a cost analysis in terms of the number of search queries that MindSearch generates, in comparison to the baselines (ReACT specifically)

questions:
1)	The discussion in line 315 is a bit confusing. The authors say “MindSearch does not yield better performance in terms of facticity, but as per Figure 4 in the paper, factuality of MindSearch is preferred 70% of the time.

2)	Please consider showing the example in Figure 5 in English.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a novel framework that combines large language models (LLMs) with a multi-agent system for complex information-seeking tasks. The authors introduce two main components: WebPlanner, which decomposes user queries into a graph of sub-questions, and WebSearcher, which retrieves information hierarchically. The multi-agent design allows for parallel processing of web information from large-scale sources, and the authors claim significant improvements in response quality over existing baselines, such as ChatGPT-Web and Perplexity.ai. The framework is evaluated on both closed-set and open-set question-answering (QA) tasks, with promising results reported in terms of depth and breadth of answers.

The paper's strengths lie in its timely and interesting problem—leveraging LLMs and search engines to handle complex queries—and the clear presentation of its methodology. Reviewers appreciated the novelty of combining dynamic graph-based query decomposition with multi-agent frameworks. The experimental evaluations are extensive, comparing MindSearch to various baselines and demonstrating its capability to outperform them in specific scenarios. The claims about the efficiency of the approach, such as completing tasks in three minutes that would otherwise require three hours of human effort, are intriguing and align with the goals of efficient AI-powered systems.

However, significant weaknesses and missing elements prevent this paper from reaching the standard required for acceptance. First, the authors fail to provide sufficient evidence for some of their claims, particularly regarding efficiency gains. There is no rigorous or systematic evaluation of the three-hour-to-three-minute claim with human benchmarks, leaving this critical aspect unsubstantiated. Second, while the paper introduces several novel elements, the lack of comparison with key baselines, such as SeePlanAct, AssistantBench, and AutoReAct, undermines the rigor of the evaluation. Reviewers highlighted that the comparisons included (e.g., ReAct, self-Ask, and Searchain) are incomplete and do not sufficiently establish the superiority of MindSearch. Additionally, the lack of discussion around failure cases and cascading errors within the system indicates insufficient robustness in the method. Key ablations, such as analyzing the necessity of WebPlanner and query decomposition, were addressed late and lacked depth. Finally, while the authors claim the framework can generalize to other models, limited validation is provided, and the testing is constrained to a few LLMs, leaving generalization across diverse architectures unconvincing.