MindSearch: Mimicking Human Minds Elicits Deep AI Searcher

## Abstract

Abstract Information seeking and integration is a complex cognitive task that consumes enormous time and effort.
Search engines reshape the way of seeking information but often fail to align with complex human intentions.
Inspired by the remarkable progress of Large Language Models (LLMs), recent works attempt to solve the information-seeking and integration task by combining LLMs and search engines.
However, these methods still obtain unsatisfying performance due to three challenges: (1) complex requests often cannot be accurately and completely retrieved by the search engine once; (2) corresponding information to be integrated is spread over multiple web pages along with massive noise; and (3) a large number of web pages with long contents may quickly exceed the maximum context length of LLMs.
Inspired by the cognitive process when humans solve these problems, we introduce MindSearch ( {CJK} UTF8gbsn思·索) to mimic the human minds in web information seeking and integration, which can be instantiated by a simple yet effective LLM-based multi-agent framework consisting of a WebPlanner and WebSearcher.
The WebPlanner models the human mind of multi-step information seeking as a dynamic graph construction process: it decomposes the user query into atomic sub-questions as nodes in the graph and progressively extends the graph based on the search result from WebSearcher. Tasked with each sub-question, WebSearcher performs hierarchical information retrieval with search engines and collects valuable information for WebPlanner.
The multi-agent design of MindSearch enables the whole framework to seek and integrate information parallelly from larger-scale ( e.g. , more than 300) web pages in 3 minute , which is worth 3 hours of human effort.
Based on either GPT-4o or InternLM2.5-7B models, MindSearch demonstrates significant improvement in the response quality in terms of depth and breadth, on both closed-set and open-set QA problems.
Besides, responses from MindSearch based on InternLM2.5-7B are preferable by humans to ChatGPT-Web (by GPT-4o) and Perplexity.ai applications, which implies that MindSearch with open-source models can already deliver a competitive solution to the proprietary AI search engine.
Code and models are available at https://github.com/InternLM/MindSearch .

## 1 Introduction

Information seeking and integration is a necessary cognitive process before analysis and decision-making in all walks of life, which usually consumes enormous human efforts and time.
The birth of search engines (Brin & Page, ; Berkhin, ) significantly has reshaped and eased the information-seeking process of human society, however, it still suffers in integrating web information based on complex human intentions.
Recently, Large Language Models (LLMs) have showcased remarkable progress in reasoning, language understanding, and information integration across a variety of domains (Achiam et al., ; Team et al., ; Touvron et al., ; Cai et al., ), whereas they struggling to deliver accurate knowledge in responses (Ji et al., ; Gu et al., ).

The complementary advantages of LLMs and search engines highlights a compelling opportunity for their combination, where the reasoning prowess of LLMs can be complemented by the extensive web information accessible via search engines, potentially revolutionizing the solution of web information seeking and integration.
Previous works (Asai et al., ; Chan et al., ) simply treat the information seeking and integration task as a vanilla retrieve-augmented generation (RAG) task (Chen et al., ; Lin et al., ).
Such a formulation, although straightforward, often results in sub-optimal performance due to a superficial engagement with the depth and complexity of web-based information retrieval, facing three major challenges for more complex user queries:

(1) Real-world problems often require in-depth analysis and proper decomposition of the question before retrieving the related information, which cannot be done by retrieving web pages at once.
(2) The overwhelming volume of searched web pages and massive information noise pose great challenges for LLMs for efficient information integration.
(3) The rapid proliferation of web search content can quickly exceed the maximum context length of LLMs, which further decreases the information integration performance.

Inspired by how human experts solve real-world problems, we propose MindSearch {CJK}UTF8gbsn思·索(^1^11The Chinese name ’{CJK}UTF8gbsn思·索‘ means thinking as human and exploring by searching), a simple yet effective LLM-based multi-agent framework, which consists of a WebPlanner (mimic human minds for problem reasoning) and multiple WebSearcher (manage the information seeking process).
Given a user query, the WebPlanner first decomposes the query into multiple atomic sub-questions that can be parallelly solved and dispatches them to the respective WebSearcher. To further enhance the reasoning ability, WebPlanner models the complex problem-solving process as an iterative graph construction: by predefining a list of standard code interfaces related to the construction of the topological mind graph, WebPlanner is able to progressively decompose the question into sequential/parallel sub-problems by adding nodes/edges in the graph via Python code generation.
Meanwhile, the WebSearcher, tasked with each sub-problem, employs a hierarchical retrieval process to extract valuable data for LLMs, which significantly improves the information aggregation efficiency facing massive search pages.
By distributing different aspects of the reasoning and retrieval process to specialized agents, MindSearch effectively reduces the load on each single agent, facilitating a more robust handling of long contexts. It seamlessly bridges the gap between the raw data retrieval capabilities of search engines and the context-understanding power of LLMs.

To validate the effectiveness of MindSearch, we conducted extensive evaluations on both closed-set and open-set question-answering (QA) problems using GPT-4o and InternLM2.5-7B-Chat models. The experimental results demonstrate a substantial improvement in response quality, both in the dimensions of depth and breadth. Moreover, comparative analysis shows that the responses of MindSearch are more preferred by human evaluators over those from existing applications like ChatGPT-Web (based on GPT-4o) and Perplexity Pro. These findings suggest that MindSearch with open-source LLMs can offer a highly competitive solution for AI-driven search engines.

## 2 MindSearch

To effectively synergize the
web information retrieval capabilities of search engines and the reasoning and information integration capability of LLMs, MindSearch consists of a WebPlanner and a group of WebSearchers (Fig. 1). WebPlanner first decomposes the user question into sequential or parallel search tasks via reasoning on the graph and determines the next step based on the search feedback (Sec. 2.1). WebSearcher is tasked with the query and performs hierarchical information retrieval on the Internet to answer sub-questions (Sec. 2.2). We also discuss the context management within the scope of the multi-agent design in Sec. 2.3.

Figure: Figure 2: A concrete example of how WebPlanner addresses the question step by step via planning as coding. During each turn, WebPlanner outputs a series of thoughts along with the generated code. The code will be executed and yield the search results to the planner. At the last turn, the WebPlanner directly provides the final response without any code generation.
Refer to caption: extracted/5762064/figs/planner_demo.png

### 2.1 WebPlanner: Planning via Graph Construction

The WebPlanner functions as a high-level planner, orchestrating the reasoning steps and coordinating other agents. However, we observed that merely prompting the LLM to plan the entire data workflow architecture does not yield satisfactory performance. Specifically, current LLMs struggle with decomposing complex questions and understanding their topological relationships, leading to coarse-grained search queries. This approach underutilizes the potential of LLMs to serve as intermediaries between humans and search engines, transforming human intentions into step-by-step search tasks and delivering accurate responses.

To enhance the capability of LLM in addressing complex questions, we model the problem-solving process as a directed acyclic graph (DAG). Given a user question $Q$, the solution trajectory is represented as $G(Q)=\langle V,E\rangle$, where $V$ is a set of nodes $v$, each representing an independent web search, including an auxiliary START node (the initial question) and an END node (the final answer). $E$ represents directed edges indicating the reasoning topological relationships between nodes (search contents). This DAG formalism captures the complexity of finding the optimal execution path, providing a more formal and intuitive representation for LLMs.

Leveraging the superior performance of current LLMs on code tasks (Guo et al., ; Roziere et al., ), we explicitly prompt the model to interact with the graph through code writing. To achieve this, we predefined atomic code functions to add nodes or edges to the graph (Step 1 and 2 in Figure 2). At each turn, the LLM first reads the entire dialogue, including previously generated code and web search results, then outputs thoughts and new code for reasoning on the mind graph, which is executed with a Python interpreter. During execution, once a node is added to the reasoning graph, it invokes a WebSearcher to execute the search process and summarize the information. Since the newly added nodes are only dependent on nodes generated in previous steps, we can parallel them to achieve a much faster information aggregation speed. When all information is collected, the planner produces the final response by adding the end node (Step 3 in Figure 2).

By integrating with the Python interpreter, WebPlanner interacts with the graph through unified code actions, dynamically constructing the reasoning path. This ”code as planning” process enables the LLM to fully leverage its superior code generation ability, benefiting control and data flow in long-context scenarios and leading to better performance in solving complex problems.

### 2.2 WebSearcher: Web Browsing with Hierarchical Retrieval

WebSearcher acts as a sophisticated RAG (Retrieve-and-Generate) agent with internet access, summarizing valuable responses based on search results (Figure 3). Due to the massive content available on the web, it is challenging for LLMs to process all related pages within a limited context length (e.g. 8K tokens). To address this, we employ a straightforward coarse-to-fine selection strategy. Initially, the LLM generates several similar queries based on the assigned questions from the WebPlanner to broaden the search content and thus improve the recall of relevant information. These queries are then executed through various search APIs, such as Google, Bing, and DuckDuckGo, which return key contents including web URLs, titles, and summaries. The search results are automatically merged based on the web URLs, and the LLM is prompted to select the most valuable pages for detailed reading. The full content of the selected web URLs is then added to the input of LLM. After reading these results, the LLM generates a response to answer the original question based on the search results. This hierarchical retrieval approach significantly reduces the difficulty of navigating massive web pages and allows to efficiently extract highly relevant information with in-depth details.

Figure: Figure 3: A detailed working pipeline of WebSearcher. It comprises at most 4 steps: query rewrite, search content aggregation, detailed page selection, and final summarization.
Refer to caption: extracted/5762064/figs/searcher.png

### 2.3 LLM Context Management in MindSearch

MindSearch provides a simple multi-agent solution to complex information seeking and integration with search engines. Such a paradigm also naturally enables long-context management among different agents, which improves the overall efficiency of the framework, especially under circumstances that require the model to quickly read plenty of web pages.
Since the WebPlanner distributes the search tasks into separate search agents and only relies on the searched results from WebSearcher, WebPlanner can purely focus on the decomposition and analysis of the user question without being distracted by the over-length web search results.
Meanwhile, each WebSearcher only needs to search contents for its tasked sub-query, without distraction from other contents.
Thanks to the explicit role distribution, MindSearch greatly reduces context computation during the whole process, delivering an efficient context management solution to long-context tasks for LLM. Such a multi-agent framework also provides a straightforward and simple long-context task construction pipeline for training single LLMs, which is also observed in (Team, ).
Eventually, MindSearch collects and integrates related information from more than 300 pages in less than 3 minute, which could take human experts about 3 hours to finish a similar cognitive workload.

Due to the explicit context state transfer across multiple agents, we need to carefully handle the context during the whole workflow. We empirically find simply focusing the decomposed query from the Planner may lose useful information during the information collection phase due to the local receptive field inside the search agent. How to effectively handle the context between multiple agents is non-trivial. We find that the constructed topological relations through the directed graph edges help us easily handle the context across different agents. More specifically, we simply prefix the response from its father node as well as the root node when executing each search agent. Therefore, each WebSearcher can effectively focus on its sub-task without losing the previous related context as well as the final goal.

## 3 Experiments

We evaluate MindSearch on two primary categories of Question Answering (QA) tasks: closed-set QA and open-set QA, which reflects both the subjective and objective judgment of MindSearch. For a fair comparison, all models only have access to the Internet through BING search API, and no extra reference sources are considered.

### 3.1 Open-Set QA

#### 3.1.1 Implementation Details

To better gauge the utility and search performance, we carefully curate 100 real-world human queries and collect responses from MindSearch (InternLM2.5-7b-chat (Cai et al., )), Perplexity.ai (its Pro version) , and ChatGPT with search plugin Achiam et al. (). We ask five human experts to manually select their preferred responses, in terms of the following three aspects:

- •
Depth: Depth refers to the thoroughness and profundity of an answer. A response with depth provides detailed information and delves into the intricacies of a question.
- •
Breadth: Breadth pertains to the scope and diversity covered by an answer. A response with breadth touches on various aspects of the question or multiple related fields, offering different perspectives or solutions.
- •
Factuality: Factuality is the degree to which an answer is accurate and fact-based. It should be grounded in reliable data and information, avoiding errors or misleading content, and ensuring the truthfulness and credibility of the information provided.

The final results are determined based on major votes. During the evaluation, the correspondence between the response and its method is invisible to the evaluators to guarantee fairness.

Figure: Figure 4: Subjective evaluation results judged by human experts on open-set QA questions. MindSearch outperforms ChatGPT-Web and Perplexity.ai Pro by a large margin in terms of depth, breadth, and facticity.
Refer to caption: extracted/5762064/figs/openset.png

#### 3.1.2 Results and Analysis

The evaluation results are depicted in Figure 4 and we also provide quantitative results in Figure 5. From Figure 4, we can observe an absolute improvement in terms of the depth and breadth of the model response, which validates the superiority of our proposed WebPlanner. By integrating code into the DAG construction phase, LLM is able to progressively decompose the complex problem into executable queries while balancing the tradeoff between time efficiency and the exploration of the search space. Besides, MindSearch goes through more fine-grained search topics about the question, therefore providing more compact and detailed responses compared to other models. However, MindSearch does not yield better performance in terms of facticity. We suspect that more detailed search results may distract the concentration of the model on the initial problem, especially when LLM holds incomplete long-context capability. Therefore, a natural future work of MindSearch is to alleviate the hallucination issues during the web browsing process.

Figure: Figure 5: Solution trajectory comparison between MindSearch and Perplexity.ai (Pro) on the same question. MindSearch provides more detailed and proper responses thanks to its fine-grained searches.
Refer to caption: extracted/5762064/figs/openset_demo.png

**Table 1: Performance comparison on various closed-set QA tasks. We select two representative LLMs: GPT-4o (close-sourced) and InternLM2.5-7b-chat (open-sourced).**
| Model | Bamboogle | Musique | HotpotQA | AVG |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 2-hop | 3-hop | 4-hop | Easy | Medium | Hard |  |  |  |
| Closed-Souced LLM (GPT-4o) |  |  |  |  |  |  |  |  |
| w/o Search Engine | 70.4 | 54.0 | 22.0 | 20.0 | 73.0 | 69.0 | 66.0 | 53.5 |
| ReAct Search | 75.2 | 48.0 | 25.0 | 13.3 | 81.0 | 73.0 | 70.0 | 55.1 |
| MindSearch | 76.8 | 60.0 | 35.0 | 14.6 | 80.0 | 74.0 | 78.0 | 59.8 |
| Open-Sourced LLM (InternLM2.5-7b-chat) |  |  |  |  |  |  |  |  |
| w/o Search Engine | 34.0 | 28.0 | 10.0 | 17.3 | 47.0 | 26.0 | 40.0 | 28.9 |
| ReAct Search | 55.2 | 38.0 | 17.0 | 16.0 | 69.0 | 56.0 | 49.0 | 42.9 |
| MindSearch | 67.8 | 46.0 | 20.0 | 18.6 | 69.0 | 66.0 | 57.0 | 49.2 |

In addition to quantitative results, we also provide a qualitative response comparison between Perplexity.ai (Pro) and MindSearch to deliver an intuitive understanding of their performance. From Figure 5, we can observe that MindSearch yields more concrete and detailed responses. We empirically find that our better responses can be attributed to the proper planning search paths compared to Perplexity.ai, which also indicates that how to decompose the human intention is the key step to the final problem.

### 3.2 Closed-Set QA

#### 3.2.1 Implementation Details

We extensively evaluate our approach on a wide range of closed-set QA tasks, including Bamboogle (Press et al., ), Musique (Trivedi et al., ), and HotpotQA (Yang et al., ). To further validate the generalization of our approach, we select both closed-source LLM (GPT-4o) and open-source LLM (InternLM2.5-7b-chat) as our LLM backend. Since our approach adopts a zero-shot experimental setting, we utilize a subjective LLM evaluator (GPT4-o) to gauge the correctness of HotpotQA.

#### 3.2.2 Results and Analysis

In Table 1, we compare our approach with two straight-forward baselines: raw LLM without search engines (w/o Search Engine), and simply treating search engines as an external tool and adopting a ReAct-style interaction (ReAct Search). We can conclude that MindSearch significantly outperforms its vanilla baselines by a large margin, validating the effectiveness of the proposed method. These advantages are amplified when transferring from closed-sourced LLMs to open-sourced LLMs, which further proves that MindSeach provides a simple approach to enhance weak LLMs with broader knowledge and alleviate hallucination issues.

## 4 Related Work

### 4.1 Tool Utilization with LLM

The Tool Learning framework empowers LLMs to seamlessly integrate with a variety of tools (Qin et al., ; Hao et al., ; Zhuang et al., ; Chen et al., ), such as search engines (Chan et al., ), databases (Parisi et al., ), and APIs (Li et al., ; Patil et al., ), offering dynamic solutions to complex problems. This integration is not only beneficial for enhancing the interpretability and trustworthiness of LLMs but also for improving their robustness and adaptability across diverse tasks, including reducing hallucinations (Ji et al., ), code generation (Gou et al., ), and question answering (Chen et al., ). Recent research has focused on enhancing the tool integration component of Tool Learning systems. Works such as (Huang et al., ; Shen et al., ; Schick et al., ) have concentrated on improving the retrieval mechanisms, ensuring that LLMs can access the most pertinent tools for a given task. Other studies, like (Qian et al., ; Yuan et al., ), aim at refining the LLMs’ ability to effectively utilize the retrieved information, optimizing the reading and comprehension processes within the framework.

### 4.2 RAG with LLM

RAG demonstrates significant advantages in addressing knowledge-intensive problems, especially in open-domain scenarios with the integration of search engines (Chen et al., ; Li et al., ). RAG allows LLMs to integrate with the retriever, providing timely information and offering effective solutions. Moreover, RAG is also applied in various tasks such as reducing hallucinations (Shuster et al., ; Gu et al., ), code generation (Zhou et al., ), and question answering (Lewis et al., ). Recently, some work (Karpukhin et al., ; Xiong et al., ; Qu et al., ) focuses on enhancing the retrieval component of RAG systems, while others (Izacard & Grave, ; Borgeaud et al., ; Yu et al., ; Lei et al., ) enhances the language model’s ability as a reader to optimize the framework.

With the advancement of LLM capabilities, some researchers have begun to reoptimize frameworks and redesign methodologies for model training. SAIL (Luo et al., ) trains LLM to be more focused on credible and informative search results. Self-RAG (Asai et al., ) enables LMMs to independently fetch, introspect, and augment their text generation capabilities. RQ-RAG (Chan et al., ) enhances query formulation by learning to refine queries through an iterative process.
Our work integrates web search capabilities into LLMs, enhancing response quality by retrieving valuable information from the Internet.

### 4.3 Web Agents

Web automation agents have evolved from question-answering tools to sophisticated systems capable of complex web interactions. Early models like WebGPT (Nakano et al., ) and WebGLM (Liu et al., ) primarily addressed QA tasks, while recent advancements have shifted towards more dynamic operations (Yao et al., ; He et al., ). MindAct (Deng et al., ) and WebAgent (Gur et al., ) represent this progression, with the latter showing exceptional web navigation despite deployment challenges due to its size. AutoWebGLM (Lai et al., ) offers a practical alternative with robust capabilities and a more compact model size. The incorporation of reinforcement learning (Bai et al., ) and behavior cloning techniques (Zheng et al., ; Patel et al., ) paves the way for even more autonomous and efficient web automation, moving the field towards scalable and versatile solutions for real-world applications.
This paper mainly focuses more on the web information-seeking and integration task with search engines instead of web browsing, and solves the main challenges with a multi-agent framework.

## 5 Conclusion

This paper introduces MindSearch, a novel LLM-based multi-agent framework for complex web information-seeking and integration tasks, by more comprehensively leveraging the strengths of both search engines and LLMs.
MindSearch conducts effective and sufficient decomposition of complex queries followed by hierarchical information retrieval to improve the precision and recall of the retrieved relevant web information, by modeling the problem-solving process as an iterative graph construction.
The multi-agent design distributes the cognitive load among specialized agents, facilitating robust handling of complex and lengthy contexts.
Extensive evaluations on closed-set and open-set QA problems using GPT-4o and InternLM2.5-7B models demonstrated significant advantages in the response quality of MindSearch.
The results that human evaluators preferred the responses from MindSearch over those from ChatGPT-Web and Perplexity.ai indicate its competitive edge in AI-driven search solutions.
We wish this work pave the way for future research on multi-agent framework for solving human-level complex cognitive tasks.

## 6 Acknowledgement

We would like to express our sincere gratitude to Jiaye Ge for her outstanding technical insights, product design skills, and coordination abilities. We also thank Zhongying Tu, Ying Zhao, Fang Fang, and Yiting Wang for their efficient execution in developing the project demo. Our gratitude extends to Xingyuan Liu, Shuaike Li, Zike Pan, Weijia Song, and Yuzhe Gu for their efforts in project suggestion.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1)	The discussion in line 315 is a bit confusing. The authors say “MindSearch does not yield better performance in terms of facticity, but as per Figure 4 in the paper, factuality of MindSearch is preferred 70% of the time.

2)	Please consider showing the example in Figure 5 in English.
soundness: 3
strengths: 1)	The paper demonstrates considerably better output responses for MindSearch, compared to proprietary AI-Search engines like Perplexity Pro and ChatGPT-Web.

2)	MindSearch also works considerably better than the closed-book and ReACT baselines on a variety of multi-hop question-answering datasets. 

3)	Extensive analysis and evaluation provided in terms of the prompting strategy for WebPlanner along with using a graph-based methodology vs JSON-based and code-based.
summary: The paper proposes MindSearch, an LLM-based multi-agent information-seeking framework for complex multi-step information-seeking questions. MindSearch includes a Web Planner which decomposes the user query into atomic sub-questions as nodes in a dynamic graph and progressively extends the graph based on results from the WebSearcher. MindSearch considerably improves in response quality in terms of depth and breadth and also improves over the baseline react-based iterative search system.
weaknesses: 1)	While the paper only evaluates for final response quality, it does not consider the attribution quality of the generated response. Popular AI search engines like Perplexity.AI and ChatGPT-web also provide citations as part of the generated output. The authors do not discuss whether MindSearch provides any kind of attribution, and if yes, what does the citation quality look like (based on automatic evaluations like ALCE [1])

2)	No analysis was provided with regard to the dynamic graph constructed by the WebPlanner. Does the number of hops in the question match the depth of the tree? How often is an incomplete graph created? Also, it would be interesting to see a cost analysis in terms of the number of search queries that MindSearch generates, in comparison to the baselines (ReACT specifically)

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. What happens when Webplanner and code style interaction is not employed ? Is query decomposition required for all queries in web-searcher ? There is also a lack of qualitative analysis of failure scenarios. What happens when response at one node of the chain is wrong ? Does it result in cascading failures. Is there ayn mechanism for the Webplanner to detect such mistakes with feedback from websearcher ? 

2. Was there any qualitative evaluation on the benchmark where several human subjects were involved in performing the task with corresponding measurement of time taken ? to compare to mindsearch ?


3. how do you respond to the first point in the weakness 1.
soundness: 2
strengths: 1. The problem is both interesting and important. Multi-agent systems for complex QA tasks that are robust and effective 

2. Easy to follow and the methods are simple and well explained.

3. Experiments that include inference cost analysis is well considered.
summary: The paper presents MindSearch, a multi-agent system for complex tasks that uses large language models (LLMs) and search engines for complex web information-seeking tasks. MindSearch addresses complex queries by decomposing them and retrieving information hierarchically, modeling the process as an iterative graph construction to enhance precision and recall. By distributing tasks across specialized agents, the framework manages complex and extended contexts effectively. The authors show that experimental results using GPT-4o and InternLM2.5-7B MindSearch outperforms benchmarks like ChatGPT-Web and Perplexity.ai, with human evaluators preferring its responses.
weaknesses: 1) The work fails to cite and compare to other relevant baselines. For complex QA tasks like HotpotQA or MusiqueQA self-ask[1] with search is a relevant baseline. Similarly Searchain[2] is particularly relevant as it also forms a global reasoning chain or graph where the query is decomposed into subquestions that comprise the nodes of the chain and this planning is similar in philosophy to Mindsearch. I think Assistantbench[3] released in July 2024 is also very relevant and useful to evaluate on. The method SeeplanAct proposed in the paper would serve as a strong baseline. SeeAct[4] is also a relevant baseline. While the authors have cited the same they have not compared to this approach. Other RAG baselines in AssistantBench are also relevant.

2) Some claims are unsupported. For instance the claim made in abstract and section 2.3 regarding the utility of Mindsearch : “Mindsearch performs in 3 minutes tasks worth 3 hours of human effort” has no related evidence cited in the paper. Was there any qualitative evaluation on the benchmark where several human subjects were involved in performing the task with corresponding measurement of time taken ? to compare to mindsearch ?.

3) The work also misses on some important ablations. What happens when Webplanner and code style interaction is not employed ? Is query decomposition required for all queries in web-searcher ? There is also a lack of qualitative analysis of failure scenarios. What happens when response at one node of the chain is wrong ? Does it result in cascading failures. Is there ayn mechanism for the Webplanner to detect such mistakes with feedback from websearcher ? The current approach is a simple tool use based approach which has been well explored in existing WebAgent based works. The additional analysis and error handling mentioned above may help strengthen and understand the core contributions of MindSearch

[1] Measuring and Narrowing the Compositionality Gap in Language Models, Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, Mike Lewis

[2] Search-in-the-Chain: Interactively Enhancing Large Language Models with Search for Knowledge-intensive Tasks, Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, Tat-Seng Chua [3] AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?, Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, Jonathan Berant [4] GPT-4V(ision) is a Generalist Web Agent, if Grounded, Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, Yu Su

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: See Weaknesses.
soundness: 3
strengths: S1: The writing and framework of this paper are clear and easy to follow.

S2: The method is novel, utilizing the agents WebPlanner and WebSearcher to perform web search tasks.

S3: Extensive experiments are conducted, demonstrating both the effectiveness and efficiency of this approach.
summary: This paper proposes a novel tool agent, MindSearch, which decomposes user queries into atomic sub-questions represented as graph nodes and progressively extends the graph based on the search results from WebSearcher. For each sub-question, WebSearcher performs hierarchical information retrieval using search engines to gather relevant information for WebPlanner. Extensive experiments are conducted, including both open-set and closed-set datasets, and using open-source models alongside close-sourced LLMs, demonstrating the its effectiveness.
weaknesses: W1: In Figure 5, the words should also be accompanied by English translations.

W2: For WebSearcher, how does the LLM select the most valuable pages from all the retrieved web content? More details should be provided. Additionally, regarding answer generation, the statement, "After reading these results, the LLM generates a response to answer the original question based on the search results," requires further elaboration, such as information on input design or specific prompt construction.

W3: For the open-set evaluation, five experts are chosen. The author should provide more details, including whether these experts had prior exposure to the answers generated by MindSearch. Furthermore, examples should be included to intuitively demonstrate the differences between the responses generated by MindSearch, ChatGPT-Web, and Perplexity.ai.

W4: The author could provide information on token consumption to help the community manage the budget when using MindSearch in their projects.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - When constructing the DAG, how does MindSearch automatically create graph nodes? What are some tips for structuring question as graph nodes?
- When large amounts of content are retrieved, how does WebSearcher reduce noise? And, as the rapid growth of web content can easily exceed the maximum context length of the LLM, how does WebSearcher effectively limit content length?
- Additionally, could the experiment include more closed-source and open-source LLMs to further validate the effectiveness of the method?
soundness: 3
strengths: - The paper presents a clear and logical approach to the problem, with a well-organized visual format that is easy to understand and read. 
- This method provides a novel question-answering retrieval method based on directed acyclic graphs, which makes the RAG more reasonable.
summary: This paper introduces a system called MindSearch, designed to emulate human cognitive processes to enhance web information retrieval and integration tasks. By combining large language models (LLMs) with search engines, the system addresses limitations in handling complex queries, fragmented information, and lengthy content through an LLM-based multi-agent framework.
* WebPlanner: Simulates the cognitive process of multi-step information seeking by breaking down user queries into atomic subproblems, represented as nodes in a graph. The graph is then progressively expanded based on search results from WebSearcher.
* WebSearcher: Conducts hierarchical information retrieval for each subproblem, using a search engine to gather valuable information for WebPlanner.

This multi-agent design enables MindSearch to search and integrate information from vast web sources within three minutes, equivalent to saving three hours of manual effort. MindSearch demonstrates significant response quality improvements in both closed-set and open-set QA tasks.
weaknesses: - The method part is not detailed enough to show the technical details. For instance, the design of DAG and the use of DAG is not clear.
- Few baseline methods from the same category are included, and many RAG-based question-answering approaches are left unexamined, such as ChatKBQA, AutoReAct, etc. 
- The backbone was only tested on GPT-4 (close sourced) and InternLM2.5 (open sourced). Under this setting, it is hard to tell if the MindSearch will work for all (at least most) LLMs.

Meta-review: The paper proposes a novel framework that combines large language models (LLMs) with a multi-agent system for complex information-seeking tasks. The authors introduce two main components: WebPlanner, which decomposes user queries into a graph of sub-questions, and WebSearcher, which retrieves information hierarchically. The multi-agent design allows for parallel processing of web information from large-scale sources, and the authors claim significant improvements in response quality over existing baselines, such as ChatGPT-Web and Perplexity.ai. The framework is evaluated on both closed-set and open-set question-answering (QA) tasks, with promising results reported in terms of depth and breadth of answers.

The paper's strengths lie in its timely and interesting problem—leveraging LLMs and search engines to handle complex queries—and the clear presentation of its methodology. Reviewers appreciated the novelty of combining dynamic graph-based query decomposition with multi-agent frameworks. The experimental evaluations are extensive, comparing MindSearch to various baselines and demonstrating its capability to outperform them in specific scenarios. The claims about the efficiency of the approach, such as completing tasks in three minutes that would otherwise require three hours of human effort, are intriguing and align with the goals of efficient AI-powered systems.

However, significant weaknesses and missing elements prevent this paper from reaching the standard required for acceptance. First, the authors fail to provide sufficient evidence for some of their claims, particularly regarding efficiency gains. There is no rigorous or systematic evaluation of the three-hour-to-three-minute claim with human benchmarks, leaving this critical aspect unsubstantiated. Second, while the paper introduces several novel elements, the lack of comparison with key baselines, such as SeePlanAct, AssistantBench, and AutoReAct, undermines the rigor of the evaluation. Reviewers highlighted that the comparisons included (e.g., ReAct, self-Ask, and Searchain) are incomplete and do not sufficiently establish the superiority of MindSearch. Additionally, the lack of discussion around failure cases and cascading errors within the system indicates insufficient robustness in the method. Key ablations, such as analyzing the necessity of WebPlanner and query decomposition, were addressed late and lacked depth. Finally, while the authors claim the framework can generalize to other models, limited validation is provided, and the testing is constrained to a few LLMs, leaving generalization across diverse architectures unconvincing.

Decision: Accept (Poster)