MindSearch: Mimicking Human Minds Elicits Deep AI Searcher
Authors: Zehui Chen
Kuikun Liu
Qiuchen Wang
Jiangning Liu
Wenwei Zhang
Kai Chen
Feng Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: language model
search engine
multi-agent system
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Information seeking and integration is a complex cognitive task that consumes enormous time and effort. Inspired by the remarkable progress of Large Language Models, recent works attempt to solve this task by combining LLMs and search engines. However, these methods still obtain unsatisfying performance due to three challenges: (1) complex requests often cannot be accurately and completely retrieved by the search engine once (2) corresponding information to be integrated is spread over multiple web pages along with massive noise, and (3) a large number of web pages with long contents may quickly exceed the maximum context length of LLMs. Inspired by the cognitive process when humans solve these problems, we introduce MindSearch to mimic the human minds in web information seeking and integration, which can be instantiated by a simple yet effective LLM-based multi-agent framework. The WebPlanner models the human mind of multi-step information seeking as a dynamic graph construction process: it decomposes the user query into atomic sub-questions as nodes in the graph and progressively extends the graph based on the search result from WebSearcher. Tasked with each sub-question, WebSearcher performs hierarchical information retrieval with search engines and collects valuable information for WebPlanner. The multi-agent design of MindSearch enables the whole framework to seek and integrate information parallelly from larger-scale (e.g., more than 300) web pages in 3 minutes, which is worth 3 hours of human effort. MindSearch demonstrates significant improvement in the response quality in terms of depth and breadth, on both close-set and open-set QA problems. Besides, responses from MindSearch based on InternLM2.5-7B are preferable by humans to ChatGPT-Web and Perplexity.ai applications, which implies that MindSearch can already deliver a competitive solution to the proprietary AI search engine.

Published as a conference paper at ICLR 2025
MINDSEARCH: MIMICKING HUMAN MINDS ELICITS DEEP AI SEARCHER
Zehui Chen"l, Kuikun Liu"?, Qiuchen Wang', Jiangning Liu?, Wenwei Zhang?
Kai Chen²t, Feng Zhao If
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, USTC
Shanghai AI Laboratory
[lovesnow, www.hhnwwnggeeaaaae email.ustc. (liukuikun, Aujiangnin ng, chenka i (arnkaai@@ppplaaa @@pjlab. fzhao95 (esooo95eeuttcuu....
ustc. edu. cn
ABSTRACT
Information seeking and integration complex cognitive task that consumes enormous time and effort. Search engines reshape the way of seeking information but often fail align with complex human intentions. Inspired by the remarkable progress of Large Language Models (LLMs), recent works attempt to solve the information-seeking and integration task by combining LLMs and search engines. However, these methods still obtain unsatisfying performance due to three chal- lenges: (1) complex requests often cannot be accurately and completely retrieved by the search engine once; (2) corresponding information to be integrated is spread over multiple web pages along with massive noise; and (3) large number web pages with long contents may quickly exceed the maximum context length of LLMs. Inspired by the cognitive process when humans solve these problems, we introduce MindSearch ( ()) to mimic the human minds in web informa- tion seeking and integration, which can be instantiated by simple yet effective LLM-based multi-agent framework consisting of WebPlanner and WebSearcher. The WebPlanner models the human mind of multi-step information seeking as a dynamic graph construction process: it decomposes the user query into atomic sub-questions as nodes in the graph and progressively extends the graph based on the search result from WebSearcher. Tasked with each sub-question, WebSearcher performs hierarchical information retrieval with search engines and collects valu- able information for WebPlanner. The multi-agent design of MindSearch enables the whole framework to seek and integrate information parallelly from larger-scale (e.g., more than 300) web pages in 3 minutes, which is worth 3 hours of human effort. Based either GPT-4o
M22.5-BB models, MindSearch demon- strates significant improvement in the response quality in terms of depth and breadth, on both closed-set and open-set QA problems. Besides, responses from MindSearch based on InternL.M2.5-7B are preferable by humans to ChatGPT Web (by GPT-4o) and Perplexity.aa ai applications, which implies that MindSearch with open-source models can already deliver competitive solution to the proprietary Al search engine. Code available at http://gfuuub.com//mmmm/////////
INTRODUCTION
Information secking and integration is a necessary cognitive process before analysis and decision- making in all walks of life. which usually consumes enormous human efforts and time. The birth
of search engines (Brin & Page, 1998; Berkhin, 2005) significantly has reshaped and eased the informatton-seeking process of human society, however, still suffers in integrating web informa- tion based on complex human intentions. Recently, Large Language Models (LLMs) have show- cased remarkable progress in reasoning, language understanding, and information integration across
"Equal Contribution
1 "ooresponndig authors.

---

Published as a conference paper at ICLR 2025
Question: Could you introduce one video game produced in China with detaills?
Web Planner
Web Searcher (video samee
need to search some video games China.
Rewrite Search Query
Q video games in China
In''t diagph
Add node and edge
Page-1 apic Chinaee Video Game That Ares' GGmahis Impact...
RRADI
Summury: But desphte theur limiattons, developen
fure
(qucstion)
gomee Chine)
geruinely stepped their gime,
Page-2 Chroese stiteo grwee expand globel reach Chiladdrily
Search API
Summary: Acconting to report recenity relensed by socal media platticm Suuterly
Genshin Impost.
Black Myths Wahang, Naraks: @lodepoint
Conchin imgant,..
Page The tap five gasning campaigns that busaed Clana''s Ges 2
2021
Based on the search results, need to find different ospects of information on about Genshin.
Sumnury: The lassh the langus of Legenis
game
tarne
Prev Graph
Add mulfiple nodes and edged
Pago Selection
Page-1, Page-6
(bankgrsund)
Node3 (roles and relationships)
Python Crumlar
Up
Page-1
Page-
Reot
Nedel
Resh
Nodel
Nade- (herners)
Impoct Myths: Wühhnng Bladepoint
Summaar
Response from #11, #2.... AN
Parallel Execution
Genchin Impoct HM Background
#1 Web Searcher @bw.hddgoosdo
#2
#8d Roles and Relationshipe
Final Responde
Web Searcher (lChan are roles infatonstoo...
Web Searcher Off hormn acGomstoo
Geasboo)
Figure 1: The overall framework of MindSearch. It consists of two main ingredients: WebPlanner and WebSearcher. WebPlanner acts as a high-level planner, orchestrating the reasoning steps and multiple WebSearchers. WebSearcher conducts fine-grained web searches and summarizes valuable information back to the planner, formalizing a simple yet effective multi-agent framework.
a variety of domains (Achiam et al., 2023; Team et al., 2024; Touvron et al., 2023; Cai et al., 2024), whereas they struggling to deliver accurate knowledge in responses (Ji et al., 2023; Gu et al., 2024).
The complementary advantages of LLMS and search engines highlights a compelling opportunity for their combination, where the reasoning prowess of L.MMs can be complemented by the exten- sive web information accessible via search engines, potentially revolutionizing the solution of web information seeking and integration. Previous works (Asai al., 2023; Chan et al., 2024) simply treat the information seeking and integration task as vanilla retrieve-augmented generation (RAG) task (Chen et al., 2017: Lin et al., 2023). Such a formulation, although straightforward, often re- sults in sub-optimal performance due to superficial engagement with the depth and complexity of web-based information retrieval, facing three major challenges for more complex user queries:
(1) Real-world problems often require in-depth analysis and proper decomposition of the question before retrieving the related information, which cannot be done by retrieving web pages once.
(2) The overwhelming volume of searched web pages and massive information noise pose great challenges for LLMs for efficient information integration.
(3) The rapid proliferation of web search content quickly exceed the maximum context length of LLMs. which further decreases the information integration performance.
Inspired by how human experts solve real-world problems, propose MindSearch ! sim- ple yet effective LLM-based multi-agent framework, which consists of a WebPlanner (mimic hu- man minds for problem reasoning) and multiple WebSearchers (manage the information seeking process). Given user query, the WebPlanner first decomposes the query into multiple atomic sub- questions that can be parallelly solved and dispatches them to the respective WebSearcher. To further enhance the reasoning ability, WebPlanner models the complex problem-solving process as an iter- ative graph construction: by predefining list of standard code interfaces related the construction of the topological mind graph, WebPlanner is able to progressively decompose the question into sequential/parallel sub-problems by adding nodes/edges in the graph via Python code generation.
The Chinese name #. means thinking as human and exploring by searching
2

---

Published as a conference paper at ICLR 2025
Meanwhile, the WebSearcher, tasked with each sub-problem, employs hierarchical retrieval pro- cess to extract valuable data for LLMs, which significantly improves the information aggregation efficiency facing massive search pages. By distributing different aspects of the reasoning and trieval process to specialized agents, MindSearch effectively reduces the load on each single agent, facilitating a more robust handling of long contexts. It seamlessly bridges the gap between the raw data retrieval capabilities of search engines and the context-understanding power of LLMs.
To validate the effectiveness of MindSearch, we conducted extensive evaluations on both closed-set and open-set question-answering (QA) problems using GPT-4o and IntemmL.M..-77BBCCM els. The experimental results demonstrate substantial improvement in response quality, both in the dimensions of depth and breadth. Moreover, comparative analysis shows that the responses of MindSearch are more preferred by human evaluators over those from existing applications like ChatGPT-Wer (based on GPT-40) and Perplexity Pro. These findings suggest that MindSearch with open-source LLMSs can offer a highly competitive solution for Al-driven search engines.
mod-
2 MINDSEARCH
To effectively synergize the web information retrieval capabilities of search engines and the reason- ing and information integration capability of LLMs, MindSearch consists of a WebPlanner and group of WebSearchers (Fig. 1). WehPlanner first decomposes the user question into sequential or parallel search tasks via reasoning on the graph and determines the next step based the search feedback (Sec. 2.1). WebSearcher is tasked with the query and performs hicrarchical information retrieval on the Internet to answer sub-questions (Sec. 2.2). We also discuss the context management within the scope of the multi-agent design in Sec. 2.3.
2.1 WEBPLANNER: PLANNING VIA GRAPH CONSTRUCTION
The WebPlanner functions as a high-level planner, orchestrating the reasoning steps and coordinat- ing other agents. However, we observed that merely prompting the LLM to plan the entire data workflow architecture does not yield satisfactory performance. Specifically, current LLMs struggle with decomposing complex questions and understanding their topological relationships, leading to coarse-grained search queries. This approach underutilizes the potential of LLMs to serve as in- termediaries between humans and search engines, transforming human intentions into step-by-step search tasks and delivering accurate responses.
To enhance the capability of LLM in addressing complex questions, we model the problem-solving process directed acyclic graph (DAG). Given user question Q. the solution trajectory is rep- resented as G(Q) = (V. E), where V is a set of nodes v, each representing an independent web search, including auxiliary START node (the initial question) and an END node (the final an- swer). E represents directed edges indicating the reasoning topological relationships between nodes (search contents). This DAG formalism captures the complexity of finding the optimal execution path, providing more formal and intuitive representation for LLMs.
Leveraging the superior performance of current LLMs on code tasks (Guo al., 2024; Roziere al., 2023), we explicitly prompt the model to interact with the graph through code writing. To achieve this, we predefined atomic code functions to add nodes or edges the graph (Step and 2 in Figure 2). At each turn, the LLM first reads the entire dialogue, including previously generated code and web search results, then outputs thoughts and new code for reasoning on the mind graph, which is executed with a Python interpreter. During execution, once node is added to the reasoning graph, invokes a WebSearcher to execute the search process and summarize the information. Since the newly added nodes only dependent on nodes generated in previous steps, achieve a much faster information aggregation speed. When all information is collected, the planner produces the final response by adding the end node (Step in Figure 2).
parallel them to
By integrating with the Python interpreter, WebPlanner interacts with the graph through unified code actions, dynamically constructing the reasoning path. This "code planning" process enables the LLM to fully leverage its superior code generation ability, benefiting control and data flow in long-context scenarios and leading to better performance in solving complex problems.
3

---

Published as a conference paper at ICLR 2025
Question
Summorize the best poper from majer ML. conferences 2023 (NeurIPS, ICMI, ICLR), and provide overseew of the trends severol papuior fields.
WebPlanner
First, need abtain
the Best Paper 2023 najor conferencesi NeurIP5, ICML, and these winning papers defermine the frends future research directions.
the research
these papars. will andlyze research Let's start building the search chart.
Frow logent, agents, sindseorch leport Searhhhhhrraph
Step 1: Groph Node
groph, add, root modeCnadee contonde" Suararize the best paper dwands from major N. conferences in 2023 (NcurIPS, IOL, ICLR), and provide on overvtew of the trends in several, poputar reeld"", nod0 root")
add model
"WeurPP paper", node, contente Wentt the best MeurIPS
28237")
groph, add, modenoded node. nON pager" nose, contonte What 15 the best IDML paper 28237"? groph, now.. mode, ndeew "ICR poper", noste. content> What 15 the best ECL paper in 20237")
add, indge mnttt_nndee root". end nade- "HeurIP5 popen*)
groph, add, edgel sttert..noe Loddd rook"" end, nade= TICM, paper")
groph, add, indgenaaaeeee antert_node "roo"".. end made- "ICU paper")
Code Execution (Pythan Interpreter)
WebSearcher (NeurIPS best popar)
WebSearcher (ICML poper)
WebSeercherr (ICLR best poper)
Environment
WiehSearcher #2 Near IFS best poparstnnlal A General Frumooork for Learning Optimize...
WierShercher ## ICML best popertinnted Leerningg Settigg--tteeeeeeeeeeeee Learning D-Adeptation
WeaSShrr ICLR beat paperstrilvl. Univeraul Few shat Learning of benae Prediction...
WebPlenner
We have cotained the Best Pepars from the 2023 machine learning conferences (NurIPS, ICML, ICLP), next, we will search for and introduce some popular fields besed the contint of theie pipers and understund their devilopment status.
groph, add rude(
"aptimiastioon goriittssssssooooooooooocooo.. deve lopreet",
Step 2: Add Node
developeent status optinization algorithes 292371) content-" are research paper", groph, (startt
upciiiaatiinn. www.ritthonooooooo dovelopeenntt"
groph, add radec
node none= "groph. www.oncuoo..... www.wookkssc.oom www.eooeeeeetttt.me and development status of GW in 20231
What the research
groph, add. (storr...oor nodeb ICLR, 2023 best popen", ond.nnode node> grrph. www.nnneul..nnee www.wrkwwooo.... developeentt")
groph,
nanes nune="text_.. 3d.nodel genenaccoo"" on"..
node. content"" What are the nesearch content and devel opeent status of text to 30 in 2823")
groph, add. wtw.eedeesstrt..oooe ICLR, 2023 ppsper" end_.nooww bt....oo.......el.ll..... wtt.....eel...eeeeettt..e...e L.penenrrtoon"
Execution
Environment
WebPlanner
We have abtained winners Best Pupers for 2023 of three major machine learning confenences (NearIP5, ICML, ICLR) reseerched the development status severci popular fields based contiett these popers, a rešpurtae node to summorize information.
grash. response, (oadle neep- "resporse")
sation elgarithe, devel
and
Add Resporce Node
grash. add esqut start node-" groph, neural notworis deve Lopment" and nude- "ess----esoonee" grash. add, edgel abart, nodo- taxt, to, rodell prodellgeeereeeeeeeeeeeeee coneration". and inde- "esporse"))
WebPlanner (Final Response)
Winners the Best Paper the Major Machine Learning Corferences in 2023 and Their Research Content
A concrete example of how WebPlanner addresses the question step by step via planning as coding. During each turn, WebPlanner outputs a series of thoughts along with the generated code. The code will be executed and yield the search results to the planner. At the last turn, the WebPlanner directly provides the final response without any code generation.
Figure 2:
2.2 WEBSEARCHER: WEB BROWSING WITH HIERARCHICAL RETRIEVAL
WebSearcher acts as a sophisticated RAG -Retriee--nnd-Geneeee agent with internet access, sum- marizing valuable responses based on search results (Figure 3). Due to the massive content available on the web, it is challenging for LLMs to process all related pages within a limited context length (e.g. 8K tokens). To address this, we employ straightforward coarse-to-fine selection strategy. Initially, the LLM generates several similar queries based on the assigned questions from the Web- Planner to broaden the search content and thus improve the recall of relevant information. These queries are then executed through various search APIs, such as Google, Bing, and DuckDuckGo, which return key contents including web URLs, titles, and summaries. The search results are au- tomatically merged based on the web URLs, and the LLM is prompted to select the most valuable pages for detailed reading. The full content of the selected web URLs is then added to the input of L.M.M After reading these results, the LLM generates response answer the original question
4

---

Published as a conference paper at ICLR 2025
based the search results. This hierarchical retrieval approach significantly reduces the difficulty of navigating massive web pages and allows to efficiently extract highly relevant information with in-depth details.
Sub Query1
Merged Results
Page Selection
Question
Sub-Query2
Summary, Titkk URL Title URL
Detailed
Response
Query
Search
Python Crowler
Web Content
Summary
Sub-Query3
API
Figure 3: A detailed working pipeline of WebSearcher. It comprises at most 4 steps: query rewrite, search content aggregation, detailed page selection, and final summarization.
2.3 LLM CONTEXT MANAGEMENT MINDSEARCH
MindSearch provides simple multi-agent solution to complex information seeking and integra- tion with search engines. Such a paradigm also naturally enables long-context management among different agents, which improves the overall efficiency of the framework, especially under cireum- stances that require the model to quickly read plenty of web pages. Since the WebPlanner dis- tributes the search tasks into separate search agents and only relies on the searched results from WebSearcher, WebPlanner can purely focus on the decomposition and analysis of the user question without being distracted by the over-length web search results. Meanwhile, each WebSearcher only needs to search contents for its tasked sub-query. without distraction from other contents. Thanks to the explicit role distribution, MindSearch greatly reduces context computation during the whole process, delivering efficient context management solution to long-context tasks for LLM.. Such a multi-agent framework also provides a straightforward and simple long-context task construction pipeline for training single LLMs, which is also observed in (Team, 2024). Eventually, MindSearch collects and integrates related information from more than 300 pages in less than 3 minute, which could take human experts about 3 hours to finish similar cognitive workload.
Due to the explicit context state transfer across multiple agents, we need to carefully handle the context during the whole workflow. We empirically find simply focusing the decomposed query from the Planner may lose useful information during the information collection phase due to the local receptive field inside the search agent. How effectively handle the context between multiple agents is non-trivial. We find that the constructed topological relations through the directed graph edges help easily handle the context across difffrrent agents. More specifically, we simply prefix the response from its father node as well as the root node when executing each search agent. Therefore, each WebSearcher can effectively focus on its sub-task without losing the previous related context as well as the final goal.
3 EXPERIMENTS
We evaluate MindSearch on two primary categories of Question Answering (QA) tasks: closed-set QA and open-set QA, which reflects both the subjective and objective judgment MindSearch. For fair comparison, all models only have access to the Internet through BING search API, and no extra reference sources are considered.
OPEN-SET QA
3.1.1 IMPLEMENTATION DETAILS
To better gauge the utility and search performance, we carefully curate 100 real-world human queries and collect responses from MindSearch LntermLM..5-7bcche (Cai et al., 2024)), Perplexity.ai (its Pro version), and ChatGPT with search plugin (Achiam et al., 2023). We ask five human experts to manually select their preferred responses, in terms of the following three aspects:
Depth: Depth refers to the thoroughness and profundity of an answer. A response with depth provides detailed information and delves into the intricacies of question.
5

---

Published as a conference paper at ICLR 2025
MindSearch
Perplexity. (Pro)
Chat Web
Depth
73%
24%
3%
Breadth
83%
13%
4%
Factuality
70%
17%
13%
Figure 4: Subjective evaluation results judged by human experts on open-set QA questions. MindSearch outperforms ChatGPT- Web and Perplexity.ai Pro by large margin in terms of depth, breadth, and facticity.
Question:
strongest
season
Perplexityy.al (Pro)
MindSearch
Solving Path
Solving Path
Which archer the strongest Areno of Valor this season?
Which season of Arena of Volar is now?
the strongest archer
in Season 36
Question: List of award-wirning films at major glabal film festivals in the past three years?
Perplexxity.at ai (Pro)
MindSearch
2021
Solving Path
Solving Path
Cannes Film Festival (2024) Aword-winning films
Connes Film Festival award post three years Benlin Festival winners the post three years
2022
Information about major global film festivals
Berlin Festival (2023)
What are the global film Festivals
2023
Award-wwiie weard-witning filled
Venice Film Festival (2023) Award-wiining filius
Verice Film Festival award winners in the past three years
Figure 5: Solution trajectory comparison between MindSearch and Perplexity (Pro) the same question. MindSearch provides more detailed and proper responses thanks to its fine-grained searches.
Breadth pertains to the scope and diversity covered by an answer. A response with breadth touches various aspects of the question multiple related fields, offering different perspectives or solutions.
Breadth:
Factuality: Factuality is the degree to which an answer is accurate and fact-based. It should grounded reliable data and information, avoiding errors or misleading content, and ensuring the truthfulness and credibility of the information provided.
The final results determined based major votes. During the evaluation, the correspondence between the response and its method invisible to the evaluators to guarantee fairness.
3.1.2 RESULTS AND ANALYSIS
The evaluation results depicted in Figure and also provide quantitative results in Figure 5. From Figure 4, we can observe an absolute improvement in terms of the depth and breadth of the model response, which validates the superiority of our proposed WebPlanner. By integrating code into the DAG construction phase, LLM is able to progressively decompose the complex problem into executable queries while balancing the tradeoff between time efficiency and the exploration of the search space. Besides, MindSearch goes through more fine-grained search topics about the question, therefore providing more compact and detailed responses compared to other models. However, MindSearch does not yield much better performance in terms of facticity, compared to breadth (70% vs 83%). We suspect that more detailed search results may distract the concentration of the model on the initial problem, especially when LLM holds incomplete long-context capability. Therefore, natural future work of MindSearch is to alleviate the hallucination issues during the web browsing process.

---

Published as a conference paper at ICLR 2025
Table 1: Performance comparison on various closed-set QA tasks. We select two representative
LLMs: GPT-4o (close-sourced) and IntemLLMM..-7--chhal (open-sourced).
Musique 3-hop
HotpotQA Medium
AVG
Model
Bamboogle
2-hop
4-hop
Easy
Hard
Closed-Souced LLM (GPT-4o)
w/o Search Engine
70.4
54.0
22.0
20.0
73.0
69.0
66.0
53.5
ReAct Search
75.2
48.0
25.0
13.3
81.0
73.0
70.0
55.1
MindSearch
76.8
60.0
35.0
14.6
80.0
74.0
78.0
59.8
Open-Sourced LLM LaternLM22.55 M2.5--b-aaatt
w/o Search Engine
34.0
28.0
10.0
17.3
47.0
26.0
40.0
28.9
ReAct Search
55.2
38.0
17.0
16.0
69.0
56.0
49.0
42,9
MindSearch
67.8
46.0
20.0
18.6
69.0
66.0
57.0
49.2
In addition to quantitative results, we also provide a qualitative response comparison between Per- plexity.ai (Pro) and MindScarch deliver an intuitive understanding of their performance. From Figure 5, we can observe that MindSearch yields more concrete and detailed responses. We empir- ically find that our better responses can be attributed the proper planning search paths compared to Perplexity.ai, which also indicates that how to decompose the human intention is the key step to final problem.
3.2 CLOSED-SET QA
3.2.
IMPLEMENTATION DETAILS
We extensively evaluate our approach on a wide range of closed-set QA tasks, including Bamboogle (Press al., 2022), Musique (Trivedi al., 2022), and HotpotQA (Yang al., 2018). To further validate the generalization our approach, we select both closed-source LLM (GPT-4o) and open- source LLM MtttemLLM..5-7b-chm experimental setting, we utilize a subjective LLM evaluator (GPT4-0) to gauge the correctness of HotpotQA.
our LLM backend. Since approach adopts zero-shot
3.2.2 RESULTS AND ANALYSIS
In Table 1, we compare our approach with two straight-forward baselines: raw LLM without search engines (w/o Search Engine), and simply treating search engines as an external tool and adopt- ing a ReAct-style interaction (ReAct Search). We can conclude that MindSearch significantly out- performs its vanilla baselines by large margin (4.7% on GPT-4o and 6.3% on InternL validating the effectiveness the proposed method. These advantages are amplified when trans- ferring from closed-sources LLMs to open-sourced LLMs, which further proves that MindSearch provides a simple approach to enhance weak LLMs with broader knowledge and alleviate halluci- nation issues. When taking closer look the difficulty level of HotpotQA, observe that most improvements are derived from the hard set, which also indicates that MindSearch are more adept solving complex questions compared to other approaches.
3.3 ABLATION STUDIES
In this section, conduct detailed ablation studies aiming to gain deeper understanding of our approach. Without loss generality, all experiments are conducted with IntemLLM2.5-7b-ccha on HotpotQA if not specified.
3.3. ABLATIONS ON WEBPLANNER
To validate the efficiency of proposed WebPlanner, compare graph-based code planning strategy with its two vanilla baselines, namely ReAct (Yao et al., 2022b), and CodeAct (Wang et al., 2024). For ReAct, prompt the WebPlanner to invoke WebSearcher with classical predefined Re-

---

Published as a conference paper at ICLR 2025
Medium
Hord
2.5-
69.0 (2.4)
(6.4
(7.6
Inference Time Coof
2.-
(9.0
66.3
5
57. (9.6)
(1.0
0.5
26.0 (1.0)
40.0
0.0
Sugle-Ture
ReAct
MineSearch
Single-Tur
ReAct
MindSearch
Single-Tum
ReAct
MineSesrch
Figure 6: Inference cost vs. search performance among various inference strategies on Hot- potQA. The scores above each bar are search performance and inference cost, respectively. We normalize the inference time cost by single-tur.
Table Ablations on the planning strategies adopted in WebPlanner. For a fair compari- son, we keep WebSearcher the same for all entries. denotes no access to web search during inference time.
Table Ablations on the components adopted in WebSearcher. 'MQG' de- notes multi-query generation and aggrega- tion, 'PPC' denotes prefix previous context, and 'PS' denotes the page selection process.
Strategy
Performance
Method
Performance
37.6
Ours w/o MQG
64.0
ReAct
58.0
60.6
CodeAct
61.3
w/o PPC
63.3
Ours
64.0
w/o PS
58.0
Act JSON format, i.e,, Thought: str; Action: WebSea (List [str]) each step. for CodeAct, we wrap WebSearcher as a function and prompt the WebPlanner to use by writing code. For fair comparison, we keep the WebSearcher the same for all entries. The results are shown Table We can observe that integrating code generation provides unified action space and therefore yields better performance compared ReAct, which proves the correctness of our adoption of code as planning. Then, when converting the original function calling process into the graph construction, the performance continues improve, especially the hard set of HotpotQA, which further validates the superiority of our proposed planning as graph construction.
3.3.2 ABLATIONS ON WEBSEARCHER
The search quality of WebSearcher determines the upper bound of the final response from WebPlan- ner. Therefore, how to efficiently aggregate highly related web pages via search engines and provide valuable summary information is of great importance. In this part, multi-query generation and aggregation, prefix previous context, and page selection processes, and report the results in Table 3. Still, we keep the WebPlanner the same for all entries for fair compar- ison. As can be seen, each module contributes to the final performance and culli-query generation has the most influence.
independently remove the
3.4 INFERENCE TIME SCALING VS. SEARCH PERFORMANCE
Properly scaling test-time computation enables LLMs improve their outputs by considerable margin (Snell et al., 2024; Kumar et al., 2024). OpenAl ol (OpenAI, 2024) has recently shown that a detailed chain-of-thoughh can dramatically enhance the reasoning ability of language models on various downstream tasks. In this section, we discuss the relationship between inference time scaling and search performance in MindSearch, which systematically analyzes such a trade-off under the Al search engine domain. Specifically, single-turn search, with only one search call for each question (adopted by most current AI search engines), (2) multi-turn search with ReAct, and (3) multi-tun turtittummm search with MindSearch. The last two patterns allow LLMs to scale up their inference computation with multiple web search interactions (during experiments, we limit the max interaction turn to 10 since observe limited performance
compare three types of search patterns in Figure 6: (1)
8

---

Published as a conference paper at ICLR 2025
gains by enlarging this hyperparameter). Single-tur search achieves a passing grade on the easy level and uses the shortest time, which actually meets the need for a large portion of real-world usage (possibly the reason why most AI search engines adopt this pattern). By enabling linear-scale reasoning through ReAct, we are able to observe that the performance improves at the cost of more inference costs. Compared to ReAct, MindSearch gets more efficient scaling performance with less inference steps, which indicates that MindSearch provides a better scaling strategy for improving the search performance.
4 RELATED WORK
4.1 TOOL UTILIZATION WITH LLM
The Tool Learning framework empowers LLMs to seamlessly integrate with variety of tools (Qin et al., 2023; Hao et al., 2024; Zhuang et al., 2024; Chen et al., 2023), such as search engines (Chan et al., 2024), databases (Parisi et al., 2022), and APIs (Li et al., 2023; Patil et al., 2023), offering dynamic solutions to complex problems. This integration is not only beneficial for enhancing the interpretabilitty and trustworthiness of LLMs but also for improving their robustness and adaptability across diverse tasks, including reducing hallucinations (Ji et al., 2024), code generation (Gou et al., 2023), and question answering (Chen al., 2024). Recent research has focused on enhancing the tool integration component of Tool Learning systems. Works such as (Huang et al., 2023; Shen al., 2023; Schick et al.. 2024) have concentrated on improving the retrieval mechanisms, ensuring that LLMs can access the most pertinent tools for a given task. Other studies, like (Qian et al., 2023; Yuan et al., 2023), aim at refining the LLMs' ability to effectively utilize the retrieved information, optimizing the reading and comprehension processes within the framework.
4.2 RAG WITH LLM
RAG demonstrates significant advantages in addressing gowwledge-intesivve problems, especially in open-domain scenarios with the integration of search engines (Chen et al., 2017; Li et al., 2017).
RAG allows LLMs integrate with the retriever, providing timely information and offering effective solutions. Moreover, RAG is also applied in various tasks such as reducing hallucinations (Shuster et al., 2021: Gu et al., 2024), code generation (Zhou et al., 2022), and question answering (Lewis et al., 2020). Recently, some work (Karpukhin et al., 2020; Xiong al., 2020; Qu et al., 2020) focuses on enhancing the retrieval component of RAG systems, while others (Izacard & Grave, 2020; Borgeaud et al., 2022; Yu al., 2021; Lei al., 2017) enhances the language model's ability as a reader to optimize the framework.
With the advancement of LLM capabilities, some researchers have begun reoptimize frameworks and redesign methodologies for model training (Luo et al., 2023a; Qiao et al., 2024). SAIL (Luo al., 2023b) trains LLM to be more focused credible and informative search results. Self- RAG (Asai et al., 2023) enables LMMs to independently fetch, introspect, and augment their text generation capabilities. RQ-RAG (Chan et al., 2024) enhances query formulation by learning to refine queries through an iterative process. Searchain (Xu et al., 2024) introduces chain-of-query (CoQ) to iteratively refine the reasoning of graph to resolve complex problems. Our work integrates web search capabilities into LLMs, enhancing response quality by retrieving valuable information from the Internet.
4.3 WEB AGENTS
Web automation agents have evolved from question- -anwerinn tools to sophisticated systems capa- ble of complex web interactions. Early models like WebGPT (Nakano et al., 2021) and WebGLM (Liu et al., 2023) primarily addressed QA tasks, while recent advancements have shifted towards more dynamic operations (Yao et al., 2022a; He et al., 2024). MindAct (Deng et al., 2024), We- bAgent (Gur et al., 2023), SeeAct (Zheng et al., 2024), and SeePlanAct (Yoran et al., 2024) represent this progression, with the latter showing exceptional web navigation despite deployment challenges due to its size. Auto WebGLM (Lai et al., 2024) offers a practical alternative with robust capabilities and a more compact model size. The incorporation of reinforcement learning (Bai et al., 2024) and behavior cloning techniques (Zheng et al., 2024; Patel al., 2024) paves the way for even more

---

Published as a conference paper at ICLR 2025
autonomous and efficient web automation, moving the field towards scalable and versatile solutions for real-word world applications. This paper mainly focuses more on the web coormatiion-eekinng and integration task with search engines instead of web browsing, and solves the main challenges with multi-agent framework.
5 CONCLUSION AND FUTURE WORK
This paper introduces MindSearch, novel LLM-based multi-agent framework for complex web information-ssekinn and integration tasks, by more comprehensively leveraging the strengths of both search engines and LLMs. MindSearch conducts effective and sufficient decomposition of complex queries followed by hierarchical information retrieval to improve the precision and recall of the retrieved relevant web information, by modeling the problem-solving process graph construction. The multi-agent design distributes the cognitive load among specialized agents, facilitating robust handling of complex and lengthy contexts. Extensive evaluations closed-set and open-set QA problems using GPT-4o and InternLM2.5-7B models demonstrated significant advantages in the response quality of MindSearch. The results that human evaluators preferred the responses from MindSearch over those from ChatGPT- Web and Perplexity. ai indicate its competitive edge AI-driven search solutions. However, there exist some limitations in this work: the citation quality of the web search references is not evaluated comprehensively, considering the extremely diverse and subjective evaluation of Al web search engines compared closed-set QA reference evaluation. Besides, MindSearch does not support visual inputs, and cannot interact with web pages, which promising and more complex scenario real-world applications. We leave them for future work and will continue to explore them in MindSearch. We wish this work pave the way for future research on multi-agent framework for solving human-level complex cognitive tasks.
iterative
6 ACKNOWLEDGEMENT
This work was supported by the Anhui Provincial Natural Science Foundation under Grant 2108085UD12. We acknowledge the support of GPU cluster built by MCC Lab Information Science and Technology Institution, USTC.
We would like to express our sincere gratitude to Jiaye Ge for her outstanding technical insights, product design skills, and coordination abilities. We also thank Zhongying Tu, Ying Zhao, Fang Fang, and Yiting Wang for their efficient execution in developing the project demo. Our gratitude extends to Xingyuan Liu, Shuaike Li, Zike Pan, Weijia Song, and Yuzhe Gu for their efforts in project suggestion.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflcction. arXiv preprint arXiv:2310.1/511 2023.
Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Di- girl: Training in-the-wil device-control agents with autonomous reinforeement learning. arXiv preprint arXiv:2406. 1896, 2024.
Pavel Berkhin. A survey pagerank computing. Internet Mathematics, 2(1):73 2005.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions tokens. In International Conference on Machine Learning, pp. 2206-2240 2022.
Sergey Brin and Lawrence Page The anatomy of a large-scale hypertextual web search engine. Computer Networks and ISDN 30(1-)),, 1998.
10

---

Published as a conference paper at ICLR 2025
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint MXXX::2403.17299 2024.
Chi-Min Chan, Chumpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. Rq-rag: Learning to refine queries for retrieval augmented generation. arXiv preprint arXiv:2404.006 2024.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open- domain questions. arXiv preprint wwXiv:/704.00055 2017.
Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, et al. T-eval: Evaluating the tool utilization capability step by step. arXiv preprint arXiv:2312 2at.:23222.4433.. 2023.
Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models. arXiv preprint arXiv:2403. 12881, 2024.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091-22 14, 2024.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprini arXiv:2406. 12793, 2024.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving, arXiv preprint arXiv:2309.17452 2023.
Yuzhe Gu, Ziwei Ji, Wenwei Zhang, Chengqi Lyu, Dahua Lin, and Kai Chen. Anah-v2: Scaling analytical hallucination annotation of large language models. arXiv preprint arXiv:2407.04660 2024.
Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder. When the large language model meets programming- the rise of code intelligence. arXiv preprint arXiv:2401. 14196, 2024.
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and pro- gram synthesis. arXiv preprint arXiv:2307. 12856, 2023.
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu, Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. Advances in Neural Information Processing Systems, 36:1-13, 2024.
Hongliang He, Wenlin Yao, Kaixin Ma. Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024.
Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al. Metatool benchmark for large language models: Deciding whether to use tools and which to use. arXiv preprint arXiv:2310.00 10.03 128, 2023.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint 2rX 2020.
Ziwei Ji, Naycon Lec, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Ziwei Ji, Yuzhe Gu, Wenwei Zhang, Chengqi Lyu, Dahua Lin, and Kai Chen. Anah: Analytical annotation of hallucinations in large language models. arXiv preprint 2xXXx:2405.203/5 2024.
11

---

Published as a conference paper at ICLR 2025
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04990 04906, 2020.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq lqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409. 12917, 2024.
Hanyu Lai, Xiao Liu, lat Long long, Shuntian Yao, Yuxuan Chen, Pengho Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: Bootstrap and reinforce a large lan- guage model-based web navigating agent. arXiv preprint arXiv:2404.0366 2024.
Xiangyu Lei, Guilin Zhang, Shuaijun Li, Huihuan Qian, and Yangsheng Xu. Dual-spring agv shock absorption system design: Dynamic analysis and simulations. In IEEE International Conference on Robotics and Biomimetics, pp. 1068-1074, 2017.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmeente genera- tion for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33: 9459-9474, 2020.
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for cool-augmented llms. arXiv preprint S.KIv:23304082244 2023.
Shuaijun Li, Guilin Zhang, Xiangyu Lei, Xiao Yu, Huihuan Qian, and Yangsheng Xu. Trajectory tracking control of a unicycle-type mobile robot with new planning algorithm. In IEEE Inter- national Conference on Robotics and Biomimetics, pp. 780-786, 2017.
Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. Ra-dit: Retrieval- -ugmeettd dual instruction tuning. arXiv preprint arXiv:2310.01335 2023.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, al. Deepseek-v2: strong, economical, and efficient mixture- of-experts language model. arXiv preprint arXiv:2405.04434 2024.
Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aohan Zeng, Zhengxiao Du, Peng Zhang, Yuxiao Dong, and Jie Tang. Webglm: Towards an efficient web-enhanced question answering system with hu- man preferences. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4549-4560, 2023.
Haoran Luo, Zichen Tang, Shiyao Peng, Yikai Guo, Wentai Zhang, Chenghao Ma, Guanting Dong, Meina Song, Wei Lin, Yifan Zhu, al. Chatkbqa: A snseraae-thhe-rettiie framework for knowledge base question answering with fine-tuned large language models. arXiv preprint arXiv:2310.08975, 2023a.
Hongyin Luo, Yung-Sun Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James Glass. Sail: Search-augmented instruction learning. arXiv preprint
arXiv:2305. 15225, 2023b.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji. Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint 1rXiv:2112.09332, 2021.
OpenAl. Learning to reason with 1lms, Sep 2024. URL https://opend: wwm/ mndm//ndex/ www.:iigg-to-reeaso--wtttt
Aaron Parisi, Yao Zhao, and Noah Ficdel. Talm: Tool augmented language models. arXiv preprint /222::2005./2255. 2022.
Ajay Patel, Markus Hofmarcher, Claudiu Leoveanu-Condre Marius- Contss-tnnntnnnin Dinu, Chris Callison- Callison-Burch, and Sepp Hochreiter. Large language models can self-improve at web agent tasks. arXiv preprint arXiv:2405. 20309, 2024.
12

---

Published as a conference paper at ICLR 2025
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305. 2023.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022.
Cheng Qian, Chi Han, Yi R Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. Creator: Tool cre- ation for disentangling abstract and concrete reasoning of large language models. arXiv preprint arXiv:2305. 14318, 2023.
Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Eleanor Jiang, Chengfei Lv, and Huajun Chen. Autoact: Automatic agent learning from scratch via self- planning. XrXiv preprint arX1v:2401.05268, 2024.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models master 16000+ real-world apis. arXiv preprint arXiv:2307.16789 2023.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering. arXiv preprini arXiv:2010.08191, 2020.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Timo Schick, Jane Dwivedi-Yuu Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539- 68551, 2024.
Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. Taskbench: Benchmarking large language models for task automation. arXiv preprini arXiv:2311. 13,08600,, 2023.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567, 2021.
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint vrXiv:2408.03314 2024.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre. Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint ar 2rXiv:2403.08295 2024.
Qwen Team. Generalizing an Ilm from 8k to Im context using qwen-agent, May 2024. URL https: /ttp:s//www/lmmmmm wwt.ub.itoooooooooo.oo wttpbog//////o/oo www.ghhw..../b.oog//eeaaaaggeete qwen-agent-24000 -2405/
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Exe- cutable code actions clicit better Ilm agents. arXiv preprint arXiv:2402.01033 010000 2024.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu. Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00800 2020.
13

---

Published as a conference paper at ICLR 2025
Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. Search-in-the-chaee Interactively enhancing large language models with search for knowledge-intent inttensivee tasks. In Proceedings of the ACM Web Conference, pp. 1362-1373, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint urXiv: (tfv:/809.00000 899,00600, 2018.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 35:20744-20757, 2022a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.06629, 2022b.
Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. Assistantbench: Can web agents solve realistic and time-consuming tasks? arXiv preprint arXiv:2407.15711, 2024.
Donghan Yu, Chenguang Zhu, Yuwei Fang. Wenhao Yu, Shuohang Wang, Yichong Xu, Xiang Ren, Yiming Yang, and Michael Zeng. Kg-fid: Infusing knowledge graph in fusion-in-decoder for open-domain question answering. arXiv preprint 1xXx::2110004330, 2021.
Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R Fung, Hao Peng, and Heng Craft: Customizing Ilms by creating and retrieving from specialized toolsets. arXiv preprint arXiv:2309. 17428, 2023.
Boyuan Zheng, Boyu Gou, Jihyung Kil. Huan Sun, and Yu Su. Gpt-4v (ision) is generalist web agent, if grounded. arXiv preprint arXiv:2401.01614 2024.
Shuyan Zhou, Uri Alon, Frank F Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. Docprompting: Generating code by retrieving the docs. arXiv preprint arXiv:220.005987, 2022.
Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for Ilm question answering with external tools. Advances in Neural Information Processing Systems, 36: 1-14, 2024.
14

---

Published as a conference paper at ICLR 2025
A COMPARISON WITH OTHER STATE-OF-THE-ARTI
In this section, we compare approach with more competitive web search counterparts, including self-ask (Press et al., 2022), CodeAct (Wang et al., 2024), and Searchain (Xu et al., 2024). All experiments conducted on the HotpotQA dataset with InternL.-76-22 and the results shown in Table 4. We can find that MindSearch consistently outperforms other state-of-the-art approaches by large margin. Specifically, Searchain shares similar spirit our approach which formulates the reasoning as query-of-chain However, at each revised step, Searchain needs to re-generate the whole reasoning chain due to its weakness in long-context reasoning, which is aome-conuuiing and fallible. Thanks to the multi-agent design, MindSearch is able to reason at each step immediately when necessary.
Table 4: Comparison with state-of-the-art approaches for web search tasks on HotpotQA dataset.
Method
Easy
Medium
Hard
Average
ReAct (Yao et al., 2022b)
69.0
56.0
49.0
58.0
self-ask (Press et al., 2022)
67.0
59.0
49.0
58.3
CodeAct (Wang et al., 2024)
70.0
63.0
51.0
61.3
Searchain (Xu et al., 2024)
70.0
61.0
54.0
61.6
MindSearch (Ours)
69.0
66.0
57.0
64.0
B TIME EFFICIENCY ON MINDSEARCH VS. HUMAN LABELERS
In this section, we provide detailed time cost measurement in terms of the human labelers and MindSearch on 10 complex research questions. We randomly distributed 10 questions to 5 human experts and asked them to collect information with web search engines. After retrieving enough data, each labeler is requested write detailed answer to the question. Human labelers spend 19h17min to accomplish the search tasks while MindSearch only takes 23 min. It can be seen that there exists relationship between MindSearch and human labelers with I min vs 1-hour time efficiency. Furthermore, we also analyze the time cost of one human labeler when labeling one question. 47 minutes are taken collect Information by searching and reading multiple (100+) web pages and another 74 minutes required to write detailed response (about 3,000 words).
C MORE ANALYSIS ON WEBPLANNER
C.I NUMBER OF HOPS VS DEPTH OF DAG
We conduct experiments on Musique with GPT-4o with 2, 3. 4 hops to study the relationships between the number of hops and the depth of the DAG. From Table 5, it can be seen that the depth of the tree increases with the number of hops monotonically, which fits our expectations. However, the number of hops is not identical to the depth of the tree, for example when the number of hops is 3, the depth of the tree is 1.2. There are two reasons: (1) MindSearch allows parallel execution, which only increases the tree by one but may resolve multiple questions at the current step, and (2) despite the question claiming 2 or more hops, there exist short-cuts simplifications in the question, resulting shorter search path.
Table 5: Experimental results on the number of hops of questions the depth of the generated DAG by WebPlanner on Musique dataset.
Num Hops
Depth
2
1.1
1.2
4
1.6
15

---

Published as a conference paper at ICLR 2025
C.2 COST ANALYSIS ON THE NUMBER OF SEARCH QUERIES
We analyze the number of queries generated by MindSearch and ReAct. Surprisingly. MindSearch generates 0.3 fewer queries on average compared to ReAct (3.2 3.5). We find that due to the weaker ability of ReAct to decompose the question, ReAct actually spends more queries repeatedly searching for some keywords, which is useless and inefficient. However, MindSearch can effectivelyy utilize the reasoning ability of LLM to search for more accurate queries, therefore, yielding fewer search times for each problem.
D GENERALIZATION TO OTHER LANGUAGE MODELS
In this section, we experiment with MindSearch on various accessible large language models to further validate the generalization of approach and the extensibility of the DAG-based code reasoning interface. We select three representative models: DeepSeekv2 (Liu et al., 2024), Qwen- 2.5-7b (Yang al., 2024), and GLM-4-9b (GLM et al., 2024), and the results are shown in Table 6. It can be seen that MindSearch can easily adapt to various models with little adaptation, which further proves the effectiveness of our approach.
Table 6: Experimental results on various LLMs for web search tasks on HotpotQA dataset with MindSearch.
Model
Easy
Medium
Hard
Average
DeepSeek-V V2 (Liu al., 2024)
70.0
71.0
68.0
69.6
Qwen-2.5-7b (Yang et al., 2024)
62.0
59.0
52.0
57.6
GLM4-9b (GLM al., 2024)
65.0
60.0
55.0
60.0
E ANALYSIS ON ERROR CORRECTION IN MINDSEARCH
In this section, we comprehensively demonstrate the error correction ability of MindSearch, which reflects the superiority of the code interface for planning, and the effectiveness of DAG protocol. We delicately select several typical examples of how MindSearch recovers from its previous mistakes, which helps the readers gain an intuitive understanding of how MindSearch deals with occasional search accidents.
E.1 ERROR CORRECTION WITH CODE EXECUTION FEEDBACK
MindSearch adopts the code as the execution protocol during the agent running, which has two ad- vantages: (1) large language models are more skillful in generating structure language, compared to natural form, and (2) code protocol provides us with validation mechanism: when WebPlanner generates wrong code invalid planning sentence, we can simply validate in the Python code interpreter and seed the exception back to the LLM for re- ce-generation. is inevitable to generate the wrong token during the text generation phase, due to the sampling techniques in seq2seq models. Therefore, it is necessary to maintain validatable protocol for self-correction. Figure 7 demon- strates how MindSearch observes the mistakes during the node exccution phase and corrects itself with the help of the error messages provided by the code executor.
E.2 ERROR CORRECTION WHEN WEBSEARCHER CANNOT FOUND RELATED INFORMATION
MindSearch introduces a multi-agent framework to address the extremely long context encountered web search scenarios. However, more communication among agents indicates that MindSearch needs to handle various responses from multiple WebSearchers. In order to avoid unexpected sponses from WebSearchers, we limit the response content of each WebSearcher to either related information about the topic information not found. In Figure 8, we demonstrate that MindSearch able to successfully recover from previous search results and regenerate the new search queries with "more information about the final match of UEFA Euro 2020", which finally find the comer
16

---

Published as a conference paper at ICLR 2025
Question
What the stock price increase on November 19, 2024, for componies with morket cxpitalizationn higher than Mite?
WebPlorner
(CoT process..)
Logent mindssiiindoe Inport ResSeancrfingph
Error Goneration: Node
grisph WetSaoncrrrggen
content> 'thot
increase November
with centtoleattion
higher then Heto?" modk nase. "roce" groph start. rodew
copi Laltzalion
Coeatte capitattzation
zottziooiionn Neua?")
(Python Interpreter)
Envirorment
Exception: keyError, "Meta cxpirclization"
line grosh eoggl__ddnnnttrrtttttooooo rede="reot".. reot". end__rode= rode:"Wets copitalizahio?) KeyError: "Mele tapi uppilaliaution
Webflorner
Sorry,
connect 'apitalization
From Lagant agints mindsesrch
Impore WitSacrenGrsph
Correct
grosh WebSearch Groph))
fiigher irdee (ndee. content- "Wht 15 the stock price increase on November 19, 2814, for conpantes with worket teattolizatton mode: namee
proph, osd_ root, mede
groph and_nagee chart_nnde "roe"" worunnd coittolsottenn of Ment capitalizztiion Neta?")
Figure 7: Illustaation of error self-correction by MindSearch with the help the error messages provided by the code executor: MindSearch regenerates the code which correctly fixes the node name error the next turn.
Question
How many corner locks were there the final match of the UEFA Euro ZU2U?
WebPlanner (CoT process..) J
Froe Lagent, www.ootonnnnneeee.. minddsoocch Liport WssSosnchi
WesSearchhGrsphe
gropp and root. mode (ncal, comfonte How eony corner intcks wore there in the F'tial salich of the UBFA Buro 20287" mode "ran""" Eroph and moder nouee cornen kickks Final nntch of UEFA, Eursoeze", mode_connaale lary conner kiicks in the rinall notch of Uhe
uroph edge Cafort_ "out". modlmm "Comee kicks, UBFA Euro0808"))
(Python Interpneter)
Enwironment
WibbSearcher (serner locks final match UEFA Euro 2020)
WebtSeorhher corner licks finsl match UBFA Euro 2030, You find the
information number of
WebPlomer
Regenercte the search queries based on curnent results
..oo process) will find rolated information searching final match of UEFA Euro 2020.
graph rocet mede. notch Burozader: content- "Nore Inforeat on about Final, metch UEFA Euro280e")
"inol natch LETA Euro2020")
Code Execuriio (Pythen Interpneter)
Envirorment
The results are found
WebSearcher (Final motch of UEFA Euro 2020)
SeaeShrr The LEFA Eurspeon Chumpiorakiip is this final, latel corner kicks were takes: by Ifdy and by Engl and.
Figure 8: Illustration of error self-correction by MindSearch when WebSearcher responds with 'not founded' feedback. MindSearch regenerates the query and successfully retrieves the expected infor- mation during the next search.
kicks in the Wikipedia pages in WebSearcher. Besides, there also some cases that cannot be found without web interaction. MindSearch enables the model to directly generate the response node when several attempts fail and directly give up the answer, which avoids the model falling into a repeatedly and meaningless loop (Figure 9).
DETAILS ON OPEN-SET EVALUATION PHASE
In this section, we provide more details on the open-set evaluation process for better reproducibility.
The labelers have no prior knowledge and information about the responses generated by the model.
17

---

Published as a conference paper at ICLR 2025
Question
What the dlasing price of Telsa on November 19th, 2024?
scueral ottempts omitted
Code Execution (Python Interpreter)
(clecing price Teslo)
WabSeurcher Sorry, find information obout clesing price
WabPlamer
Regenerate the search queries based current resilts [GeT process) con not relsted information obout Tedio clasing price on Nev. 19th, 2024, give my final nesponde now.
groph. add. response. node(riodz modi nano- "rosponse")
WebPlonner (Final Response)
November 2094,
Skoobbery
price
Figure 9: Illustration of the self-conclusion of MindSearch when the information cannot be found within the current search engine. It guarantees that MindSearch will not get stuck in certain tasks and can finally give the answer.
Question Question Content...
Prev
Next
Model 1 response
Model
Model response
response
Please Select
Factuality:
Model
Model
Model
Depth:
Model 1
Model 2
Model 3
Breath:
Model
Model 2
Model
Figure 10: Open-set evaluation labeling interface illustration. We do not expose the source of the
response to the labeler by replacing them with an anonymous model number, by randomly sampling from 1,2.3 for each question.
When we ask the labeler to choose the best response, we remove the source method by replacing it with "modell", "model2", and "model3". In order to avoid the labeler making any assumption of the sequence of models 1,2,3, we randomly shuffle the order of the responses before presenting them to the labeler.
The labelers will be asked to choose the best response from the three responses generated by the model according three aspects: factuality, depth, and breath.
In order gain straightforward understanding of our evaluation process, we provide an illustration of our labeling interface (see Figure 10), where human experts can select the best response based on the factuality, depth, and breath with simple choice. Besides, we allow labelers to move back the previous question to re-check/correct their original choices. Apart from the interface demonstration, we also provide a response sample from MindSearch, Perpexity.ai Pro, and ChatGPT-Web in the following:
Question: Recently, Chang e-b6 returned samples from the far side of the Moon. First, analyze the technical challenges of sampling and returning from the lunar far side, then detail how each chal- lenge was overcome. Compare this achievement with the Apollo 11 lunar landing mission, assessing China's 's contribution international space exploration through this successful lunar mission.
18

---

Published as a conference paper at ICLR 2025
Response from MindSearch
The recent success of China's Chang's 6-6 mission in returning samples from the far side of the Moon marks significant milestone in human space exploration. This achievement not only showcases China's technological advancements but also highlights contributions to international space exploration. Below, we delve into the technical challenges faced during the mission, how these challenges were overcome, and compare this achievement with the Apollo 11 lunar landing mission.
### Technical Challenges and Overcoming Them
#### 1. Communication Difficulties
The lunar far side is permanently shielded from Earth's view due to the Moon's synchronous rotation. This means that direct communication between the lunar far side and Earth is impossible. To address this, China deployed the Queqiao 2 relay satellite in lunar orbit. This satellite acts sent to the Chang 6 probe on the lunar far side [[6]][[2]]-
intermediary, enabling data transmission and control signals to be
#### 2. Navigation Challenges
Navigating to the lunar far side poses significant difficulties due to its lack of direct Earth-based communication. This necessitates the use of autonomous navigation systems. The Chang'c e 6 mission employs a combination of orbiter, lander, ascender, and reentry capsule components to perform its mission. The Kander-ascender combination will make soft landing in the South Pole-Aitken Basin, the largest and oldest crater on the lunar far side. The navigation system uses relative measurements between a halo orbiter and surface assets to achieve accuracies of tens of meters on the lunar far side surface [[]]][[]]].
#### 3. Power Supply and Energy Management
Power availability is critical enabler for lunar activities. The Chang'e 6 mission relies on a combination solar power and onboard energy storage solutions. The LunaGrid service, which consists of power-generating stations and power transmission cables, ensures continuous power supply to lunar surface assets for multi-year operations [[9]][[]]].
#### 4. Sample Collection and Return
Collecting samples from the lunar far side involves advanced robotic systems and mecha- nisms. The Chang'e 6 probe is equipped with drill and mechanical arm to gather surface and underground samples. The collected samples are vacuum-sealed in metal container and transferred to the reentry capsule for return to Earth. The entire process designed to be efficient and automated, minimizing the risk of sample contamination and ensuring the integrity the samples [[6]][[]]].
#### 5. Scientific and Technical Advancements
The lunar far side offers unique scientific opportunities due to its distinct geological features and electromagnetic environment. The samples collected by Chang'e e 6 are expected to provide valuable insights into the Moon's 's early evolution, volcanic activities, impact history, and the record of galactic activity [[4]][[[]].
### Comparison with Apollo
The Apollo 11 mission, which successfully landed astronauts Neil Armstrong and Buzz Aldrin the Moon in 1969, remains of the most iconic achievements in human his- tory. While Apollo 11 focused on landing humans on the Moon and conducting scientific experiments, Chang's e-6's primary objective was collect samples from the Moon's 's far side
19

---

Published as a conference paper at ICLR 2025
and return them to Earth. Both missions represent significant advancements in space exploration, but they differ in their goals and methodologies.
#### Technical Challenges
Both Chang's e-6 and Apollo 11 faced significant technical challenges in their missions. The Apollo 11 mission required precise calculations and engineering to ensure a safe landing on the Moon's surface, while Chang's e-6 had to navigate the lunar far side, which is not visible from Earth due to the Moon's 's synchronous rotation. The far side of the Moon presents additional challenges, including the lack of communication with Earth during the mission's critical phases.
To overcome these challenges, Chang's e-6 utilized relay satellite, Queqiao-2, to maintain communication with Earth during the mission's critical phases. This satellite played crucial role in ensuring the successful return of the samples to Earth. Similarly, Apollo 11 relied on extensive ground based tracking and communication systems to monitor the spacecraft's 's progress and ensure safe return.
### China's Contribution to International Space Exploration
China's involvement the Chang c-6 mission highlights its growing role in international space exploration. The mission carried four international payloads, including one developed by Pakistan, which marked Pakistan's 's first venture into lunar exploration. This collaboration demonstrates China's commitment to fostering international cooperation in space science and technology.
Furthermore, China's lunar exploration program, including the Chang'e series of missions, has provided a platform for international scientific research. The samples collected by Chang's e-6 are expected to contribute significantly the global understanding of lunar geology and the Moon's 's formation and evolution. This collaborative approach underscores China's role as a key player in the global space community.
### Conclusion
The success of Chang's e-6 collecting samples from the Moon's far side and returning them to Earth represents significant achievement in human space exploration. While the Apollo 11 mission remains a landmark in human history for its successful lunar landing, Chang's e- 6's focus sample collection from the far side of the Moon showcases China's technolog- ical prowess and commitment to international cooperation in space exploration. Through these missions, China has made substantial contributions to the global understanding of the Moon and the broader field of space science.
Response from Perplexity.ai
The recent success the Chang' e-6 mission in returning samples from the far side of the Moon marks a significant achievement in lunar exploration. This endeavor presented several technical challenges due to the unique conditions and requirements of sampling and returning from this less-explored region.
## Technical Challenges of Sampling and Returning from the Lunar Far Side
#Communication Limitations** **; The far side of the Moon is always facing away from Earth, making direct communication impossible. This necessitated the use of relay satellites maintain contact.
2. **Landing * Achieving a precise landing on the far side, where terrain is rugged and less understood, posed significant risks for the lander.
20

---

Published as a conference paper at ICLR 2025
3. **Sample Collection and Preservation Collecting samples without contamination and ensuring their integrity during transit back to Earth was crucial.
**Thermal Contro]**; The lunar environment experiences extreme temperature fluctua- tions, which could affect both the lander and the samples.
P*wwr Supply** The mission required a reliable power source for operations, especially during the long lunar night.
## Overcoming the Challenges
1. *Communication Limitations**:
To address communication issues, Chang'e-6 e-6 utilized the Tianhe relay satellite, which orbited the Moon and facilitated continuous communication between the lander and mission control on Ea
** *anddin Precision**:
Advanced navigation systems were employed, including high-resolution terrain mapping and autonomous landing technology, to ensure accurate touchdown on the targeted site[1].
3. *ammple Collection and Preservation**:
The mission utilized a specialized robotic arm designed for lunar soil collection. This arm was equipped with tools to minimize contamination and preserve sample integrity during collection and storage[1].
4. **herraal c
Chang'e e-6 was equipped with thermal insulation materials and active heating systems to protect sensitive instruments and samples from extreme temperatures, ensuring operational stability throughout its mission duration[1].
5. **Power Supply**:
Solar panels were integrated into the lander's 's design to provide a consistent power supply during lunar days, supplemented by rechargeable batteries support operations during lunar nights[1].
Overall, Chang'e-6" successful navigation of these challenges not only demonstrates China's growing capabilities in space exploration but also contributes valuable scientific data from an area of the Moon that remains largely unexplored.
Response from ChatGPT-Wa Web
The Chang's e-6 mission's successful retrieval of samples from the Moon's far side represents a significant achievement in space exploration, overcoming several technical challenges:
##1. Communication Constraints
The Moon's far side is perpetually hidden from Earth's direct line of sight, complicating real-time communication with the spacecraft. To address this, China deployed the Queqiao- 2 relay satellite into lunar orbit in March 2024, facilitating data transmission between Earth and the Chang's e-6 lander.
##2. Autonomous Operations
Due to communication delays and potential signal loss, Chang'e required advanced autonomous systems for navigation, landing, sample collection, and ascent. These systems enabled the spacecraft to perform critical operations without immediate human intervention. #*3. Precision Landing in Rugged **
The South Pole-Aitken Basin, the mission's target, features challenging topography with deep craters and uneven surfaces. Achieving precise landing in this environment necessitated sophisticated guidance and control technologies.
#*4. Sample Collection and Preservation**
Chang's e-6 was equipped with specialized instruments to collect approximately 2 kilograms operating in the Moon's
of lunar material, including a mechanical arm and drill capable of low-gravity conditions. Ensuring the integrity of these samples during collection, storage, and transport was crucial for scientific analysis.
21

---

Published as a conference paper at ICLR 2025
*#5. Ascent and Orbital Rendezvous**
After sample collection, the ascent vehicle needed to launch from the lunar surface, ren- dezvous, and dock with the orbiting return module-a complex maneuver requiring precise timing and control. This process was successfully executed, marking milestone in China's space capabilities.
Overcoming these challenges underscores the mission's technical sophistication and con- tributes valuable insights into lunar science.
G
SYSTEM PROMPT
System Prompt for WebPlanner
Character Profile
You are a programmer capable of Python programming in Jupyter environment. You can utilize the provided API to construct a Web Search Graph, ultimately generating and executing code.
API Description
Below is the API documentation for the WobSearchGraph class, including detailed attribute descrip-
tions:
Class: WebSearchGraph
This class manages nodes and edges of web search graph and conducts searches via weh proxy. Initialization Method
Initializes an instance of WebSearchGraph.
** *ttributes:
nodes (Dictfstr, Dict[str, str]]): A dictionary storing all nodes in the graph. Each node is indexed by its name and contains content, type, and other related information.
adjacency (Dict[strr List[str]]): An adjacency storing the connections between all nodes in the graph. Each node is indexed by its name and contains a list of adjacent node names.
Method: add wootooooooooooooo .ndde Adds the initial question as the root node.
**Parameters:**
node content (str): The user's question.
node name (str, optional): The node name, default 'root'.
Method: add_node
Adds a sub-question node and returns search results.
**Parameters:**
node name (str): The node name.
node_content (str): The sub-question content.
t*Rtturns:***
str: Returns the search results.
Method: reddrrssposeen mooe
Adds a response node when the current information satisfies the question's 's requirements.
s*Parameters:
node name (str, optional): The node name, default is "response".
Method: add
Adds an edge.
**
start node (str): The starting node name.
node (str): The ending node name.
22

---

Published as a conference paper at ICLR 2025
6. Method: reset
Resets nodes and edges.
Method: node
Get node information.
def node(self, node_name: name: str) -> str ** Parameetess:
node_name (str): The node name.
*Returns:**
str: Returns a dictionary containing the node's 's information, including content, type, thought process (if any). and list of predecessor nodes.
Task Description
By breaking down question into sub-questions that can be answered through searches (unrelated questions be searched concurrently), each search query should be single question focusing specific person, event, object, specific time point. location. or knowledge point. should not compound question (c.g., a time period), Step by step, build the search graph to finally answer the question.
Considerations
1. Each search node's content must be single question; do include multiple questions (e-g.. do not ask multiple knowledge points or compare and filter multiple things simultaneously, like asking for differences between A, B, and C, or price ranges -> query each separately).
2. Do not fabricate search results; wait for the code to return results.
Do not repeat the question; continue asking based on existing questions. When adding response node, add it separately: do not simultaneously.
response node and other nodes
5. In single output, do not include multiple code blocks; only one code block per output.
6. Each code block should be placed within code block marker, and after generating the code, add faction. end tag shown below: act iio.inttrrrtrooeeoeeeeeeeeeeeiee start interpretez 'python
# Your code block (Note that the 'Get new added node information' logic must be added at the end of the code block, such as roaph.node("...
*'< act 1 Lon_end
The final response should add a response node with node name 'response", no other nodes should added.
System Prompt for WebSearcher
Character Introduction
You are intelligent assistant that can call search tools. Please collect information reply to the question based on the current problem. You can use the following tools: itool
Reply Format
When calling the tool, please follow the format below:
...
Your thought process...
< <laction.start Iplugin "name": "tool_name", "parameters": "paraml": "valuel end
Requirements
Each key point in the response should be marked with the source of the search results ensure the credibility of the information. The citation format is [[int]]. If there multiple citations, multiple [[] to provide the index, such as [[i 1111]]1 [[dd 2]].
Based on the search results of the "current problem"". write detailed and complete reply to answer the "current problem"..
23


Reviews:
[{"review_id": "EbGJQXVrsy", "reviewer": "Reviewer_BgrW", "summary": "The paper proposes MindSearch, an LLM-based multi-agent information-seeking framework for complex multi-step information-seeking questions. MindSearch includes a Web Planner which decomposes the user query into atomic sub-questions as nodes in a dynamic graph and progressively extends the graph based on results from the WebSearcher. MindSearch considerably improves in response quality in terms of depth and breadth and also improves over the baseline react-based iterative search system.", "questions": "1)\tThe discussion in line 315 is a bit confusing. The authors say “MindSearch does not yield better performance in terms of facticity, but as per Figure 4 in the paper, factuality of MindSearch is preferred 70% of the time.\n\n2)\tPlease consider showing the example in Figure 5 in English.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1)\tThe paper demonstrates considerably better output responses for MindSearch, compared to proprietary AI-Search engines like Perplexity Pro and ChatGPT-Web.\n\n2)\tMindSearch also works considerably better than the closed-book and ReACT baselines on a variety of multi-hop question-answering datasets. \n\n3)\tExtensive analysis and evaluation provided in terms of the prompting strategy for WebPlanner along with using a graph-based methodology vs JSON-based and code-based.", "weaknesses": "1)\tWhile the paper only evaluates for final response quality, it does not consider the attribution quality of the generated response. Popular AI search engines like Perplexity.AI and ChatGPT-web also provide citations as part of the generated output. The authors do not discuss whether MindSearch provides any kind of attribution, and if yes, what does the citation quality look like (based on automatic evaluations like ALCE [1])\n\n2)\tNo analysis was provided with regard to the dynamic graph constructed by the WebPlanner. Does the number of hops in the question match the depth of the tree? How often is an incomplete graph created? Also, it would be interesting to see a cost analysis in terms of the number of search queries that MindSearch generates, in comparison to the baselines (ReACT specifically)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DByvX0xN7G", "reviewer": "Reviewer_8bqD", "summary": "The paper presents MindSearch, a multi-agent system for complex tasks that uses large language models (LLMs) and search engines for complex web information-seeking tasks. MindSearch addresses complex queries by decomposing them and retrieving information hierarchically, modeling the process as an iterative graph construction to enhance precision and recall. By distributing tasks across specialized agents, the framework manages complex and extended contexts effectively. The authors show that experimental results using GPT-4o and InternLM2.5-7B MindSearch outperforms benchmarks like ChatGPT-Web and Perplexity.ai, with human evaluators preferring its responses.", "questions": "1. What happens when Webplanner and code style interaction is not employed ? Is query decomposition required for all queries in web-searcher ? There is also a lack of qualitative analysis of failure scenarios. What happens when response at one node of the chain is wrong ? Does it result in cascading failures. Is there ayn mechanism for the Webplanner to detect such mistakes with feedback from websearcher ? \n\n2. Was there any qualitative evaluation on the benchmark where several human subjects were involved in performing the task with corresponding measurement of time taken ? to compare to mindsearch ?\n\n\n3. how do you respond to the first point in the weakness 1.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The problem is both interesting and important. Multi-agent systems for complex QA tasks that are robust and effective \n\n2. Easy to follow and the methods are simple and well explained.\n\n3. Experiments that include inference cost analysis is well considered.", "weaknesses": "1) The work fails to cite and compare to other relevant baselines. For complex QA tasks like HotpotQA or MusiqueQA self-ask[1] with search is a relevant baseline. Similarly Searchain[2] is particularly relevant as it also forms a global reasoning chain or graph where the query is decomposed into subquestions that comprise the nodes of the chain and this planning is similar in philosophy to Mindsearch. I think Assistantbench[3] released in July 2024 is also very relevant and useful to evaluate on. The method SeeplanAct proposed in the paper would serve as a strong baseline. SeeAct[4] is also a relevant baseline. While the authors have cited the same they have not compared to this approach. Other RAG baselines in AssistantBench are also relevant.\n\n2) Some claims are unsupported. For instance the claim made in abstract and section 2.3 regarding the utility of Mindsearch : “Mindsearch performs in 3 minutes tasks worth 3 hours of human effort” has no related evidence cited in the paper. Was there any qualitative evaluation on the benchmark where several human subjects were involved in performing the task with corresponding measurement of time taken ? to compare to mindsearch ?.\n\n3) The work also misses on some important ablations. What happens when Webplanner and code style interaction is not employed ? Is query decomposition required for all queries in web-searcher ? There is also a lack of qualitative analysis of failure scenarios. What happens when response at one node of the chain is wrong ? Does it result in cascading failures. Is there ayn mechanism for the Webplanner to detect such mistakes with feedback from websearcher ? The current approach is a simple tool use based approach which has been well explored in existing WebAgent based works. The additional analysis and error handling mentioned above may help strengthen and understand the core contributions of MindSearch\n\n[1] Measuring and Narrowing the Compositionality Gap in Language Models, Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, Mike Lewis\n\n[2] Search-in-the-Chain: Interactively Enhancing Large Language Models with Search for Knowledge-intensive Tasks, Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, Tat-Seng Chua [3] AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?, Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, Jonathan Berant [4] GPT-4V(ision) is a Generalist Web Agent, if Grounded, Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, Yu Su", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dWyBxr09Xv", "reviewer": "Reviewer_XR8b", "summary": "This paper proposes a novel tool agent, MindSearch, which decomposes user queries into atomic sub-questions represented as graph nodes and progressively extends the graph based on the search results from WebSearcher. For each sub-question, WebSearcher performs hierarchical information retrieval using search engines to gather relevant information for WebPlanner. Extensive experiments are conducted, including both open-set and closed-set datasets, and using open-source models alongside close-sourced LLMs, demonstrating the its effectiveness.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "S1: The writing and framework of this paper are clear and easy to follow.\n\nS2: The method is novel, utilizing the agents WebPlanner and WebSearcher to perform web search tasks.\n\nS3: Extensive experiments are conducted, demonstrating both the effectiveness and efficiency of this approach.", "weaknesses": "W1: In Figure 5, the words should also be accompanied by English translations.\n\nW2: For WebSearcher, how does the LLM select the most valuable pages from all the retrieved web content? More details should be provided. Additionally, regarding answer generation, the statement, \"After reading these results, the LLM generates a response to answer the original question based on the search results,\" requires further elaboration, such as information on input design or specific prompt construction.\n\nW3: For the open-set evaluation, five experts are chosen. The author should provide more details, including whether these experts had prior exposure to the answers generated by MindSearch. Furthermore, examples should be included to intuitively demonstrate the differences between the responses generated by MindSearch, ChatGPT-Web, and Perplexity.ai.\n\nW4: The author could provide information on token consumption to help the community manage the budget when using MindSearch in their projects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UMkCyqU5He", "reviewer": "Reviewer_iC9F", "summary": "This paper introduces a system called MindSearch, designed to emulate human cognitive processes to enhance web information retrieval and integration tasks. By combining large language models (LLMs) with search engines, the system addresses limitations in handling complex queries, fragmented information, and lengthy content through an LLM-based multi-agent framework.\n* WebPlanner: Simulates the cognitive process of multi-step information seeking by breaking down user queries into atomic subproblems, represented as nodes in a graph. The graph is then progressively expanded based on search results from WebSearcher.\n* WebSearcher: Conducts hierarchical information retrieval for each subproblem, using a search engine to gather valuable information for WebPlanner.\n\nThis multi-agent design enables MindSearch to search and integrate information from vast web sources within three minutes, equivalent to saving three hours of manual effort. MindSearch demonstrates significant response quality improvements in both closed-set and open-set QA tasks.", "questions": "- When constructing the DAG, how does MindSearch automatically create graph nodes? What are some tips for structuring question as graph nodes?\n- When large amounts of content are retrieved, how does WebSearcher reduce noise? And, as the rapid growth of web content can easily exceed the maximum context length of the LLM, how does WebSearcher effectively limit content length?\n- Additionally, could the experiment include more closed-source and open-source LLMs to further validate the effectiveness of the method?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a clear and logical approach to the problem, with a well-organized visual format that is easy to understand and read. \n- This method provides a novel question-answering retrieval method based on directed acyclic graphs, which makes the RAG more reasonable.", "weaknesses": "- The method part is not detailed enough to show the technical details. For instance, the design of DAG and the use of DAG is not clear.\n- Few baseline methods from the same category are included, and many RAG-based question-answering approaches are left unexamined, such as ChatKBQA, AutoReAct, etc. \n- The backbone was only tested on GPT-4 (close sourced) and InternLM2.5 (open sourced). Under this setting, it is hard to tell if the MindSearch will work for all (at least most) LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)