WebWalker: Benchmarking LLMs in Web Traversal
Authors: Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, Fei Huang
Accepted at: acl25

WebWalker: Benchmarking LLMs in Web Traversal

Jialong Wu1,4*, Wenbiao Yin4, Yong Jiang4†, Zhenglin Wang1, Zekun Xi4, Runnan Fang4, Linhai Zhang2, Yulan He3,3, Deyu Zhou1†, Pengjun Xie1, Fei Huang4
1 School of Computer Science and Engineering, Key Laboratory of Computer Network and Information Integration, Ministry of Education, Southeast University, China
2 Department of Informatics, King’s College London, UK
3 The Alan Turing Institute, UK 4 Tongyi Lab, Alibaba Group
* wujialongml@gmail.com, d.zhou@seu.edu.cn, yongjiang.yj@alibaba-inc.com

Abstract

Retrieval-augmented generation (RAG) demonstrates remarkable performance across tasks in open-domain question-answering. However, traditional search engines may retrieve shallow content, limiting the ability of LLMs to handle complex, multi-layered information. To address it, we introduce WebWalkerQA, a benchmark designed to assess the ability of LLMs to perform web traversal. It evaluates the capacity of LLMs to traverse a website’s subpages to extract high-quality data systematically. We propose WebWalker, which is a multi-agent framework that mimics human-like web navigation through an explore-critic paradigm. Extensive experimental results show that WebWalkerQA is challenging and demonstrates the effectiveness of RAG combined with WebWalker, through the horizontal and vertical integration in real-world scenarios.1

1 Introduction

Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range of natural language processing tasks (Ouyang et al., 2022; OpenAI, 2022b). While their knowledge base remains static post-training, integrating external search engines via retrieval-augmented generation (RAG) allows LLMs to retrieve up-to-date information from the web, enhancing their utility in dynamic, knowledge-intensive scenarios (Lewis et al., 2020; Fang et al., 2025). However, traditional online search engines, e.g., Google or Bing, perform horizontal searches of queries and may not effectively trace the deeper content embedded within websites.

Interacting with the web pages and digging through them can effectively address this issue. Previous works related to web pages focus on addressing action-based requests, such as Mind2Web (Deng et al., 2023) and WebArena (Zhou et al., 2024a); these HTML-based instruction-action benchmarks face challenges such as excessively noisy information and overly long inputs, which can significantly hinder performance due to limitations in long-context understanding. Additionally, they fail to capture the complexities of real-world scenarios where relevant information is buried deep within web pages and requires multiple layers of interaction.

To fill this gap, a new task Web Traversal is proposed, given an initial website corresponding to a query, systematically traverses web pages to uncover information. We propose WebWalkerQA, designed specifically to evaluate LLMs on their ability to handle queries embedded in complex, multi-step web interactions on a given root website. WebWalkerQA focuses on text-based reasoning abilities, using a Question-Answer format to evaluate traversal and problem-solving capabili-

Figure 1: A multi-source QA2 example from WebWalkerQA that requires traversing web pages to gather information for answering the given question.

2In our paper, multi-source refers to the requirement of information from multi distinct web pages.
ties in web scenarios. We constrain actions to “click” ⏲️ to evaluate the agent’s navigation and information-seeking capabilities. This paradigm is more targeted and aligns better with practical applications. WebWalkerQA reflects real-world challenges, emphasizing the depth of the source information across education, conference, organization, and game domains, where official sources are published and paths to information are more structured with clickable buttons and reasoning logic. Several types, including multi-source and single-source QAs, are developed to evaluate the ability of LLMs to mimic different human web-navigation paradigms.

Additionally, we introduce a strong baseline WebWalker, a multi-agent framework designed to emulate human-like web navigation through vertical exploration. The framework consists of an explorer agent and a critic agent. Given the need for reasoning capabilities to navigate and interact with web pages effectively, the explorer agent is built upon the ReAct framework (Yao et al., 2023), leveraging a thought-action-observation paradigm, while the critic agent is responsible for maintaining memory and generating responses based on the exploration conducted by the explorer agent. We evaluate the performance of the WebWalker, built on various mainstream LLMs, including both closed-source and open-sourced, using WebWalkerQA as the benchmark. However, even with the most powerful LLMs as the backbone, its performance on WebWalkerQA remains suboptimal, thereby validating the challenge posed by WebWalkerQA.

We then conduct further experiments to validate the integration with the RAG for information-seeking QA tasks. Our findings are as follows: (i) Web navigation still requires efforts in tasks that demand planning and reasoning; (ii) By combining RAG with the WebWalker, this horizontal and vertical coordination proves effective; (iii) Vertical exploration of pages offers a promising direction for scaling inference time in RAG systems.

The contributions of our work are as follows:

• We construct a challenging benchmark, Web-WalkerQA, which is composed of 680 queries from four real-world scenarios across over 1373 webpages.
• To tackle the challenge of web-navigation tasks requiring long context, we propose Web-Walker, which utilizes a multi-agent framework for effective memory management.
• Extensive experiments show that the Web-WalkerQA is challenging, and for information-seeking tasks, vertical exploration within the page proves to be beneficial.

2 Related Work

2.1 Web-Oriented Benchmark

Before the era of LLMs, several web-oriented benchmarks had already been proposed (Liu et al., 2018; Xu et al., 2021; Humphreys et al., 2022; Yao et al., 2022; Mialon et al., 2024; Xu et al., 2024). LLMs are capable of interacting with complex environments, like the open web in HTML or DOM format (Tan et al., 2024), leading to the development of an increasing number of benchmarks aimed at evaluating the interaction capabilities of LLMs with web content. The widely used benchmark today, Mind2Web (Deng et al., 2023), is a dataset designed for evaluating web agents that follow instructions to complete complex tasks, typically through multiple-choice questions. Subsequent works have extended the interaction to the vision domain, incorporating information from screenshots (Zheng et al., 2024a,b; He et al., 2024a; Koh et al., 2024a; Cheng et al., 2024). The web-oriented benchmark is becoming progressively more human-like, vision-centric, and increasingly broad, complex, and realistic (Liu et al., 2024; Hong et al., 2024; Kim et al., 2024; Zhang et al., 2024c). The most closely to ours are the MMInA (Zhang et al., 2024c) and Assistant-Bench (Yoran et al., 2024), both of which focus on time-consuming tasks that require navigation across multiple pages. In our work, WebWalkerQA takes the form of QA pairs. Unlike all previous works, we construct both single-source and multi-source queries from the width perspectives of the website, aiming to simulate two types of page exploration patterns typically exhibited by humans. The comparison between WebWalkerQA and other benchmarks is shown in Table 2.

2.2 Agents on Web-Navigation

Based on web-oriented benchmarks, numerous web agents have been proposed (Nakano et al., 2021; Liu et al., 2023; Zhou et al., 2023; Lai et al., 2024; Zhou et al., 2024b; Xi et al., 2025; Zhang et al., 2025). Web agents primarily follow two lines of development: one leverages a small language model trained specifically to filter actions or identify relevant HTML elements (Zheng et al., 2024a; Deng
<table>
  <tr>
    <th></th>
    <th>Language</th>
    <th>Format</th>
    <th>Depth</th>
    <th>Width</th>
    <th>Hop</th>
    <th># Pages</th>
  </tr>
  <tr>
    <td>Mind2Web (Deng et al., 2023)</td>
    <td>En</td>
    <td>Multi-choice</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>100</td>
  </tr>
  <tr>
    <td>WebArena (Zhou et al., 2024a)</td>
    <td>En</td>
    <td>Action</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>6</td>
  </tr>
  <tr>
    <td>AssistantBench (Yoran et al., 2024)</td>
    <td>En</td>
    <td>QA</td>
    <td>x</td>
    <td>✓</td>
    <td>✓</td>
    <td>525</td>
  </tr>
  <tr>
    <td>MMInA (Zhang et al., 2024c)</td>
    <td>En</td>
    <td>Action</td>
    <td>x</td>
    <td>✓</td>
    <td>✓</td>
    <td>100</td>
  </tr>
  <tr>
    <td>GAIA (Mialon et al., 2024)</td>
    <td>En</td>
    <td>QA</td>
    <td>x</td>
    <td>✓</td>
    <td>✓</td>
    <td>-</td>
  </tr>
  <tr>
    <th>WebWalkerQA</th>
    <th>En&Zh</th>
    <th>QA</th>
    <th>✓</th>
    <th>✓</th>
    <th>✓</th>
    <th>1373</th>
  </tr>
</table>

Table 1: Comparison between WebWalkerQA and other benchmarks. Depth refers to the extent of exploration required on a given website. Width denotes whether answering a query necessitates multiple sources. Hop indicates whether multiple steps are required to complete the task. #Pages refers to the number of webpages involved.

et al., 2024; Furuta et al., 2024). The other line focuses on prompting LLMs (Reddy et al., 2024; Song et al., 2024; Koh et al., 2024b), where different agentic modules are used to guide the model in accomplishing complex web navigation tasks more effectively. In addition, with the rise of visual web-oriented benchmarks, many agents now use screenshots as sensory input (He et al., 2024b; Abuel-saad et al., 2024; Iong et al., 2024). Unlike previous works, WebWalker specializes in information-seeking by reasoning over HTML button data. It emulates human-like page interactions with web pages to access reliable, authoritative information utilizing a multi-agent framework.

3 WebWalkerQA

We present WebWalkerQA in this section, starting with an overview of the data collection process to ensure quality (\S3.1), followed by a discussion of WebWalkerQA’s statistics (\S3.2) and introduction of a new task, Web Traversal (\S3.3). Finally, we describe the evaluation metrics for WebWalkerQA(\S3.4).

3.1 Data Collection

To make the annotation process cost-efficient and accurate, we employ a two-stage funnel annotation strategy, combining LLM-based and human annotation. In the first stage, GPT-4o (OpenAI, 2022a), performs initial annotations, followed by a second stage, where crowd-sourced human annotators conduct quality control and filtering to refine the final results. The overall data collection pipeline is illustrated in Figure 2.

LLM-based Annotation The collection pipeline is outlined as follows:

• Step1: Traverse official websites recursively, collecting information on accessible sub-links and their respective pages.

• Step2: Construct queries based on the provided page information and specified role, such as focusing on the solo page or considering both pages simultaneously.

• Step3: Verify and filter for legitimate queries that deviate from natural, human-like phrasing, retaining only QA pairs with short answers containing entities.

The additional details, including step-specific prompts and case examples, are provided in Appendix D. As illustrated in Figure 2 (b), our dataset construction includes both multi-source and single-source types, corresponding to two types of human information-seeking behaviours within web pages. The single-source type simulates a user deeply exploring a single piece of information hidden within web pages, while the multi-source type simulates multi-source scenarios where users rely on multiple pages to solve a query. Notably, the multi-source QA tasks can not be easily exploited by search engine shortcuts (Mavi et al., 2024).

Human Annotation After the synthetic queries are generated by LLM, human annotators can rewrite and calibrate the questions and answers to ensure the QA pairs are correct and consistent.

3.2 Data Statistics

Through such data construction method with LLM and human participation, we obtain 680 question-answer pairs for WebWalkerQA. The annotated case is shown in Figure 10. We will provide comprehensive statistics on WebWalkerQA, categorized by type, domain, and language.

Type WebWalkerQA contains two types of data: multi-source and single-source QAs. Single-source QAs are labeled as single-source_i, where i \( \in [2,4] \), denoting the depth of the corresponding subpage.
Figure 2: Data Generation Pipeline for WebWalkerQA. We first collect root official websites across conference, organization, education, and game domains. Then we mimic human behavior by systematically clicking and collecting subpages accessible through sublinks on the root page. Using predefined rules, we leverage GPT4o to generate synthetic QA-pairs based on the gathered information, followed by manual verification to ensure accuracy and relevance.

<table>
  <tr>
    <th></th>
    <th colspan="3">Single-source QAs</th>
    <th colspan="3">Multi-source QAs</th>
  </tr>
  <tr>
    <th></th>
    <th>Easy</th>
    <th>Medium</th>
    <th>Hard</th>
    <th>Easy</th>
    <th>Medium</th>
    <th>Hard</th>
  </tr>
  <tr>
    <td></td>
    <td>80</td>
    <td>140</td>
    <td>120</td>
    <td>80</td>
    <td>140</td>
    <td>120</td>
  </tr>
</table>

Table 2: Dataset statistics on data difficulty level.

Similarly, Multi-source QAs are labeled as multi-source_i, where \( i \in [2, 8] \), representing the sum of the depths of the two associated subpages\footnote{Taking multi-source_6 as an example, it may refer to a query constructed from two 3rd level pages or from one page at the 2nd level and another at the 4th level.}. In other words, answering this query requires reading both pages simultaneously.

**Difficulty Level** We categorize the questions into three difficulty levels: easy, medium, and hard, based on the value of \( i \). Specifically, single-source_2, single-source_3, and single-source_4 correspond to the easy, medium, and hard levels, respectively. Similarly, for multi-source questions, multi-source_2–4, multi-source_4–6, and multi-source_6–8 correspond to the easy, medium, and hard levels, respectively. The data statistics for the different data types are presented in Table 2.

**Domain** WebWalkerQA encompasses four real-world domains: conference, organization, education, and game. These domains are selected because they provide authoritative information relevant to their respective fields, and their pages contain rich clickable content, offering substantial depth for exploration.

**Language** WebWalkerQA is a bilingual dataset that includes both Chinese and English\footnote{Classification based on the language of the root webpages.}, reflecting the most widely used and universal languages in real-world web environments.

The statistics of WebWalkerQA on domain and language are illustrated in Figure 3. The proportions of the conference, organization, education, and game domains are 24.0%, 7.9%, 46.3%, and 24.0%, respectively. In terms of language distribution, Chinese and English account for 60.5%, 39.5%, respectively. WebWalkerQA features a diverse distribution of languages and domains to ensure a comprehensive evaluation.

![The language and domain distribution.](page_1012_370_388_246.png)

Figure 3: The language and domain distribution.

3.3 Web Traversal Task

Formally, given an initial website URL \( U_{root} \) and a query \( Q \), which needs to be answered by exploring the website. The goal of this task is to gather enough information through page traversal to ultimately answer the query \( Q \). The task is to navigate the website to find the corresponding information.

3.4 Evaluation

WebWalkerQA can be evaluated from both *performance* and *efficiency* perspectives. using question-answering *accuracy (acc.)* as the performance metric and the *action count (A.C.)* of successful agentic executions answering correctly as the efficiency metric. Due to the varying lengths of generated text, it is challenging to perform exact match evaluation, even though we have controlled for short answers. We use GPT-4 as the evaluator, which determines the *correctness* of responses by com-
paring the predicted answer with the ground truth using CoT prompting strategy (Wei et al., 2022)5.

4 WebWalker

We introduce WebWalker, a multi-agent framework designed to interact with web environments to answer queries. The WebWalker framework consists of two agents: an explorer agent and a critic agent. As illustrated in Figure 4, the explorer agent traverses the web pages in Thought-Action-Observation (\( \mathcal{T}, \mathcal{A}, \mathcal{O} \)) paradigms. The critic agent updates the memory until sufficient information is accumulated to effectively address the query. The details regarding prompts for both agents are presented in Appendix D.3.

4.1 Think then Explore

The explorer agent explores the subpages by interacting with HTML buttons on the page. At time step \( t \), the explorer agent receives an observation \( \mathcal{O}_t \) from the web environment and takes an action \( \mathcal{A}_t \), following the policy \( \pi(\mathcal{A}_t|\mathcal{H}_t) \). The observation \( \mathcal{O}_t = (p_t, l_t) \) consists of the information from the current page \( p_t \) and a set of clickable sublinks \( l_t = \{button_i\}_{i=1}^K \), where each \( button_i \) describes HTML button information for one of the \( K \) sublinks and have an associated URL. The action \( \mathcal{A}_t \) involves selecting a URL of a subpage to explore and does not encompass answering the question. Specifically, we utilize the web page’s markdown content along with clickable HTML buttons (and corresponding URL) extracted using Beautiful Soup as the observation for the current page. The context \( \mathcal{H}_t = (\mathcal{T}_1, \mathcal{A}_1, \mathcal{O}_1, \cdots, \mathcal{O}_{t-1}, \mathcal{T}_t, \mathcal{A}_t, \mathcal{O}_t) \) represents the sequence of past observations and actions leading up to the current step \( t \). The context will be updated, and this exploration process will continue until the critic agent determines to answer the query or the maximum number of steps is reached.

4.2 Think then Critique

Due to the policy \( \pi(\mathcal{A}_t|\mathcal{H}_t) \) being implicit and the potentially large size of \( \mathcal{H}_t \), motivated by pair programming (Williams et al., 2000; Noori and Kazemifard, 2015), we incorporate a critic agent into the WebWalker framework to address these challenges.

The critic agent operates after each execution of the explorer agent. Its input consists of the query and the explorer’s current observation. The critic initializes a memory to incrementally accumulate relevant information. Formally, at each step, \( t \), following the execution of the explorer agent, the critic agent takes the query \( Q \) and the explorer’s
<table>
  <tr>
    <th rowspan="2">Backbones</th>
    <th rowspan="2">Method</th>
    <th colspan="3">Single-source QA</th>
    <th colspan="3">Multi-source QA</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>Easy</th><th>Medium</th><th>Hard</th>
    <th>Easy</th><th>Medium</th><th>Hard</th>
    <th>acc.</th><th>A.C.</th>
  </tr>
  <tr>
    <th colspan="10">Closed-Sourced LLMs</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>ReAct</td>
    <td>53.75</td><td>2.53</td><td>45.00</td><td>3.34</td><td>30.00</td><td>5.61</td><td>32.50</td><td>2.34</td><td>31.43</td><td>3.97</td><td>15.00</td><td>6.77</td><td>33.82</td><td>3.83</td>
  </tr>
  <tr>
    <td></td>
    <td>Reflexion</td>
    <td>56.25</td><td>2.91</td><td>51.43</td><td>3.88</td><td>30.83</td><td>5.75</td><td>35.00</td><td>3.67</td><td>27.14</td><td>4.13</td><td>16.67</td><td>7.05</td><td>35.29</td><td>4.27</td>
  </tr>
  <tr>
    <td></td>
    <td>WebWalker</td>
    <td>55.00</td><td>2.97</td><td>50.00</td><td>3.43</td><td>30.00</td><td>6.02</td><td>47.50</td><td>4.06</td><td>34.29</td><td>3.85</td><td>15.83</td><td>6.57</td><td>37.50</td><td>4.67</td>
  </tr>
  <tr>
    <td>Qwen-Plus</td>
    <td>ReAct</td>
    <td>48.75</td><td>1.67</td><td>48.57</td><td>2.69</td><td>28.33</td><td>4.00</td><td>35.00</td><td>2.60</td><td>27.86</td><td>3.11</td><td>14.17</td><td>6.55</td><td>33.08</td><td>3.03</td>
  </tr>
  <tr>
    <td></td>
    <td>Reflexion</td>
    <td>53.75</td><td>3.66</td><td>40.00</td><td>3.79</td><td>24.17</td><td>5.88</td><td>47.50</td><td>3.28</td><td>30.00</td><td>4.07</td><td>15.00</td><td>7.11</td><td>33.23</td><td>4.32</td>
  </tr>
  <tr>
    <td></td>
    <td>WebWalker</td>
    <td>55.00</td><td>3.72</td><td>47.14</td><td>3.19</td><td>30.00</td><td>6.13</td><td>35.00</td><td>3.89</td><td>27.14</td><td>4.39</td><td>15.00</td><td>7.38</td><td>33.82</td><td>4.36</td>
  </tr>
  <tr>
    <th colspan="10">Open-Sourced LLMs</th>
  </tr>
  <tr>
    <td>Qwen-2.5 -7B</td>
    <td>ReAct</td>
    <td>37.50</td><td>3.36</td><td>18.57</td><td>4.88</td><td>9.17</td><td>5.45</td><td>17.50</td><td>3.42</td><td>11.43</td><td>3.62</td><td>5.83</td><td>4.57</td><td>16.02</td><td>2.99</td>
  </tr>
  <tr>
    <td></td>
    <td>Reflexion</td>
    <td>37.50</td><td>4.03</td><td>23.00</td><td>3.48</td><td>11.67</td><td>4.57</td><td>30.00</td><td>2.66</td><td>15.71</td><td>5.45</td><td>4.17</td><td>7.8</td><td>19.11</td><td>4.07</td>
  </tr>
  <tr>
    <td></td>
    <td>WebWalker</td>
    <td>41.25</td><td>3.39</td><td>24.71</td><td>3.86</td><td>12.50</td><td>5.93</td><td>18.75</td><td>4.00</td><td>20.71</td><td>3.34</td><td>5.83</td><td>7.28</td><td>19.85</td><td>3.94</td>
  </tr>
  <tr>
    <td>Qwen-2.5 -14B</td>
    <td>ReAct</td>
    <td>36.25</td><td>1.86</td><td>32.14</td><td>2.75</td><td>15.00</td><td>3.61</td><td>27.50</td><td>2.31</td><td>22.86</td><td>3.00</td><td>5.00</td><td>5.00</td><td>22.35</td><td>2.76</td>
  </tr>
  <tr>
    <td></td>
    <td>Reflexion</td>
    <td>46.25</td><td>2.21</td><td>34.29</td><td>2.83</td><td>15.00</td><td>4.44</td><td>36.25</td><td>2.51</td><td>22.86</td><td>3.34</td><td>5.83</td><td>5.42</td><td>25.14</td><td>3.01</td>
  </tr>
  <tr>
    <td></td>
    <td>WebWalker</td>
    <td>41.25</td><td>2.42</td><td>41.43</td><td>3.24</td><td>23.33</td><td>4.42</td><td>30.00</td><td>3.95</td><td>22.86</td><td>3.56</td><td>10.00</td><td>6.16</td><td>27.50</td><td>3.60</td>
  </tr>
  <tr>
    <td>Qwen-2.5 -32B</td>
    <td>ReAct</td>
    <td>47.50</td><td>2.21</td><td>35.71</td><td>3.20</td><td>16.67</td><td>3.55</td><td>36.25</td><td>2.68</td><td>18.57</td><td>3.00</td><td>8.33</td><td>3.70</td><td>25.44</td><td>2.93</td>
  </tr>
  <tr>
    <td></td>
    <td>Reflexion</td>
    <td>42.50</td><td>2.52</td><td>32.86</td><td>2.65</td><td>16.67</td><td>3.90</td><td>31.25</td><td>2.84</td><td>23.57</td><td>3.12</td><td>5.83</td><td>5.00</td><td>23.26</td><td>3.00</td>
  </tr>
  <tr>
    <td></td>
    <td>WebWalker</td>
    <td>41.25</td><td>2.69</td><td>34.29</td><td>4.14</td><td>22.50</td><td>5.14</td><td>27.50</td><td>3.11</td><td>25.00</td><td>3.51</td><td>10.00</td><td>6.08</td><td>26.02</td><td>3.90</td>
  </tr>
  <tr>
    <td>Qwen-2.5 -72B</td>
    <td>ReAct</td>
    <td>47.50</td><td>1.68</td><td>38.57</td><td>2.79</td><td>20.00</td><td>4.04</td><td>45.00</td><td>2.25</td><td>32.14</td><td>3.13</td><td>10.00</td><td>5.41</td><td>30.73</td><td>2.86</td>
  </tr>
  <tr>
    <td></td>
    <td>Reflexion</td>
    <td>57.50</td><td>3.04</td><td>44.29</td><td>3.88</td><td>28.33</td><td>5.82</td><td>36.25</td><td>3.62</td><td>25.00</td><td>3.60</td><td>12.50</td><td>6.26</td><td>32.50</td><td>4.09</td>
  </tr>
  <tr>
    <td></td>
    <td>WebWalker</td>
    <td>58.75</td><td>2.70</td><td>48.57</td><td>3.07</td><td>25.83</td><td>5.77</td><td>35.00</td><td>3.57</td><td>29.29</td><td>4.87</td><td>15.00</td><td>7.38</td><td>33.26</td><td>4.32</td>
  </tr>
</table>

Table 3: Main results of three methods across closed-sourced and open-sourced LLMs as the backbone. Acc. and A.C. refer to accuracy and action count, respectively.

current observation and action \((\mathcal{O}_t, \mathcal{A}_t)\) as input. It then updates the memory \(M\), evaluates whether the gathered information is sufficiently complete to answer the query, and provides an answer once the required information is deemed sufficient.

5 Experiment

5.1 Experimental Setting

Baselines We choose widely recognized state-of-the-art agent frameworks, ReAct and Reflexion, as our baselines. ReAct (Yao et al., 2023) is a general paradigm that combines reasoning and acting with LLMs by multiple thought-action-observation steps. Reflexion (Shinn et al., 2024) is a single-agent framework designed to reinforce language agents through feedback.

Backbones To thoroughly assess the web traversal capabilities of existing LLM-based agents, we select models with a context window of at least 128K to accommodate the extensive length of page information. Given the inherent complexity of the task, we opt for models with at least 7B parameters. We validate a total number of nine models, including both closed-sourced and open-sourced ones:
Closed-sourced LLMs GPT-4o\(^6\) (OpenAI, 2022a); Qwen-Plus\(^7\) (Team, 2024); Open-sourced LLMs Qwen2.5 series models (Yang et al., 2024) specifically, Qwen2.5-(7,14,32,72)B-Instruct.\(^8\)

Implementation Details Considering the context limitation of models, our proposed WebWalker, along with two baselines, all operate in a zero-shot setting. We limit the number of actions \(K\) for the explorer agent to 15, meaning that the explorer

\footnotetext{6https://platform.openai.com/docs/models#gpt-4o}
\footnotetext{7https://www.alibabacloud.com/help/en/model-studio/}
\footnotetext{8The LLaMA series models (Dubey et al., 2024) demonstrate limited ability to handle react-format instructions in our preliminary experiments.}

![A scatter plot showing the relationship between accuracy and action count for various models and methods.](page_1012_1342_410_312.png)

Figure 5: ▲ represents WebWalker using various models as backbones, ■ represents Reflexion with different backbone models, and ● denotes ReAct employing various backbone models.
agent can explore at most 15 steps. More implementation details are presented in Appendix A.

5.2 Main Results
The main results across six LLMs are presented in Table 3. The closed-source models outperform the open-source models in both performance and efficiency. For open-source models, performance and efficiency improves as the model size increases. Our proposed WebWalker framework outperforms Reflexion, which in turn outperforms React. We only counted the action count (A.C.) from correct executions, and as the model size increases, the A.C. grows, indicating that larger LLMs have enhanced long-range information-seeking ability. Even the best-performing WebWalker using GPT-4o as its backbone does not surpass 40%, highlighting the challenge posed by WebWalkerQA. It can be observed that as the depth increases or the number of sources required increases, the difficulty of acquiring the information needed to resolve the query becomes greater, resulting in a decline in accuracy performance.

The performance distribution of accuracy and action count for different methods across various models is shown in Figure 5. The further towards the top-right corner, the more effective and prolonged the web traversal becomes. We observe that increasing the model size or introducing reflection on the process of each action can address certain problems requiring multi-step solutions, thereby enabling long-distance task-solving capabilities in web traversal tasks.

5.3 Results across Domains and Languages
WebWalkerQA is a bilingual dataset encompassing both Chinese and English and spans multiple domains, including games, conferences, education, and organizations. The performance across different domains and languages is shown in Figure 6. In the domain of conference, the framework demonstrates relatively superior performance, likely due to the more explicit and directive nature of the button information, which facilitates more straightforward inferences. The framework performs similarly in both Chinese and English, as the models we employed are both pre-trained and supervised-fine-tuned in a bilingual setting.

5.4 Error Assessment
In Table 3, we only report the action count of the successful agentic executions (A.C.). For incorrect execution, errors can also be categorized into three types: refusal to answer or locating wrongly, reasoning error, and exceeding the maximum number of steps \( K \). The prediction distribution is shown in Figure 7. The model with a relatively small number of parameters using the ReAct framework lacks the capacity to explore the depth of information, making judgments within just a few iterations of taking action, regardless of whether relevant information has been found. It tends to “give up” and exhibits characteristics of _impatience_. Introducing memory to manage the long context, along with an increase in model parameters, provides evidence that this phenomenon stems from the interference of long contexts having noisy information and the inherent capabilities of the model itself, consistent with the analysis drawn in §5.2. Some errors are categorized as reasoning errors, where the golden page has been found in the visited pages but is still incorrectly marked. This underscores the challenge of reasoning on page information in certain cases.\footnote{The corresponding case is presented in Appendix F.1.}

Figure 6: Performance across domains and languages of WebWalker building upon Qwen-14B and Qwen-Plus.

Figure 7: Predication distribution of WebWalker and React method building on Qwen-14B and Qwen-Plus.
<table>
  <tr>
    <th rowspan="2">Systems</th>
    <th colspan="3">Single-source QA</th>
    <th colspan="3">Multi-source QA</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>Easy</th>
    <th>Medium</th>
    <th>Hard</th>
    <th>Easy</th>
    <th>Medium</th>
    <th>Hard</th>
  </tr>
  <tr>
    <th colspan="8">Close Book (No Retrieval)</th>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td>
    <td>12.50</td>
    <td>7.86</td>
    <td>8.33</td>
    <td>11.25</td>
    <td>6.43</td>
    <td>5.00</td>
    <td>8.08</td>
  </tr>
  <tr>
    <td>o1-preview</td>
    <td>16.25</td>
    <td>10.00</td>
    <td>9.17</td>
    <td>7.50</td>
    <td>10.71</td>
    <td>6.67</td>
    <td>9.85</td>
  </tr>
  <tr>
    <th colspan="8">Commerical Systems</th>
  </tr>
  <tr>
    <td>Doubao</td>
    <td>45.00</td>
    <td>15.00</td>
    <td>18.33</td>
    <td>13.75</td>
    <td>8.57</td>
    <td>10.00</td>
    <td>16.76</td>
  </tr>
  <tr>
    <td>Gemini-Search</td>
    <td>40.00</td>
    <td>32.14</td>
    <td>29.17</td>
    <td>30.00</td>
    <td>23.57</td>
    <td>17.50</td>
    <td>27.94</td>
  </tr>
  <tr>
    <td>ERNIE-4.0-8K</td>
    <td>52.50</td>
    <td>30.00</td>
    <td>28.33</td>
    <td>21.25</td>
    <td>18.57</td>
    <td>30.00</td>
    <td>28.97</td>
  </tr>
  <tr>
    <td>Kimi</td>
    <td>77.50</td>
    <td>41.43</td>
    <td>40.83</td>
    <td>26.25</td>
    <td>26.43</td>
    <td>22.50</td>
    <td>37.35</td>
  </tr>
  <tr>
    <td>Tongyi</td>
    <td>41.25</td>
    <td>45.00</td>
    <td>41.67</td>
    <td>40.00</td>
    <td>41.43</td>
    <td>34.17</td>
    <td>40.73</td>
  </tr>
  <tr>
    <th colspan="8">Open-Sourced Systems</th>
  </tr>
  <tr>
    <td>Naive RAG</td>
    <td>37.50</td>
    <td>25.71</td>
    <td>24.17</td>
    <td>20.00</td>
    <td>14.29</td>
    <td>12.50</td>
    <td>20.73</td>
  </tr>
  <tr>
    <td>MindSearch</td>
    <td>15.00</td>
    <td>11.43</td>
    <td>10.83</td>
    <td>8.75</td>
    <td>12.14</td>
    <td>10.00</td>
    <td>11.32</td>
  </tr>
  <tr>
    <th>Avg.</th>
    <td>37.50</td>
    <td>24.29</td>
    <td>23.42</td>
    <td>19.86</td>
    <td>18.02</td>
    <td>16.48</td>
    <td>-</td>
  </tr>
</table>

Table 4: Accuracy results on Commercial and Open-sourced Searched-enhanced RAG systems.

whether they can retrieve deep information, presented in Table 4.

We first evaluate the performance under Close Book settings using the state-of-the-art model OpenAI o1 (OpenAI, 2024) and Gemini-1.5-Pro without retrieval. We then access the performance of several commercial and open-sourced RAG systems\footnote{The commercial RAG systems are accessed through business-oriented API. The details of RAG systems are provided in Appendix B.}. Without performing the search, even the strongest models exhibit very poor performance. WebWalkerQA is built on official websites with dynamically updated information, while pre-trained models rely on static knowledge limited by a cutoff date and lack dynamic updates\footnote{The case study is shown in Appendix F.2.}. Both commercial and open-sourced RAG systems exhibit relatively poor performance on WebWalkerQA, with the best result coming from Tongyi, which only reaches 40%. Commercial RAG systems are typically modular, consisting of various components such as rewrite, router, reranker, and others.

Some systems, like ERNIE, may have stronger search capabilities for Chinese, resulting in higher values. For open-sourced RAG systems, Multi-source queries have lower accuracy than Single-source queries, which validates the challenge posed by WebWalkerQA, as search engines are unable to retrieve all relevant information in one or several single horizontal search attempts. Furthermore, as the difficulty increases, e.g. the depth of information growing deeper, the performance tends to deteriorate. Overall, search engines still face challenges when retrieving content that is buried deeper.

Findings (i): *RAG systems struggle with key challenges that require effective web traversal.*

6.2 WebWalker Combined with RAG System

![Performance under standard RAG and RAG combined with WebWalker configurations. SS and MS denote single-source and multi-source QAs.](page_1012_1342_410_246.png)

Figure 8: Performance under standard RAG and RAG combined with WebWalker configurations. SS and MS denote single-source and multi-source QAs.

The standard RAG system can be viewed as a *horizontal search* for relevant documents in response to a query, while WebWalker can be considered as a vertical exploration approach. WebWalker can seamlessly integrate into standard RAG systems to acquire deep information and enhance problem-solving capabilities. We integrate WebWalker building upon Qwen-2.5-Plus into the naive RAG system, and the detailed results are shown in Figure 8. The core contribution of WebWalker is providing useful information for question answering; specifically, the memory \( \mathcal{M} \) of the critic
agent is append to the relevant documents to aid in generation. It is observed that, after the integration, performance has improved across all difficulty levels, especially in the multi-source category.

Findings (ii): WebWalker can be a module in agentic RAG system, enabling vertical exploration.

6.3 Scaling Up on Action Count K

![Bar and line chart showing overall performance on WebWalker and RAG combined WebWalker at varying values of K, using Qwen-Plus as backbones.](page_246_573_518_246.png)

Figure 9: Overall performance on WebWalker and RAG combined WebWalker at varying values of \( K \), using Qwen-Plus as backbones.

Previous work (Yue et al., 2024) explored the inference scaling laws for the RAG system by examining the impact of increasing retrieved documents. We scale up the amount of \( K \in \{5, 10, 15, 20, 25\} \) to study the impact of scaling during the inference phase when tracing source information. Figure 9 shows the results of scaling up, where larger values of \( K \) lead to better performance, validating the feasibility of vertical scaling within a certain range.

Findings (iii): Scaling the process of digging through links could represent a potential direction for vertical exploration in RAG systems.

7 Conclusion

We introduce WebWalkerQA, a benchmark for evaluating LLMs’ web traversal abilities in complex, multi-step information-seeking tasks. We also proposed WebWalker, a multi-agent framework that mimics human-like web navigation, combining exploration and critique. Experiments show that WebWalkerQA effectively challenges RAG systems, and combining RAG with WebWalker improves web navigation performance. Our work highlights the importance of deep, vertical exploration in web-based tasks, paving the way for more scalable and reliable LLM-based information retrieval integrated with RAG.

Limitations and Discussion

We discuss the following limitations:
Dataset Size: Due to the complexity of queries in the web-agent domain, similar to benchmarks such as AssistantBench (Yoran et al., 2024) (214) and MMIna (Zhang et al., 2024c) (1,050), GAIA (Mialon et al., 2024) (466), our proposed WebWalkerQA currently comprises 680 high-quality QA pairs. Additionally, we possess a collection of approximately 14k silver QA pairs, which, although not yet carefully human-verified, can serve as supplementary training data to enhance agent performance, leaving room for further exploration.
Multimodal Environment: In this work, we only utilize HTML-DOM to parse clickable buttons. In fact, visual modalities, such as screenshots, can also assist and provide a more intuitive approach (Nguyen et al., 2024; Zhang et al., 2024a; He et al., 2024b). We leave this for future work.
Agent Tuning: WebWalker is driven by prompting without additional training. We can use agent tuning to help LLMs learn web traversal. This involves fine-tuning models with golden trajectories, enabling them to take effective actions for completing information-seeking tasks (Zeng et al., 2024; Chen et al., 2024b; Zhang et al., 2024b; Qiao et al., 2024; Zhu et al., 2024; Wu et al., 2025).
Better Integration with RAG Systems: In §6.2, the root url is provided for the WebWalker to execute. To better integrate with the RAG system, one approach could be to first rewrite the query within the RAG system to refine the search, directing it to the query’s official websites likely to contain relevant information. The WebWalker can then be used to extract useful information. Both the knowledge retrieved from the RAG system and the information mined by the WebWalker can be combined as augmented retrieval knowledge for generation, leading to a better result.

WebWalker can function independently as a web information retrieval assistant for a given webpage or seamlessly integrate with RAG systems to expand their scope. Under the agentic RAG paradigm, the click action proves to be highly effective.
Acknowledgement
The authors would like to thank the anonymous reviewers for their insightful comments. This work is funded by the National Natural Science Foundation of China (Grant No.62176053).

References
Tamer Abuelsaad, Deepak Akkil, Prasenjit Dey, Ashish Jagmohan, Aditya Vempaty, and Ravi Kokku. 2024. Agent-e: From autonomous web navigation to foundational design principles in agentic systems. In NeurIPS 2024 Workshop on Open-World Agents.

Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, and Feng Zhao. 2024a. Mindsearch: Mimicking human minds elicits deep ai searcher. arXiv preprint arXiv:2407.20183.

Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahu Lin, Kai Chen, and Feng Zhao. 2024b. Agent-FLAN: Designing data and methods of effective agent tuning for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9354–9366, Bangkok, Thailand. Association for Computational Linguistics.

Kanzhi Cheng, Qushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. Seecllick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935.

Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Yang Deng, Xuan Zhang, Wenxuan Zhang, Yifei Yuan, See-Kiong Ng, and Tat-Seng Chua. 2024. On the multi-turn instruction following for conversational web agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8795–8812, Bangkok, Thailand. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravanakumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, and et al. 2024. The llama 3 herd of models. CoRR, abs/2407.21783.

Runnan Fang, Xiaobin Wang, Yuan Liang, Shuofei Qiao, Jialong Wu, Zekun Xi, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. 2025. Synworld: Virtual scenario synthesis for agentic action knowledge refinement. Preprint, arXiv:2504.03561.

Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. 2024. Multimodal web navigation with instruction-finetuned foundation models. In The Twelfth International Conference on Learning Representations.

Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024a. WebVoyager: Building an end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6864–6890, Bangkok, Thailand. Association for Computational Linguistics.

Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Hongming Zhang, Tianqing Fang, Zhenzhong Lan, and Dong Yu. 2024b. Openwebvoyager: Building multimodal web agents via iterative real-world exploration, feedback and optimization. arXiv preprint arXiv:2410.19609.

Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2024. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290.

Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramsom, Petko Georgiev, Adam Santoro, and Timothy Lillicrap. 2022. A data-driven approach for learning to control computers. In International Conference on Machine Learning, pages 9466–9482. PMLR.

Iat Long Iong, Xiao Liu, Yuxuan Chen, Hanyu Lai, Shuntian Yao, Pengbo Shen, Hao Yu, Yuxiao Dong, and Jie Tang. 2024. Openwebagent: An open toolkit to enable web agents on large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 72–81.

Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2024. Language models can solve computer tasks. Advances in Neural Information Processing Systems, 36.

Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024a. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. In ICLR 2024 Workshop on Large Language Model (LLM) Agents.

Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. 2024b. Tree search for language model agents. arXiv preprint arXiv:2407.01476.

Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang,
Xiaohan Zhang, Yuxiao Dong, et al. 2024. Autowbglm: A large language model-based web navigating agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5295–5306.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.

Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. 2018. Reinforcement learning on web interfaces using workflow-guided exploration. arXiv preprint arXiv:1802.08802.

Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aoahan Zeng, Zhengxiao Du, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023. Webglm: Towards an efficient web-enhanced question answering system with human preferences. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4549–4560.

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aoahan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2024. Agent-bench: Evaluating LLMs as agents. In The Twelfth International Conference on Learning Representations.

Vaibhav Mavi, Anubhav Jangra, and Adam Jatowt. 2024. Multi-hop question answering. Preprint arXiv:2204.09140.

Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2024. GAIA: a benchmark for general AI assistants. In The Twelfth International Conference on Learning Representations.

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332.

Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyoung Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, et al. 2024. Gui agents: A survey. arXiv preprint arXiv:2412.13501.

Fariba Noori and Mohammad Kazemifar. 2015. Simulation of pair programming using multi-agent and mbti personality model. In 2015 Sixth International Conference of Cognitive Science (ICCS), pages 29–36. IEEE.

OpenAI. 2022a. Gpt-4 system card.

OpenAI. 2022b. Introducing ChatGPT.

OpenAI. 2024. Introducing openai o1.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.

Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Jiang, Chengfei Lv, and Huajun Chen. 2024. AutoAct: Automatic agent learning from scratch for QA via self-planning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3003–3021, Bangkok, Thailand. Association for Computational Linguistics.

Revanth Gangi Reddy, Sagnik Mukherjee, Jeonghwan Kim, Zhenhailong Wang, Dilek Hakkani-Tur, and Heng Ji. 2024. Infogent: An agent-based framework for web information aggregation. arXiv preprint arXiv:2410.19054.

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36.

Yueqi Song, Frank Xu, Shuyan Zhou, and Graham Neubig. 2024. Beyond browsing: Api-based web agents. arXiv preprint arXiv:2410.16464.

Jiejun Tan, Zhicheng Dou, Wen Wang, Mang Wang, Weipeng Chen, and Ji-Rong Wen. 2024. Htmlrag: Html is better than plain text for modeling retrieved knowledge in rag systems. arXiv preprint arXiv:2411.02959.

Qwen Team. 2024. Qwen2.5: A party of foundation models.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Laurie Williams, Robert R Kessler, Ward Cunningham, and Ron Jeffries. 2000. Strengthening the case for pair programming. IEEE software, 17(4):19–25.

Jialong Wu, Baixuan Li, Runnan Fang, Wenbiao Yin, Liwen Zhang, Zhengwei Tao, Dingchu Zhang, Zekun Xi, Yong Jiang, Pengjun Xie, Fei Huang, and Jingren Zhou. 2025. Webdancer: Towards autonomous information seeking agency. Preprint, arXiv:2505.22648.

Zekun Xi, Wenbiao Yin, Jizhan Fang, Jialong Wu, Runnan Fang, Ningyu Zhang, Jiang Yong, Pengjun Xie, Fei Huang, and Huajun Chen. 2025. Omthink: Expanding knowledge boundaries in machine writing through thinking. Preprint, arXiv:2501.09751.
Kevin Xu, Yeganeh Kordi, Tanay Nayak, Ado Asija, Yizhong Wang, Kate Sanders, Adam Byerly, Jingyu Zhang, Benjamin Van Durme, and Daniel Khashabi. 2024. Tur[k]ingbench: A challenge benchmark for web agents. Preprint, arXiv:2403.11905.

Nancy Xu, Sam Masling, Michael Du, Giovanni Campana, Larry Heck, James Landay, and Monica Lam. 2021. Grounding open-domain instructions to automate web support tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1022–1032, Online. Association for Computational Linguistics.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, and et al. 2024. Qwen2 technical report. CoRR, abs/2407.10671.

Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations.

Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. 2024. AssistantBench: Can web agents solve realistic and time-consuming tasks? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8938–8968, Miami, Florida, USA. Association for Computational Linguistics.

Zhenrui Yue, Honglei Zhuang, Aijun Bai, Kai Hui, Rolf Jagerman, Hansi Zeng, Zhen Qin, Dong Wang, Xuan-hui Wang, and Michael Bendersky. 2024. Inference scaling for long-context retrieval augmented generation. arXiv preprint arXiv:2410.04343.

Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2024. AgentTuning: Enabling generalized agent abilities for LLMs. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3053–3077, Bangkok, Thailand. Association for Computational Linguistics.

Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Qingwei Lin, Saravan Rajmohan, et al. 2024a. Large language model-brained gui agents: A survey. arXiv preprint arXiv:2411.18279.

Dingchu Zhang, Yida Zhao, Jialong Wu, Baixuan Li, Wenbiao Yin, Liwen Zhang, Yong Jiang, Yufeng Li, Kewei Tu, Pengjun Xie, and Fei Huang. 2025. Evolvesearch: An iterative self-evolving search agent. Preprint, arXiv:2505.22501.

Wenqi Zhang, Ke Tang, Hai Wu, Mengna Wang, Yongliang Shen, Guiyang Hou, Zeqi Tan, Peng Li, Yueting Zhuang, and Weiming Lu. 2024b. Agentpro: Learning to evolve via policy-level reflection and optimization. arXiv preprint arXiv:2402.17574.

Ziniu Zhang, Shulin Tian, Liangyu Chen, and Ziwei Liu. 2024c. Mmina: Benchmarking multithop multimodal internet agents. Preprint, arXiv:2404.09992.

Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024a. Gpt-4v (ision) is a generalist web agent, if grounded. In Forty-first International Conference on Machine Learning.

Boyuan Zheng, Boyu Gou, Scott Salisbury, Zheng Du, Huan Sun, and Yu Su. 2024b. WebOlympus: An open platform for web agents on live websites. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 187–197, Miami, Florida, USA. Association for Computational Linguistics.

Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abhishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024a. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations.

Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, Jing Chen, Ruipu Wu, Shuai Wang, et al. 2023. Agents: An open-source framework for autonomous language agents. arXiv preprint arXiv:2309.07870.

Wangchunshu Zhou, Yixin Ou, Shengwei Ding, Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang, Xiaohua Xu, Ningyu Zhang, et al. 2024b. Symbolic learning enables self-evolving agents. arXiv preprint arXiv:2406.18532.

Yuqi Zhu, Shuofei Qiao, Yixin Ou, Shumin Deng, Ningyu Zhang, Shiwei Lyu, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen. 2024. Knowagent: Knowledge-augmented planning for llm-based agents. Preprint, arXiv:2403.03101.
A Implementation Details

In this study, we utilize Qwen-Agent\(^{12}\) as the foundational codebase for building and developing the baselines proposed WebWalker. The details of LLM hyperparameters for generation are as follows: \(top_p = 0.8\). We sincerely thank the contributors and maintainers of ai4crawl\(^{13}\) for their open-source tool, which helped us get web pages in a Markdown-like format. We release the code of WebWalker in our GitHub Codebase.

B Details for RAG Systems

We select five mainstream commercial systems and two open-source systems for evaluation.

B.1 Commercial Systems

Doubao\(^{14}\), ERNIE-4.0-8K\(^{15}\), Tongyi, Kimi, and Gemini-Search are all accessed through their business-oriented API interfaces to ensure reproducibility. The detailed configuration of each API can be found in our codebase.

B.2 Open-sourced Systems

(a) Mindsearch (Chen et al., 2024a) is to mimic the human minds in web information seeking and integration, which can be instantiated by a multi-agent framework consisting of a WebPlanner and WebSearcher. (b) Naive RAG built from scratch
We use Google to query the relevant terms and concatenate the information from the Top-10 returned links with the query to provide instructions for the Qwen-Plus to generate a response.

C Annotated Case

An annotated case is shown in Figure 10. The WebWalkerQA dataset is available at HuggingFace Datasets.

D Details on Annotation

D.1 Sources of Root Page

The root page is initially identified through a Google search using keywords such as “conference official website” or “game official website”, followed by manual filtering. For the education domain, we choose the official websites of various university computer science departments, closely

\(^{12}\)https://github.com/QwenLM/Qwen-Agent
\(^{13}\)https://github.com/unclecode/crawl4ai
\(^{14}\)https://www.volcengine.com/docs/82379/1302004
\(^{15}\)https://cloud.baidu.com/doc/WENXINWORKSHOP/s/clntwmv7t

Annotated Data Format

```json
{
    "Question": "When is the paper submission deadline for the ACL 2025 Industry Track, and what is the venue address for the conference?",
    "Answer": "The paper submission deadline for the ACL 2025 Industry Track is March 21, 2025. The conference will be held in Brüne-Kreisky-Platz 1.",
    "Root_Url": "https://2025.aclweb.org/",
    "Info": {
        "Domain": "multi-source",
        "Domain": "Conference",
        "Language": "English",
        "Difficulty_Level": "Medium",
        "Source_Urls": ["https://2025.aclweb.org/calls/industry-track/", "https://2025.aclweb.org/venue/"],
        "Golden_Path": ["root->call>student_research_workshop", "root->venue"]
    }
}
```

Figure 10: A JSON-format case in WebWalkerQA.

reflecting real-world scenarios. The distribution of the domain is shown in Figure 3.

D.2 Details on Prompts for Annotation

The prompts for GPT-4o-based initial annotation are presented below.

<table>
  <tr>
    <th>Prompts for Multi-source Data Annotation</th>
  </tr>
  <tr>
    <td><b>Question Generate</b><br>You are a professional web content analyst. Based on the provided material, construct a query statement:<br>Sublink 1 URL: Sublink 1 INFO<br>Sublink 2 URL: Sublink 2 INFO<br>...<br>Sublink n URL: Sublink n INFO<br>### Requirements:<br>1. **Core Goal of the Query**: Create a multi–step standalone query where the user needs to integrate information from at least two sublinks to find the final answer. The answer should be a single, clear, concise, and precise entity.<br>2. **Relevance of Sublinks**: The selected sublinks must have an intrinsic connection, and the answer should be derived by combining information from these two sublinks.<br>3. **Logical and Complex**: The constructed query should be as complex and specific as possible , challenging, and can leverage time, sequence, or commonly mentioned topics to construct a naturally coherent reasoning process. Avoid questions about browsing history, browsing paths, etc., which have no practical value.<br>4. **Accuracy of the Answer**: Ensure the answer is accurate, concise, and closely connected to the logical chain constructed in the query.<br><br>Please return in JSON format, structured as follows:</td>
  </tr>
</table>
{
    "sublink_reason": "Describe why these specific sublinks were chosen and how they are interrelated.",
    "sublinks": ["Selected sublink URL", "Selected sublink URL"],
    "reason": "Explain the reason for designing this query and how it encourages the user to engage in multi–step reasoning.",
    "query": "Your query statement",
    "answer": "The answer to the query"
}
Sublink 1 URL; Sublink 1 INFO
Sublink 2 URL; Sublink 2 INFO
...
Sublink n URL; Sublink n INFO

Question-Answer Verify
You will act as a strict judge. You need to evaluate whether the given query can be accurately answered only by combining the information from two documents (doc1 and doc2) and the provided answer. Additionally, check if the answer is concise (as an entity or a judgment) and correct.

If the answer is incorrect, can be answered using only one document, or is not concise enough, you should return false.
If any document (doc1 or doc2) does not contain the necessary key information for the answer and only provides context for the query, you should return false.
If any document merely provides query background information unrelated to the answer and does not require combining information from both documents, you should return false.
If the answer is a long answer and not of an entity type, you should return false.
If the query is unnatural, doesn’t appear as a complete query, or has a harsh tone, you should return false.
Each question should require combining information from both documents, meaning the answer results from multi–hop reasoning or multi–step reasoning, and it is concise for you to return true.
You are very strict, and any case failing to meet the above criteria should result in a false. Please return your result in JSON format as follows:
{
    "reason": "Consider each of the conditions above in sequence to assess whether the query and answer meet the criteria. If they do meet the criteria, list the helpful parts from each doc for answering the question.",
    "decision": "true/false"
}
{Doc1 INFO}; {Doc2 INFO}

Prompts for Single-source Data Annotation

Question Generate

Question-Answer Verify
You will act as a strict judge. You need to assess whether current knowledge from doc2 is required to accurately answer the given query based on the two provided documents (doc1 and doc2) and the given answer. Doc1 represents known knowledge, while doc2 represents current knowledge. Your task is to determine if the answer relies on doc2 to be accurately provided. Additionally, evaluate whether the answer is short (an entity or judgment) and correct.

If the answer is incorrect or not concise, return false .
If the necessary key information is found in the known knowledge doc1, also return false.
If the answer is a long answer and not of entity type , return false.
If the query is unnatural, not a complete query, or awkwardly phrased, return false.
The answer should result from multi–hop reasoning or multi–step reasoning, where multi–step reasoning indicates that the generated query is challenging and requires reasoning or calculation to answer, and only if the answer is concise should you return true.
You are extremely strict, and any requirements not met should result in a return of false.

Please return the result in JSON format as follows:
{
    "reason": "Evaluate against the above conditions step by step, considering whether the query and answer meet the conditions. Use English to justify, and if they do, list the sections from doc2 that assist in answering the query.",
    "decision": "true/false"
}

D.3 Details Prompts for Agents
The prompts for the Exploer Agent 🧭 and Critic Agent 🧵 are shown below.

<table>
  <tr>
    <th>Prompts for WebWalker</th>
  </tr>
  <tr>
    <td><b>The Exploer Agent 🧭</b><br>Digging through the buttons to find quality sources and the right information. You have access to the following tools:<br><br>{tool_descs}<br><br>Use the following format:<br><br>Question: the input question you must answer<br>Thought: you should always think about what to do<br>Action: the action to take, should be one of [[ tool_names ]]<br>Action Input: the input to the action<br>Observation: the result of the action ... (this Thought/Action/Action Input/Observation can be repeated zero or more times)<br><br>Begin!<br><br>{query}</td>
  </tr>
  <tr>
    <td><b>The Critic Agent 🧵</b></td>
  </tr>
</table>
Critic
You are a critic agent. Your task is to analyze the given observation and extract information relevant to the current query. You need to decide if the observation contains useful information for the query. If it does, return a JSON object with a "usefulness" value of true and an "information" field with the relevant details. If not, return a JSON object with a "usefulness" value of false.

**Input:**
  – Query: "<Query>"
  – Observation: "<Current Observation>"

**Output (JSON):**
{
  "usefulness": true,
  "information": "<Extracted Useful Information>"
}
Or, if the observation does not contain useful information:
{
  "usefulness": false
}
– Query: {Query}
– Observation: {Observation}

Answer
You are a critic agent. Your task is to evaluate whether the accumulated useful information is sufficient to answer the current query. If it is sufficient, return a JSON object with a "judge" value of true and an "answer" field with the answer. If the information is insufficient, return a JSON object with a "judge" value of false.

**Input:**
  – Query: "<Query>"
  – Accumulated Information: "<Accumulated Useful Information>"

**Output (JSON):**
{
  "judge": true,
  "answer": "<Generated Answer>"
}
Or, if the information is insufficient to answer the query:
{
  "judge": false
}
– Query: {Query}
– Accumulated Information: {Information}

E  Details for Evaluation

E.1  Evaluator

The evaluator prompt is shown in Figure 11.

F  Case Study

F.1  Reasoning Error

As shown in Table 5, this question requires first locating the webpage related to the Inclusive Connections Lounge, followed by a comprehensive understanding of the information on the page to calculate the required time. In such cases, it is also necessary to account for the system’s ability to perform time calculations or reasoning. Consequently,

CoT-QA Evaluator

You are a teacher grading a quiz.
You are given a question, the context the question is about, and the student's answer. You are asked to score the student's answer as either CORRECT or INCORRECT, based on the context.
Write out in a step by step manner your reasoning to be sure that your conclusion is correct. Avoid simply stating the correct answer at the outset.

Example Format:
QUESTION: question here
CONTEXT: context the question is about here
STUDENT ANSWER: student's answer here
EXPLANATION: step by step reasoning here
GRADE: CORRECT or INCORRECT here

Grade the student answers based ONLY on their factual accuracy. Ignore differences in punctuation and phrasing between the student answer and true answer. It is OK if the student answer contains more information than the true answer, as long as it does not contain any conflicting statements. Begin!

QUESTION: {query}
CONTEXT: {answer}
STUDENT ANSWER: {result}
EXPLANATION
GRADE:""

Figure 11: The prompt for evaluation.

even when the source page is successfully located, errors might still occur if the system fails to process the time correctly.

F.2  Time Cut-off

As shown in Table 6, the cutoff date for o1’s temporal data is October 2023, rendering it unable to provide answers regarding web information published beyond this point.
<table>
  <tr>
    <th>Root Url</th>
    <td><a href="https://www.mrs.org/">https://www.mrs.org/</a></td>
  </tr>
  <tr>
    <th>Question</th>
    <td>How many hours in total would a person spend if they attended the <b>Inclusive Connections Lounge</b> activities from December 1 to 6, 2024, at the MRS Fall Meeting?</td>
  </tr>
  <tr>
    <th>Answer</th>
    <td>66 hours</td>
  </tr>
  <tr>
    <th>Source Website</th>
    <td><a href="https://www.mrs.org/meetings-events/annual-meetings/2024-mrs-fall-meeting/meeting-events/broadening-participation/inclusive-connections-lounge">https://www.mrs.org/meetings-events/annual-meetings/2024-mrs-fall-meeting/meeting-events/broadening-participation/inclusive-connections-lounge</a></td>
  </tr>
</table>

Website Information

Table 5: The case requiring reasoning capability in web traversal task.

<table>
  <tr>
    <th>Question</th>
    <td>Where and when will the <b>2025</b> MRS Fall Meeting take place?</td>
  </tr>
  <tr>
    <th>Answer</th>
    <td>Boston, Massachusetts: November 30 to December 5, 2025.</td>
  </tr>
  <tr>
    <th>Prediction</th>
    <td>As of my knowledge cutoff in <b>October 2023</b>, the MRS has not yet announced the exact dates or location for the 2025 MRS Fall Meeting.</td>
  </tr>
</table>

Table 6: The case of time cutoff in predictions generated by o1.

Reviewer paper summaries: This paper introduces WebWalkerQA, a benchmark designed to evaluate the ability of large language models (LLMs) to perform systematic web traversal, enabling them to extract high-quality, complex information from website subpages. The authors propose WebWalker, a multi-agent framework employing an explore-critic paradigm that mimics human-like web navigation. Experimental results demonstrate that WebWalkerQA poses significant challenges and highlights the effectiveness of combining RAG with WebWalker in handling real-world, multi-layered information tasks.

This paper introduces WebWalkerQA, a dataset designed to assess the ability of LLMs to perform human-like web traversal for answering questions that require web searching. Additionally, the authors propose WebWalker, a multi-agent framework that simulates human-like navigation to explore websites and retrieve relevant information. Extensive experiments and analyses highlight the challenges of the WebWalkerQA dataset and demonstrate the effectiveness of the multi-agent framework.
Summary of strengths: - Handling real-world, multi-layered information tasks is highly complex and reqiured task, the paper proposed a timely benchmark and multi-agent framework.
- The paper introduces WebWalkerQA to evaluate LLMs' web traversal capabilities, addressing multi-layered information retrieval challengesm, which is novel.
- The proposed WebWalker employs an explore-critic paradigm that mimics human-like web navigation, which is technical sound.

1. This paper introduces an interesting new task called Web Traversal, which mimics human-like web navigation for information-seeking. The WebWalkerQA dataset is carefully curated, high-quality, and presents varying levels of difficulty. This dataset has the potential to be highly valuable to the NLP community.
2. The proposed multi-agent framework, WebWalker, is effective and can serve as a baseline or benchmark for future research.
3. The experiments are robust, involving a range of both open-sourced and closed-sourced models. Additionally, the extensive analysis offers valuable insights.
Summary of weaknesses: - The main contribution is the proposing a benchmark and a multi-agent framework, however, neither the WebWalkerQA nor WebWalker are released, or not even promise to release, which concerns me.
- This reserach require human effort to annotate the data, however, the human annotation details are missing.
- Thes is limited exploration of reasoning errors and potential solutions to address them. Some case study will be helpful.

As a dataset and resource paper, I do not identify any major weaknesses. However, the paper could be improved by providing more detailed information about the data sources. While the paper mentions the domains (conference, education, organization, and game), it lacks specifics on the exact data sources used.
Comments, suggestions and typos: - Increase the dataset size and include a wider range of domains and languages to ensure robustness and generalizability.
- Release code and datasets to ensure reproducibility.

Typo:

1. L079, “Yao et al.” wrong citation format, missing year. Similarly, L132, L135, L155, L164, L348, etc.
2. L246, “Table 1” → “Table 2”
Overall assessment scores: 3

3.5