Self-Guided Hierarchical Exploration for Generalist Foundation Model Web Agents
Authors: Qianlan Yang
Xiangjun Wang
Danielle Perszyk
Yu-Xiong Wang
Venue: NeurIPS 2025 poster ()
Keywords: Web Agent
Foundation Models
VLM
Reinforcement Learning
Primary area: deep_learning
TL;DR: We propose SAGE, a self-guided hierarchical exploration framework that trains generalist foundation model web agents without human supervision, achieving state-of-the-art performance

Abstract: Foundation models have recently shown strong potential as web agents, capable of interpreting high-level instructions and interacting with complex web interfaces. However, existing training paradigms for these agents often rely on predefined task datasets and curated demonstrations, limiting their scalability, adaptability, and capacity for self-improvement. In this work, we introduce *Self-guided hierArchical exploration for Generalist wEb agents* (SAGE), a new training framework designed to support autonomous skill acquisition through self-guided hierarchical exploration. Our method introduces a three-tier exploration strategy: a pre-exploration phase to build structural understanding of web environments, a top-level exploration strategy to generate a self-evolving curriculum of tasks from easy to hard, and a low-level exploration mechanism that combines planning-based rollouts with step-wise learning to improve policy efficiency. Together, these components form a scalable, supervision-free framework for web agent training. Experimental results on WebVoyager and WebArena demonstrate that our method significantly outperforms prior approaches, enabling foundation model agents to learn complex web tasks with greater generalization and robustness.

Self-Guided Hierarchical Exploration for Generalist Foundation Model Web Agents
Qianlan Yang University of Illinois Urbana Champaign qianlan2011linois edu
Xiangjun Wang Amazon xj Camazon com
Danielle Perszyk
Amazon
perszdan@amazon.coom com
Yu-Xiong Wang University of Illinois Urbana Champaign yxweillinois.ed.
Abstract
Foundation models have recently shown strong potential as web agents, capable interpreting high-level instructions and interacting with complex web interfaces. However, existing training paradigms for these agents often rely on predefined task datasets and curated demonstrations, limiting their scalability, adaptability, and capacity for self-improvement. In this work, we introduce Self-guided hierArchical exploration for Generalist wFth agents (SAGE), to support autonomous skill acquisition through self-guided hierarchical explo- ration. Our method introduces three-tier exploration strategy: pre-exploration phase to build structural understanding of web environments, a top-level explo- ration strategy to generate self-evolving curriculum of tasks from easy hard, and low-level exploration mechanism that combines planning-base with step-wise learning to improve policy efficiency. Together, these components form scalable, supervision-free framework for web agent training. Experimental results on Web Voyager and WebArena demonstrate that our method significantly outperforms prior approaches, enabling foundation model agents to learn complex web tasks with greater generalization and robustness. Our project can be found https://yanqval. http:://yangall.giuubi////AAAA github.io/SAGEE.
training framework designed
rollouts
Introduction
Foundation models have demonstrated impressive generalization across domains such as natural language processing, code generation, and multimodal reasoning [1-3]. With their strong capabilities understanding and generating structured content, foundation models, particularly laagun-language models (VLMs) have recently shown promise web agents, capable of interpreting high-level instructions and interacting with visual user interfaces [4, 5]. These agents have the potential to automate real-wort world tasks the web, such as booking tickets, extracting information, and navigating software systems, without nomain-speciff engineering or fine-tuning for each application.
Despite this promise, existing approaches to training foundation model web agents face significant limitations. Most methods rely on static datasets composed of numan-authores tasks and curated expert demonstrations [6-9], which are expensive to collect and fail to capture the long-tail complexity of real web interactions. Furthermore, these methods often lack mechanisms for continuous self- improvement, making it difficult for agents to scale their capabilities beyond the data they were trained on.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

State
Task Generator
Task Buffer
Action
Action
Action
Pre-Explorer
State 2
Stake
State 4
Task 1. Find mes's wallet with RFID blocking, under $00.
Check avsilable (Kttl-dhliwry.
Taakk Pind men's wallet with RPID blecking,
Action 1
Action
Taak Filter olt mon's wallet under 5S0
Bollout
Check the Zun availeble for FREE-dddivrry
State
State 6
State n1
Rallott
Palli
Trajectories
Agent
State +
Soccess!
Figure 1: Framework of SAGE. Left: In the pre-exploration phase, the agent automatically collects structural information from the environment. Middle: In the top-level exploration phase, the task generator proposes new tasks based on the agent's performance in the previous iteration. Right: In the low-level exploration phase, the agent uses MCTS explore and solve individual tasks efficiently, We also provided case study to show SAGE works in Appendix Section F.
To address these challenges, we propose a new training framework for generalist foundation model web agents that emphasizes self-guided exploration as the core learning principle. Rather than depending on fixed task templates or expert demonstrations, our framework enables agents to autonomously explore real-world websites, generate their own instructional goals, and acquire skills through structured interaction. At the heart of our framework is a three-stuge exploration hierarchy that supports curriculum-driven learning from the ground up.
As illustrated Figure. 1, we begin with pre-exploration phase, in which the agent autonomously traverses the structure of the website before training begins. This phase allows the agent to construct semantic map of the environment, by capture page layouts, interactive elements, and navigation paths, which serves as the foundation for later skill generation. Next, we introduce a top-level exploration strategy that governs how instructional tasks are proposed and evolved throughout training. By analyzing past performance, this component adaptively generates new tasks from easy to hard by composing or decomposing instruction templates, thus enabling a adaptive learning curriculum. Finally, we develop a low-level exploration stralegy to enhance the efficiency of within-task learning. Using Monte Carlo Tree Search (MCTS) [10] and a multi-action agent policy, agent explores promising action sequences for each task and refines its behavior through structured rollout and preference- learning.
Together, these components form an end-to-end system Self-guided hierArchical exploration for Generalist wEb agents (SAGE), in which agents progressively discover how to interact with and reason about complex web environments. Our experiments demonstrate that this adaptive exploration framework leads to stronger generalization and performance compared prior approaches that rely on static task definitions or imitation data. By enabling agents to learn from their own interaction cycles, our method takes a step toward scalable, self-improving web agents grounded foundation model capabilities.
Our Contributions are three-fold. (i) We propose a top-level exploration strategy that enables agents to autonomously generate and adapt instructional tasks from easy to hard, forming a self-evolving curriculum without relying on predefined task sets. (ii) We introduce low-level exploration strategy that combines MCTS- planning and stepwise learning, allowing agents to efficiently discover viable action sequences and improve under sparse rewards. (iii) We unify all components into SAGE, generalist web agent training framework that leverages multi-level exploration to progressively acquire skills and generalize across complex, real-world internet environments. As demonstrated by our empirical results, SAGE achieves state-of-the-an performance on web-based tasks. On WebArena [1]] []]] it outperforms all open-source baselines by 26%, and even surpasses proprietary models by 11% %.

---

2 Related Work
Foundation Model Web/GUI Agents. The emergence of Large Language Models (LLMs) and LasguuLa Models (VLMs) has spurred the development of agents capable of interacting with structured interfaces such as web pages, software GUIs, and mobile applications [1, 4, 12, 13]. Prior approaches generally fall into two groups: training-free systems that rely on prompt engineering and orchestration of frozen models [3, 5, 14], and fine-tuning-hasee muning-hased methods that adapt model parameters using supervised trajectories [6-9]. While training-free systems offer flexibility, they are often bounded by the static capabilities of the underlying models. On the other hand, supervised fine-tuning improves task-specific performance but typically depends on curated task datasets and lacks adaptability. To mitigate this, recent methods introduce automated evaluators that estimate task completion as a reward signal [15, 16], or propose instruction-bass self-supervision [17]. However, most of these pipelines still rely on static method introduces a structured, hierarchical exploration framework that adaptively generates and organizes tasks from easy hard, enabling agents to acquire skills and improve autonomously without relying on static benchmarks or curated supervision.
manchmark-definee instruction pools. In contrast. our
Reinforcement Learning for LLMs. Reinforcement learning (RL) has increasingly been applied to align LLM behaviors with downstream tasks, especially in domains that require sequential decisions and delayed feedback. While RL was initially used to fine-tune models based on human preferences or scalar rewards, recent work has shown that it can be extended to environments with interface- interactions [13, 18]. Actor-criti methods and policy optimization frameworks such as Proximal Policy Optimization [19] and Direct Preference Optimization (DPO) [20] have proven effective for training agents to operate
complex action spaces. However, most existing methods assume fixed task distribution and lack a mechanism for adaptively shaping the learning trajectory. Our work complements this line by introducing a self-adaptive task curriculum that evolves in response the agent's 's learning progress, enabling continual policy improvement under a changing task landscape.
Instruction Generation and Skill Discovery. Generating training signals autonomously has emerged as a key mechanism for improving the scalability and generality of foundation model agents. In- struction generation approaches such Self-Instruct [21] and Meta- Prot-prropting [22] enable models to bootstrap themselves by producing and solving their own tasks, while others fine-tune models in simulated planning environments [23]. Parallel work in unsupervised reinforcement learning has focused on discovering diverse skills without access to task labels [24, 25], but these approaches often prioritize behavioral diversity over goal-directed utility. In the domain of web agents, recent work PAE [26] attempts to train agents via instruction sets either statically defined or updated from interaction history, but still relies initial human-written instructions define the scope of training. Our method departs from this design by constructing a fully self-evolving task curriculum that requires no external instruction corpus and enables agents grow their capabilities from primitive operations to compound, high-level behaviors through direct interaction.
3 Background
Problem fomulation. We formalize the training of generalist web agents contextual Markov Decision Process (MDP) defined by the tuple M = (S,A, T,R, R. H, I), where S is the state space, A the action space, T the environment dynamics, R the reward function, H the episode horizon, and the task instruction space.
At the start of each episode, task instruction 1 E I is compleddttypceee command (c.g.. "add the product to the cart"). The agent begins interacting with a web environment and receives state 8t E S at each timestep & E {1, ... H) }. comprising the current webpage content and interaction history. Conditioned on 8 and I, the agent selects an action a E A, such as clicking or typing. The environment transitions to St+1 T( St, az), and the process continues.
natural language
The agent follows a policy (a 8c, I), parameterized by 0, and receives a scalar reward at the end of the episode indicating task success. Crucially, the true instruction distribution I is unknown during training, and the agent must learn from proxy tasks or self- self-generated curricula. The objective is to maximize expected return over task and trajectory distributions:
3

---

max I (11.01...... y
r Ce. 1)
0
Multimodal Environments. To enable lightweight and efficient training for interactive web agents, we build upon the web environment developed in PAE [17]. which integrates both visual and structural information. the original PAE setup, agents receive screenshot of the current webpage annotated with set-of-markss marks that help the agent precisely identify interactive elements. Each marked element is paired with textual description to provide semantic grounding for action decisions.
We further extend this environment by incorporating the webpage's accessibility tree as an additional input modality. The accessibility tree captures the full structure of all interactive elements in hierarchical form, including those not explicitly annotated in the screenshot. To bridge the visual and structural modalities, we establish one-to-one alignment between the set-of-marks labels and corresponding nodes in the accessibility tree. This alignment allows the agent to use consistent numerical identifiers to reference web elements across both visual and semantic representations, enabling seamless cross-modal reasoning and interaction. We provided a more detailed setting of our environment in Appendix Section B.
4 Method
In this section, we propose SAGE, a new training framework for developing generalist foundation model-based web agents that can adaptively explore and interact with real-world Internet environ- ments, while continuously self-improving in a self-guided manner. Our objective is to enable agents not only to complete individual tasks but also to continually acquire new skills through experi- ence- stae-sstaiing from simple goals and gradually advancing more complex, real-world objectives. At the core of our method is hierachical exploration framework where the agent is self- self-guided in understanding its environment, generating meaningful tasks for practice, and discovering effective strategies for solving them.
To support this process, we introduce three levels of exploration, each operating at different func- tional scope: (1) a pre-exploration strategy, which enables the agent to build an initial understanding of the workspace and identify the interactive structure of the environment; (2) top-level exploration strategy, which helps the agent discover and organize tasks from easy to hard, establishing structured path for learning and skill acquisition; and (3) low-level exploration strategy, which allows the agent to efficiently explore fine-grained actions within a specific task, thereby enhancing its ability to solve concrete objectives. Finally, we conclude this section with an overview of how these components integrate into unified framework for scalable skill learning and generalization. The entire framework of SAGE is illustrated in Figure I. We also provided additional algorithm details in Appendix Section A.
4.1 Pre-Exploration Strategy
To enable the agent to develop an initial understanding of website's structure and functionality prior to training, we introduce a pre-exploration phase. The goal of this phase is to allow the agent to autonomously investigate the interactive landscape of the invirommnntiidenifying key elements, navigation paths, and potential functionality-as that it may later generate and reason about feasible tasks relevant to the site.
In this stage, the agent receives only the URL of the website's 's homepage as input. Without any task-speeific instruction or supervision, the agent begins exploring the site from this entry point using a Breadth-Fin First Search (BFS) strategy. We maintain a queue of states, where each state corresponds to unique URL discovered during the exploration. Initially, the queue contains only the homepage URL. At each step, the agent dequeues a state and decides whether the page warrants further action, especially it has been previously visited. If so, the agent selects up to actions that are likely to lead to unexplored semantically distinct pages. These actions are executed, and the resulting new URLS are enqueued as candidate states for further exploration. The process continues until the queue is empty predefined limit on the total number of states is reached.

---

Throughout this exploration, the agent logs detailed information about the discovered environment. Specifically, records: (I) the screenshot of each visited state; (2) structural transitions between states, described in a textual format such as "page A transitions to page B via action click [Search]" "; and (3) page-specific textual metadata, such as product categories on commercial sites or section labels on documentation pages. These collected data form knowledge base of the environment, which will be used by subsequent modules to generate tasks and guide agent learning.
4.2 Top-Level Exploration Strategy
To guide the agent toward learning complex tasks in a structured and feasible manner, we introduce a top-level exploration strategy that selects and adapts tasks dynamically across training iterations. This module is composed of a self-adaptive task generator and task replay buffer, working together to provide an evolving curriculum that matches the agent's ' current capabilities.
During the initial training iteration, the task generator relies exclusively on the knowledge acquired in the pre-exploration phase. It samples tasks grounded in the observed website structure and content, such retrieving product information, locating links, or filling out forms. In subsequent iterations, the generator is conditioned not only on the pre-exploration information but also on the agent's s past execution trajectories, partieularly their success and failure patterns. Based on this history, the generator proposes new tasks using three complementary strategies.
First, similar to the initial iteration, the generator continues to introduce new tasks derived from the structural information collected during gre-exploration, expanding the task space beyond previously attempted goals. Second, the generator identifies tasks with low success rates, typically those that the agent struggles to complete and produces simplified variants to help the agent learn more incrementally. For example, complex instruction like "What is the price of the table which I bought yesterday?" "" may be decomposed into subtasks such as "What is the table bought yesterday?" and "What the price of this given item?". In other cases, the generator may relax constraints from difficult task to form a more tractable one. For instance, simplifying "Find a recipe of apple pie which needs less than 30 minutes, has more than 4 stars, and at least 50 likes" to "Find recipe of apple pie which needs less than 30 minutes and at least 50 likes." " Third, for tasks that the agent consistently solves correctly, the generator increases difficulty by proposing more challenging variants. These may be constructed by composing simpler instructions into compound task, or by tightening the conditions in the original query, thereby advancing the curriculum.
Task Buffer. To store and manage the generated tasks across iterations, we employ a double-buffer replay system. The first buffer contains only tasks proposed during the initial iteration, preserving a stable sample of the original task distribution. The second buffer stores tasks generated in later iterations as part of the self-adaptive curriculum. At training time, tasks are sampled from both buffers: portion is drawn from the initial buffer to retain exposure to foundational skills, while the remainder is drawn from the adaptive buffer to promote progressive learning. As new tasks are generated, they randomly replace entries in the second buffer, ensuring freshness while bounding the total number of tasks considered in any given iteration.
This replay mechanism balances the agent's exposure to both early-stage and evolving tasks, ensuring that its foundational abilities retained while new capabilities are continually acquired. By dynamically adjusting task difficulty based on performance feedback, this top-level exploration strategy enables the agent to follow self-paced learning trajectory, from mastering basic interactions to solving complex, real-world tasks.
4.3 Low-Level Exploration Strategy
To enable efficient decision- decision-making during the execution of individual tasks, we introduce a low-level exploration strategy based on Monte Carlo Tree Search (MCTS) [10]. This component improves the agent's 's ability to explore diverse action sequences, reason over possible outcomes, and prioritize high-potential decisions, which is particularly important in sparse-reward environments with long task horizons.
Instead of rolling out trajectories linearly from the initial state to the end of an episode, the agent performs structured exploration using MCTS. Each search begins with a root node representing the task's 's starting state, which is typically defined by the entry URL. At each iteration, the agent traverses
5

---

the tree by selecting child nodes estimated to maximize the likelihood of task success, continuing until a leaf node is reached. This leaf is then expanded using up to k candidate actions proposed by the agent, resulting in k new states. Based on the estimated success probability, the most promising of thes selected to grow the tree. From the final candidate node, the agent conducts a standard rollout using the remaining step budget of the episode. The outcome of this rollout (success or failure) is recorded and backpropagated through the tree to refine future selection decisions. This iterative process enables the agent to concentrate exploration on high-potential trajectories, improving task success rates while reducing inefficient exploration.
Multi-Action Agent Policy. To support both based planning and linear rollouts, we design multi-action agent policy. Given a single state, the policy outputs up to k candidate actions, each associated with confidence score. These candidates are used differently depending on context. During MCTS-based rollout, all candidate actions are treated equally, regardless of their confidence, to encourage broader exploration. During standard execution (i.e., non-tree rollout), the agent selects a single action according to the highest-confidence prediction. This dual-mode usage enables the policy to be both exploratory and decisive, depending on the operational stage.
Training-Baa -aaeed Outcome Evaluator. We adopt a sparse reward structure in which the agent receives a binary reward only at the end of trajectory, indicating whether the task was successfully completed. To evaluate this outcome reliably, we introduce learned outcome evaluator. Similar to prior approaches [17, 15]. the evaluator takes as input the task instruction, the agent's final output, and the history of the last three steps leading to the final state. This condensed representation captures the trajectories
agent's s recent reasoning and actions. The evaluator is trained using both and a limited number of human-and annootated examples, allowing it to generalize effectively across a wide range of task formats. A well-trained evaluator improves the accuracy of success detection and serves as reliable signal for downstream learning.
Stepwise Learning Strategy. To update the agent's policy based on observed success and failure cases, adopt learning approach inspired by the stepwise variant of Direct Preference Optimization (Step-DPO) [20, 27]. Given pair of trajectories, one successful and one failed, we identify the divergence point at state Sk, where the two trajectories share common prefix up to Sk-1. The policy is then trained to prefer the successful continuation from that point onward. Our use of MCTS facilitates the collection of diverse trajectory pairs by sampling multiple decision paths during tree expansion and rollout. This provides ample training signal for the Step-DPO objective and enables the agent to improve policy performance through graindd localized credit assignment.
Together, the low-level MCTS planner, the multi-action policy, the outcome evaluator, and the preference-based learning strategy form a coherent execution framework. These components enable the agent to efficiently explore within-task decisions, accurately evaluate outcomes, and learn from partial successes and failures, thereby accelerating the acquisition of robust web-interaction skills.
4.4 Overall Framework
Inspired by prior work such as PAE [17], we adopt a modular training pipeline that integrates the components described above into a unified system. While PAE establishes general iterative self- training structure, it relies on static instruction distribution without mechanisms for adapting task difficulty or exploration strategy. Building upon this paradigm, our framework introduces two key modificationss First, we explicitly incorporate a pre-exploration phase before training begins, allowing the agent to construct structured understanding of the website environment. Second, we build self-guided learning loop that interleaves curriculum-drivve task generation, efficient planning- based exccution, and preference-based policy updates, enabling continuous adaptation and improvement.
Each training iteration proceeds as follows. We begin by using the top-level exploration strategy to propose new tasks based on the agent's current performance and the structural knowledge obtained from the pre-exploration phase. These tasks are added the replay buffer, which maintains a mix of early-stage and adaptively generated goals. Next, a batch of tasks is sampled from the replay buffer, and the agent executes them using the low-level exploration strategy. This execution involves MCssd planning guided by the multi-action policy, as well as evaluation using the learned outcome model. Finally, the collected trajectories from these rollouts are used to update the policy via a stepwise learning algorithm inspired by Step-DPO. This cycle is repeated across iterations,

---

Allrecipes
ArXWv
GarHuu
Preprietary
Scanct
Quen2.5V1-
2.3
Open source
Qwen2-5VL_32BB
LLaYa-78
2.3
0
18.4
17.5
6.2
0
4,7
0
0.0
0.0
7.1
LLaVa.341
0
2.3
0
2.4
0
0.0
LLaNa-78 PAF
14.3
37.5
17.5
19.0
14.6
0.0
35.3
LLaVa-7B SAGE
16.6
32.9
22.5
21.2
15.3
6.3
36.2
22.7
Fine-Tuning
Qwen2: 5V1-7B
28.7
25.3
17.9
Qwen2.5V VVL SAGE:
34.5
40.8
24.7
28.9
20.4
12.2
40.3
Quen2 5VL PAE
40.3
59.5
45.1
30.3
39.8
16.4
49.7
VL- SAGE
4366
64.7
51.
35,0
56.2
18.2
62.4
Cambridge Dicsionary
BBC News
Google Mips
Google Search
HugzingFace
Wolfiam Alpha
Avenie
Proprictary
Clunde Sounet
86.0
30.2
44.2
66.7
Qwen2.5V SVL-7B
Open sourcce
Qwen2.5N1_-23
LLaVa:7B
0
0
4.6
12.6
0.0
LLaVa-34B
0
2.3
2.3
0.9
LLaVa-7B PAE
52.4
18.6
22.5
23.3
19.0
24.4
22.3
L1a99-7B SAGIF
68.0
34.1
24.3
25.0
21.5
24.4
26,0
LLaVa=34BB PAE
74.4
39.0
22.0
18.5
25.6
42.9
33.0
Tuning
Qwen2
71.3
Qwen2 SV1 SAGIE
12.7
22.1
Awenn.5VVL:3328 PAE
774
30.1
26-4
30.9
55.8
40.8
Qwen2.5VL SVL-32B SAGE
83.6
42.8
34.1
37.1
34,8
64.5
47,5
Table I: Results on Web Voyager. Success rates reported in the table reflect full task completion rather than individual step accuracy. Each row corresponds to specific website, and the Average column indicates the overall success rate across all tasks. The results demonstrate that SAGE consistently improves agent performance across diverse open-internet web environments.
enabling the agent to continuously refine its skills, explore tasks, and improve performance in self-guided manner.
5 Experiments
We design a series of experiments to evaluate the effectiveness of our adaptive exploration framework for training generalist foundation model web agents. Specifically, we aim answer the following key questions: (1) Can the agent acquire meaningful web interaction skills from scratch, without any aanttan-d demonstrations task supervision? (2) How effectively
the agent solve complex, multi-step tasks that require conditional reasoning? (3) What is the contribution of each component in our framework to the overall performance?
(4) How does the hierarchical exploration strategy compriinn pre-exploration, top-level task generation, and low-level rollout-impact the agent's ability learn and generalize web tasks? We leave some additional experiments in Appendix Section C.
5.1 Environments
We evaluate on two realistic web-based benchmarks: Web Voyager [5] and WehArena [11]-
Web Voyager consists of 643 instructional tasks spanning 15 real-world websites, including plat- forms such as Amazon and GitHub. Following the setup in PAE [17], we exclude two domains Google Flights and Google Bookings due major structural changes that render the original tasks unachievable. As a result, we evaluate on the remaining 13 websites, comprising a total of 557 tasks.
WebArena is a closed-domaie benchmark hosted on internal servers, featuring five realistic web environments: OpenStreetMap (Map). Reddit, GitLab, a content management system (CMS) for an online store, and OneStopShop (OSS). While WebArena originally included 812 tasks, many are unsuitable training autonomous web agents due to execution constraints. For a fair comparison, we follow existing practice and adopt a combined task set drawn from WebArena-Fasy [1 [17] and WebArena-LLit [28] as our evaluation suite.
5.2 Baseline Comparisons
In this paper, we compare our method against two major categories of baseline models: (1) pro- prietary VLMs and (2) state-of-the-art open-source VLMs. For the first category, we use Claude 3.5 Sonnet [29] as a representative proprietary model. It prompted using the same setup as prior work [5], including met-of-marks augmented screenshots and chain-of thouthttt action outputs.

---

For the open-source category, we evaluate several strong VLMs, including LLa Qwen2.5VL-77B3221 [30, 311. Within this category, we include three training variants: (1) SFT- based models, which are fine-tuned on trajectories generated by Claude 3 Sonnet over tasks proposed in PAE [17]: (2) models trained using the PAT framework [17]: and (3) models trained with our proposed method, which introduces multi-level adaptive exploration and self-supervised learning without relying on pre-defined instruction sets. This comparison allows us to assess the contributions of our exploration-driven framework relative to imitation-based and instruction-guide baselines across both closed- and open-domain web interaction tasks.
5.3 Main Results
As shown in the main experimental results presented in Tables and 2, our method consistently outperforms strong baselines across both Web and WebArena. On WebVoyagee SAGE achieves a notable performance improvement of 8%-17% over PAE when using the same underlying vision-language model, demonstrating the benefit of our adaptive exploration framework in open- domain, real-world websites. In WebArena featuring server-hosted, closed-domale environments with more structured and rigid Interfaces-SAe continues to deliver strong gains, outperforming PAE by 19%-256% across multiple model backbones.
In addition to outperforming PAE, we observe that SAGE consistently im- proves the performance of base mod- els relative to their original, untrained versions. Notably, Qwen2VL-72B trained under our framework matches or even exceeds the performance of the proprietary Claude 3.5 Sonnet model in several benchmark scenar- ios. This finding underscores the po- tential of large open-source VLMs when coupled with structured, self- supervised training pipelines such as ours- the scalability and accessibility of our approach across different model sizes and training setups.
Map
Reddit
Gitlab
CMS
Avg
Praprictary
Claude 3.5 Sonnet
40.1
46.9
42.0
24.2
25.1
35,6
1.LaVa-7B LL4Va-34B
0.0
0.0
0.00
0.0
0.0
0.0
1.3
0.0
0.0
0.0
0.0
0.6
Open source
Qwen 2.5VL-7B
0.9
10.2
20.2
833
7.4
9,4
Qwen 2.5VL-5228
13.4
18.4
26.4
11.6
8.3
18.6
Qwee 2.5VL-7B PALE
23.8
24.6
37.4
15.3
12.4
22.7
Fine- Fime-Taming
Owen 2.5VL- SAGE
28.1
29.7
43.2
18.9
15.6
27.1
Qwen 2.5V1L-32R PAE
34.5
41.0
22.0
31.5
Qwen 2.5VL-32B SAGE
42.3
43.7
46.1
31.4
35.2
39.7
Table 2: Results on WebArena. Success rates indicate full task completion across five sandboxed, server-hosted environments. Each column represents a distinct website domain, and the Avg column reports the weighted average success rate based on the num- ber of tasks per domain. The results show that SAGE significantly improves performance across all model variants.
To further analyze model behavior across varying levels of task complexity, we categorize the tasks in WebArena into three difficulty levels based on the minimum number of steps required for completion, as determined by human annotators. Specifically, tasks requiring fewer than 7 steps are labeled Easy (25% of all tasks), those requiring between and 15 steps are labeled Medium (54%), and those with more than 15 steps are categorized as Hard (21%). We evaluate model performance separately within each bucket to assess their ability to generalize to long-horizon tasks.
As shown in Table 3, performance gener- ally declines as task complexity increases across all models. However, the relative improvement of SAGE over baselines be- comes more pronounced on harder tasks. trained with our method achieves comparable perfor- mance to Claude 3.5 Sonnet on Hard tasks and significantly outperforms other base- lines. These findings highlight the effee- tiveness of our hierarchical exploration framework, particularly the curriculum- aware task proposer and MCTS -guided low-level strategy- in enabling agents to handle complex, multi-step reasoning tasks that are typically beyond the reach of imitation-bas based fine-tuning.
Easy
Medium
Hard
Avg
Proprictary
Claude
Sonnet
30,0
15.0
35.6
Qwen2.5VVL 5VL-7B PAE
40.0
21.6
5.0
22.7
Fine-Tuning
Qwen2.5VL-7B SAGE
27.2
9.0
27.1
Qwen2. SVL-32B PAE
48.0
32.6
9.0
31.5
For instance, Qwen2.5VLL-33
Qwen2.5V1 SAGE
52.0
43.6
15.0
39.7
Table 3: Success rate comparisons by task difficulty. Tasks are grouped into Easy (<7 steps), Medium (7-15 steps), and Hard (>15 steps), covering 25%. 54%, and 21% of the We- bArena task set, respectively. SAGE consistently improves performance across all difficulty levels, with especially strong gains harder, long- long-horizon tasks.
8

---

5,4 Ablation Studies
In this section, we conduct several ablation studies to show the effectiveness of different components of our SAGE.
Impact of the Top-Level Exploration Strategy. To assess the importance of our self-adaptive task proposer, we conduct an ablation study in which it is replaced by static task proposer that samples from fixed distribution derived solely from the pre-exploration phase. In this setting, the agent receives tasks drawn from the same initial distribution across all training iterations, without any adjustment based on its evolving performance. While this ablated version still benefits from task diversity due to the breadth of information collected during pre-exploration, it lacks the ability to gradually increase task complexity or respond to the agent's learning progress.
As shown in Table 4 removing the self-adaptive mechanism results in substantial drop in performance. This highlights the importance of dynamic task generation in driving curriculum progression. The ability to promote simpler tasks when the agent strug- gles and introduce more challenging ones as it improves proves essential for effective skill acquisition in com- plex web environments.
Model
Map
Reddit
OSS
Gitlab
CMS
Avg
Qwen 2.5VL-7B SAGE w/o top
25.6
25.3
40.1
17.5
14.2
24.5
Qwen L-5 SAGE
28.1
29.7
43.2
18.9
15.6
27.1
Qwen 2.5V1.-328 SAGE w/o top
40.5
37.1
42.9
28.6
28.4
35,5
Qwen 2.5V SAGE
42.3
43.7
31.4
35.2
39.7
Table 4: Ablation study on the top-level exploration strategy. We compare performance with and without the self-adaptive task proposer on WebArena. The results show that removing top-level exploration-i.e., replacing the adaptive task curriculum with static task distribution Iaaiiiioon--addddddee to consistent drop in performance across all domains and model sizes. This highlights the importance of self-adape adaptivd task generation in driving skill acquisition.
Impact of the Low-Level Explo- ration Strategy. We further evaluate
the contribution of our low-level exploration strategy by replacing both the MCTS-based rollout and the step-wise learning algorithm with a conventional learning pipeline. In this ablation, the agent samples task instructions uniformly from the task set and performs standard rollouts without search- planning. The collected trajectories are then filtered based on task success, and a behavior cloning objective applied to fine-tune the agent using only successful trajectories.
As shown in Table 5, removing the low-level exploration module leads to a clear drop in both sample efficiency and final task success rate. This result underscores the importance of plan- ning and structured trajectory selec- tion in sparse-reward environments. The MCTS- guided rollout encourages the agent to actively search for viable action sequences, whereas the normal incorrect behaviors. By enabling targeted exploration and better credit assignment, the low-level exploration module substantially enhances performance, particularly on long- horizon and more complex tasks.
Model
Map
Rediãt
OSS
Gillab
CMS
Avg
Qwen 2.5VL-7B SAGE w/o low
27.3
26.3
42.5
15.2
14.6
25.2
Qwen 2B SAGE
28.
29.7
43.2
18.9
15.6
27.1
Qwen 2.5VL-328 SAGE wio low
41.0
42.3
44.8
27.1
29.3
36.9
Qwen 2.5VI SAGE
43.7
46.1
35.2
39.7
Table 5: Ablation study on the low-level exploration strategy. Removing the MCTTS-based rollout and step-wise learning leads to consistent performance drops, highlighting their importance for efficient planning and long-horizon reasoning.
rollout strategy often causes the agent to get stuck repeating
Accuracy of the Beaining-Based Evaluator. To evaluate the effectiveness of our learned outcome evaluator, we compare the evaluation accuracy across different models before and after training. We first collect a set of trajectories by rolling out Qwen2.5VL-32B SAGE on held-out tasks, and then manually annotate the success or failure of each trajectory using human judgment. These auman--nnottted outcomes serve as ground-truth labels for evaluating different evaluator models. In our setup, we fine-tune Dwen2.5VVL-7B to serve as the learned evaluator, trained using both gener-egee and human-verified supervision signals.
As shown in Table 6, we observe that proprietary models initially outperform open-source models in evaluator accuracy. However, after training, our Qwen2.5VL-7B evaluator achieves the highest accuracy overall. Notably, the trained evaluator is significantly more precise in handling challenging cases, such recognizing task infeasibility, which are critical for providing reliable supervision during agent training. These results highlight the importance of robust, adaptive evaluator in enabling self-supervised learning.
5.5 Error Analysis

---

Droi Type Charibbtion bbn Across Model
Figure 2: Error type distribution across models. Each donut chart illustrates the proportions of different eπror types and success cases on WebVoyager tasks, annotated for cach model variant. SAGE effectively reduces both low-level and high-level error rates across model sizes. For Qwen2.5VL-7BB we also includc an intermediate checkpoint ("mid") which shows a significant reduction low-level errors carly training. As training progresses, high-level reasoning errors also diminish, suggesting that SAGE enables agents to first master simple skills before tackling harder reasoning challenges. On Qwen2.5VL-322 SAGE achieves the lowest error rates and highest success rate, demonstrating its robustness at scale.
To better understand how SAGE facilitates skill acquisition from easy to hard tasks, we conduct detailed error analysis the WebVoyager [5] benchmark. We randomly sample 200 tasks from the test set and evaluate multiple models, and then manually annotate the resulting trajec- tories based on failure modes.
Claude 3.5 Sonnet
Qwen 2.5VI.7B
SAGE
Accuracy
82%
75%
90%
Table 6: Evaluator accuracy comparison. Accu- racy measured against human labels. We fine-tune Qwen2.5VI.-7B as the learned evaluator.
Following the error types introduced in prior work [17], we categorize each failure into six distinct types:
Low-level execution errors: The agent has a plausible plan but fails to interact correctly with the interface, such as clicking the wrong element or failing to navigate to the intended page.
High-level reasoning or planning errors: The agent fails to formulate effective strategy or to reason correctly through the interface in order to solve the task.
Visual hallucinations: The agent fabricates answers that are not supported by the visible content, e.g., referencing information that not present or misidentifying page content.
Timeouts: The agent is on the correct path but exceeds the maximum number of interaction steps before completing the task.
Technical issues:
Errors resulting from environment- problems such as broken links, loading failures, or website outages, not attributable to the agent itself.
Miscellaneous cases that do not fall into the above categories, including fundamen- tally infeasible tasks.
Other:
6 Conclusion
We present new self-guided training framework for generalist foundation model web agents, centered around multi-level hierachical exploration strategy. By decomposing the learning process into three functional phases: pre-exploration for environment understanding, top-level exploration for aurricluumddrven task generation, and low-level exploration for efficient interaction- agent is able to autonomously acquire and refine skills directly through interaction with real-world websites. Our framework eliminates the need for static task datasets or expert demonstrations, instead allowing the agent to generate its own tasks, evaluate its progress, and improve over time. Empirical results demonstrate that this self-guided training approach leads to more scalable and robust skill acquisition, enabling agents to solve complex web tasks with stronger generalization and higher efficiency. We hope this work takes a meaningful step toward the development of foundation model agents that learn from experience and adapt to diverse digital environments autonomously.
10

---

Acknowledgments
This work was supported part by the Amazon-Iliinos Center on AI for Interactive Conversational Experiences, NSF under Grants 2106825 and 2519216, the DARPA Young Faculty Award, and the This work used computational resources, including Amazon Web Services (AWS), the NCSA Delta and DeltaAl supercomputers through allocation CIS230012 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program. as well as OpenAl API through the National Artificial Intelligence Research Resource (NAIRR) Pilot.
ONR Grant 100014-22-1-22
References
[1] OpenAL. GPT-4 technical report. arXi::2303.08774 2024. 1,3
[2] Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseck-coder: When the large language model meets programming the rise of code intelligence. arXiv:240I. 14196, 2024.
[3] An Yan, Zhengyuan Yang. Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang. Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, and Lijuan Wang. GPT-4V in won- derland: Large multimodal models for zero-shot smartphone gui navigation. arX5v:2311.07522, 2023. 1,3
[4] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. GPT-4v(ision) is generalist web agent, if grounded. arXiv: 2401.01614, 2024. 1,3
[5] Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Hongming Zhang, Tianqing Fang, Zhen- zhong Lan, and Dong Yu. Openwebvoyager: Building multimodal web agents via iterative real-world exploration, feedback and optimization. arXiv: 2410. 19609, 2024. 1, 7, 10
[6] Izzeddin Gur, Hiroki Furuta, Austin V. Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. arviv: 2307.12856, 2021. 1,3
[7] Wenyi Hong. Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang. Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents. arXiv: 2312.08914, 2023.
[8] Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for 1lms. arXiv: 2310.12823, 2023.
[9] Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. arXiv: 2310.05915, 2023. 1,3
[10] Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In Computers and Games, 2007. 2,5
[11] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Weharena: realistic web environment for building autonomous agents. In ICLR, 2024. 2,
[12] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan. Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv: 2005. 1 14165, 2020.
[13] Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. In NewrIPS, 2024. 3
11

---

[14] John Yang. Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. In NeurlPS, 2024. 3
[15] Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Au- tonomous evaluation and refinement of digital agents. In COLM, 2024. 3, 6
[16] Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani. Chelsea Finn, Divyansh Garg, and Rafael Rafailov, Agent Q: Advanced reasoning and learning for autonomous agents. arXiv: 2408.07199, 2024. 3
[17] Yifei Zhou, Qianlan Yang. Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xiong Wang. Sergey Levione, and Erran Li. ProposerAAgent-vvllcee (PAE): Autonomous skill discovery for foundation model internet agents. In ICML, 2025. 3, 4, 7, 8, 10, 18, 23, 28
[18] Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, and Sergey Levine. Fine-tuning large vision visionllagguage models as decision-making agents via reinforcement learning. In NeurIPS, 2024.
[19] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. 2017. 3
[20] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.
In NeurlPS, 2023. 3, 6
[21] Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions. In ACL, 2023. 3
[22] Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv:
2024.
[23] Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jianguang Lou, Qingwei Lin, Ping Luo. Saravan Rajmohan, and Dongmei Zhang. Agentgen: Enhancing planning abilities for large language model based agent via environment and task generation. In KDD, 2024. 3
[24] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv: 1802.06070 2018. 3
[25] Scohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllabiliy-awwar unsuper- vised skill discovery. In ICML, 2023.
[26] Zehan Qi, Xiao Liu, lat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, and Yuxiao Dong. Webrl: Training 11m web agents via self-evolving online curriculum reinforcement learning. In ICLR, 2024. 3
[27] Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-DPO: Step-wise preference optimization for long-chain reasoning of 1lms. arXiv: 14629, 2024.
[28] Xiao Liu, Tianjie Zhang, Yu Gu, lat Long long, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. Visualagentbench: Towards large multimodal models as visual foundation agents. In ICLR, 2024. 7
[29]
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. 2,
[30] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaVA-NeXT: Improved reasoning, ocr, and world knowledge. arXiv: 2401. 13601, 2024. 8
[31] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge. Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv 250. 2025. 8
12

---

[32] Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets Lim self-training via process reward guided tree search, arXiv preprint arXiv:2406.03811 2024.
21
[33] Andy Zhou, Kai Yan, Michal SilapenntokhRotthman Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models, 2023. 21
[34] Alexandra Carpentier, Alessandro Lazaric, Mohammad Ghavamzadeh, Rémi Munos, Peter Auer, and András Antos. Upper-confidence-bon bound algorithms for active learning in multi-armed bandits, arXiv: 1507,04523, 2015. 21
[35] An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang. Jin Xu. Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, 7eyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. arXiv: 2407, 10671, 2024. 22
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. We claimed in Sec 1. that devloped framework of training foundation model web agents achieveing state-of-the-are performance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes. We discussed the limitations in the supplementary materials.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
We didn't 't make any theory assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification: Yes. we provided all the details in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:
[No]
Justification: We are not allowed to release data and code. We will release the code and data upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https: /nips. cripsscc/public//uu guwdee/iee///ee CCodeSubeeiiaceecc sisiissiinn ssssonoliccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes. We listed all the details in Section 5. and the supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our experiments take too much computational resources so that we cannot run it too many times to get the error bar. Also, typically in LLM research, people don't 't care too much about the error bar.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
16

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes. We provided this in the supplemantary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eerrpp.ee/ee ctt.pplbc///hccccuiddeeeeeee
Answer: [Yes]
Justification:
We believe we followed the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes. We discussed this the supplementary materials.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our method is not designed for generation.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited all the works we used in our work, including data and code from [17].
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We didn't create new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Yes. we have described all the usage of LLMs in our paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https //tp:///eeuupppeeeeee htttp::/neurips..ccoonfeee / for what should or should not be described.
20

---

A Algorithm Details
In this section, we provide additional details on the core algorithmic components of SAGE. We first present the Monte Carlo Tree Search (MCTS) rollout procedure used during low-level exploration. followed by the evaluator training setup that enables robust supervision without relying on ground- truth rewards.
A.1 MCTS Rollout Procedure
Our MCTS-based rollout strategy is inspired by prior work on structured decision-making and planning [32, 33], with key adaptations for the web-based setting and multimodal agent interaction. The search tree is initialized with single root node representing the starting state of the task, typically corresponding to the homepage URL.
Trainable and fixed components. During training, the task generator and evaluator models remain fixed and only adjust their contextual inputs dynamically based on the agent's recent per- formance. The agent model is the only component optimized with gradient- based updates, while MCTS functions purely as planning and sampling mechanism to collect high-quality trajectories and is not directly optimized. This design ensures stable and reproducible learning while allowing adaptive task progression.
Training objective. Since only the agent is trainable, we use step-level Direct Preference Opti- mization objective. Given a task prompt and two trajectories Twin and Tios sharing the same prefix (k)) where one succeeds and the other fails, the loss
L(0) (E Pome)) log (B log
To (Twin
2, 81.k) log
Tp ( Those $1:k)
1)]
Tref Twin
$1:k) Treel Tose
It 81.k)
where Te denotes the agent policy, Tiet a frozen reference model, and a temperature coefficient. This objective encourages the agent prefer trajectories leading successful completions.
Training procedure. Before the first epoch, conduct pre-exploration phase to map website structures and collect structural affordances. In each epoch, the task generator produces a batch of tasks conditioned on these structures. The agent performs MCTS -CTTS-guided rollouts to explore both successful and failed trajectories. After collecting sufficient data, the agent policy is updated via the step-level DPO loss above. Task-level success statistics are then fed back into the task generator context, allowing adaptive adjustment of task difficulty for the next epoch.
MCTS rollout. Each MCTS iteration proceeds through four phases: selection, expansion, simula- tion, and backpropagation.
Selection. Starting from the root, the agent traverses the tree using the Upper Confidence Bound (UCB) score [34]:
In parent
UCB(u) Uu + cV
Th
where v. is the value estimate of node u, n. its visit count, and Nquaent the visit count of its parent node. The coefficient balances exploration and exploitation.
Expansion. Upon reaching a leaf node, the agent expands it by sampling multiple candidate actions from (Q, s), each producing new child state. The value of each child u. is initialized using the evaluator to estimate task success likelihood, providing a prior for downstream planning.
Simulation. From the most promising child node, rollout is performed up fixed depth L. At each step, the agent greedily selects the top-confidence action. The trajectory terminates upon task completion or reaching the step limit.
Backpropagation. After the rollout, binary success reward is propagated upward to update value estimates and visit counts. The MCTS process continues for fixed number of iterations.
21

---

end for Algorithm Low-Level Exploration via MCTS
Input: Task Q, agent policy и, initial page Po. value model V, max iterations N, episode length L
Initialize: T InitializeTree(pp)
1: for = to N do
2: f RRoo(())
3: while is not leaf node do
4:
arg max, Ust V
log
n1,
5:
end while
6:
if is not terminal then
7:
01:k + (Q,
8:
$1 Applonctioss(s, a1a))
9:
Ug <- ((,)) for all j
10:
s' arg max, U22
f- RolloutFrom(s' depth = depth(s'))
12: UpdateNode(s', reward = r)
13: end
14:
BackPropagate(s)
15:
Low-level exploration pseudocode. For clarity, we summarize the low-level MCTS exploration procedure below.
This integration of MCTS planning with gradient- DPO training enables SAGE to explore promising trajectories more efficiently and improve success rates across complex, long-horizon web tasks.
A.2 Confidence Prediction Training.
The agent policy my is augmented with scalar confidence head that outputs [0, for each action. For initialization, we prompt the proprietary model Claude Sonnet 3.5 to generate demonstration trajectories where each action is annotated with self-estimated confidence score. These annotated trajectories are used during supervised fine-tuning to initialize the confidence prediction capability of base model.
During the subsequent SAGE training phase, the agent's confidence estimates are further refined within the step-level DPO framework. Since the MCTS-based sampling process does not always follow the action with the highest predicted confidence, realignment step is applied to calibrate confidence with actual success outcomes. Specifically, in successful trajectories, the executed action assigned a confidence value of ensuring that the model learns to associate high confidence with decisions that lead to successful task completion. This refinement improves the calibration between model certainty and real-world decision reliability, yielding more stable long-horizon exploration.
A.3 Evaluator Training
To enable reliable supervision without access to ground-truth reward functions, we train task outcome evaluator using the open-source model Qwen2.5VL-7B [35]. The evaluator is designed to predict task success failure based evidence collected during agent interaction. Its input consists of: (I) the task instruction, (2) the final answer generated by the agent, (3) the full interaction trajectory, and (4) multimodal evidence composed of the final three screenshots and their associated accessibilityy tree. The output is binary success label, optionally accompanied by a natural language explanation.
We construct the training dataset in two stages. First, we bootstrap the evaluator using over 10,000 evaluation traces generated by Claude 3.5 Sonnet [29], treating its responses pseudo-labels. These include both reasoning and binary outcome labels. This initial phase enables the evaluator to inherit strong generalization capabilities from the proprictary model.
22

---

In the second stage, we refine the evaluator using a smaller, high-quality human annooated dataset. Specifically, we randomly sample 1,200 trajectories from the initial dataset and manually annotate their ground-truth success labels. Among these, we identify approximately 400 instances where Claude's predictions disagree with human judgment. For each of these error cases, we provide corrected binary label and, when relevant, brief explanation describing the evaluation mistake. The format remains consistent with the initial dataset, enabling seamless fine-tuning. This two-stage process enhances the evaluator's precision and robustness, particularly for nuanced failure-prone cases.
B Environment Details
In this section, describe the multimodal web environments used in experiments, including implementation details, observation formatting, and action space definitions.
Environment Overview. Our environment is implemented using combination of Selenium' and Playwright', and wrapped into a standardized Gym interface to support reinforcement learning workflows. In addition to basic functionalities required for agent-environment interaction, we extend the framework with support for saving and restoring environment states at arbitrary time steps. This feature is critical for enabling tree-based search strategies such MCTS, which require repeated rollouts from intermediate checkpoints.
Observation Details. As illustrated in Figure 3, each observation consists of two aligned modal- ities: rendered screenshot with overlaid set-of-marks, and structured accessibility tree. In our implementation, we first extract the set-of-marks and the accessibility tree independently. We then compute matching between elements in the two modalities and align the numeric labels in the set-of-marks with the corresponding nodes in the accessibility tree. This alignment ensures that agents can refer to web elements consistently across visual and structural representations.
Action Space. Following the setup in PAE [17], we define discrete action space that includes the following interaction types:
Click: Click on a labeled element such button link.
Type: Enter text into an input field.
Scroll: Scroll a scrollable container or the entire page.
Return: Navigate back the previous page in browser history.
Answer: Return the final answer.
C
Additional Experiments
In this section, we present supplementary experimental results to further analyze the effectiveness of individual components in our framework.
C.1 Ablation Study: Pre-Exploration Strategy
To evaluate the impact of the pre-exploration strategy, we conduct an ablation experiment where this phase removed. In this variant, the task generator only receives the base URL of the website along with trajectories collected from previous training iterations. Without the structured understanding provided by pre-exploration, the generator lacks information about the page layout and interaction flow.
As shown in Table 7, removing pre-exploration leads to noticeable degradation in performance. The task generator in this setting frequently proposes irrelevant or infeasible tasks due to its limited knowledge of website structure, demonstrating the importance of pre-exploration in grounding curriculum design.
hhap:://pppiorrghmnoccc/steetiio hatps://paaywrihh..eww
23

---

[333] link 'Skip content"
[360] alert "Announcement"
[o] link 'Learn more'
[1] button 'Close'
[387] heading 'Navigation Menu'
[2] link 'Homepage"
Build and ship software
[434] navigation 'Global'
[3] button 'Product'
single, collaborative platform
[4] button 'Solutions'
[5] button 'Resourcea'
[6] button 'Open Source'
[7] button 'Enterprise'
[8] 1ink 'Pricing'
[9] button 'Search jump to....
[10] link 'Sign in'
[11] link 'Sign up'
Figure 3: Multimodal environment input. Left: Screenshot of a rendered webpage with set-of- marks. Right: Accessibility tree snippet showing the hierarchical structure of interactive elements. This multimodal input enables agents understand both visual and structural content for accurate interaction. The agent can find buttons both from the vision input and the text input.
C.2 Ablation Study: Evaluator
We conduct an ablation study to assess the importance of the training-based evaluator in our framework. Specifically, we replace the learned evaluator with alternative mod- clsude 3.5 Sonnet and Qwen2.5VL 32B-and use their raw outputs to super- vise agent training. As shown in Table 8, our training-baat based evaluator leads to better task success rates than both baselines, de- spite being based on a smaller open-source model. This highlights the benefit of tar- geted evaluator fine-tuning on trajectory outcomes, which enhances the quality and reliability of training supervision.
Map
Ang
Qwen Qwen Owen 2.5VL-328 pee
18.3
27.1
Owen 2.5VL SAGE
31.4
Table 7: Ablation study on the pre-exploration strategy. We evaluate the impact of removing the ere-xxplooation phase, where the task generator receives no structural information about the website beforchand. Without this step, the task gen- erator becomes less grounded and often infeasible, resulting in significant performance drop across all categories. These results demonstrate the importance of pre-exploration for derstanding environment structure and supporting meaningful task generation.
We present detailed error breakdown in Figure to examine how SAGE facilitates progressive skill acquisition across training stages and model scales. Across all mod- els, we find that SAGE significantly im- proves task performance by reducing two major sources of failure: low-level execu- tion errors and high-level planning errors. Notably, in the Qwen2.5VI--7B model, we analyze an intermediate checkpoint after training iterations and observe marked decrease in low-level
Map
Avg
Qwen SAGE Owen
Owen V1-
2.5VT
2.511
Table 8: Ablation study on the evaluator module. We compare training- baseed evaluator against two alternatives: Claude 3.5 Sonnet and Qwen2.5VL-32B. both used fixed evaluators without additional fine-tuning. Despite its smaller scale, our trained evaluator consistently yields higher task suc- cess rates, demonstrating the importance of domain- adapted supervision for reliable reward estimation.
that the agent learns precise actions early on in the training process, largely through practicing easier tasks. As training progresses, the proportion high-level planning errors decreases as well, reflecting the agent's 's transition to mastering more complex tasks. This learning trajectory aligns with the curriculum induced by our top-level exploration strategy. Furthermore, on the larger Dwen2.5VLL-321 model, SAGE consistently reduces both types of errors and achieves the highest success rate among all variants, highlighting scalability and effectiveness even with stronger backbones.
24

---

C.3 Evaluator Fine-tuning and Cross-Framework Comparison
We further analyze the role of limited hu- man annotations in SAGE. Only small amount of human-labcled data (less than eight hours of effort by one author) is used for fine-tuning the outcome evaluator the final stage, while all other components are trained fully autonomously. This small supervision improves the evaluator's ac- curacy but has minimal effect on overall system performance-SAGE main gains stem from its adaptive curriculum and ex- ploration strategies. The comparison Table 9 also shows that even when PAE adopts the same trained evaluator, the performance gap remains large, underscoring that SAGE effectively leverages stronger evaluators through its hierarchical design.
Model
Map
Redii
OSS
Cittsh
CMS
Arg
Qwwn2. SVLL-13 QV PAB
23.5
28.6
37.8
15.3
12.4
22.7
Qwen2 MMLL-7H PAI SAGE
24.1
24.4
17.6
18.9
12.9
12.9
Owes2 SVL-7B
28.3
18.9
13.6
SVL-328 SAGE Evai
43.1
26.4
Quee22 328
43.7
31.4
35.2
39.7
Table 9: Evaluator fine-tuning and cross-framework com- parison. We compare PAE and SAGE under identical evalua- lors to isolate the effect of evaluator quality, Even when using the same trained evaluator, SAGE substantially outperforms PAE, confirming that the main performance gain arises from our hierarchical exploration and curriculum mechanism rather than evaluator differences.
C.4 Efficiency of Low-Level Exploration
To quantify the efficiency of our low-level exploration strategy, compare the num- ber of successful trajectories generated un- der the same sampling budget of 10k trajec- tories per model. As shown in Table 10, the low-level exploration consistently produces a higher proportion of successful rollouts, demonstrating improved sample efficiency and faster policy improvement over stan-
Model
Successful Trajecoores (10k Samples)
Qwar2 SVL-78 + Nocmal
1937
Qwen2 SVL-7B + LLw- Exploration
2102
Qwen2 SVL-328 - Nommal
2578
Qwin SVI-328 1wt.128--oooooonn Low-level Exploration
2805
Table 10: Sample efficiency of low-level exploration. We compare number successful trajectories collected under identical sampling budgets. The proposed low-level MCTS- based exploration consistently yields more successful rollouts than standard rollouts, confirming its superior sample effi- ciency and stronger training signal.
dard rollouts.
C.5 SFT Baseline Comparison
For completeness, report the perfor- mance of the supervised fine-tuned (SFT)
Model
Map
Reddit
OSS
Gidab
CMS
Avg
baselines that serve as initialization for both PAE and SAGE. As shown in Ta- ble 11, SAGE consistently achieves higher success rates than both SFT and PAE across all benchmarks, further validating the effectiveness of our hierarchical train- ing paradigm.
Qwen2.5VL.22 SFT
16.5
21.3
32.6
13.7
10.1
19.4
Qwen2.5VL-78
Owen2.5VL.7B. SAGE
28.1
29.7
43.2
18.9
15.6
27.1
Qwea2.5V7 5VL-32B SFT
27.3
27.8
38.0
18.2
11.7
25.7
Dwwn2.5VL-3226 PAE
34.5
35.9
41.0
24.1
22.0
31,5
Qwen2.5VLL32B SAGE
Table 11:
Comparison with SFT baselines. We report results of the supervised fine-tuned (SFT) models for pleteness. SAGE consistently outperforms both SFT PAE across all environments and model scales, confirming the hen- efit of hierarchical exploration over static imitation learning.
D Hyperparameters
We include the hyperparameters that we have used in Table 12.
E
Prompts
We include all prompting templates used throughout our framework. In particular, Figures 5 and show the prompt used during the pre-exploration phase, where the agent instructed to explore and describe the structure and functionality of given website starting from its homepage. Figures 7 and provide the prompt template used in the top-level exploration phase, where the task generator proposes instructional tasks based on previous trajectories and structural understanding of the
25

---

Hyperparameter
Value
number of actions
maximum exploration step
100
number of tasks pre-generated
10, 000
number of tasks generated in iteration
000
number of iterations in MCTS number of trajectories actor update epochs per iteration batch size DPO /3
25
2, 048
0.45
Table 12: Hyperparameters used for SAGE.
Task: most recently updated Agent Project
Train Task: Find Web Apest Project
Train Task: Search the moet roantly updated peojedt this page
contributurs of project
Infer
Start
Action: Type Into
Contributors
Action:
Type ioto
Cantrbbutor
Sourre
Srowppate
Priding
Actio: Click
Action: Click
Contributor2
Rnd
Infer
Action Scroli DOWN
Action: Action Click
Figure 4: An example showing how SAGE facilitates skill acquisition through hierarchical exploration. The agent initially fails complete the task "Find the recently updated Web Agent project and list its contributors." In subsequent iterations, the task generator offers simpler subtasks with the top-level exploration. The agent learns solve each subtask individually through the low-level exploration, ultimately enabling it to succeed on the original, more complex task. The blue line shows a trajectory the agent generated during the inference phase.
environment. All prompts are designed to be general and domain- ensuring compatibility across both open-source and proprietary models.
A Case Study of SAGE
We provide a case study that illustrates how SAGE enables an agent to acquire complex skills through structured, hierarchical exploration. We examine challenging task from the GitHub website: "Find the most recently updated Web Ageni project and list its contributors." As shown in Figure 4, prior to training, the agent consistently fails to initiate the correct actions, struggling even to identify how search for relevant projects.
Through the top-level exploration module. SAGE's self-guided task generator decomposes the difficult instruction into three simpler subtasks: (1) locating a Web Agent project, (2) identifying the most recently updated entry, and (3) retrieving its contributors. These tasks are short-horizon and require only few well-defined interactions, allowing the agent to explore and acquire each skill individually via low-level exploration.
Once the agent demonstrates competence on subtasks, the task generator reintroduces the original complex task, allowing the agent to practice solving it end-to-end. Empowered by prior experience and learned sub-skills, the agent successfully completes the original instruction-ddmmonstiio demonstrating the effectiveness of SAGE in enabling curiculum-driven skill composition and hong-horizon reasoning.
26

---

G Limitations and Future Work
While SAGE achieves strong performance across diverse web environments, it is not without limita- tions. First, framework assumes access to stable and scriptable browser environment, which may not hold in fully open-web scenarios that include CAPTCHAs, login requirements, or highly dynamic JavaScript content. Although states-enabling reproducible resets similar to prior works [17? -his assumption limits the applica- bility of our MCTS--assee exploration in real-world settings where actions may be irreversible or pages non-deterministic. Future extensions may incorporate conservative planning, safe policy filtering, or human-in-the-le oversight enable safer deployment in uncontrolled web environments.
Playwright wrappers store and reload webpage
Second, SAGE introduces several environment-specifn augmentations -uchh as leveraging webpage accessibility trees, BFS-based initial trajectory collection, and full environment resets- that simplify training in standardized benchmarks but may not generalize to arbitrary or uninstrumented websites. Improving robustness under weaker assumptions (e.g., partial observability or limited environment control) remains an important avenue for future research.
Third, our current experiments are conducted in standardized, resettable environments that require substantial computational resources for large-scale MCTS rollouts and seh-fiielitty webpage render- ing. This high barrier to replication may hinder broader community adoption re-implementation without comparable infrastructure. We plan to mitigate this by releasing lightweight simulation setups and environment wrappers to facilitate partial reproduction under modest compute budgets.
Finally, although SAGE is designed for research and sandboxed enviromments, applying it directly to live websites may raise privacy concerns, agents could unintentionally access or leak user-sensitive information during open-ended exploration. Ensuring privacy-safe deployment and extending our approach beyond web navigation-e, e.g., desktop or mobile human-compunn interaction reppeseennrrereeennnt key directions for future work.
H Broader Impact
This work aims to advance the development of autonomous web agents that can interact with websites using vision-language foundation models. Such systems have the potential to automate a wide range of digital tasks tasks-ranging from information retrieval to workflow aasistanee-thereby improving accessibility and efficiency for users. At the same time, their interaction with real-world websites introduces important safety and ethical considerations.
Controlled research setting. All experiments in SAGE are conducted within strictly controlled enviromments. WebArena is a fully sandboxed simulator, and Web Voyager provides access only to publicly available webpages without user authentication private data. All web interactions are executed inside a secure, prewall-protecte requests. The agent is never provided with personal information, and all tasks involving user accounts or sensitive data are explicitly excluded. These measures ensure that SAGE operates entirely within a research context and cannot cause real-world effects.
network that monitors and blocks potentially unsafe
Safety and misuse considerations. We recognize that autonomous web agents raise broader safety risks, including unintended interactions with real systems, information leakage, or malicious misuse. Although such risks are absent in our current setup, future large-scale deployments will require robust safeguards, such as: (1) decoupling policy outputs from execution through a controlled intermediary layer with safety filters, (2) adding dedicated safety-monitor models detect and block risky actions, (3) enforcing rate limiting, permission gating, and anomaly detection, and (4) maintaining human-in in- the-loop oversight during execution. We emphasize that our work focuses on algorithmic capability under safe, offline environments, but we strongly support the establishment of community standards for safe deployment of web agents.
Ethical and privacy implications. Interactions with real-world websites can raise privacy or data protection concerns. While our experiments do not involve any private or user-generated content, future work must ensure that autonomous web agents respect consent boundaries and data-use policies. We also acknowledge potential social risks, such unintended automation of sensitive or
27

---

economically impactful tasks. These issues call for broader interdisciplinary research that combines technical safeguards with ethical and policy frameworks to ensure responsible deployment.
Outlook. Our primary goal is to improve the technical understanding and safe design of autonomous web agents. We believe that future progress in this field must go hand in hand with advances safety, interpretability, and accountability mechanisms. We are committed to contributing to the development of transparent, privacy-conscione and human-aligned web agent systems that can benefit society while minimizing risk.
I Ethical and Safety Considerations
This work develops SAGE as a research framework for studying autonomous web agents controlled environments. All experiments are conducted entirely within WebArena and Web Voyager, following prior work [17]. WebArena is a fully synthetic sandbox that simulates websites locally without any network connection real data. Web WoyVoyaeer contains only small set of manually curated, static webpages from publicly accessible domains (e.g.. Google Maps, Wikipedia), without user authentication, private data, executable operations. Agent activities occur behind firewall, and all interactions are logged for analysis and accountability. Consequently, SAGE does not access or collect any sensitive personalized information.
We acknowledge the broader ethical and security concerns associated with deploying fully au- tonomous agents the open web. Although SAGE poses privacy misuse risk in its current form, future upplications should incorporate additional safeguards such as restricted execution priv- ileges, rate limiting, security monitoring, and human-in-the-lonn approval before critical actions. Detailed action logs and auditing mechanisms should also be used to ensure accountability and prevent unintended behaviors.
Finally, no crowd-sourced or external annotation was used in this project. All labeled examples for evaluator calibration were created internally by the authors. We oppose any unauthorized or adversarial use of this system and encourage future research to adhere to responsible, transparent, and preecc-preeeriing development practices.
28

---

Task Generator Prompt for SAGE (Part A) ("web_name": "Amazon",
"web": htttps//www.amazzn.ccom
"structure-info" ("Homepage includes sections such as 'Electronics', 'Books', and 'Deals'.", "Prod- uct pages contain price, reviews, delivery info, and variants.", "Search results are filterable by price, rating, and brand."].
"previous_trapee trajectories""
("instruction": "Search for headphones under $50 with at least 4 stars.", "success rate": 0.32), ("instruction": "Compare features between AirPods Pro and Galaxy Buds.", "success rate": 0.00),
("instruction": "Find a gift card worth exactly $100.", "success rate": 0.90)
])
We are training a web agent that learns to complete real-world tasks by interacting with web interfaces. Your job is to generate figh-quality, instructional tasks that help the agent improve over time. The agent has already explored this website and attempted previous tasks. Based on this prior experience and structural understanding of the site, you must generate a new batch of 25 training instructions. Step 1: Reasoning Instructions
Before listing your output, write your reasoning on how you adapted your tasks. Your strategy must consider the following:
Simplify low-success tasks:
For instructions with low success (c.g., below 40%), generate 1-2 simpler variants by relaxing filters breaking down subtasks.
Harder from high-success tasks: For instructions with high success (e.g., above 80%), generate harder variants by adding filters composing with other subtasks.
Figure 5: Task Generator Prompt Part A.
Task Generator Prompt for SAGE (Part B)
Structure-aware task design: Use the provided structurejinfo to ensure realism.
Curriculum balance: Ensure a mix of easy, medium, and hard tasks.
Step 2: Output Format
Output exactly 25 instructions in JSONL format.
Each line must be valid JSON object with single field "instruction".
Rules and Constraints
Do not require login, user data, or current time.
Ensure task is verifiable via screenshots.
Completion should require 2-8 steps.
In-Context Examples:
Simplified: ("instruction": "Search for headphones under $50.")
Harder: ("instruction": "Compare Galaxy Buds and AirPods, then find a cheaper rated alternative.")
Figure 6: Task Generator Prompt Part B.
29

---

Pre-Exploration Prompt (Part A)
You are an intelligent web browsing agent designed to explore and understand new websites before learning specific tasks. In this phase, you will only receive the homepage of a new website, and your goal discover and record meaningful pages and their functions. This phase is unguided-you not solving any tasks.
In each step:
You will observe a screenshot, accessibility tree, and numerical element labels.
Recall visited pages to avoid redundant navigation.
Propose up 10 distinet actions that could lead to semantically different states.
Record what the page reveals (e.g., filters, product listings, FAQs).
Figure 7: Pre-Exploration Prompt Part A: Instructions for agent behavior and high-level goals.
Pre-Exploration Prompt (Part B)
Output Format per step:
Thought: Reason about the state and exploration plan.
Top-10 Actions: Propose 10 diverse actions (e.g., Click [3], Scroll WINDOW; down).
Summary: Describe the page's ' purpose or findings.
Exploration Guidelines:
Avoid login, signup, or non-navigational elements.
Maximize cerssity-explore filters, categories, products, help pages, etc.
Use accessibility tree guide semantic understanding.
Avoid repeated redundant actions.
Example Output:
Thought: The homepage includes categories and search bar. haven't explored "Books" filtered
search results yet.
Top-k Actions:
Click [3]
Click [4]
Scroll WINDOW; down
Type [10]: "usb-c hub"
Summary: This is the homepage with links to major categories and search bar. begin mapping product pages and filter paths.
Figure 8: Pre Pre-Explorntion Prompt Part B: Format, examples, and behavior guidelines for site exploration.
30


Reviews:
[{"review_id": "rh40Ui3XaO", "reviewer": "Reviewer_L48z", "summary": "The paper introduces a new training framework for Web Agents. The framework consists of three components: first, initial trajectories are collected using BFS; then, tasks are generated by a self-evolving curriculum based on the collected trajectories; the curriculum continues to evolve based on new environment trajectories and performance; and finally, MCTS-inspired learning and inference are used for the Web Agent. The framework outperforms all baselines by a substantial margin.", "questions": "* Is the curriculum learning in the paper based on PAE, or did you add something on top of it?\n\n* How is confidence trained?", "limitations": "See the limitations in weaknesses.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Weaknesses**\n\n1. Methodological opacity\n    \n    The Method section does not explain the core components of the pipeline in sufficient detail. In particular:\n    \n    * Curriculum learning, one of the key contributions, is only sketched. It feels to be build on PAE, yet PAE itself is never introduced in the Background section.\n    * The terms *hierarchical* and *top‑level* are used to describe the curriculum, but the text never clarifies what hierarchy is being referenced or how these levels interact.\n\n2. The paper claims that learning, sampling, and inference are based on MCTS, but it does not provide any pseudocode, equations, or detailed illustrative figure (there is fig. 1 but it is not very detailed). It also references the use of the Claude model for initial value estimation, but without sufficient explanation how later this value interact with the learning pipeline. As a result, the procedure is difficult to follow, and several key aspects remain unclear:\n\n    * Which components of the pipeline rely on fixed neural networks and in‑context adaptation, versus which involve gradient-based learning.\n\n    * What objectives or loss functions are used during training.\n\n    * How the overall pipeline operates in detail.\n\n3. Fragmented presentation\n\n    Critical details are scattered between the main text, footnotes, and appendices. For example, the description of the *Evaluator*, essential for interpreting results, appears only in the appendix. Putting all important elements of the pipeline (e.g., curriculum updater, MCTS module, evaluator) into the main text would improve readability.\n\n4. Environment‑specific augmentations\n    \n    The authors introduce several augmentations that may not generalise to other Web environments:\n    \n    * Access to the web page’s accessibility tree.\n    * BFS‑based initial trajectory collection with full environment resets.\n5. Accessibility tree needs clarification\n    \n    The manuscript never defines what it calls the web page’s *hierarchical accessibility tree* or explains why the tree is considered hierarchical. Moving the screenshot that is now in the appendix into the main text and adding a concise explanation, would eliminate this.\n    \n**Suggested improvements**\n\n* Provide a concise background on PAE and formally state how the proposed curriculum extends or differs from it.\n* Add pseudocode or a flowchart for the MCTS‑inspired loop, specifying which neural components are frozen, which are trained, and the corresponding objective functions.\n* Relocate the Evaluator description and any other pipeline‑critical details to the main body.\n* Clarify the practicality and limits of using the accessibility tree and BFS in larger or non‑resettable environments, possibly by discussing mitigation strategies or future work.\n\n**Strengths**\n\nThe proposed method outperforms the baseline PAE pipeline.", "was_revised": true, "final_justification": "My concerns about clarity have been addressed, and I trust the authors will update the main text accordingly based on their rebuttal.", "rebuttal": "Thanks a lot for your insightful and inspiring comments! We provide the following clarifications in response to your concerns:\n\n1. **Core contribution of SAGE and comparison with PAE**\n - SAGE builds upon the proposer–agent–evaluator (PAE) paradigm but addresses key limitations in curriculum learning and long-horizon task decomposition. While PAE establishes a general structure for autonomous training, it uses a static instruction distribution and lacks mechanisms for adjusting task difficulty or improving exploration efficiency.\n\n - PAE’s self-improving loop, where a proposer generates tasks, the agent attempts them, and an evaluator provides feedback, demonstrates that web agents can be trained without human-written data. However, it does not support dynamic task adaptation or skill progression.\n\n - SAGE introduces a *hierarchical exploration* framework that redefines task generation and learning. It includes a task generator that decomposes failed tasks and synthesizes harder ones from successful trials, enabling adaptive curricula. At the execution level, SAGE employs MCTS-based low-level planning and step-wise policy refinement to improve the quality and diversity of collected trajectories.\n\n - While SAGE retains the core PAE infrastructure for autonomous interaction, it adds a *new exploration strategy*, including curriculum-based task generation and planning-driven trajectory selection. These components fundamentally reshape training, enabling agents to gradually acquire complex skills and achieve much stronger performance under the same autonomous framework.\n2. **Hierarchical explanation**\n - By hierarchy, we refer to the multi-level process that enables agents to solve complex web tasks. At the *top level*, the agent decomposes tasks into subtasks and learns transferable skills; at the *low level*, it executes precise interactions (e.g., button clicks or typing) to complete each subtask.\n\n - These levels interact closely: pre-exploration provides structural knowledge for curriculum generation, and low-level rollouts feed back successful trajectories to refine future tasks for the top-level exploration. For example, in Fig 12, the agent fails at a complex task but succeeds after breaking it into simpler goals like locating a project and identifying its contributors. We will revise the paper to clarify this hierarchical structure and add more detailed explanations and cross-references.\n3. **MCTS algorithm details**\n - We apologize for the lack of clarity in explaining the details of our MCTS-based training pipeline. Below, we address the reviewer’s questions one by one, and we are happy to provide further clarifications if needed.\n - Regarding which components of the pipeline rely on fixed neural networks versus gradient-based learning: In our proposer-agent-evaluator system, both the *task generator* and *evaluator* models along with the value remain fixed during the training phase and only adjust the context of their inputs dynamically. The *agent* model is the only component that undergoes gradient-based updates throughout training. Notably, MCTS serves purely as a planning and sampling strategy to improve the quality of rollouts, but it is not directly involved in optimizing the agent’s parameters.\n - Regarding the training objectives: Since we train only the agent, we adopt step-level DPO as our training loss. Let $x$ denote the task prompt, and let $\\tau_{\\text{win}}$​ and $\\tau_{\\text{lose}}$​ be two trajectories with the same prefix steps $s_1, s_2, \\dots , s_k$, where one trajectory is successful and the other one is fail. Our objective is:\n\n$L(\\theta) = - E_{(x, s_1, \\dots, s_k,\\tau_{win}, \\tau_{lose})~D} [ \\log ( \\beta * \\log ( \\pi_{\\theta}(\\tau_{win} | x; s_1, \\dots, s_k) / \\pi_{ref}(\\tau_{win} | x; s_1, \\dots, s_k) ) - \\beta * \\log ( \\pi_{\\theta}(\\tau_{lose} | x; s_1, \\dots, s_k) / \\pi_{ref}(\\tau_{lose} | x; s_1, \\dots, s_k) ) ) ].$\n\nwhere $\\pi_{\\theta}$ denotes the agent policy and $\\pi_{\\text{ref}}$ is a fixed reference policy.\n\n - Regarding how the overall pipeline operates: Before the first epoch, we perform a pre-exploration phase to analyze the website’s structure and collect environment-specific information. In each epoch, the task generator produces a set of tasks using the structural information as context. We then randomly sample tasks and use MCTS rollouts to generate successful and failed trajectories. After collecting a sufficient number of trajectories for the current epoch, the agent is updated using the step-DPO objective above. Finally, the success statistics of tasks are fed back into the task generator’s context for the next epoch, allowing it to adaptively refine task difficulty.\n\n - To help clarify the interaction between the agent and MCTS, we have added the following pseudocode to illustrate our low-level exploration strategy:\n\n**Algorithm: Low-Level Exploration Algorithm**\n\n**Input:** Question *Q*, agent model $\\pi$, initial page $p_0$, max iterations *N*, value model *V*, episode length *L*\n\n\n1. $T \\leftarrow$ Initialize($p_0$)  \n2. **for** $i = 1$ to $N$ **do**  \n3. &nbsp;&nbsp;&nbsp;&nbsp;$s \\leftarrow$ root($T$)  \n4. &nbsp;&nbsp;&nbsp;&nbsp;**while** $s$ is not a leaf node **do**  \n5. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$s \\leftarrow \\arg\\max_{s'} \\left( v_{s'} + \\epsilon \\sqrt{ \\frac{\\log n_s}{n_{s'}} } \\right)$  \n6. &nbsp;&nbsp;&nbsp;&nbsp;**end while**  \n7. &nbsp;&nbsp;&nbsp;&nbsp;**if** $s$ is not a terminal state **then**  \n8. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$a_1, \\dots, a_k \\leftarrow \\pi(Q, s)$  \n9. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$s'_1, \\dots, s'_k \\leftarrow$ Apply actions to $s$  \n10. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$v_{s_1'}, \\dots, v_{s_k'} \\leftarrow V(s_1'), \\dots, V(s_k')$  \n11. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$s' \\leftarrow \\arg\\max_{s_j'} v_{s_j'}$  \n12. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$r \\leftarrow$ Rollout from $s' $ up to depth $L - depth(s')$  \n13. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Update $v_{s'} $ and $n_{s'}$ with result $r$  \n14. &nbsp;&nbsp;&nbsp;&nbsp;**end if**  \n15. &nbsp;&nbsp;&nbsp;&nbsp;BackPropagate($s$)  \n16. **end for**\n\n\n\n\n - We agree that these technical details should be presented more systematically. In the revised version of the paper, we will improve the organization of the methodology section, adding a pseudocode and additional illustrative figures to clearly show the interactions between MCTS, the agent, and the task generator.\n\n4. **Fragment presentation**\n - We appreciate the reviewer’s feedback. Due to space limitations, we initially placed some algorithmic details in the appendix; in the revised version, we will reorganize the content and incorporate key components into the main text to improve clarity and coherence.\n\n5. **Accessibility tree explanation**\n - We would like to clarify that our paper does not mention the term “hierarchical accessibility tree.” We only refer to the standard accessibility tree, which is a browser-defined structure. The accessibility tree is essentially a subset of the DOM tree that contains elements relevant to displaying the content of a webpage. Each element is represented by its role (e.g., link or button), text content, and properties. Compared to the raw DOM, the accessibility tree preserves the structural information of a webpage while being much more compact and directly interpretable.\n - We appreciate the reviewer’s suggestion. In the updated version, we will move the figure and explanation currently placed in the appendix into the main text for better clarity.\n\n6. **Environment-specific augmentations**\n - Our approach is designed to work with most modern web environments that are based on standard HTML. The accessibility tree we use is automatically derived from the HTML DOM by the browser, meaning that for any HTML-based website, we can directly obtain both the DOM and the corresponding accessibility tree. Similarly, our BFS-based initial trajectory collection relies on the save and load functionality provided by Playwright, which is generalizable to the vast majority of websites that support session resets or navigation history.\n - We acknowledge that our current method may not be fully applicable to certain highly dynamic websites, such as those built entirely with custom JavaScript logic or virtualized rendering where the DOM structure is not fully exposed. Handling these specialized environments remains an interesting direction for future work. In future extensions, we may consider more generalizable state representations—for example, relying solely on raw webpage screenshots (without set-of-marks) and using absolute coordinate-based actions for interaction—so that the framework can adapt to a broader range of web technologies.\n\n\n7. **Confidence training clarification**\n - In our implementation, the confidence score is represented as a scalar in the range [0,1]. For the initial model, we obtain confidence annotations by prompting Claude Sonnet 3.5 to generate action sequences where each action is accompanied by a confidence score. These annotated trajectories are then used during supervised fine-tuning to initialize the confidence prediction capability of our base model.\n - During the subsequent SAGE training process, we further refine the agent model using the step-DPO framework. A refinement is necessary because the MCTS-based sampling process does not always follow the action with the highest predicted confidence from the agent’s policy. To correctly align confidence with actual successful decision paths, we manually adjust the confidence labels in successful trajectories: the executed action is assigned a confidence of 1. This approach ensures that the agent learns to calibrate its confidence scores according to actions that lead to successful task completion."}, {"review_id": "h8mME4MTPb", "reviewer": "Reviewer_Aznj", "summary": "The paper presents SAGE, a framework designed for generalist foundation model web agents. The goal is to have agents that can autonomously explore web environments, generate their own instructional goals, and continuously refine themselves. The architectural backbone of SAGE uses a three-tier exploration strategy: a pre-exploration phase for structural understanding, a top-level exploration strategy for self-evolving curriculum generation, and a low-level exploration mechanism that integrates planning-based rollouts with stepwise learning.   \nThis exploration tackles one of the biggest limitation of model web agents, i.e., relying on static, expensive, and limited human-authored task datasets and curated expert demonstrations.", "questions": "See above.", "limitations": "See above.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strenghts**\n\nThe paper is well written and presents clearly all the components. It also seems to cover relevant related work, although I am not very familiar with the topic and I can't say if more references are needed.\n\nI find the key idea of the proposed self-exploration strategy well presented and interesting. In particular, the idea of building a curriculum of tasks from easy to hard seems appropriate and useful for self-exploring the web.\nThe paper's core contribution, \"self-guided hierarchical exploration\" for generalist web agents, demonstrates significant originality. While established techniques like MCTS and DPO are leveraged, their innovative integration into a unified, self-supervised framework that autonomously generates its own training curriculum -- without relying on static, human-authored task sets -- is novel. This self-generation of tasks and the multi-level exploration strategy fundamentally differentiate SAGE from prior approaches like PAE.\n\nFrom the presented results, finetuning models with SAGE clearly outperforms benchmarks on all baseline, which I find impressive. \nWhile its performance declines with increasing task complexity (easy, medium, hard) for all models, SAGE's relative improvement over baselines is more pronounced on harder tasks.\nThe ablation studies also effectively demonstrate the critical contribution of SAGE's core components.\n\n**Weaknesses**\nThe paper acknowledges a lack of error bars and statistical significance for the experimental results, justifying this by citing the high computational resources required for multiple runs. While understandable given the scale of foundation model research, this remains a methodological weakness for a NeurIPS submission, where statistical robustness is highly valued. \n\nThe paper's discussion on broader impacts appears limited to a checklist affirmation, stating that safeguards against misuse are \"NA\" because the method is \"not designed for generation.\" However, an autonomous web agent, even if not a generative model in the content-creation sense, interacts with real-world web environments and can perform actions with significant consequences.\nWhile I don't think this falls under any of the \"major ethical concerns\", it should be addressed and discussed by the authors.\n\nSimilarly, autonomous agents can be vulnerable to prompt injection attacks, unauthorized data access, and privilege escalation. The paper does not explicitly discuss safeguards against such security threats in an operational context.\nOr, given that these agents explore autonomously without the need of human-labeled tasks, what prevents them to pose security threats? In the checklist, the authors say that no safeguards are needed, but I disagree.\n\nFinally, as agents become more autonomous, their decision-making processes can become opaque. The paper does not delve into how accountability would be ensured when an autonomous SAGE agent makes errors or causes unintended harm, nor does it discuss mechanisms for explaining its actions. While this is true for any web agent, I believe this is more relevant to SAGE given its autonomous exploration mechanism.\nA more comprehensive discussion of these potential harms and concrete mitigation strategies (e.g., human-in-the-loop oversight, rate-limiting autonomous actions, anomaly detection, ethical filters in the policy) is warranted for a work with such significant real-world interaction capabilities.\n\nThe paper explicitly states that the experiments require \"too much computational resources,\" hindering replicability and further extensive experimentation. This high barrier to entry poses a practical limitation for the broader research community to build upon or verify SAGE's findings without substantial computational infrastructure.", "was_revised": true, "final_justification": "I maintain my positive rating.\nMy concerns regarded 1) limited seeds, and 2) safety and ethics, but since they were \"easy fix\" they did not impact my final rating (borderline accept). If the authors would have not addressed my concerns in the rebuttal, I would have lowered my score.", "rebuttal": "Thanks a lot for your insightful and inspiring comments! We provide the following clarifications in response to your concerns:\n\n\n1. **High computational resources requirements and error bar**\n\n - We thank the reviewer for raising concerns regarding the lack of error bars and statistical analysis. We fully agree that reporting variance is important. However, as in prior work on web agents [1,2], collecting multiple runs for all experiments is computationally prohibitive due to the high cost of real web interactions. This is an inherent challenge in training and evaluating real-world web agents at scale, where each run requires lots of browser-based operations with full model inference.\n\n - To provide evidence of robustness, we conducted 5 independent runs on WebArena for our largest open-source model and the proprietary model. The observed standard deviation across different websites was consistently low, suggesting that performance differences are reliable rather than stochastic. Detailed results are provided below:\n\n| Model                 | Map            | Reddit         | OSS            | GitLab         | CMS            | Avg           |\n|----------------------|----------------|----------------|----------------|----------------|----------------|---------------|\n| Claude 3.5 Sonnet    | 40.1 ± 1.4     | 46.9 ± 0.8     | 42.0 ± 0.9     | 24.2 ± 0.5     | 25.1 ± 1.1     | 35.6 ± 1.0     |\n| Qwen2.5VL-32B SAGE    | 42.3 ± 0.4 | 43.7 ± 0.6 | 46.1 ± 0.7 | 31.4 ± 1.2 | 35.2 ± 0.8 | 39.7 ± 0.7 |\n\n - The high computational requirements primarily arise from the intrinsic difficulty of web agent training: strong base models and sufficient task diversity are necessary for meaningful evaluation. While our current compute budget limits the scale of repeated evaluations, we emphasize that our results remain reproducible. To help the community build upon this work, we will release all code, trained models, and necessary configurations upon acceptance. We also believe that larger-scale evaluations can be conducted by organizations with greater compute resources, and we welcome such extensions in future work.\n\n[1] Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents. ICML 2025.\n\n[2] WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning. ICLR 2024.\n\n2. **Broder Impact**\n - We acknowledge that our discussion of broader impacts in the original submission was limited, and we appreciate the reviewer’s suggestion to elaborate further. While SAGE focuses on training web agents, the interaction with real-world websites (e.g., during WebVoyager evaluation) could raise potential security concerns, such as accidental access to unsafe websites or unintended information leakage. In our experiments, we mitigated these risks by deploying the agents within a controlled network environment equipped with firewall-like mechanisms and strict network policies to block potentially harmful requests. Furthermore, the agents were not provided with any personal or sensitive information, and all tasks involving user data or authentication were restricted to sandboxed environments (e.g., WebArena).\n\n - It is important to emphasize that our work primarily aims to improve the technical capabilities of web agents in a research context, rather than deploying them for open-ended, real-world usage. For future large-scale deployment, additional safeguards would be necessary, such as enhanced firewall protections, strict access control, human-in-the-loop validation, and anomaly detection mechanisms. We thank the reviewer for raising this point and will add a more detailed discussion of these considerations in the revised version.\n\n3. **Safety Issue**\n - We thank the reviewer for emphasizing the importance of safety concerns. We fully agree that ensuring the safe and responsible deployment of autonomous web agents is a critical issue—not only for our framework but for the field at large.\n\n - In our current work, SAGE is purely a *research framework* designed to enhance the autonomous learning capabilities of web agents. All experiments are conducted in strictly controlled environments: WebArena is a fully sandboxed simulator, and WebVoyager provides access only to publicly available pages, without involving any private or sensitive user data. Moreover, all web access is routed through a secure firewall layer that restricts and monitors outbound traffic, ensuring that the agent cannot interact with external websites beyond the experimental scope or perform actions with real-world consequences. These safeguards ensure that the agent operates safely within research boundaries.\n\n - That said, we recognize that general safety concerns surrounding web agents, such as unintended interaction with real-world systems, information leakage, or malicious misuse, are of paramount importance. We agree with the reviewer that these concerns warrant deeper discussion. In particular, the potential for LLM-based agents to autonomously explore the web raises novel safety risks that differ from traditional AI systems. These include challenges around authentication spoofing, unintentional automation of sensitive tasks, and reinforcement of harmful behaviors through reward hacking.\n\n - We believe these are important open problems in the design and deployment of general-purpose web agents, and we are committed to addressing them in future work. In the revised version, we will expand our discussion of these issues in a dedicated section. This will include considerations such as: (1) decoupling policy outputs from execution via an intermediate control layer with safety filters; (2) adding a dedicated safety-monitor model to detect and intervene on risky behaviors; and (3) incorporating human-in-the-loop oversight and runtime constraints such as rate limiting and permission gating.\nWe reiterate that our current research focuses on algorithmic capability in safe offline environments, but we strongly agree that future deployment must be accompanied by rigorous safety audits and infrastructure-level safeguards. We thank the reviewer again for highlighting this essential topic."}, {"review_id": "AX8h6mLugW", "reviewer": "Reviewer_ZYML", "summary": "This paper presents SAGE, a fully autonomous framework for training LLM-based web agents without any human-written instructions or demonstrations. It introduces a hierarchical exploration process with three levels: 1) Pre-exploration to map website structure, 2) Top-level exploration to generate a curriculum of tasks that evolve based on agent performance, and 3) Low-level exploration using MCTS and preference-based learning to improve execution efficiency. Experiments on WebVoyager and WebArena benchmarks show that SAGE outperforms previous self-supervised methods like PAE and proprietary model like Claude 3.5 Sonnet.", "questions": "- How does the task generator ensure generated tasks are valid and non-trivial?\n- How does SAGE compare with GPT-4V?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The proposed method is fully autonomous without relying on predefined instructions or expert demos.\n- The hierarchical structure (pre-exploration, curriculum learning, MCTS planning) is well-motivated and technically sound.\n- SAGE achieves strong empirical results particularly on long-horizon tasks.\n\nWeaknesses:\n- While SAGE is fully autonomous, the core framework (propose-act-evaluate) remains largely similar with previous work PAE, making the novelty incremental. \n- The fully-autonomy increases exploration space and computational cost. The system may be inefficient or brittle without human instructions.\n- Missing comparison with GPT-4V and other strong prompt-based agents, which are widely used for autonomous web exploration.", "was_revised": true, "final_justification": "I think my concern about novelty compared to PAE is addressed to a good extent by the rebuttal. However, I still have concerns about the validity and quality of the generated tasks. Therefore my ratings remain the same.", "rebuttal": "Thanks a lot for your insightful and inspiring comments! We provide the following clarifications in response to your concerns:\n\n1. **Novelty compared with PAE**\n - We appreciate the reviewer’s comment on the novelty relative to PAE. While our work inherits the general proposer–agent–evaluator pipeline introduced by PAE, the core contributions and motivations of SAGE are fundamentally different. PAE’s primary goal is to provide *a usable framework for web agents to train without relying on predefined task sets*, showing that autonomous web training is feasible. In contrast, our focus is on introducing a novel technical approach that allows web agents to learn progressively from simple to complex tasks through *hierarchical exploration*. This is a key distinction: while PAE establishes the feasibility of self-training, it does not provide an effective mechanism for skill acquisition on long-horizon or complex tasks, often leading to stagnation when the agent fails to make progress on difficult instructions.\n\n - To address this, SAGE introduces a *hierarchical exploration strategy* that bridges this gap. Our pre-exploration phase systematically builds structural knowledge of the website, enabling the task generator to propose relevant and feasible tasks. The top-level exploration module dynamically adjusts the curriculum, by decomposing tasks with low success rates into simpler subtasks and composing new, harder tasks from high-success ones, while the low-level exploration module, built on MCTS rollouts and step-wise learning, enables the agent to actively search for solutions under sparse rewards. This design allows the agent to bootstrap its skills, mastering simpler operations before tackling complex reasoning tasks.\n\n - Empirically, this technical innovation translates into significant performance gains: SAGE improves over PAE by 7–15% on WebVoyager and 15–25% on WebArena, demonstrating that our hierarchical exploration framework substantially enhances both adaptability and final success rates beyond what PAE can achieve.\n\n2. **System efficiency without human instructions**\n\n - While fully autonomous learning may seem to enlarge the exploration space, SAGE mitigates this challenge through a structured *pre-exploration* phase, which systematically maps the interactive elements and navigational pathways of each website. This phase drastically reduces irrelevant exploration by building a compact structural representation of the environment before training begins. Moreover, our top-level exploration module dynamically adapts task difficulty, ensuring that the agent progresses from simple tasks to complex objectives in a guided manner rather than exploring blindly.\n - Regarding the use of human annotations, our framework relies on extremely limited human input. The only human-labeled data we use is for fine-tuning the evaluator. Importantly, SAGE remains effective even without human annotations. As shown in the following table (also reported in Table 2 of the supplementary), SAGE trained with only model-generated evaluator signals performs strongly, achieving results close to or better than using human corrections.\n\n| Model                          | Map  | Reddit | OSS  | Gitlab | CMS  | Avg  |\n|--------------------------------|:----:|:------:|:----:|:------:|:----:|:----:|\n| Qwen2.5VL-7B SAGE (Claude)     | 24.3 | 25.3   | 38.4 | 16.3   | 14.2 | 23.0 |\n| Qwen2.5VL-7B SAGE (Qwen)       | 23.2 | 26.3   | 36.2 | 16.0   | 14.6 | 22.7 |\n| Qwen2.5VL-7B SAGE              | 28.1 | 29.7   | 43.2 | 18.9   | 15.6 | 27.1 |\n| Qwen2.5VL-32B SAGE (Claude)    | 41.7 | 40.8   | 41.3 | 29.6   | 32.4 | 36.4 |\n| Qwen2.5VL-32B SAGE (Qwen)      | 39.9 | 40.7   | 41.6 | 25.8   | 30.2 | 34.5 |\n| Qwen2.5VL-32B SAGE             | 42.3 | 43.7   | 46.1 | 31.4   | 35.2 | 39.7 |\n\n\n3. **Comparisons with other strong prompt-based agents.**\n - We appreciate the reviewer’s suggestion to include comparisons with other prompt-based agents such as GPT-4V. In our experiments, we have already compared against Claude 3.5 Sonnet, which is widely recognized to be on par with or even** stronger** than GPT-4V in web reasoning tasks. Our results show that SAGE-trained open-source models, such as Qwen2.5VL-32B, achieve comparable or even better results than Claude 3.5 Sonnet on both WebVoyager and WebArena benchmarks, demonstrating the strong effectiveness of our approach.\n - To further address this point, we additionally provide GPT-4V results on WebArena. Even when compared to GPT-4V, SAGE-trained models achieve competitive or superior performance despite using significantly smaller base models without any manual prompt engineering.\n - We also note that SAGE is fundamentally a self-improvement framework rather than a single static agent, and thus its contribution lies in the training methodology rather than the absolute performance of the base model. Our approach can, in principle, be applied to even larger closed-source models to achieve further gains.\n| Model                  | Map  | Reddit | OSS  | Gitlab | CMS  | Avg  |\n|------------------------|:----:|:------:|:----:|:------:|:----:|:----:|\n| GPT-4V                 | 38.3 | 45.6   | 42.5 | 25.0   | 24.6 | 34.8 |\n| Claude 3.5 Sonnet      | 40.1 | 46.9   | 42.0 | 24.2   | 25.1 | 35.6 |\n| Qwen2.5VL-32B SAGE     | 42.3 | 43.7   | 46.1 | 31.4   | 35.2 | 39.7 |\n\n4. **Quality of generated tasks**\n - Regarding the validity of generated tasks, we do not guarantee that every task is fully valid, as this issue also exists in the original WebVoyager and WebArena datasets. For tasks that turn out to be invalid or infeasible, the agent will output a response indicating that the task cannot be completed, and the evaluator will verify whether this judgment is correct. As a result, invalid tasks do not negatively impact the training process or the final policy quality.\n - For the non-triviality of tasks, our top-level exploration strategy adaptively adjusts task difficulty based on the agent’s performance. Tasks with high success rates are merged into more complex ones, while failed tasks are decomposed into simpler subtasks. This approach ensures that tasks remain diverse, challenging, and effective for skill acquisition. Although the task generator is not intended to perfectly match the real-world task distribution, it successfully guides the agent to progress from simple to complex tasks, which is validated by our significant improvements on both WebVoyager and WebArena benchmarks."}, {"review_id": "qcMKhJfQx7", "reviewer": "Reviewer_3FCb", "summary": "The paper describes a protocol for collecting training data for web agents by largely unsupervised self-guided exploration. The protocol is rather complex, it involves three stages. The first phase is the structural exploration of website that results in a knowledge base of accessible URLs. The second phase is adaptive task generation that takes into account the agent’s current performance. The third phase is fine grained exploration to solve as many tasks as possible using MCTS. Experiments show that the proposed protocol helps achieve higher performance compared to a strong baseline.", "questions": "- My understanding is that MCTS requires tight control over the state of the environment? How were able to do this in the case of the open-web WebVoyager task?\n- lines 378 - 381: \"removing the low-level exploration module leads to a clear drop in both sample efficiency and final task success rate\" -  I did not find sample efficiency results in Table 5\n- lines 297 - 300: the paper mentions SFT results with the SFT data coming from Claude; I did not find these results in the paper.\n- can you talk more about the human annotation efforts in the paper? I see some information in the supplementary material, but I think it is important enough to be discussed in the main text. \n- what would be SAGE's performance without human annotation?\n- I think a better explanation of the baseline PAE method would improve the paper", "limitations": "How would this method work for pure JavaScript web pages that mostly use the same URL?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The main strength of the paper is that it reports very strong empirical results. Increasing the performance of Qwen32B from 49.7 to 62.4 is quite impressive. But the method is very complex and the presentation is rather high-level, it would be very hard to reproduce the results from this paper. Additionally, the paper glosses over some crucial details, such as e.g. “a limited number of human-annotated examples” that were used to train the reward model. One can see from Table 6 that the finetuned Qwen2.5VL-7B evaluator is in fact stronger than the frontier models. I’d guess it required a substantial data collection, and hence the exploration was not really purely self-guided in the presented study. \n\nThere are other things that are not very clear from the paper, see “Questions” section. \n\nOverall, the paper's results are worth sharing, but I encourage the authors to provide more details to increase reproducibility.", "was_revised": true, "final_justification": "I will maintain my original assessment.", "rebuttal": "Thanks a lot for your insightful and inspiring comments! We provide the following clarifications in response to your concerns:\n\n\n\n1. **Human-annotated data**\n\n - We would like to clarify that our framework is largely self-guided and relies only on a very small amount of human-annotated data. Specifically, we collected 1,200 trajectories with human annotations to fine-tune the outcome evaluator in the final stage of training. These annotations do not form the **core** of the agent training pipeline.\n\n - To demonstrate that SAGE remains effective even without any human annotations, we provide an ablation study. As shown in the table below (also refer to the table 2. In the appendix), the performance drop is minor when removing the human-annotated data, confirming that our method’s strong performance is driven by autonomous exploration rather than manual supervision:\n\n| Model                          | Map  | Reddit | OSS  | Gitlab | CMS  | Avg  |\n|--------------------------------|:----:|:------:|:----:|:------:|:----:|:----:|\n| Qwen2.5VL-7B SAGE (Claude)     | 24.3 | 25.3   | 38.4 | 16.3   | 14.2 | 23.0 |\n| Qwen2.5VL-7B SAGE (Qwen)       | 23.2 | 26.3   | 36.2 | 16.0   | 14.6 | 22.7 |\n| Qwen2.5VL-7B SAGE              | 28.1 | 29.7   | 43.2 | 18.9   | 15.6 | 27.1 |\n| Qwen2.5VL-32B SAGE (Claude)    | 41.7 | 40.8   | 41.3 | 29.6   | 32.4 | 36.4 |\n| Qwen2.5VL-32B SAGE (Qwen)      | 39.9 | 40.7   | 41.6 | 25.8   | 30.2 | 34.5 |\n| Qwen2.5VL-32B SAGE             | 42.3 | 43.7   | 46.1 | 31.4   | 35.2 | 39.7 |\n\n - This result shows that even using raw model evaluators without human annotations, SAGE still performs competitively, and the additional fine-tuning only provides a modest improvement.\n\n2. **MCTS in open-web environments**\n\n - We agree with the reviewer that MCTS typically requires tight control over the environment state. In our framework, we address this by using Playwright to wrap the webpage state, allowing us to store and reload the web page. This enables the reset functionality required by MCTS. Regarding open-web WebVoyager versus the sandboxed WebArena, there is no fundamental difference in how MCTS operates: while some webpages in WebVoyager may occasionally change, they generally remain stable over the short time horizon of a single search process. If a specific branch of the search tree cannot be reliably replayed due to dynamic content, we simply discard that branch and continue the search. We acknowledge that our current approach is less robust for pure JavaScript single-page applications that heavily reuse the same URL, and we consider improving MCTS handling for such dynamic websites an interesting direction for future work.\n\n3. **Efficiency of low-level exploration**\n\n - We appreciate the reviewer’s question regarding the efficiency of our low-level exploration strategy. In our context, sample efficiency refers to the success trajectories achieved by the agent when generating trajectories under the same number of tasks tried. Compared to a standard rollout strategy, our low-level exploration prioritizes promising action sequences and avoids repeatedly sampling failed trajectories. This results in a higher proportion of successful trajectories for the same number of samples, which translates into a more effective training signal and faster policy improvement.\n\n - To provide direct evidence, we conducted an additional ablation study on WebArena using the same initial SFT models. For each model, we sampled 10k trajectories using either standard rollouts or our low-level exploration. As shown in the table below, the low-level exploration strategy consistently produced more successful trajectories, which validates its superior sample efficiency:\n\n\n| Model                  | # Successful Trajectories (10k Samples) |\n|------------------------|-----------------------------------------|\n| Qwen2.5VL-7B + Normal  | 1937                                    |\n| Qwen2.5VL-7B + Low     | 2102                                    |\n| Qwen2.5VL-32B + Normal | 2578                                    |\n| Qwen2.5VL-32B + Low    | 2895                                    |\n\n - By enabling targeted search over the action space, our low-level module improves not only the learning speed but also the final task success rates, as confirmed by our main results and ablations in Table 5.\n\n\n4. **SFT claude data**\n\n - Different from PAE, which emphasizes demonstrating that the proposer–agent–evaluator framework can improve performance based on SFT models, our focus is on showing that SAGE provides a more effective training paradigm compared with PAE. Due to space constraints, we did not include the SFT baseline results in the main text. For completeness, we now report the SFT performance on WebArena, compared with PAE and SAGE in the table below.\n\n| Model                | Map  | Reddit | OSS  | Gitlab | CMS  | Avg  |\n|----------------------|:----:|:------:|:----:|:------:|:----:|:----:|\n| Qwen2.5VL-7B SFT     | 16.5 | 21.3   | 32.6 | 13.7   | 10.1 | 19.4 |\n| Qwen2.5VL-7B PAE     | 23.8 | 24.6   | 37.4 | 15.3   | 12.4 | 22.7 |\n| Qwen2.5VL-7B SAGE    | **28.1** | **29.7** | **43.2** | **18.9** | **15.6** | **27.1** |\n| Qwen2.5VL-32B SFT    | 27.3 | 27.8   | 38.0 | 18.2   | 14.7 | 25.7 |\n| Qwen2.5VL-32B PAE    | 34.5 | 35.9   | 41.0 | 24.1   | 22.0 | 31.5 |\n| Qwen2.5VL-32B SAGE   | **42.3** | **43.7** | **46.1** | **31.4** | **35.2** | **39.7** |\n\n5. **Comparison with PAE**\n\n - The baseline method PAE establishes a general *self-improving framework* for web agents by combining three components: a proposer that generates diverse task instructions, an agent that attempts to solve these tasks, and an evaluator that judges task completion to provide rewards. This design successfully removes the reliance on manually curated datasets, enabling web agents to improve through autonomous interaction with web environments. However, PAE samples tasks from a static instruction distribution and does not explicitly address the progression from simple to complex tasks or the challenges of long-horizon exploration.\n\n - Building on this foundation, our work introduces a *hierarchical exploration strategy* as the **core** of SAGE. Specifically, we propose 1) a self-guided task proposer that adaptively adjusts task difficulty by analyzing the agent’s performance and decomposing difficult tasks into simpler subtasks, and 2) a low-level exploration mechanism using MCTS-based planning and step-wise learning to improve trajectory quality and sample efficiency. These contributions transform the training process into a dynamic curriculum that allows agents to acquire fundamental skills first and then tackle increasingly complex tasks. As a result, SAGE significantly improves performance compared to PAE, while maintaining the fully autonomous training paradigm.\n\n6. **Complexity and reproducibility**\n\n - We acknowledge that SAGE involves multiple components, but each module is designed to address a specific challenge inherent to training autonomous web agents. The pre-exploration stage reduces the enormous task space by systematically mapping website structures; the top-level exploration module tackles the difficulty of long-horizon tasks by generating a self-evolving curriculum from easy to hard; and the low-level exploration module improves the agent’s ability to explore and solve complex tasks efficiently.\n\n - To ensure reproducibility, we have already provided extensive technical details in both the main paper and the supplementary material, including full descriptions of the pre-exploration process, task generator prompts, pseudocode for the MCTS rollout, hyperparameter configurations, and evaluator training procedures. Furthermore, we will release the full codebase and trained models upon acceptance to facilitate reproducibility and further research."}]

Author rebuttal:

Meta-review: The submission proposes an algorithm for training web agents to solve tasks in a particular environment. First, it explores the website to build a map. Then, it generates tasks starting with lower complexity. Finally, MCTS search is used to improve the policy. 

The reviewers concur on the value of the strong results. Most of the questions were clarified in the rebuttal.

For instance, reviewer 3FCb said: 

> the method is very complex and the presentation is rather high-level, it would be very hard to reproduce the results from this paper. Additionally, the paper glosses over some crucial details, such as e.g. “a limited number of human-annotated examples” that were used to train the reward model.  

Indeed, MCTS requires an evaluator. The supplementary material says that training the evaluator required manual annotation of 400 cases where Claude 3.5 Sonnet and humans disagreed. 

One key issue was pointed out by reviewer ZYML, who, in their final justification, said:
> I still have concerns about the validity and quality of the generated tasks

The task generator is an LLM prompted to generate tasks of increasing difficulty. That probably worked because the benchmarks are well-known user experiences on the web.

Considering the empirical results, the quality of the ablation study, and despite the lack of analysis of the task-generated data, the anecdotal discussion of the evaluator's quality, and the possible missing details for reproducing the work, I consider that the submission should be accepted.

Please follow the recommendation of the Ethics Reviewer
Decision: Accept (poster)