Proposer-Agent-Evaluator (PAE): Autonomous Skill Discovery For Foundation Model Internet Agents
Authors: Yifei Zhou
Qianlan Yang
Kaixiang Lin
Min Bai
Xiong Zhou
Yu-Xiong Wang
Sergey Levine
Li Erran Li
Venue: ICML 2025 poster (2025)
Keywords: Web agents
LLM Agents
Autonomous skill discovery
Primary area: deep_learning->large_language_models
TL;DR: We proposer PAE, a framework that automatically discovers and learns web navigation skills without manual instruction

Abstract: A generalist foundation model agent needs to have a large and diverse skill repertoire, such as finding directions between two travel locations and buying specific items from the Internet. If each skill needs to be specified manually through a fixed set of human-annotated instructions, the agent’s skill repertoire will necessarily be limited due to the scalability of human-annotated instructions. In this work, we address this challenge by proposing Proposer-Agent-Evaluator (PAE), an effective learning system that enables foundation model agents to autonomously discover and practice skills in the wild. After a context-aware task proposer generates instructions based on website information, the agent policy attempts those tasks in the real world with resulting trajectories evaluated by an autonomous VLM-based success evaluator. The success evaluation serves as the reward signal for the agent to refine its policies through RL. We validate PAE on challenging vision-based web navigation, using both real-world and selfhosted websites from WebVoyager and WebArena. Our results show that PAE significantly improves the zero-shot generalization capability of VLM Internet agents (around 50% relative improvement)
to both unseen tasks and websites.

Proposer-Agent-Evalutee (PAE): Autonomous Skill Discovery for Foundation Model Internet Agents
Yifei Zhou 1
Qianlan Yang"
Kaixiang Lin
Min Bai
Xiong Zhou² Yu-Xiong Wang? ? Sergey Levine' Erran Li3
Abstract
also elucidate some of the deepest questions in AI. Recent advancements in foundation models provide with com-
A generalist foundation model agent needs to have a large and diverse skill repertoire, such as find- ing directions between two travel locations and buying specific items from the Internet. If each skill needs to be specified manually through a fixed set of numan-annotater instructions, the agent's skill repertoire will necessarily be limited instruc-
pelling possibility for how this vision could be realized (Ope)
nAl 2024] GeminiTeam 2024). Pre-trained foundation models have been used to explore generalist agents (Liu
ct al. 2023b) in real-world decision decision-making scenarios, such as navigating websites to make travel plans (He et al. 2024a)
and solving real-world Github issues (Jimenez et al. 2024 To succeed in these decision- making domains, goal directed post-training is often needed to elicit long-horizon reward- maximizing behaviors such as information seeking (Hong)
due to the scalability of tions. In this work, we address this challenge by proposing Aeepottr-Ageet--iee Evaluator (PAE), an effective learning system that enables foundation model agents to autonomously discover and prac- tice skills in the wild. After a context-aware task proposer generates instructions based website information, the agent policy attempts those tasks in the real world with resulting trajectories evalu- ated by an autonomous VLM-based success eval- uator. The success evaluation serves as the reward signal for the agent to refine its policies through based
et al. 2022a) and recovery from mistakes (Bai et al. 22024). instead of only imitating the most probable actions in the pre-training corpus.
A crucial requirement for a successful post-training ap- proach is to endow the generalist agent with large and di- verse goal-directed skill repertoire. This can include finding directions between two travel locations and buying specific items from the Internet, which the agent can then exploit to solve real-world tasks proposed by users. However, manu- ally specifying the skills (Deng et al., (023) (i.e. through static set of aamaa--nnnttted instruction templates such as "Find the driving directions and estimated time to travel from Location A to Location B") will likely result in a lim- ited skill repertoire. In particular, generating high-quality task templates can be expensive, making it impractical to scale up. The use of a small set of task templates fails to capture the range of skills an agent needs for the full breadth of the real world, leading to distribution shift problems when deployed at the test time. Therefore, it naturally raises the research question: instead of requiring users to manually define tasks for foundation model agents, can these agents autonomously discover and practice po- tentially useful skills on their own?
RL. We validate PAE on challenging web navigation, using both real-world and self- hosted websites from Web Voyager and WebArena. Our results show that PAE significantly improves the zero-shot generalization capability of VLM In- ternet agents (around 50% relative improvement) to both unseen tasks and websites.
1. Introduction
The vision of a broadly capable and goal-directed agent has long captured the imagination: from agents that can browse the web to fulfill user instructions robots that could per- form diverse tasks in the home, AI systems that can indepen- dently accomplish open-wont world goals would not only present tremendous practical value, but their development would
To answer this question, we first note the observation of the asymmetric capabilities of SOTA VLMs as skill proposer- slevaluators and agents (ablation experiment results in Section 5) in many realistic task settings such as web agents. Intuitively, VLMs are very good at conceiving common tasks that people would like to perform on shopping web- site (c.g. adding a specific product to cart) and confirming
Equal contribution 'University of California, Berkeley Uniiersityy of Illinois, Urbana- Champaign Amazon. Correspon- dence to: Yifci Zhou zyifci @ berkeley.edu>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Proposer-Agent--aalalle (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Agent Environments
RL Training
Agent Policy
air a
nOw NEWS
Roll-out Trajectories
Thought:
C
E
Action: ...
Autonomous Evaluator
CHAMIIC
Gather context information
Generate proposal tasks
Names
User Demos
Context-aware Task Proposers
Reddit
Discover the available options for accessing materiols lectures in different languages on Coursero.
Google Map
HuggingFace
Figure 1. An overview of Proposer-Agent-A Evaluato showing the main components of our autonomous skill discovery framework, that endows the agent with autonomously discovered skills for future human requests.
whether specific product has been added to the shopping cart (e.g. by looking at the final screenshot to see if the shopping cart contains the product), while worse at actually navigating through the web to find the product and add it to the cart. If our system can be designed to maximally exploit this asymmetric gap, we may bootstrap the performance of the agent capability by reliable signals from task proposers and evaluators. However, merely discovering possible skills is not enough, since the ultimate goal is to do well for un- seen realdworld human requests. The right design choices of the system also need to be carefully validated to ensure that the autonomously discovered skills are actually useful and can generalize well to unseen human requests.
different sources and are shown to be effective, such as user demos and even website name alone! Moreover, to obtain the most robust reward signal without accessing the hidden state information, we apply an image-based evaluator that only provides sparse 0/1 rewards based on the final outcome. Finally, we design an additional reasoning step before the agents output actual actions, which enables the agents better reflect on their skills and results in significant im- provement in its generalization to unseen tasks.
The scope of our experiments covers challenging end-to- end vision- web navigation, where the observation space simply contains the screenshot of the current web page and the action space contains primitive web operations such as clicking on links and typing into text boxes. We validate the effectiveness of PAE framework with realis- tic web-navigation benchmarks, including more than 100 domains both from online websites like Amazon from We- b boyager (He et al., 2024a) and self-hosted websites like PostMill from WebArena (Zhou et al. 2024a). In our ex- periments, we find that PAE with LLa Va- 1.6 (Liu al., 2024) the agent policy can autonomously discover useful skills through interactions with various websites without any human supervisions, even when the task proposer and evaluator model is significantly worse than the agent model. More importantly, our results demonstrate that these skills can zero-shot transfer to unseen test instructions and even unseen test websites. On websites from Web Veyaer and WebArena, PAE attains around 50% relative improvement in success rate, enabling L.La Va-1.66666-78 to achieve perfor- mance comparable with LLaVa- Va-1.6-344 fine-tuned with demonstration data and Qwen2VL-725 Yang et al. (024a).
In this work, our main contribution is propose an effec- tive learning system, Proposer-Agent EvagttttEvlluutr(PPAE) for agents (in particular, Internet agents) based on foundation models to autonomously discover new skills without any human supervision and these new skills can be effectively exploited to solve unseen real-world Human--nnnoatee tasks in zero-shot manner. At a high level, PAE identifies the interesting skills with task proposer, attempts them with an agent policy, and performs online RI loop based the reward provided by an autonomous evaluator. PAE iden- tifies the following design choices to obtain reliable signals from the task proposer and evaluator, and ensure that the agent can extrapolate from autonomously discovered to un- seen human requests. First of all, to propose feasible and realistic tasks, PAE employs class of context-awee task proposers where the context of functions and constraints crucially define what actions are supported by the specific environments (e.g., creating reddit post) while others may not be supported (e.g., checking the protected information of other users). Such context can implicitly defined from
2

---

Proposer-Agent--Valuttor (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
2. Related Works
choices are. While some concurrent works (Q et al. 2024
He al. 2024b, Murty al. 2024) also apply autonomous task proposals to training web agents where instructions and evaluation rewards are generated from stronger proprietary models, our PAE system focuses on exploiting the asymmet- ric capabilities of foundation model web agents to achieve self-improvements, where even weaker models can be used to improve the performance of stronger agents.
Foundation model agents. Thanks to the generalization ca-
pabilities of Large Language Models (LLMs) (Brown et al.
2020
Laaa3 Team
2024, Gemini
2024) and Vision
Langguage Models (VL.Ms)
(OpenAl 2024 Liu al. 2024
Wang et al.
c024c Liu et al. 2023a Zhai et al.
2024). re-
cent works have successfully extended such agents to more
general real-world use cases (Bai et al.
2024
Zheng et al.
2024, He et al.]
2024a Zhang et al.]
2023a
Zhou et al.
Unsupervised skill discovery in deep RL. Unsupervised skill discovery has been an important research direction in the field of traditional deep RL literatrue (Achiam et al., 2018 Eysenbach et al.] 2018) where various algorithms have been developed to discover new robotic skills such as humanoid walking without the need of explicitly deñindd reward functions. Common algorithms in this field aim to discover every possible skill (both meaningful skills like walking and less meaningful ones like random twisting) through either maximizing the mutual information between different states and skill latent vectors (Campos et al. 2020 Laskin et al. 2022
2024a Koh et al.
2024, Gur et al.
2021 Furuta et al. 2024
Lin et al. 2024 Xu et al.
2024a b). Besides constructing
prompting wrappers around proprietary VLMS (Zhang et al.
2023a He al. 2024a| Zheng al.
2024 Xie al. 2024
Yang et al. 2024b Wang et al.
2023a) and fine-tuning open-
source VLMs with expert demonstrations (Gur et al. [021
Hong et al.
2024, Zhang & Zhang 2023b; Furuta et al.
2024|Zenge et al. 2023 Chen al. 2023). a recent trend has
emerged involving interactive improvement of L.LM/VI.M .MM in particular web/GUI agents, through autonomous evalua-
tor feedback (Pan al. 2024: Bai et al.] [2024] Putta al.
Sharma et al. 2020). or maximizing the divergence of each skill as measured in a metric space (Park] et 2022 2023 2024). In contrast, our work only dis- covers meaningful skills as specified through natural lan- guage instructions with the help of pre-trained foundation models, significantly reducing the search space of skills in LLM/VLM agent applications with complex state spaces.
2024, |aan et al. 2024b), where evaluator LLMs/VLMSs
are prompted to evaluate the success of the agents to serve as the reward signal. This approach aims to elicit goal-oriented and reward-opting reward-optimizing behaviors from foundation models with minimal human supervision. However, these methods still depend on a static set of human-curated task templates, constraining their potential and scalability. Our work intro- duces framework where agents can discover and practice the skills they find useful, thereby eliminating the reliance on predefined and human-curated task templates and open- ing up new possibilities for scalability and adaptability in training generalist autonomous LLM/VLM agents.
3. Areposer-Agent--vallllee Autonomous Skill Discovery system For Foundation Model Agents
(PAE):
Next, we will explain the technical contributions of this paper. In this section, we will define the general system of PAE including a task proposer, an agent policy, and an autonomous evaluator. We will begin by formalizing the learning goal of this system and then detail the roles of each key component. In the section to follow, we will explain how we apply PAE to VLM Internet agents.
Self-generated instructions. Self-generated instructions for improving LLMss have been shown to be effective in single-turn LLM alignment (Wang et al., 2023b Yuan et al. 2024a) and reason-
2024; |Wu et al.) 2024; Wang et al.
ing (Pang et al.) 2024) domains without interactions with
an external environment. AgentGen (Hu et al., 2024) takes a step further to fine-tune LLM agents with expert trajec- tories in self- generated environments and tasks. However, its feasibility in the self-play agent setting with RL and autonomous evaluators has not been understood. On the other hand, the closest works to ours employ autonomous RL and foundation model task proposers to simplified envi- ronments such as games /hang et al. 2024a Faldor et al.
Problem setup We begin by formalizing the problem setup of autonomous skill discovery for real-world agents. The learning goal of PAE is to find deward-maximizing pol- icy parameterized by 0 in contextual Markov De- cision Process (MDP) environment defined by M = (S, A, R., H,C), where S, A are the state space and ac- tion space respectively, and H is the horizon within which the agent must complete the task. We assume that the agent has access to the environment and can collect online roll-out trajectories through accessing the dynamics model T as a function of determining the next states given the current states and actions. Crucially, we assume that the ground- truth task distribution C and the reward function R are hidden during training and we have to use a proxy task distribution Ć and reward function R instead. Consider the
2024| Colas et al., 2023 2020) and robotics settings with
limited number of scenes (Du et al., 2023 Zhang et al.
2023b Zhou et al. 2024c). While they have shown that the use of autonomous RL combined with foundation model task proposers can help the agent learn diverse skills, this work takes an important step forward to study when those skills can generalize to human requests in realistic bench- marks in the context of web agents and what the best design
3

---

Proposer-Agent--aalaaro (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
setting of training a real-world Internet agent. The dynamics
-errof-Markk Observation
Action Types
model 'I would be simulated browser environment that the Internet agent can interact with. The ground-truth task dis- tribution C might be the distribution of tasks that would be asked by the real users when the Internet agent is deployed and a possible choice for the reward function R might be whether the agent has satisfactorily completed the tasks for the real users. In such a world setting, although the agent can freely access resources from the Internet through simulated browser environment during training, assuming knowledge of the ground-truth task distribution and reward function is impractical. Therefore, we employ VLM-based task proposers C and reward model R as proxies. The de- sired outcome is that improving the policy wo with C and R can lead to an improved policy that can successfully gen- eralize to the ground-truth task distribution and reward functions which are only used as evaluations.
Artion
Deseription
Click
Clids on un dlarmnnt 16 thks and buttine).
Type
1ypw to un sieement
(6.g. search bar)
ScroW/
Soxll sciol ber)
Wait
for page load. Go to the previcus page.
Back
Google
Go namepage of google com Firssh the trapcctory Submii an anowor
Answer
Figure 2. An illustration of the observation space and action space of our vision-based web navigation environment.
tigate in the Hugging Face documentation how to utilize the 'Trainer' API for training model on custom dataset, and note the configurable parameters of the Trainer class". As illustrated in Figure 2 each observation from the observation space contains only the screenshot of the last web page just like how humans interact with the Internet. To provide better action grounding. we follow the practice from prior works (Zheng et al., 2024 He et al.) 2024a) to augment the observation space with number marks on top of each interactive element such as web links and text boxes. To execute a web browsing action, the Internet agent can directly output the number of the element to interact with and the corresponding action such as clicking and typing, without the need of locating the coordinates of each web element. Therefore each web action a, contains the type of the action to perform and the number of the element to interact with. Each episode finishes either when the agent chooses to finish through the "Answer" action or when a maximum number of 10 steps have been reached. In our ex- periments, we use ground-truth success detectors (based on either human annotations or functional verifiers) and human annotated tasks from WebArena (Zhou et al., 2024a) and Web Voyager (He et al. 2024a) to evaluate the performance of different policies. Crucially, both the ground-trutt truth suc- cess detector and the distribution of human tasks are kept hidden, which challenges the generalization capability of the learnt skills to generalize to a hidden reward function and task distributions.
Key components Figure 1 shows the interplay between the key components in our framework, including a context- aware task proposer, an agent policy, and an autonomous evaluator. The role of the task proposer é is to serve as a proxy improve on the ground-truth task distribution C during the learning process. However, it might be unreal- istic to expect the task proposer to generate feasible tasks without knowledge of the environment. To provide more context of the functions and constraints of the environment, we assume access to some key information of the environ- ment ZM based on which the tasks ((xM) are proposed. In the Internet agent example, this key information can be screenshots of the websites from user demos, or even just the name of the website itself if it is well-known website such as Amazon.com Similarly, the autonomous evaluator R serves as a proxy of the ground-trum -tuth reward function R. The input to the autonomous evaluator is the current state, the current action from the agent policy, and current task that the agent is attempting. In principle, any RL algorithm can be used update the agent policy π using a dataset D that stores all the autonomous interaction data. In practice, we instantiate VLM-based task proposers and autonomous evaluators by prompting foundation models and they are kept unchanged throughout training.
4.2. Context-Awand Task Proposer
In order to generate diverse set of feasible tasks, we frame task proposing é as a conditional auto-regressive generation based on the context information of the websites. Thanks to the vast pre-training knowledge of relevant context for popular websites like Amazon.com, we find it suffice to use only website name as ZM- However, for less common or access restricted websites such as self-hosted websites in WebArena, it is necessary supply the task proposer with richer context. In the cases of user demos being avail- able, we consider an alternative to sample some additional screenshots from the user demos to serve as the context information. In our experiments, we consider both using proprietary models such as Claude 3-Sonet (Anthropic
4. PAE for VLM Internet Agents
With the general framework set up, we are now ready to dis- cuss the concrete instantiation of PAE in the setting of VLM Internet agents. We start by introducing the environment of vision-based web navigation and then explain how we implement the key components from PAE in this setting.
4.1. Vasion Web Browsing Environment
We consider the general vision-based web browsing environ- ment (He et al., 2024a Koh et al., 22024). The goal for VLM agents in this environment is to navigate through realistic web pages to complete some user tasks o such as "Inves-
4

---

Proposer-Agent--Valuator (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
and open-source models such as Qwen2VL--B YYng et al. 2024a) for the task proposers, with prompts in Ap-
2024
tions: (1) Can our autonomous skill discovery framework successfully discover skills useful for zero-shot transfer to tasks from evaluation task distribution unseen to the task proposer? (2) How does the models trained with PAE com- pare with other open-source VIM agents? (3) How does the effectiveness of PAE scale with the size and performance of the base model? (4) Whether the effectiveness of PAE is limited by the performance of the task proposer and evalua- tor m5del?(5) How does the use of different contexts (e.g. website names and user demos) affect the performance?
pendix D
4.3. Image-Basee Baree-Baedddo Outcome Evaluator
To take full advantage of the asymmetric capability of SOTA VLMs as agents and as evaluators (experimental evidence presented in Section 5). we find most robust for the au- tonomous evaluators to complete the easiest evaluation: eval- uating the success of the final outcome (Bai et al. 2024 He
et al. 2024a) based on the final three screenshots and the agents' final answers to provide only 0/1 response in the end. Other alternatives such as code-based (Zhang et al. 2024a) or step-based evaluations (Pan et al. (024) are either impractical without access to hidden state information or too noisy because of the hallucination issues present even in SOTA VLMs. In our experiments, we also consider both us- ing proprietary models such as Claude-3-Sonee (Anthropic
5.1. Environments
Web Voyager (He et al. 2024a) contains a set of 643 tasks spanning 15 websites in the real world such as ESPN and Arxiv. As tasks in Google Flights and Booking domain are longer feasible due to website updates,
use the subset of 557 tasks spanning the other 13 websites. Human annotations are carried out for evaluating the success of each trajectory as the ground-truth performance measure.
2024) and open-source models such as Qwen2VL-7B (Yang
et al. 2024a). with prompts presented in Appendix
WebArena Zhou et al. 2024a) is a sand-boxed environ- ment that kept an archived version of popular websites from different domains, including OpenStreetMap, GitLab, PostMill, store content management system (CMS), and an E-commerce website (OneStopMarket), It includes in total 812 hand-written tasks with functional verifications as the ground-truth reward function. Since Gitl and CMU do not support multi-thread data collection necessary for RL fine-tuning, our experiments are carried out using the task subsets on SpenStreetMap, PostMill, and OneStopMar- ket. As open-source VLM agents fail to achieve non-trivial performances on PostMill and OneStopMarket (Zhou et al., 2024a). we hand rewrote tasks in those two websites and supplement them with verification functions. Due to these practical constraints, the resulting WebArena Easy con- tains 108 original tasks on OpenStreetMap and 50 rewritten tasks on PostMill and OneStopMarket each.
4.4. Chain-of- -hinugght- Agent Policy
Crucially, as the ultimate goal for the agent policy is to complete human requests, the agent should not only learn diverse skills on the proposed tasks but also reflect on the skills learnt so that they can be helpful for unseen human requests. Therefore, we incorporate an additional reasoning step to outputs the agent's 's -f-thugghht before the actual web operation. This reasoning step is optimized with the any online RL algorithm just like the actual web operation. For its simplicity and effectiveness,, we employ use the REINFORCE objective (Williams) 2004) to optimize the policy with the following loss:
"
[(T) E,
Σ (8,, az))
e log ग (arls:)),
where is an on-policy trajectory with H steps, and r(s;, a.) the reward at i-th step with the action a, at the state s;- Because the 0/1 terminal reward structure, in this work it turns out to be equivalent the most simple online policy optimization algorithm Filtered Behavior Cloning (Filtered BC) that simply imitates all thoughts and actions in success- ful trajectories with the negative log- eigliiki loss. This simple method has been widely adopted in prior literature of RL+LLM such as (Zhou et al., 2024b) [Snell et al.) (2023) This simple policy optimization objective can already lead to superior generalization capability of the learnt agent. In our experiments, our agent policy is initialized from Va-
5.2. Baseline Comparisons
We validate the effectiveness of PAE by comparing it with (1) proprietary VLMs, (2) ofate-of-thh-aas art open-source VI.Ms, and (3) an alterative supervised fine-tuning (SFT) approach. We consider Claude 3 Sonnet and Claude 3.5 Sonnet (Anthropic) (024) for proprietary VLMs, and Qwen2VL-7B, Qwen2VL-72B (Yang et al. 2024a). (Zhang et al. 2024b), and LLa LLaVa-Next-7BB344B (Liu et al. 2024) for SOTA open- source VL.Ms. All models are prompted similar to He et al. (2024a) using set-of-marks augmented screenshot observa- tions and including chain-of offthoughtt in the action outputs. The prompts are included in Appendix [ As SOTA open- source models struggle to achieve non-trivial performance in the challenging web navigation benchmarks except the largest Qwen2VL-72B, we include another baseline LLaVa- SFT that fine-tunes L.LaVa with Claude Sonnet (Anthropic
1. 1.6-Mistral-7B and
Yi-34B (Liu et al. 2024
5. Experiments
The goal of our experiments is to understand the effective- ness of PAE to complete real-world visual web tasks. Specif- ically, we design experiments to answer the following ques-
5

---

Proposer-Agent-Evaluatoo (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Allrecipes
Amazon
Apple
ArXiv
GitHlub
ESPN
Coursera
Cladde 3.5 Sonnet
50.0
68.3
60.4
46.5
58.5
27.3
78.6
Proprietury
Claude 3 Sonnet
15.9
46.3
51.1
39.5
41.4
11.3
45.2
Qwen2V1.-79
0.0
0.0
2.3
2.3
0.0
0.0
2.3
Qwen2VL-72B
0.0
29.0
28.1
9.7
5.9
Open-
V1.25.1.2..588
0
0
0
LLaVa-7B
0
0
0
0
0
0
LLaVa-34B
0
2.3
2.4
LLaVa-7B SFT
4.5
39.0
18.6
16.3
4.9
5.1
16.7
LLaVa-7B PAE
14.3
37.5
17.5
14.6
0.0
33.3
Ours
LLaVa-7B PAE (Qwen7B)
26.5
45.2
16.7
18.2
15.6
0.0
37.5
LLaVa-34B SFT
6.8
26.8
23.3
16.3
4.9
8.6
26.8
LLaVa-34B PAE
22.7
53.7
38.5
25.6
14.6
13.6
42.9
Camhridge Dictionary
Google Map
Gongle Search
HuggingFace
Wolfram Alpha
Average
Claude 3.5
86.0
44.2
66.7
Proprietary
Claude 3 Sonnet
79.1
40.5
41.5
41.9
37.2
61.9
42.4
Qwen2V1.-7B
2.3
0.0
0.0
4.7
0.0
4.8
1.4
Qwen2VL-72B
60.6
12.5
16.1
21.2
9.1
36.4
22.6
Open-source
1nternVL...-888
0
0
2.3
0.2
LLaVa-7B
1LaVa-34B
0
2.3
0
2.3
2.3
0.9
LLaVa-7B SFT
41.9
7.1
19.5
9.3
0
11.9
14,9
LLaVa-7B PAE
52.4
18.6
22.5
23.3
19.0
24.4
22.3
LLaVa-7B (Qwen7B)
LLaVa-34B SFT
67.4
16.7
12.2
23.3
20.9
38.1
22.2
LLaVa-34B PAE
74.4
39.0
22.0
18.6
25.6
42.9
33.0
Table I. Success rate comparisons on Voyage. The results are automatically annotated by Claude Sonnet 3 and human alignment reported Figure For PAE, running average of the ovaluation results at each iteration is reported. The final column is weighted average by the number of tasks on different websites. The results may be different from reported in other papers due to the dynamic nature of online websites.
2024 agent trajectories on self- seff-eenrrated tasks on 85 real- world websites not included in Web Voyager and WebArena. More details in the data generation for SFT can be found in Appendix F To investigate the impact of the capability of task proposer and evaluator model on the effectiveness of PAE, we include two additional baselines LLLVV-34B PAE (Qwen7B) and LLaVa-344 PAE (Qwen72B) where we use Qwen2VL-7B/Qwen2VV
on WebArena Easy, surpassing the prior best performing model Qwen2VL--2BB with 10x more parameters. More importantly, our analysis shows that PAE can enable Internet agents to learn general web browsing capabilities that zero- shot transfer to unseen websites.
How does existing open-source and proprietary models perform in vision-based web navigation? First, we note the difficulty and significance of real-world vision-based web navigation, source VLM agents such as Qwen2VL-7B and Intern VL2.5- 8B with -frkss augmented observations and chain-of- thought prompting. In particular, on the WebVoyager bench- mark, among open-source VLM agents, only the largest Qwen2VL-72B can achieve a non-trivial average success rate of 22.6% on WebVoyager, while all other open-source agents completely fail this benchmark with average suc- rate under 2%. On the other hand, closed-source propri- etary models start to show promise in becoming generalist Internet agent with Claude 3.5 Sonnet achieving an aver- age success rate at 50.5% and 50. 1 on Web WebVoyager and WebArena Easy. Comparing V.L.Va-77 SFT and LLaVa- 7B, we find that supervised fine-tuning on demonstration data can significantly improve the general web browsing capabilities of open- VLM agents. Even if the SFT demonstration data is collected on out-of-distribatioo on- line websites, the general web browsing capabilities can zero-shot transfer to Web WebVoyager websites, resulting in a performance improvement from 0% to 14.9%.
as both task proposers and evaluators. Finally, to study the effects of different contexts for our task proposer, we compare the performance of two variants from PAE as discussed in Section 4.2 LLaVV- 34B PAE and LLa Va-BB PAE uses only the name of the website as the context, while LLaVa-7B-PAE (User Demos) uses additional screenshots per website from human collected user demos.
for state-of-the-ar medium-size open-
5.3. Main results
We present main baseline comparisons of PAE with other baselines in Table 2 and Overall, comparing to the SFT checkpoint using demonstration data, LLaVa-7B PAE can achieve an average of 7.4% and 10.8% absolute improvement terms of success rates on WebVoyager and WebArena Easy respectively. A similar improvement of 10.4% on Web is observed for VL-34B PAE well, indicating favorable scaling performance of PAE. As result, our resulting model LLaVa-344 PAE achieves absolute success rate of 10.4% on Web Voyaer over the prior state-of-the-ar open-source VLM agents. Similarly, La- 7B PAE also establishes a new state-of-the-ar performance
6

---

Proposer-Agent-Vvalutee (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
OpenStreetMap
PostMill
OneStopMarket
Average
Claude 3.5 Sonnet
38.3
70.0
53.0
50,1
Claude Sonnet
24.3
55.8
41.7
36,0
Qwen2VI.-7B
0.7
10.2
20.2
7.5
Qwen2VI.72BB
16.0
32.8
32.7
23.9
IntemVVL2.5888
1.8
0.5
6.0
2.5
LL
0.0
0.0
0.0
LLaVa-34B
0.9
0.5
LLaVa-7IB SIFT
15.2
16.8
25.4
18.0
LLaVa-7B PAE
19.5
21.1
42.3
24.6
LLaVa-7B PAE (Qwen72B)
LLaVa-7B PAE (Qwen7B)
20.2
25.0
28.6
23.1
LLaVa-7B PAE (User Demos)
21.7
21.5
42.1
25.7
Table Success rate comparisons on WebArena Easy. Success and failure are detected with ground-truth verification functions. For PAE, running average of the evaluation results at each iteration is reported. The final "Average" column weighted average by the number of tasks on different websites.
Proposed Set Performance
Evaluation Set Performance (Ground-Truth, Evaluator)
(Autonomous Evaluator)
LLaVa-7B PAE
Surcoss Rate 99
70
25
LLaVa-7B PAE (No Reasoning)
60
50
20
LLaVa-7B (w/ QwenZVL-728)
40
15
LLaVa-7B PAE (w/ Qwen2VL-78)
30
QwenZVL-728
20
10
Qwen2VL-78
10
LLaVa-7B PAE (Step Reward)
10
15
20
0
5
10
15
20
LLaVa-7B PAE (Function Reward)
Trajectories (1K)
Trajectories (1K)
Figure 3. Ablation experiments on WehArena Easy. The left figure measures the performance on the set of proposed tasks by different models with autonomous evaluators while the right figure measures the performance on WebArena Easy with the ground -ruth evaluator.
Midel Claude 3 Sounet
Websites
Unseon Welsites
WebArena Easy. Notably, all of the improvements from PAE are achieved in self-play setting without any human intervention, only knowing the names of the websites!
42.4
25.0
Qwen2VL-7B
1.4
QwenIVLL-238
22.6
8.3
LLaVE-78 SPT
14.9
9.1
LLaVa-7B
LLaVa-7B PAE (Qwen7Bi
21.7
13.7
Is the improvement of PAE bounded by the performance of the evaluator and task proposer model? To under- stand whether the improvement of PAE is bounded by the performance of the model used as the task proposer and evaluator, we replicate the experiments of PAE using open- source VLMs (Qwen2VL-7B and Qwen2VL-72B) as task proposers and autonomous evaluators, thus completely elim- inating the dependence of PAE proprietary models. Re- sults are included in Table I lar, on WebArena Easy, we found that LLava-7B PAE using Qwen2VL-72B as the task proposer and evaluator achieved a similar performance as using Claude 3 Sonnet as the task proposer and evaluator, despite their significant difference in agent performances (23.9% compared 36.0% average suc- cess rate). As a result of this improvement, LLava-7B PAE using Qwen2VL-72B as the task proposer and evaluator achieved better performance compared to Qwen2VL-72B itself. Perhaps more surprisingly. even Qwen2VL-7B with inferior agent performance compared to L-LaVa-BB SFT (7.5% compared to 18.0%) can be used to make significant improvements (from 18. 18.0% to 23).... similar conclusion is observed on WebVoyager experiments as well, where even Qwen2VL-7B (with agent performance of only 1.4%) can be used task proposers and evaluators to improve the per-
(Qwen728)
LLaVa-348 SPT
22.2
16.1
LLaVa-348 PAF.
33,0
Table 3. Task success rate comparisons on unseen websites that PAE never interacts with. We select 85 unseen real-world online websites and generate 500 synthetic tasks similar to the procedure VeyVer (He el al. 2024a). Seen websites are online websites in WebVoyager. Results show that PAE can discover general web browsing skills useful for unseen websites.
and Figure 3 In particu-
Is PAE able to autonomously discover and practice skills useful for unseen evaluation instructions? On top of the performance gain from downstream fine-tuning, Va-7B PAE additionally improves the success rate by more than 30% relatively (14.9% to 22.3% on WebVoyager and 18.0% to 24.6% on WebArena Easy). In particular, LLaVas VB PAE beats Va-BB SFT across the board with substantial im- provements on 10 out of 13 websites from Voyager and all websites from WebArena Easy, showing the robust- ness of the PAE framework. In fact, LLaVa-7B PAE even beats the 3.L SFT (22.3% compared to 22.2%) model more than 5x larger, resulting a better model with 5x less test-time compute. The release of our models en- ables medium-size VLMs such as LLaVa-7B to beat the prior SOTA Qwen2VL--7BB with 10x more parameters on

---

Proposer-Agent-vaalatoo (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
formance of LLa Va7B-SFF from 14.9% on seen websites and 9.1% unseen websites to 21.7% and 13.7% on unseen websites. These results demonstrate that the improvements from PAE root in the asymmetric capabiliiies of -f- the-art VLMs as agents and as task proposers/evaluutors, instead of naively imitating a stronger VLM.
function-based evaluator, we provide 3 examples of verifi- cation functions as used by WebArena (Zhou et al. 2024a) and ask Claude 3 Sonnet to also come up with verification functions to functionally verify the final task success rate (e.g. checking if the final wehsite url is the same as the ground-truth url). As shown in Figure 3 both step-based evaluator and function- based evaluator perform worse than the outcome-based evaluator, where the use of step-based evaluator even leads worse performance compared to the SFT checkpoint to start with. We found that the step- based evaluator hallucinated more often and tended to be too "generous" in terms of considering the success of each step, potentially because the task of evaluating the success of each step is significantly harder compared to only evaluat- ing the success of the final outcome. Furthermore, we found that oftentimes the function-based evaluator hallucinates the success criterion for the verification function (e.g. making up a non-existing url that the agent needs to go to), therefore resulting in most tasks being impossible to learn. In contrast, the design choice of an outcome based evaluator can best provide reliable reward signals for the policy to improve, resulting in better performances.
Does PAE scale well with larger and more eapable base models? To test the scaling performance of PAE, we repeat our experiments
Web Voyager with a larger and more capable VLM base model LLa 1.634B (Liu et al. 2024). With a better base model, we still find a similar performance gain of PAE despite the model size change from 7B to 34B (7.4% compared to 10.8% absolute success rate improve- ment). Again, Va-34B PAE beats VL PAE on 12 out of 13 websites from WebVoyager. Our scaling experi- ments suggest favorable scaling property that can similarly improve better and larger base VLM agents if available.
Do the skills learnt by PAE generalize to unseen environ- ments? To understand the generalization of PAE to websites that it has never interacted with, we apply the workflow from He et al. 2024a) to generate 500 tasks using Claude 3 Son- net on 85 unseen online websites and test the checkpoints from Web Voyager experiments. Results are presented in Table and list of the websites is included in Appendix F We observe that PAE for both LLaVa-7B and LLava-34B enable the agents to learn general web-browsing skills that can be zero-shot transferred to unseen websites, with 7.2% and 5.3% improvement in absolute success rate respectively.
A Alignment with human judgements. We demonstrate the effectiveness of our autonomous evaluator with a user study, We randomly select 200 trajectories for each method and present all screenshots in the trajectories, the corresponding actions at each step, and the task descriptions to the human annotator to decide the task has actually been completed or not. As shown in Figure 4 a)) there is high correlation between our evaluator and human assessments across dif- ferent models with an average misalignment of 1.7% at the system level and 8.6% at the instance level. The effective- ness of PAE as judged by human annotators is consistent with what is reported in Table 1
6. Discussions
The effect of additional reasoning step. We also perform an additional ablation on the effect of the PAE design choice of asking the VLMS to output their thoughts first prior to the actual web operations. We consider an additional baseline of directly outputting the web operations without thoughts, and carry out the similar SFT and Filtered BC experiments using the same setup described in Section 5.2 As reported in Figure 3 although PAE without reasoning can also achieve improvements in the proposed set, the lack of additional reasoning step results in a significantly inferior performance in its generalization to the unseen evaluation set.
Comparison of different choices of contexts. We present our study on the effects of using different contexts on We- bArena in Table 2 and Figure 5 By comparing the success rate between VL PAE and VL-7B PAE (User De- mos), we find additional information significantly improves the performance in the original WebArena task set Map (19.5% to 21.7%) but does not make a big difference in the rewritten easier task sets on PostMill and OneStopMarket. By manually inspecting the tasks proposed with and without user demos, we find that many tasks proposed with website names alone are too hard or even impossible given the sup- ported features of OpenStreetMap. For example, a task like "Locate the closest movie theater to the address 456 Oak Street, Chicago. Illinois, and provide the theater's name, address, and current movie showtimes." is impossible to be completed on OpenStreetMap as it does not contain infor- mation related to the current movie showtimes. As shown in the learning curve in Figure indeed the agent achieves significantly lower performance on the training tasks of
The effect of choice of evaluators. Finally, we present ablation results on the effect of different design choices of evaluators in Figure 3 We compare the outcome- mased eval- uator included in PAE with other choices of evaluators in the related literature such as step-based evaluators (Pan et al. 2024) and function-based evaluators (Zhang et al. 2024a Faldor et al. 2024). In our implemantation of step-based evaluator, we ask Claude 3 Sonnet to evaluate whether each step is correct or not (i.e. whether it gets the agent closer to the goal) and behavior clone all the steps considered cor- rect by the baspd evaluator. In our implementation of
8

---

Propeser-Agent-Evllutto (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
50.0 56.0
Claude-3.0 Evaluator
Human
% Success Rate
36.535.5
24.026.5
15.5
25.0
19.0 20.0
Humen Labels Portive Hegative
70.8%
4.2%
0.6
12.5
1.0
Cloude Sonnet
QwenZVL-77 788
Qwann 728
Ma MoVa-7B.8 SPT
LLav 7B PAE
LLAV 348 SPT
LLave-34B PAE
4,4%
20.6%
Negalivv Auto Evaluator Labels
(a) Correlation between Human and Autonomous Evaluator.
(b) Confusion Matrix
Figure 4. Correlation and confusion matrix analysis of different models in Webvoyager. (a) Correlation hetween human evaluations and autonomous evaluator across various models at the system level. (h) Confusion matrix of the overall correlation between human evaluations and our autonomous evaluator at the instance level. Both results show strong correlation between our autonomous evaluator and human evaluations.
Succwss Rata (%)
Map-Train
Map-Test
PostMill-Train
PostMill-Test
Market- Train
Market-Test
40
22
20
40
22
60
70
40
20
18
16
30-
20
18
40
50
35
30
10
20
30
0
10
20
30
0
10
20
30
10
20
30
10
30
0
10
20
30
Trajectories (1K)
Trajectories (1K)
Trajectories (1K)
Trajectories (1K)
Trajectories (1K)
Trajectories (1K)
LLaVa-7B PAE
LLaVa-7B PAE (User Demos)
LLaVa-7B SFT
Figure 5. Online sample complexity comparisons on different websites in WebArena Easy between PAE using different contexts.
Note that PAE with different contexts for task proposers uses different training tasks. Learning curves are smoothed with exponential running averages.
PAE compared to that of PAE (User Demos). On the con- trary, this gap in terms of training set performances is much reduced on PostMill and OneStopMarket. We hypothesize that this is because our simplified tasks on PostMill and On- eStopMarket only examine the basic usages of the websites such as "Go to a forum related to relationship advice" and "Browse the Patio and Garden shopping category" and such tasks can be easily proposed with rudimentary understand- ing of the websites inferred from the names of the websites alone. As the tasks get harder and involve more complicated interactions with elements different websites, expect the use of context information to play more important role.
issues-reducing hallucinations in 7B from 37% to 23%, and execution errors in 34B from 45% to 21%-by by fos- tering more grounded reasoning and enriching the agent's repertoire of web interaction skills. These improvements emerge from our system's 's ability to adaptively guide agents toward learning missing capabilities rather than overfitting to demonstrations. Compared to other VLMs, open- open-source models such as Qwen2VL primarily suffer from execution failures, whereas proprietary models like Claude 3.5 Sonnet exhibit broader, more balanced distribution of error types. These results demonstrate that PAE complements different model capabilities by adaptively targeting their dominant weaknesses. Detailed breakdowns and example trajectories are provided in Appendix B and K
Cause Strner
1L SFT
LL PA
(Lxis 318 PAC
7. Conclusions
In this paper, we introduced an effective learning system, PAE, for autonomous skill discovery with foundation model agents, addressing the limitations of using a static set of human-annotated instructions for fine-tuning agents. In- stead of manually specifying what the agents should learn, our system enables the agents to explore, practice, and refine new skills autonomously through open-ended interactions with various environments. The framework's 's key compo- nents-task proposer, action policy, and autonomous eval- uator-work together to generate, attempt, and evaluate tasks without any human intervention, leading more than 10% improvement across web agent benchmarks. This work paves the way for more capable open-source foundation model agents.
Drrens
Figure 6. Failure mode analysis for different models in Weh Wey ager environment.
Error Analysis. To understand where the improvements of PAE come from, we categorize common failure types in We- boyager into six classes: missing low-level skills, incorrect high-level planning, visual hallucinations, timeouts, tech- nical issues, and others. As shown in Figure LLLaVa-7B SFT frequently hallucinates due to limited reasoning ability, often generating confident but unsupported answers, while LLaVB-34B SFT struggles with low-level execution despite better awareness of task correctness. PAE addresses both

---

Proposer-Agent-Vvalutee (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Impact Statement
Colas, C., Teodorescu, L., Oudeyer, P.-Y., Yuan, X., and Côté, M.-A. Augmenting autotelic agents with large language models, 2023. httths://sxxipi/
This work aims to enhance autonomous Internet agents through open-ended interactions with the web. However, the irresponsible or unrestricted use of such agents may pose risks, including personal data leaks or vulnerabilities to malicious attacks. To mitigate these risks, it is crucial to implement robust precautionary measures. In our experi- ments involving open-ended web navigation, we ensure that the agent is restricted from accessing personal accounts and employ appropriate firewalls to block DNS requests to sus- picious websites. These safeguards help prevent unintended consequences and protect sensitive information.
abs/2305.12487.
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2web: Towards generalist agent for the web, 2023. URL htttttsss///axiiti
abs/ aba/2306. 06070
Du, Y. Watkins, O., Wang, Z., Colas, C., Darrell, T., Abbeel, P., Gupta, A., and Andreas, J. Guiding pretraining in reinforcement learning with large language models, 2023. 2230. 06692
References
Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. Diversity is all you need: Learning skills without reward func- tion, 2018. 06070
Achiam, J., Edwards, H., Amodei, D., and Abbeel, P. Variational option discovery algorithms, 2018. URL
org abs/ / 1802
https://arxiv. org//bgg/......oooo.... abs/ / 1807. 00188001.991111 10299
Faldor, M., Zhang, J.. Cully, A., and Clune, J. Omni-epic Open-endedness via models of human notions of inter- estingness with environments programmed in code, 2024, URL /R//ttpps///zzx....rr///a.. wwtpg///aaaa///a00 http://wwww.....//a222211.....5.
Anthropic
The claude 3 model fam- ily: Opus, sonnet, haiku, 2024.
URL
https: www cdn ctmp com debbabbblcccbbcccaabbbbbbbbbb
Model_Caree Clauud___auud_______
Furuta, H., Lee, K.-H., Nachum, O., Matsuo, Y., Faust, A., Gu, S. S., and Gur, I. Multimodal web navigation
Bai, H., Zhou, Y., Cemri, M., Pan, J., Suhr, A., Levine, S., and Kumar, A. Digirl: Training in-the-wild device- control agents with autonomous reinforcement learn- ing, 2024. UNL/:ctpss//arxxivorr///bb///// 11896
with http///awwi..rrrraaa//0..0.000.
foundation models, 2024. URL
https:
GeminiTeam. Gemini: family of highly capable mul- timodal models, 2024, URL URL httl/ttss///rxiv.orr
abs / 2222232.222 12312.1180S5 11805
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
Gur, I., Furuta, H., Huang, A. V., Safdari, M.. Matsuo, Y., Eck, D., and Faust, A. A real-world webagent with plan- ning, long context understanding, and program synthesis, 2021.
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radfordddd A., Sutskever, I., and Amodei, D. Language models are few-shot learners, 2020. URL https:/7 arxiv. wtt. at0.aabb//00011111
He, H., Yao, W., Ma, K., Yu, W., Dai, Y., Zhang, H., Lan, Z., and Yu, D. Webvoyager: Building an end-to-end web agent with large multimodal models, 2024a. URL
13919
Campos, V., Trott, A., Xiong, C., Socher, R., Nieto, X. G., and Torres, J. Explore, discover and learn: Unsupervised discovery of state-covering skills, 2020. URL https: //arxiv.orge wwarg//ab/////
He, H., Yao, W., Ma, K., Yu, W., Zhang, H., Fang, T., Lan, Z., and Yu, D. Openwebvoyager: Building mul- timodal web agents via iterative real-world exploration. feedback and optimization, 2024b. URL https://
abs/ 2002. 03647
arxiv. abaa 19609
Chen, B., Shu, C., Shareghi. E., Collier, N., Narasimhan, K., and Yao, S. Fireact: Toward language agent fine- tuning, 2023. URL
Hong, J., Levine, S., and Dragan, A. Zero-shot goal-directed dialogue via r1 on imagined conversations, 2023a. URL
2310.05913 05915
05584
Colas, C., Karch, T., Lair, N., Dussoux, J.-M., Moulin-Frier, C., Dominey, P. F., and Oudeyer, P.-Y. Language as a cognitive tool to imagine goals in suriosity-driver ex- ploration, 2020. URL https: /ttp://aavvvv.. org/ abs/ 2002. 09253
Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Jí, J., Wang, Y. Wang, Z., Zhang, Y., Li, J., Xu, B., Dong, Y., Ding, M., and Tang, J. Cogagent: A visual language model for gui agents, 2023b. URL https: /tap:/:///xxv.oo/// abs/2312.088
10

---

Proposer-Agent--valtttor (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Hu, M., Zhao, P., Xu, C., Sun, Q. Lou, J., Lin, Q. Luo, P., Rajmohan, S., and Zhang, D. Agentgen: Enhancing planning abilities for large language model based agent via environment and task generation. 2024. URtht 1/arxiv. wrw. abs/ /tabb//44440 220. 00764
OpenAL. Gpt-4 technical report, 2024. URL https: //
a axxiv.org/abs/23035 08774
Pan, J.. Zhang. Y., Tomlin, N., Zhou, Y., Levine, S., and Suhr, A. Autonomous evaluation and refinement of digi- tal agents, 2024.
Jimenez, C. E., Yang. J., Wettig, A.. Yao, S., Pei, K.. Press, O., and Narasimhan, K. Swe-bench: Can language models resolve real-world github issues?, 2024. URL httts:///ariin org. abs/////////// 231 0. 06770
2404_ 06474
Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimiza- tion, 2024. 19733
patttps:/arrxtt wwtp:xwwwwwvgggax/o////
Koh, J. Y., Lo, R., Jang, L.., Duvvur, V., Lim, M. C., Huang, P.-Y., Neubig. G., Zhou, S., Salakhutdinov, R., and Fried, D. Visualwebarena: Evaluating multimodal agents realistic visual web tasks, 2024. https
Park, S., Choi, J., Kim, J., Lee, H., and Kim, G. Lipschitz- constrained unsupervised skill discovery. 2022. URL
https://arxiv. httpss//arrv..oorrgbbb///22020.0
7/arxiv. org/ abs/ / 2401. 13649
Park, S., Lee, K., Lee, Y., and Abbeel, P. Controllabilitt- aware unsupervised skill discovery, 2023, URL https
Laskin, M., Liu, H., Peng, X. B., Yarats, D., Rajeswaran, A., and Abbeel, P. Unsupervised reinforcement learning with contrastive intrinsic control. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 344911 Curran Associates, Inc., 2022. URL.
//arxtw.org//ass/2302.. 0ttp://222200010..333
Park, S., Rybkin, O., and Levine, S. Metra: Scalable un- supervised rl with metric-aware abstraction, 2024. URL
Putta, P., Mills, E.,, Garg, N., Motwani, S., Finn, C., Garg, and Rafailov, R. Agent q: Advanced reasoning and and
cc/ [aaaerrrfLLes//pprr////////e
Papes
Conter CE learning for autonomous ai agents, 2024. URL https
pdf
//aaxiv.org/abs/22222
07199
Lin, K. Q.. Li, L., Gao, D., Yang, Z., Wu, S., Bai, Z., Lei, W., Wang, L., and Shou, M. Z. Showui: One vision- language-action model for gui visual agent, 2024. URL /ettp://arriv.. org. abs/ 2411 17465
Qi, Z., Liu, X., long. I. L., Lai, H., Sun, X., Zhao, W., Yang, Y., Yang. X., Sun, J., Yao, S., Zhang, T., Xu, W., Tang, J., and Dong, Y. Webrl: Training Ilm web agents via self- evolving online curriculum reinforcement learning, 2024. URL htt.:htts://// /tttps://arxiv.. .rgg abs/ /2111 02337
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tuning, 2023a. URL /tttpsi//ariiv... org/obgg//......ooo 2304.08485
Sharma, A., Gu, S., Levine. S., Kumar, V., and Hausman, K.
Dynamics-aware unsupervised discovery of skills, 2020. 01657]
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL bo/blo//
/ttpssts:://xxxv.... http:/::/w/wwwwwoo//aa///a abs 907. 000.006677777
Snell, C., Kostrikov, I., Su, Y., Yang. M., and Levine, S. Offline rl for natural language generation with implicit language q learning, 2023. URL https:://aaat: org/ abss/200 11371
Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H. Men, K., Yang. K., Zhang, S., Deng, X., Zeng, A., Du, Z., Zhang, C., Shen, S., Zhang, T., Su, Y., Sun, H., Huang, M., Dong, Y., and Tang, J. Agentbench: Evaluating llms as agents, 2023b. URL wtwgabb////oooo https: /wtp:vaaivvv...///aa/........av. org/ abs/ 2308 03688
Wang. G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An open-ended embodied agent with large language models, 2023a. URL
16291
Wang. T., Kulikov, I., Golovneva, O., Yu, P., Yuan, W., Dwivedi- Yu, J., Pang, R. Y., Fazel-Zarandi, M., Weston, J., and Li, X. telft evaluators, 2024a. URLbtttps
Llama3Team. The llaaa 3 herd of models, 2024. URL https://arxxv org/ abs 2407.440700 21783
//axxiv. abs /400 02666
Murty, S., Bahdanau, D., and Manning, C. D. Nnetscape navigator: Complex demonstrations for web agents with- out a demonstrator, 2024. URL https: /arxiv hrg/abs//2e 02907
Wang. T., Wu, Z., Liu. J., Hao, J., Wang, J., and Shao, K. Distrl: An asynchronous distributed reinforcement learning framework for on-device control agents, 2024b. URL /ttp:://axxvvvvo http///xxv..oo/a/////////// UKLhhttpes//ariir..or/////////11 24411/43331111131 14803.

---

Proposer-Agent--Vallattoo (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Wang, W., Lv, Q. Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L.., Song, X., Xu, J., Xu, B., Li, J., Dong, Y., Ding, M., and Tang, J. Cogvlm: Visual expert for pretrained language models, 2024c. URL https: 1/arxiv.orge abs/ /aaas/2311 1111111111111111 03079
Yuan, W., Pang, R. Y., Cho. K., Li, X., Sukhbaatar, S., Xu, J., and Weston, J. Self-rewarding language models, 2024. URL BRLhttps::22 pattts::/arxiv.orrt wwbpa//wo..aaaaao.aeaaaaabbaaaaa wwwaa//000 /20p..401... /000
/ass
Zeng, A., Liu, M. Lu, R., Wang, B., Liu, X., Dong, Y., and Tang, J. Agenttuning: Enabling generalized agent abilities for 1lms, 2023.
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self- generated instructions, 2023b. htarrg/bb//2211 URL //t:://rxxii httt:ss//arrxl..orrra///////////
abs/ foss23110 0. 1.....8.00 12823.
Zhai, Y., Bai, H., Lin, Z., Pan, J., Tong, S., Zhou, Y., Suhr, A., Xie, S., LeCun, Y., Ma, Y., and Levine, S. Fine- tuning large vision-language models as decision-making agents via reinforcement learning, 2024. URL https: //arxiv.org/abs: 2405. 02292
10560
Williams, R. J. Simple statistical gradient-following al-
gorithms for connectionist reinforcement learning. Ma- chine Learning, 8:229-256, 2004. URL htthtts:::///
semanticscholar.com http://ohhh.oot...oooooooooooaaa
Zhang, C., Yang, Z., Liu, J., Han, Y., Chen, X., Huang, Z., Fu, B., and Yu, G. Appagent: Multimodal agents as smartphone users, 2023a. URL httts: org /ba//bb2...21177
Wu, T., Yuan, W., Golovneva, O., Xu, J., Tian, Y., Jiao, J.. Weston, J., and Sukhbaatar, S. Meta-rewarding lan- guage models: Self-improving alignment with lmm-as- a- judge, 2024. URL sittps://ariis org/
Zhang. J., Zhang, J., Pertsch, K., Liu, Z., Ren, X., Chang, M.. Sun, S.-H., and Lim, J. J. Bootstrap your own skills: Learning to solve new tasks with large language model guidance, 2023b. URL htt.hhtss//aaii.oorg////////
abs/ abs/2407. 9594
Xie, T., Zhang. D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., and Yu, T. Osworld: Benchmarking multimodal agents for open- ended tasks in real computer environments, 2024. URL https: //tpx//aavvvv//// http://wwwrrabb///bbb// 07972
2310.10021 10021
Zhang, J., Lehman, J., Stanley, K., and Clune, J. Omni: Open-endedness via models of human notions of interest- ingness. 2024a. URL htt.ps://axxvv.rrg///
Xu, Y., Lu, D., Shen, 7., Wang. J., Wang, 7., Mao, Y., Xiong, C., and Yu, T. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials, 2024a. URL h
2306 01711
Zhang, P., Dong, X., Zang, Y., Cao, Y., Qian, R., Chen, L., Guo, Q. Duan, H., Wang, B., Ouyang, L., Zhang, S. Zhang, W., Li, Y., Gao, Y., Sun, P., Zhang, X., Li, W., Li, J., Wang, W., Yan, H., He, C., Zhang, X., Chen, K., Dai, J., Qiao, Y., Lin, D., and Wang, J. Internim- xcomposer-2.5: A versatile large vision language model supporting long-contextuall input and output, 2024b. URL abs wttp://ggbbbb///.....om 2200 03320
//arxiv. org/ abs 22420.9605
Xu, Y., Wang, Z., Wang. J., Lu, D., Xie, T., Saha, A., Sa- hoo, D., Yu, T., and Xiong, C. Aguvis: Unified pure vi- sion agents for autonomous gui interaction, 2024b. URL https: /eltps://arxio /arxiv. www.xii.vgggaaaaaaa
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C.,
Li, C., Li, C., Liu, D., Huang. F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N.. Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Liu, X., Fan, Y., Yao, Y., Zhang, Y. Wan, Y., Chu, Y., Liu, Y., Cui, Z., Zhang, Z., Guo, Z., and Fan, Z. Qwen2 technical report, 2024a. URL https: //arxv// www.ooog//aoooooo abs////s//////// 2407. 10671
Zhang, Z. and Zhang, A. You only look at screens: Mul- timodal chain-of of-action agents, 2024. URL https
//arxiv.org/abll httprr///////00.11111 1143661.11166
Zheng, B., Gou, B., Kil, J., Sun, H., and Su, Y. Gpt- 4v(ision) is a generalist web agent, if grounded, 2024. / // /ttp//aaavv////////////.///..a./ org. /abg/bb//////.a 2201.0111111111
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., and Neubig, G. Webarena: A realistic web environment for building autonomous agents, 2024a. URI. https! //arxiv.org/abs//30/
Yang, J., Jimenez, C. E., Wettig. A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. Swe-agent: Agent- computer interfaces enable automated software engi- neering, 2024b. URL httph: /ttpav.///rxvvvvoo http://rrxxx.oooa/////// 2405. $405.15793 15793
Zhou, Y., Zanette, A., Pan, J., Levine, S., and Kumar, A. Archer: Training language model agents via hierarchical multi-turn rl, 2024b. URL https: haapps///arae /aaaaaivvvo/// org/
abs/2402.19445
12

---

Proposer-Agent--Valaatoo (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Zhou, Z., Atreya, P., Lee, A., Walke, H., Mees, O., and
Levine, S. Autonomous improvement of instruction
following skills via foundation models, 2024c. URL https://arxxn wwtpxxxwwwogg////xxxxe wttpg//////.../////// 20635
13

---

Proposer-Aeent-Vaalaaor (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Appendices
A. Algorithm
In Algorithm we include a formal definitions of our practical algorithm of PAE as presented in Section 3
Algorithm Proposer-Agent--Vallaade Practical Algorithm
Require: Context information 2,,, task proposer ć, autonomous evaluator R.
1: Initialize policy from pre-trained checkpoint.
2: Initialize replay buffer D + (-
3: ## Propose tasks based on the context information.
4: Obtain proposal task distribution ((=M).
5: for each global iteration do
6: for each trajectory be collected do
Sample task from the task proposer e C(zM). Reset the environment obtain the initial observation 8o for each environment step do Sample aę ~ π (-188e c), 81+1 T(1(s), C) if done then
8:
9:
10:
11:
12:
## Autonomously evaluate the outcome the agent rollout. rt R(8,, c).
else
Tt O. end if
16:
17:
D - D
18:
end for
19:
end for
20: ## Update the agent policy with any RL algorithm.
- RL..uddat(Tr, D)
22: end for
B. Error Analysis
Claude 3.5 Sonnet
Qwen2VL-7B
LLaVa-7B SFT
LLaVa-7B PAE
LLaVa-34B SFT
LLaVa-34B PAE
Low-level skills Technical issues
High-level planning and reasoning Others
Visual hallucinations
Timeouts
Task success
Figure 7. Failure mode analysis for different models in WebVoyager environment.
To understand where the improvement of PAE comes from, we conducted a user study to analyze different error types across various models. We classified the error types into the following categories: (1) Low-level skills missing error refer the cases where the agent has reasonable plan to solve the problem but fails to execute precise actions the website, such as not knowing which button to click to navigate the desired page. (2) High-level planning or reasoning errors refer to the cases where the agent fails to generate a plan in its thoughts to solve the given task or cannot arrive at the correct answer through reasoning with the website's 's screenshots.
(3) Visual hallucinations refer to the cases where the agent generates responses with made-up up information that are not supported by the screenshot. For example, the agent may claim that has found product that the task asked for while remaining the homepage of Google search, or the agent may produce
14

---

Proposer-Agent-Eaalaee (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
wrong answer while being on the right page. (4) Timeouts refer to the cases where the agent is on the right track to solving the tasks, but couldn't it complete the task within the maximum number of steps. (5) Technical Issues are not the fault of the agent but caused by environmental problems such as websites being out of service or connection issues. (6) Others include other less common error types, such as the task itself being impossible.
We present the failure mode statistics across models in Figure 7 with further example trajectories shown in Appendix K Compared to LLaVa-34B SFT, Va SFT tends to make more hallucination errors, often generating fabricated answers when it lacks sufficient reasoning capability. In contrast, the 34B model demonstrates better understanding of correctness but fails more frequently in executing web interactions, leading higher rate of low-level execution errors. PAE improves both models by reducing their dominant failure makeng the smaller model less prone hallucination and the larger model more effective at executing detailed actions. Comparing our models with other VLM agents, we find that other open-source VLM agents such as Qwen2VL-7B and Qwen2VL-72B mostly struggle with low-level web navigation skills while the error types for more advanced proprietary models such as Claude 3.5 Sonnet are more spread out.
C. Details on Human Annotations and User Demos
Details on User Demos. User demos from experiments in Figure J are collected by the authors without appealing to actual users. For each website, the authors attempt 10 tasks that we think are representative of the use of the particular website, and 10 most distinctive web pages are identified in the process of attempting those 10 tasks.
Details on Human Annotations. Five annotators of PhD students participate in the user study and the entire process takes around 40 annotator hours with the help of a designated user interface programmed in Gradio. To clarify the precise definition of the different error categories used in 6ection we provide the following instruction to give more comprehensive explanations with example trajectories:
(1) Lowelevel skill missing errors refer to cases where the agent has reasonable plan to solve the problem but fails to execute precise actions on the website, such as not knowing which button to click to reach the desired page. We classify trajectories where the agent seems to follow reasonable plan but struggles with specific operations into this category. For example, in Figure
[] the task is "Find the Easy Vegetarian Spinach I Lasagna recipe on Allrecipes and tell me what the latest review says." The agent attempts to search for the desired item but fails to click the correct button to reach the detailed page in the search results.
(2) High-level planning reasoning errors occur when the agent fails to generate complete plan or cannot reason correctly with the website's 's screenshots to solve the task. Trajectories where the agent cannot devise plan for complex tasks or misinterprets the screenshot's 's content are categorized as such. For instance, in Figure 9 the task is "Give 12 lbs of c-yaanoidooee, converted to molar and indicate the percentage of C, H. N." The agent should first search on Google about the chemical definition of 4-cyanoindole, then use WolframAlppa to calculate the result. However, the agent fails to get the precise definition of 4-cyanoindole, and doesn't know how to solve the task.
(3) Visual hallucinations refer to instances where the agent generates fabricated responses not supported by the screenshot. The agent might, for example, claim to have found requested product while still the Google homepage or provide an incorrect answer even when on the correct page. In Figure T0, the task is "Find out the trade-in value for an iPhone 13 Pro Max in good condition on the Apple website".. The agent claims with a very detailed answer but actually it never access any page related to the trade-in on the website.
(4) Timeouts occur when the agent is on the right track to solving the task but cannot complete it within the maximum number of steps. This error indicates that the agent did nothing wrong but was constrained by the environment's step limits. For example, in Figure [1] the task is "Go the Plus section of Cambridge Dictionary, find Image quizzes, and complete an easy quiz about Animals. Tell me your final score." The agent reaches the maximum time step limit (10) while attempting to finish the quiz.
(5) Technical issues are not caused by the agent but by environmental problems, such as websites being down or connection failures. In Figure 12, the ChromeDriver crashes after a valid operation.
(6) Others include less frequent error types, such as when the task itself is impossible to complete.
15

---

Proposer-Agent-Vaalatte (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Thought The current
Amazon's Laber Day sale
Including a oet can be
Alredpes hamepage. To Spinsch Laragra recioe and get
dkilltss grllic and parpper
This
relerant
rewlew use the search Accions (2): Bady Vegatamin nach
Amazon's Labor Sale Has 49% Off Cast Iron Cookware, Including Lodge and Le Creuset
finsiing the Budy
aay Sale Has to 49% Off
Vogetarie Scinach
gatting
re, Including Lodge and Le
the latast restew
taak
chourd return main Click (9)
spologiie, let me take resurts Thought: Upon rrspectinn the result Alrestpes,
Thought The search results 'Bary etariin
Spinach Labogna" shoun, top result I'm
recice mram
leaiing for based on and
snippst from Google
the ateet restewr chould
Athectped rec page. Redper button next to that (11)
Lan Vestados Sprich
Siapl spred uargea
Spitach Laagge
Day Hugaarar Spltach
Single Safrcc Laagna
spled iagpa
Action Click (606]
Thought: I'm back the
Logie
Thought: sppeart
"eary vegetarien spinach. top fwo recutss stil seem to
logiting mighiingiitmee page on which necessary tnd the should dld return the continue looking for the full
lazágna recipes, not the full reciee
should bry clicking ane lower numpered lino hypelull the comple ple
Leans comes
Spinach
Lasagra recia. Actione
atredpo
Thought: have gated
the hull rectoe debals
Rol Up? pilrecpes This
atrrccped Send Redipt
pop up window blooking swarch resula page wes diamiss this and access aearch back from pap-ur up Action!
Troupht The
Spinach Lasagna Roll Ups
rec from the one
Spinach Lasage
Eaty Vegetanen Spinach
specitic rec pe will need is and try clickng on lower resuit link Action: GoBack
need
Thought: The search roulu now show beo fifeent vegetanin lasegne from Alincipess and the Vegatarian each recipe decalss for the fromt previsus atcenpt, To wither rec pa will nead to indivs vicuellll recioe page.
Trought: have navigated
Log
Lasegra recipe page dap aying the hull recipe soiity to log or jonn Facebonk, or Google. Honever:
Laugee
tabeshh Leugua
create account and review this specitc Act Gottack
Figure& 8, Extra full trajectories of fail trajectory 1, with error type Low-level Operational error, executed by model LLaVa-7B SFT. The task is 'Find the Easy Vegetarian Spinach Lasagna recipe on Allrecipes and tell me what the latest review says'.
16

---

Pnoposer-Agent-EVaauatoo (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
WoltremAphh
numbardi of moles, compane il and in los of the ch emiiaa formulh tools whllram Arpha to end caiculare
Thougho: The chem ca senesed Now (need to clek- 18] bo Wolfram Alpha from mass mo les and the garcentage compostion of N. Actian: Llick (8)
WolframmA/phit
mase meles bape compraii Action:
carvhha
thought: Alber ciicing the button [8). Wichram Alpha pernentigg compos on of Cit PNG0 in terms and orygen The resuts g subolance to 86, 1% boo, hydrogam. mieggnn 0.1% ourpen maae. The siigtll
Thought After clicking the Welhram App has from mase to moles for the CS4Th30, The result shows tut
nirrgen
WolframApiss
WolframA
subsiance has carbons, 7 tydhopent, intogaas. and crugan pecartage calcu Bon. need alementa ratio
rouncing. Action: ANDWER: The compastt (5.45 mall d-cyandirdof
result. Action: Click (9)
chemical famuula CSh7N30 -yyloogam 4,5% 12.2 g)
Figure 9. Extra full trajectories fail trajectory 2, with error type Planning or Reasoning error, executed by model LLaVa-7B PAE The task is 'Give 12 lbs of 4-cyanoindole, converted to molar and indicate the percentage of C, H, N'.
D. All prompts in the experiments
For completeness, we include examples of the prompts that we have used in this section. In particular, in Figure 13 we have provided the prompt that we used for the Claude-Sonnet-3 autonomous evaluator to evaluate the success for the task completion for all tasks in WebArena. A similar one is used for all tasks in Web Voyager. In Figure [4 included the prompts that we used for generating the proposal tasks for each domain. We used the same prompts with additional website screenshots appended the messages for PAE + User Demos. are domain-genent general and have little domain customizations. particular, for all 13 real-world websites from Web Voyage,, we use the same prompt to generate tasks except with the placeholder of can easily scale to multiple websites without the need for domain -spaiin-peccfic knowledge. The prompt for zero-shot VLM agents are included in Figure 17
15 16
we have
worth noting that our task proposers
This shows that our PAE framework
18 and 19
E. Prompts For Zero-Shot VLM Agents
We also append the prompts (Figure 17 18 and 19p that we used for the zero-shot baselines including Claude-Sonnet-3, Claude-Sounet-3.. Qwen2VL, InternVL2bs La-Va- 1.6- and LLa 1.6-4BB. The prompt for Web tasks largely follow from that used in the prior literature (He et al. 2024a). We include additional necessary domain knowledge of the WebArena tasks and evaluation protocols in the prompt that we used for WebArena.
F. Details for SFT
SFT for WebVoyager As shown in Table I unlike proprietary VLMs, none of the open-source VLM agent able to follow the instructions and achieve non-trivial performances real-world web navigation tasks in the zero-shot manner. Such models can rarely get success rewards in the process of RL, thus leading to very slow convergence. To "warm-up" the open-source VLM agent to achieve non-trivial performance the start of RL training. we turn to enhancing the performances with SFT before RI. Note that the SFT process may not be needed if the base VL.M agent model can already achieve non-trivial performances such as Claude 3 Sonnet. To prevent data contamination, we gather 85 out-of-distribution real-world websites (listed in Figure 20 and 21.. and collect 11220 trajectories in total using Claude 3 Sonnet with the prompt specified in Figure 17 The average trajectory success rate is 25% as measured by our Claude 3 Sonnet evaluator. Each action in the trajectories contains both thoughts and actual web actions shown in Figure 2 All 11220 trajectories are
17

---

Troposer-Agent-Evalaaee (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Shop iPhone for Education
Thought bask to
iPhone Hav the the screenchok there contems nu information about Action: click (24)
phone section has been mosels To find the trade-in
weache,
Almmdets Take your pict.
Mas, Inved to nasipelu suction. Action GoBlck
Education Store. The most powerful learning.
ihhre PD #hore Pie
Phone 15a iPone IGPMM
Phoee Proes
Thought The currett
ection was incoriec Let me Thought: Alter fre model, should now sowefic deta this pape, incluting any information,
ihone section on the App
phone 15 tro Max model
fiterr, Click ng that check the trude-nn value [25155]
Education Store. The most powerful tools for learning.
Education Store. The most powerful tools for learning.
don't need clikk on any elements, action Scroll
powerrul tools For rearning.
Thought page product Max con oee the priding, opt oduocation pricing and under the "imber Simbed-ee lome
Aduanced looks mrutles Chocse from treep
AMSIVER: The trace Tro Max cood
exbsito are "aade
later
5450 the 099 mobel any Phore $1.009
Hax $1.05s, $26. per monch 24...
Figure Extra full trajectories of fail trajectory 3, with emor type Visual Hallucination, executed by model LLaVa-7B SFT. The task 'Find out the trade-in value for an iPhone 13 Pro Max in good condition the Apple website'.
18

---

Proposer-Agent-Vaalatto (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Cambridge
Combridgge Bictiondry
iPiza
Thought new on the
Thought The task recures
naxipaing to
Cambnoge Dictionary, heating image
Engiih Jaus
eia page which offers Nowever, Idont tae an gy lutee or Anima ou some. sost allle content find pes we're looking for, The "Comzen"" bettom might provids menu wile resouroes Action Ciick
easy aue about (nimais. honrepage, Innes to cldce top menu to access the [S]
Wed Scren dk
Exp the the Cambridge Distenary
Thought We'
sucsonafully nao gated
Thought: The Contents oi oowing vantous ophons Image which what we're loking ig for need option find animal quiz. Accions Click
can the "Anjmai
section.
which ecactly
Irage qurres
what we loocing for The
fiee
chiu do an easy outz, and
on of suis tühe eleailant
sale
with one dei, Thie the
gride
Action: Click [22]
Curdrridge Dictenary APlln
Cambarddge
Dictonary sPau
Thouphe: We ron on the Animals qui: The image queetion adic "Whvi woed connect anower arong current score Lss
Thought: We started the Anmais quez. frrt
e echart "Which matches mmae?? The comeet hannt"
pogeis
among the given octions.
indicating we ensemered first queation After selesting the ersiver, need nwnt question. Action Click
we need move bo the
(24)
thirl quaction of the shows a cheen, and the matches the image?* The correct "Shces" Our spare io die.
fourth cuection of the thoer whie ret, and the motches the in" The co
Outin Artmads
Dutir Arkrub
optiono "rot". Our carrend score
indicating nowered the cuestions correctly. After the ansseer, need the next Action: Click (25)
ansivrred the firtt thres comeat semectly. ecting
we'l need move
question. Actiont Click [25]
Cambridge
Cambridg
Dicsionrry 4PAux
Thought Wa're now the
Dicainary 4PAI
Thought: We're now on the boymar que The mage quedioo sdke Nyitch ward matches image? answer options "mouse""
in mank qutr The mage
mundion seks "Wiich word matches image?" enswer options
curment
4%, indiaat
316.
wa''e arrovereed quastions correcty.
indi cating we ve amsvered
questiors
acting armoer. welll need next quediion. Action
conwecly oter selecting antweer our scone Actior: Click
Figure 11. Extra full trajectories of fail trajectory 4, with error type Timeouts, executed by model Claude 3.5 Sonnet. The task is 'Go to the Plus section of Cambridge Dictionary, find Image quizzes and do easy quiz about Animals and tell me your final score".
19

---

Proposer-Agent--aaualle (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
CounsrO
Learn how to make Al work for you
Thought cument page cou humressge
new visilee search
30%
payenatooy; functionally
intreduction
intreduction to paychology. intrcdicctinn to peychology
Click [2]
AI ERROR
prov des decaks about thehesiona Ganetica
Thought: search tecults some
ptychology. "Irrrduucion betavioral serice University Ninnecota which seema promiinn to it cemation Action Click
from the University by of tex attort toubleshocting steps However does not provice instructer's namee epprova
action my provious
erfomaaioon provided.
Thouphe: "Cloudfrane" text lieely part
webrite cannot
directly
find the isstructon naine look for more relevort
ANDSiiE: The matrucer's to Human Sahavioral anovided, along with the detal's indicato requires complete trypttchhhoccng staps belore
Figure Extra full trajectories of fail trajectory 5, with error type Technical issues, executed by model LLa Va-BB PAE. The "Identify course Coursera that provides an introduction to Psychology, list the instructor's name, the institution offering it, and how many hours will approximately take complete'.
20

---

Proposer-Agent-Eaauaae (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Autonomous Evaluator Prompt
You are an expert evaluating the performance of a web navigation agent. The agent designed to help human user navigate website complete task. Your goal decide whether the agent's execution is successful or not.
As an evaluator, you will be presented with three primary components assist you in your role:
1. Web Task Instruction: This is clear and specific directive provided in natural language, detailing the online activity to be carried out.
Result Response: This a textual response obtained after the execution of the web task. serves as textual result in response to the instruction.
Result Screenshots: This visual representation of the screen showing the result intermediate state of performing web task. It serves visual proof of the actions taken in response the instruction.
You SHOULD NOT make assumptions based on information not presented in the screenshot when comparing it to the instructions.
Your primary responsibility is to conduct a thorough assessment of the web task instruction against the outcome depicted in the screenshot and in the response, evaluating whether the actions taken align with the given instructions.
NOTE that the instruction may involve more than one task, for example, locating the garage and summarizing the review. Failing complete either task, such as not providing summary, should be considered unsuccessful.
NOTE that the screenshot is authentic, but the response provided by LLM is generated at the end of web browsing, and there may be discrepancies between the text and the screenshots.
NOTE that if the content the Result response is not mentioned on or different from the screenshot, mark it as not success. You should explicilt consider the following criterions:
Whether the claims in the response be verified by the screenshot. E.g. if the response claims the distance between two places, the screenshot should show the direction. YOU SHOULD EXPECT THAT THERE IS A HIGH CHANCE THAT THE AGENT WILL MAKE UP AN ANSWER NOT VERIFIED BY THE SCREENSHOT.
Whether the agent completes EXACTLY what the task asks for. E.g. if the task asks to find a specific place, the agent should not find similar place.
In your responses: You should first provide thoughts EXPLICITLY VERIFY ALL THREE CRITERIONS and then provide definitive verdict whether the task has been successfully accomplished, either 'SUCCESS' or 'NOT SUCCESS' A task is 'SUCCESS" only when all of the criteria are met. If any of the criteria are not met, the task should be considered 'NOT SUCCESS'.
Figure 13. The prompt used by the autonomous evaluator for 3Taude-Sonnet-3, Same prompt is used to evaluate tasks from WebArena websites. The evaluator takes as inputs the task description, the response from the agent's ANSWER action, and last three scrcenshots in the trajectory. The evaluation result is a binary verdict of "SUCCESS" or 'NOT SUCCESS'.
21

---

Proposer-Agent--vaauator (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Task Proposer Prompt for VoyVge
("web_name": "Apple". "id": "Apple-40", "ques": "Find the pricing and specifications for the latest Mac Studio model, including the available CPU and GPU options.", "web": htttps:/wwww.apple.com We are training a model to navigate the web. We need your help to generate instructions. With the examples provided above, please give 25 more example tasks for the model to learn from in the domain of [web_name) You should imagine tasks that are likely proposed by a most likely user of this website. A few demos of users navigating through the web are provided above.
YOU SHOULD MAKE USE OF THE DEMOS PROVIDES TO GENERATE TASKS, SO THAT YOUR TASKS ARE REALISTIC AND RELEVANT TO THE WEBSITE.
Please follow the corresponding guidelines:
First output your thoughts first on how you should come up with diverse tasks that examine various capabilities on the particular website, and how these tasks reflect the need of the potential user. Then you should say 'Output:" and then followed by the outputs STRUCTURED IN ISONL FORMAT. You should not say anything else in the response.
2)PLEASE MAKE SURE TO HAVE 25 examples in the response!!!
Your proposed tasks should be DIVERSE AND COVER A WIDE RANGE OF DIFFERENT POSSIBILITIES AND DIFFICULTY in the domain of (web name). Remember, your job is to propose tasks that will help the model learn to navigate the web to deal with various real world requests.
Your task should be objective and unambiguous. The carry-out of the task should NOT BE DEPENDENT on the user's personal information such as the CURRENT TIME OR LOCATION.
You should express your tasks in diverse expressions as possible help the model learn to understand different ways of expressing the same task.
Your tasks should be able to be evaluated OBJECTIVELY. That is, by looking at the last three screenshots and the answer provided by an agent, it should be possible to tell without ambiguity whether the task was completed successfully or not. Your tasks should require a minimum completion steps from 3 to steps, your tasks should have a diverse coverage in difficulty as measured by the minimum completion step. I.E. You should propose not only tasks that may take more than steps to complete but also tasks that can be completed within 3 steps.
H)Hmass should have 100% success rate in completing the task.
Your tasks should be able to be completed without having to sign in to the website.
Figure 14. Prompts used by Claude-Sonnet- for proposing tasks in WebVoyager experiments. For PAE User Demos, the same prompt with additional user demos appended to the message.
used for SFT. RL training is carried out top of the SFT checkpoint.
SFT for WebArena. In our preliminary experiments, we found that the SFT checkpoint trained on real-world websites do not generalize well to self-hosted websites on WebArena. This is potentially because of the distribution shift between real-world commercial websites and self-hosted websites. For example, most real-world map websites such as Google Maps and Apple Maps support advanced fuzzy search capabilities such as "pittsburgh to new york" while OpenStreetMap from WebArena will not return any results with such queries. Therefore, we collect 3000 Claude Sonnet generated trajectories each from OpenStreetMap, Reddit, and OneStopMarket websites from WebArena. We use the prompts from Figure [8| and 19 for the Claude agent. The average trajectory success rate is 27% measured by Claude 3 Sonnet evaluator. The SFT checkpoint for WebArena is fine-tuned from the SFT checkpoint for Web Voyager.
G. Additional Results on WebArena
For completeness, we have also provided additional experiment results of different models from Table 2 in the original task split of WebArena (Zhou et al. 2024a). As shown the comparison results presented in Table 4. even SOTA proprietary
22

---

Proposer-Aeent-Vaaluato (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
VLM agents like Claude Sonnet struggle with the tasks in WebArena with success rate of only 14.6% with set-of-marks observations and chain-of-thought prompting- After performing SFT using the demonstrations generated by Claude 3 Sonnet, LL SFT can only achieve 1.4% and 5.8% success rate on PostMill and OneStopMarket. By manually inspecting the roll-out trajectories generated by LLaVa-SFT, we found that around half of the successful trajectories on those two websites are false positives from the WebArena evaluator. In these trajectories, the agent simply guessed the answer to be "no" or "N/A" where the ground truth happens to be that the task is not executable. As a result, the actual success rate on those two websites lower than 2%, leaving very sparse reward signals for RL to make meaningful improvements. We therefore rewrote the tasks on PostMill and OneStopMarket to be easier and report the performances of PAE Table 2
OpenStreetMap
PostMill
OneStopMarket
Average
Proprietary
Claude Sonnet
24.3
10.6
11.2
14.6
Qwen2VL-7B
0.7
0.0
1.3
0.7
Open-source
Intern VL2.5-8B
2.6
0.2
3.3
2.3
LLa Va-BB
0.0
0.0
0.0
0.0
Ours
LLa Va-7B SFT
15.2
1.4
5.8
7.2
Table Success rate comparisons across different domains from WebArena. Success and failure detected with ground-truth verification functions. All tasks from OpenStreetMap are kept unchanged from WebArena task splits.
H. Limitations
Despite the progress of PAE for open-source VLM agents, there are still some limitations due to practical constraints. First of all, due to the limitations in fundamental capabilities of open-source base VLM models, our models trained with PAE are still inferior to state-of-the-art proprietary models in realistic web navigations, where advanced reasoning and planning capabilities are required. Moreover, because the dynamic nature of the real websites that we are using, some of our results may not be produced exactly, although a significant improvement from PAE should still be observed.
Hyperparameters
We include the hyperparameters that we have used in Table 5 have on top of standard supervised fine-tuning are number of trajectories to collect in each global iteration in Algorithm [] number proposed tasks from the task proposer before RL training, and the number of seen screenshots for the evaluator. In our experiments, we found that PAE relatively not sensitive the choices of these hyperparameters, showing the robustness of PAE.
As shown in the table, the only hyperparameters that PAE
J. Qualitative Comparisons
To qualitatively understand the benefits of PAE, we present snippets of example trajectories in Figure 22 from evaluations on WebVoyager where VL-7B PAE and LLB SFT attempt the same tasks. Full trajectories are included in Appendix In the first example, we find that while VL-BB knows SFT that should use search bar to find models related to error correction, it fails to choose the correct search bar (should be [18] instead [1]). However, Va-7B PAE learns the skill of using the search bar through typing into the correct index [1] and executes its plan complete the task. In the second example, the agent needs to navigate to the Advanced Security page of Github. While both models are able to navigate to the Security page of Github first, there turns out to be no direct links from the Security page to the Advanced Security page. As result, LLaVa-7B SFT ends up wandering in Github without finding the Advanced Security page. In contrast, LLaVa-7B PAE learns the skill of using Google Search in the absence of a direct link and it successfully navigates to the right page with its help. In both cases, we observe qualitative evidence of PAE teaching the agent a diverse repertoire to effectively complete unseen tasks.
23

---

Proposer-Agent-Vaalaator (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Table 5. Hyperparameters for All Experiments
Environment
Hyperparameter
Considered
Chosen
learning rate
{2e-5, 5e-5, 2e-4]
2e-5
rollout trajectories
{512, 1024, 2048, 4096)
4096
rollout temperature
{0.4, 1.0, 2.0}
1.0
maximum gradient norm
{0.01}
0.01
Web Voyager
actor updates epochs per iteration batch size gradient accumulation size number of proposed tasks
{1, 2, 4, 8, 20) {8}
4
8
{16, 32)
32
{10000, 50000, 100000)
100000
number of seen screenshots for evaluator
{1.3}
3
learning rate rollout trajectories rollout temperature
{2e-5, 5e-5, 2e-4)
2e-5
{512, 1024, 2048, 4096)
2048
{0.4, 1.0, 2.0} {0.01}
1.0
maximum gradient norm
0.01
WebArena
actor updates epochs per iteration batch size
{1, 2, 4, 8, 20} {8}
2
Easy
8
gradient accumulation size number of proposed tasks number of seen screenshots for evaluator
{16, 32)
32
{10000, 30000, 100000)
30000
(1.3]
3
Table Hyperparameters for PAE for WehVoyager and WebArena Easy experiments.
K. More Qualitative Examples
In this section, we present additional qualitative examples of agent trajectories while performing tasks to further demonstrate the effectiveness of our PAE will also release the full dataset for further analysis.
Full trajectories of examples in Section 6 Here, we provide the complete trajectories for the examples discussed in the qualitative comparisons in Section 6 shown in Figures 26 We detail the agent's thoughts and actions at each time step throughout the entire trajectory.
Some representative successful trajectories. We also showcase representative successful trajectories generated by the LLaVa-7B PAE model to highlight the strengths of our method. In Figure 27 the task is "Show the most played games on Steam, and tell me the number of players currently in-game." " In Figure 28, the task "Find out the starting price for the most recent model of the iMac on the Apple website." In Figure 29 the task is "Look up the use modal verbs grammar section for expressing possibility (e.g., 'might', 'could', 'may') and find examples of their usage in sentences on the Cambridge Dictionary." Finally, in Figure 30 the task is "Search for plumbers available but not open 24 hours in Orlando, FL.."
24

---

Proposer-Agent--aalalle (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Task Proposer Prompt for WebArena Map
("web name": "map", "id": "map-2", 2", "ques": "Tell me the full address of all international airports that are within a driving distance of 50 km University of California, Berkeley")
"map". "id": "map-10", "ques": "I will arrive San Francisco Airport soon. Provide the name of Hilton hotel in the vicinity, if available. Then, tell me the the shortest walking distance to supermarket from the hotel.") ("web name": "map", "id": "map-17", "ques": "Check if the ikea in pittsburgh can be reached in one hour by car from hobart street")
We are training model navigate the web. We need your help to generate instructions. With the examples provided above, please give 25 more example tasks for the model to learn from in the domain of OpenStreetMap. You should imagine who is the most likely user for the website and propose tasks that are likely to be proposed by this user. Please follow the corresponding guidelines:
F)First output your thoughts first on how you should come up with diverse tasks that examine various capabilities on the particular website, and how these tasks reflect the need of the potential user. Then you should say 'Output:' and then followed by the outputs STRUCTURED IN JSONL FORMAT. You should not say anything else in the response.
2)PLEASE MAKE SURE TO HAVE 25 examples in the response!!!
Your proposed tasks should be DIVERSE AND COVER A WIDE RANGE OF DIFFERENT POSSIBILITIES AND DIFFICULTY in the domain of OpenStreetMap. Remember, your job is to propose tasks that will help the model learn to navigate the web to deal with various real world requests. Your task should be objective and unambiguous. The carry-out of the task should NOT BE DEPENDENT on the user's personal information such the CURRENT TIME OR LOCATION.
You should express your tasks in diverse expressions as possible to help the model learn to understand different ways of expressing the same task.
6) Your tasks should be able to be evaluated OBJECTIVELY. That is, by looking at the last three screenshots and the answer provided by an agent, it should be possible to tell without ambiguity whether the task was completed successfully or not. Your tasks should require a minimum completion steps from 3 to steps, your tasks should have diverse coverage in difficulty as measured by the minimum completion step. L.E. You should propose only tasks that may take more than steps to complete but also tasks that can be completed within 3 steps.
Hummnss should have 100% success rate in completing the task. Your tasks should be able to be completed without having to sign in to the website.
Figure 15. Prompts used by Claude 3 for proposing WebArena Tasks for Map. For PAE User Demos, the prompt with additional user demos appended to the message. For this domain only, we provide three hand-written in-domain examples to set the right difficulty for the task proposer. Such in-domain examples not needed for all other domains including Reddit and OneStopMarket, and other real-world WeyVoy websites.
25

---

Proposer-Agent-Baauuale (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Task Proposer Prompt for WebArena Reddit and OneStop.arret
("web_name) name": "Apple", "id"; "Apple-40", "ques": "Find the pricing and specifications for the latest Mac Studio model, including the available CPU and GPU options.", "web": "https:/wwww.apple.comm We are training a model to navigate the web. We need your help to generate instructions. With the examples provided above, please give 25 more example tasks for the model to learn from in the domain of {web_name). You should provide tasks in the DOMAIN OF (web_name))
Please follow the corresponding guidelines: F)risst answer how many screenshots are provided and describe in detail the functions of the website that you see from each of the screenshot. Then output your thoughts first. Then you should say 'Output:' and then followed by the outputs STRUCTURED IN JSONL FORMAT. You should not say anything else in the response.
2)PLEASE MAKE SURE TO HAVE 25 examples in the response!!!
Your task should start from the home page of the website instead of the shown screenshots.
Your task does not need to be the same as real users would do, but it should examine diverse capabilities of the agent to do web navigartion.
Your tasks should examine the VERY BASIC functions of the website and should not require complicated web page operations. They can be completed within 5 steps.
7YTHIS DOMAIN IS SELF-HOSTED STATIC DOMAIN AND DIFFERENT FROM POPULAR WEBSITES, DO NOT ASSUME ANY INFORMATION NOT PROVIDED IN THE SCREENSHOTS.
Your tasks should examine the capability of the web agent to find some information on the website, navigating some specific web pages. Do not propose tasks that involve making actual modifications to the websites.
9) Your tasks should result in the agent landing in a single groundtruth web page or finding a single grounth truth answer. The landed webpage can be some specific categories, a drafted post, some search results, or even the homepage of website, When the task to find some information, specify exactly what information the agent should find such as the price, the number of comments, the title, etc. It can also be information about the current account.
Figure 16. Prompts used by Claude-Sonnet-3 for proposing WebArena Tasks for Reddit and OneStopMarket. For PAE + User Demos, we the prompt with additional user demos appended to the message.
26

---

Proposer-Agent--valttor (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Zero-Shot VLM Agent Prompt for WebVoyager (1/2)
Imagine you are robot browsing web, just like humans. Now you need to complete task. each iteration, you will receive an Obscrvation that includes a screenshot a webpage and some texts. This screenshot will feature Numerical Labels placed in the TOP LEFT corner of each Web Element. Carefully analyze the visual information to identify the Numerical Label corresponding to the Weh Flement that requires interaction, then follow the guidelines and choose one of the following actions:
1. Click Weh Flement.
2. Delete existing content textbox and then type content.
3. Scroll up down. Multiple scrolls are allowed browse the webpage. Pay attention!! The default scroll the whole
window, If the scroll widget located in a certain area of the webpage, then you have to specify a Web Element that area.
would hover the mouse there and then scroll.
4. Wait. Typically used to wait for unfinished webpage processes, with a duration of 5 seconds.
5. Go hack, returning to the previous webpage.
6. Google. directly jump to the Google search page. When can't find information in some websites, try starting over with Google.
Answer. This action should only chosen when all questions the task have been solved.
Correspondingly. Action should STRICTLY follow the format:
- Click (Numerical_Laeee
-
TType [Numerical Label]; [Content]
Scroll [Numerical_Lne Label or WINDOW]; [up or down]
Wait
GoBack
Google
ANSWER; [content]
Key Guidelines You MUST follow:
Action guidelines
1) To input text, NO need to click textbox first, directly type content. After typing. the system automatically hits "ENTER" key. Sometimes you should click the search button apply search filters. Try to use simple language when searching.
2) You must Distinguish between textbox and search button, don't type content into the button! textbox found, you may need to click the search button first before the textbox displayed.
3) Execute only one action per iteration.
4) STRICTLY Avoid repeating the same action the webpage remains unchanged. You may have selected the wrong web element or numerical label. Continuous use of the Wait is also NOT allowed.
5) When a complex Task involves multiple questions or steps, select "ANSWER" only at the very end, after addressing all of these questions (steps). Flexibly combine your own abilities with the information in the web page. Double check the formatting requirements in the task when ANSWER.
Web Browsing Guidelines
I) Don't interact useless web elements like Login, Sign-in, donation that appear Webpages. Pay attention Key Web Elements like search textbox and menu.
2) Vsit video websites like YouTube is allowed BUT you can't play videos. Clicking to download PDF allowed and will be analyzed by the Assistant API.
3) Focus on the numerical labels in the TOP LEFT corner of each rectangle (element). Ensure you don't 't mix them up with other numbers (e.g. Calendar) on the page.
4) Focus the date task. you
look for results that match the date. may necessury find the correct year,
month and day at calendar.
5) Pay attention to the filter and functions the page, which, combined with scroll, can help you conditions like 'highest', ; 'chcapest', 'lowest', ; 'carliest', etc. Try your best to find the answer that best fits the task.
Your reply should strictly follow the format:
Thought: [ YYour brief thoughts (briefly summarize the info that will help ANSWER))
Action: {One Action formal you choose)
Then User provide:
Obscrvation: (A labeled screenshot Given by User)
Figure 17. The prompt used for all zero-shot VLM agents for WebVoyager websites, including Claude-Sonnet-3, Claude-Sonnet-3.4. Qwen2-VL, VL, Intern Intern C.C.oose LLLVa-1.6-7B and LLaVa-1.6-34B.
27

---

Proposer-Agent--Vallatoor (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Zero-Shot VLM Agent Prompt for WebArena (2/2)
Imagine you robot browsing the web, just like humans. Now you need complete task. In each iteration, you will receive an Observation that includes a screenshot of a webpage, some texts and the accessibility tree of the webpage. This screenshot will feature Numerical Labels placed in the TOP LEFT corner of each Web Element. The accessbility tree contains information about the web elements and their properties. The numrical labels in the screenshot correspond to the web elements in the accessibility tree.
Carefully analyze the visual information to identify the Numerical Label corresponding to the Web Element that requires interaction, then follow the guidelines and choose one of the following actions:
1. Click Web Element.
Delete existing content textbox and then type content.
3. Scroll up or down, Multiple scrolls are allowed to browse the webpage. Pay attention!! The default scroll is the whole window. the scroll widget is located in certain area of the webpage, then you have to specify a Web Element area.
I would hover the mouse there and then scroll.
4. Wait. Typically used to wait for unfinished webpage processes, with duration of 5 seconds.
Go back, returning to the previous webpage.
Answer. This action should only be chosen when all questions in the task have been solved.
Correspondingly, Action should STRICTLY follow format:
Click [Numerical.LLbee]]
"Tppe [Numerical_Laber [Content]
Scroll [Numerical WINDOW]; (up or down]
Wait
GoBack
ANSWER; [content]
Key Guidelines You MUST follow:
* Action guidelines
1) To input text, NO need to click textbox first, directly type content. After typing, the system automatically hits 'ENTER'
key. Sometimes you should click the search button to apply search filters. Try to use simple language when searching.
2) You must Distinguish between textbox and search button, don't 't type content into the button! If no textbox is found, you may need to click the search button first before the textbox is displayed.
3) Execute only one action per itcration.
STRICTLY Avoid repeating the same action if the webpage remains unchanged. You may have selected the wrong web
element or numerical label. Continuous use of the Wait is also NOT allowed.
5) When complex Task involves multiple questions or steps, select "ANSWER" only at the very end, after addressing all of these questions (steps). Flexibly combine your own abilities with the information in the web page. Double check the formatting requirements in the task when ANSWER.
6) If you can't 't find the answer using the given website because there no such information on the website after some attempls, you should report "N/A" the answer to represent that the task is impossible to solve with the given website. You may have 15 steps to try to solve the task.
Only provide answer based on the information from the image, make sure the answer consistent with the image, don't hallucinate any information that is not based on image.
* Web Browsing Guidelines *
1) Focus on the numerical labels in the TOP LEFT corner of each rectangle (element). Ensure you don't 't mix them up with other numbers (e.g. Calendar) on the page.
Pay attention the filter and sort functions on the page, which, combined with scroll, help you solve conditions like "highest', 'ceaapss', 'lowest', 'carliest,, etc. Try your best to find the answer that best fits the task.
Figure 18. The prompt used for all zero-shot VLM agents for WebArena websites, including Claude-Sonnet-3, Claude-Sonnet-3.4, VW, Intern 2.2.2.5..Cooopose LLaVa-L16-7B, and LLaa--1663443 To be continued in Figure [9
28

---

Propeser-Agent--Vallatto (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Zero-Shot VLM Agent Prompt for WebArena
OpenStreetMap Usage Guidelines
I) When you need to search the address of a location, you can just type the location in the "search" bar. You don't need to use the directions button to get the address. The directions button is only used when you need to find the distance/wall//diive time between two locations.
2) When you are trying to search for a location, you may get no results. This is because the OpenStreetMp does not support approximate search. You may try search some alteraative keywords or try to find the location by yourself. Note that openstreet map does not support search phrase like 'Cafe near CMU", you should try to find it by yourself.
3) When you need to find the distance/wall/ddrie time between two locations, you should FIRST CLICK ON THE DIRECTIONS BUTTON (drawn as arrows), the right of the 'Go' Button and usually labeled as [10] or [11]. AND ONLY INPUTTING THE TWO LOCATIONS AFTER CLICKING ON THE DIRECTIONS BUTTON WHEN THE DIRECTIONS SEARCH BARS ARE SHOWN.
4) When you are trying type some locations in the directions search bar, sometimes you may receive an alert of 'couldn't it locate' followed by the location you typed. This means the location you typed is not found in the map. Do not immeadiately try something else. You need to quit the direction and find the precise name of this location by searching it in the map first.
5) When you search the walk/drive/bike time, make sure that you are USING THE RIGHT MODE OF TRANSPORTATION. The default mode is usually set to 'Drive'.
6) When you need get the DD of some location, you need to click the location shown in the search result in the left part of the screen. The DD will be shown then starting with 'Location:'.
7) When you need to answer the zip code of some location, you should directly answer the 5-digit zip code. The answer shoule be "15232" instcad of "The zip code of the location is 15232". Note that the zipcode will be displayed in the search result, you don't 't need to click the location to the information page to find the zip code.
8) When you need to answer the phone of some location, please omit the part of the country code. The answer should be
"41 "4122683259" instead of "1 412 268 3259".
* Reddit Usage Guidelines
1) You are already in the reddit website, though you may not see the 'reddit" in any part of the screenshot. You do not need to further navigate to the reddit website.
2) When you want to find subreddit, you need to first navigate to Forums the list of subreddits. Under forums, you will see only a subset of subreddits. To get the full list of subreddits, you need to navigate to the Alphabetical option. To know you can see the full list. of subreddits, you will see 'All Forums' the observation. Often you will not find a focused subreddit that exactly matches your query. In that case, go ahead with the closest relevant subreddit. To know that you have reached a subreddit successfully, you will see //f/subreddit name" in the observation.
3) When you want to post forum in reddit, remember to fill up all the content, then click the button 'Create forum'. The button maybe located below out of the screenshot, you need to scroll down to find it.
4) When you want to ask or post something in a subreddit, you need to first find that subreddit and then finish the work. 5) forums and subreddits are the same thing.
Your reply should strictly follow the format:
Thought: Your brief thoughts (briefly summarize the info that will help ANSWER) Action: One Action format you choose
Then the User will provide:
Observation: A labeled screenshot Given by User
Remember only execute one action in cach step. For example, 'Action: Type [8]: CMU, Type [9] Pittsburgh' not allowed. You should execute the action "Type [8]; CMU" first, then "Type [9] Pittsburgh" in the next step.
Remember to always make your answer simple and clear. For example, if you want to report the zip code of some location, always say "ANSWER; 06516" instead of "Thh zip code of the location is 06516".
Figure 19. The prompt used for all zero-shot VLM agents for WebArena websites, including Claude-Sonnet-3, Claude-Sonnet-3.4, Qwen2-VL, LL, Intern MitemmLL-2.-XCooopseee LLaVa-1.6-7B, and LLaVa-1 1.-44BB Continued from Figure [8]
29

---

Proposer-gget-EVvaltttr (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Out-of-distributioo Websites of Web for SFT (1/2)
Allrecipes:
Simply Recipes: httpss//w...snnoyyreeppsssss
Food Network: http:///ww..occhwwwrt.ccooo
Taste of Home: httpss//www..assteoohooess
Yummly:
ht http:://w/wwwwwwm.y.yyyycoo
Food.com:
httpp::/ww..ooo..ooo
Amazon:
eBay: https://www.ebay.oo con
Walmart: https::/ww..w.mmatttt http:://wwwwwwwalaaaaaa. com
Target: tttps.//ww..aragtt.ccco
Best Buy: htt.ptpp../www.bbsbbbyyyco
Alibaba: http::/www.allDagtt
Apple:
Samsung: http://w....ssssssccccc
Microsoft: http:://www.mmmrrrtrtttcco
Sony: httpps:/www.sonn) www.:eeeee.oooe..ooooooo
Google Store: http::/sss/s/tt...gooo/o////
Dell: ftttps:/www.dell.com
ArXiv:
SSRN: https://ww..sm..cmm
bioRxiv: httkkw//tttss///.bbboo......ggg
IEEE Xttoorr/ttppp///eeeees.rr..e...r
PubMed: P http::///iiimee.................
GitHub:
GitLab: httt:s://abctt.gittab..om
Bitbucket: httsst//////ttbbc.ttttogrgg
SourceForge: http:////////////oooooooooreeet
Codebase: http:://www..ccbbbbecccccco
Gitea: htttp:://gita..10
ESPN:
CBS Sports: http:::www.dssspttt..ccm
Fox Sports: https://ww.ffopports.ccom
NBC Sports: https://www.nbcppprtscccoo
Bleacher http:///ttsssss//...aaaaaaeeeee.
Sky Sports: http:://w..sssspott...oo
Coursera:
edX: htt.tts://w.ee....../
Udacity: http:/tss/www..uuc.tty.ccooooo
Udemy:
httpp:://www.uddmyyy
Futureleeam: htw.a//.w..ta////ooo.....t.aaaaa
Khan http:::/www.haaaaaaaattteeee
Figure 20. A of websites that we used to collect demonstration trajectories with Claude Sonnet. total 11220 trajectories were collected with different tasks. These websites also used for testing the zeroshot generalization of PAE dustt--ditttibution websites in Section List continued in Figure 21
30

---

Proposer-Agent--Vaauatoo (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Out-of-distribution Websites of Web WebVoyager for SFT (2/2)
Cambridge Dictionary:
Merriam-
com
Dictionary.comt
Oxford
s httpa//awwaaaaaaaaa//oaoaaaaaata
Collins English /ttpoo/rrrr.sso/..oooooooooooaor con
YourDictionary:
BBC News:
CNN: https::/ww..com com
Al Jazeera: http::://www.w..t httpa//s...p///....aaaaeeeeaaa wwwp:/awawwaeeaa.oom
Reuters: httpp:::ww..rutteeeettee/ com
The Guardian: htt.://nnnng//////...uuuuuuaaaaa
NBC htwp:/tttts//wwmmmneee..oo
Google Maps:
Apple Maps: http:::/maps..pplee...
Bing Maps: https::////// https::/www.bing. www.ppa..................a
MapQuest: http:t/sst///ww...pppuss........
Waze: httpp:::/wwwwwwwss...mm
Here WeGo: htt.://tttts//gg..eee...oo
Google Search:
Bing: https: http://wwww.ngg.../
Yahoo Search: httpp:://eercc..aabbo.o
DuckDoo http:///tt.tp/////cccccco...... com
Baidu: httpa://ww.baidd..c com
Yandex: htttps://aadex..
Hugging Face:
OpenAl: https::/openai... com
TensorFlow: httpps::wwww.tenooffllo... org
PyTorch: ht::s:://pytorcchoogg
Kaggle: https://ww.kaggge..com
SpaCy: https://spacy.it
Wolfram Alpha:
Google Scholar: httpss:/scholaaa.ggogll....
M httpss//ww..nattwwyyycooom
Symbolab: bttps://xww..ymmbollabb.
Microsoft Math Sathho/oooo//tttssaaaatttsssseee www.osoft..ccmmmmmm.mm
http://matttsseeeeemooooooooooo com
Desmos: https://www.desnot/
Figure 21. A list of 85 websites that used to collect demonstration trajectorics with Claude 3 Sonnet. In total 11220 trajectories collected with different tasks. These websites were also used for testing zeroshot generalization PAE to ant-of-distribution websites in Section ??. List continued from Figure 21
31

---

Proposer-Agent-Vaalaale (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Task:
Find the most recently updated machine learning model Huggingface which focuses on Error Correction.
LLaVa-78 78 PAE
LLava-7B SFT
X
Step1 1
Step 2
Step
Step Thought: (Hallucination) The search results show several models related to error corraction,
To Thought:
Thought:
Thought:
recently updated machine learning nodel focusing
The page shows details about the "polish-sinple error- conrection" nodel. seens to be the most recently updated model focused on Error Correction that looking for.
Now that I'n in the Models section, should search for models related Error Correction. can try filtering results by adding "error correction"
Correction, I need to search for relevant keywords like
"eroor cornection 11 and hype sed11". To find the most recently updated model
"error correction' and filter the results by the highest update date.
Action: Answer: polish-
Action: (Wrong)
correction, I may need to further refine the search check the filters.
Action: Type [1] Error correction
Type [18] Error correction
simple-error-
correction [13]
Task:
Find the Security topic in GitHub Resources and answer the role of GitHub Advanced Security.
LLaVa-7B 7B PAE
LLaVa-78 SFT
Security
Security
Step
Step 5
Step
Step 8
Thought:
Thought:
Thought: (Planning error)
Thought:
The "Security" section subsactions. learn nore about Gittlub Advanced Security ifycllll, mey need search navigate
The search results provide sone relevant information about GitHub Advanced Security. The top result seens
of Gitilub's advanced security. There highlighting
page appears
Thought: The search "shift left testing" did not return any relevant results Gitilub, as indicated the results" nessage. To find information related to this topic, I may need to different: search query or look in other Action: GoBack
main the benefits and capabilities The "what is shift left testing?" section likely provides an introduction to GitHub's 's advanced security testing features. Exploring those sections in detail should help role of GitHush Advanced Security, Action: (Wrong)
official documentation from GitHub on topic. Reading this page should give me good understanding of role and features included in Gittub Advanced Security. Action: Click [17]
further within the
Resources saction.
the
Action:
Google: github advanced security role
Gittlub.
Type [17] What is shift left testing?
Figure 22, Qualitative comparison between LLa Va-7B PAE and LLaVa-7B SFT on the same tasks. LLaVa-7B PAE model successfully completed tasks using learned skills from the RL training.
32

---

Proposer-Agent-Vvaluaee (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Thought: To hnd the moet leaming model thousing on search for relevant
Thought fidd
recently updaced mach Enor Correctinn need section un dugging Aoc
The community building the future.
conrection and fiterthe the update dete action ener
Thought: The shews "polish sbout the
Thought search
"caith maiith-mzlee carach
Updated moree This teems most focused Ermy Connection Action Antswent The most leaming model focused on Face the model which was leet ago.
Eror Conrection However divious information about when up updated. find the most may need futtner fiber additonna data is each (W knoww] down
Figure 23. Full trajectories of success trajectory 1 Figure 22 with task 'Find the most recently updated machine learning model on Huggingface which focuses on Error Correction' executed by model LLaVa-7B PAE.
33

---

Proposer-Agent--vaatttr (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
recently upcated mach na
Thought: New that the Models stouls models related to Emor Corretion, can ty the sdiling emer cormecion starch Type (16): erorr correction
Huggingtace that facuses
The community building the future.
chould first navigace to the
Models from
Action Click
Hugping
"Other" has expancedd to show
show several models such as the
"aaeety promotatie" Waie namoxtin cown the model
Toytestee dance hype sio information on about when updated. To find the most recently
informa ontio seliially when erorr corection the losk updated. may need check the apdated ep each Action: Goflack
mode: Correcion.1 reed further rehne search the hers Click (17)
Thought: Afer going back resuts pape, don't see the liote arrer cormect oo The Last Undateed fiter poter Guily surface the most sowe me
find the most recently
model on Huggingface
findhinnlltee i appooach soarch dinccty
community building the future.
but
Google, Perhapo Google lase more up-lo information altermetive
Sher option. le the most revera my epproach aearch Google,
rearted this teak.
[8] recently updated ener model
huppingtica
Thought: The Google
sereling doan lutther the Google recent discussions and facee models, but aill dhe't which specific encer model trest recently updated one. resuit covit but
search resula relevent ressurces from Hupping related tools. rodals and not
entinely cleat which specific updated one focused on suem sover some genemeting promoted definitive answer on the (WNNDOM) down
clear information comection models. Actions Try again with a cuery
Huzginng face to hopenully to an coer the tack.
sededd
Thought The search or Google thow toma
recently upditted errir
modele from Hugging Race. one. i-hooud go drrccty to focused soarch there. the Luria Emor
Hupging Face. should
more specific search query their webs tawif. action: [1]:
sttchuppingges ca carract
Speech Recogn bo on 147
Figure 24. Full trajectories of fail trajectory 1 in Figure 22 with task 'Find the most recently updated machine learning model on Huggingface which focuses on Error Correction' executed by model VL SFT.
34

---

Proposer-Agent--Vaatttr (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Thought Rescurces been expordedd revading
Let's build from here
Thought The lask to find the Security
Let from here
Pepzorress ane role Gatmub Advenced begin, lahould ravicate to the Rezources website. Actian: Click [9]
the
"DenOppr Security" informat advanned the ghouldd subsecttion Action Click
develgerglatoo plaction
Thought The "Seurityy
Security
oubsections, What 21"
testing
not refum This malles that the current Resources section
gthii advanced security
coftitine Bon
you (SCA)" These
exerviey and left taating
intormation speciically about thib
and development
zottuare are
Secumyy and derats topic,
leam more about Oitius
different segroach like navigabiig to fffieert security and compllence.
specifically by may need
within the Resources
Ben fownd doddur begadd
g glruu edvenced security
The pray de some relevent Advanced Security The top omiial documentatin from Reasing through this page undordanding the role Github Acvenced Security Action: [17]
thought: am now the Google search pape Advanced decuthy, should nearth quary the teuron. advenced security rile
featune
bibb makes oo aeculty featre ab
About Gitllbb Advancnd Security
Thought: The surrent page Decs tboumentates on provides an overvew and more informaa Information about the answer the mite Oitiuu Acvanced the related dssumennatt aections Action: click [12]
Thought: The "About features: eact provides suieerbility ecann ng. coding features included in This appears to directly role capab loes the offering.
caloeees unint Adreeeed enel Decullly imares Theoe Coleeluore
by.
should
Action: Click [12)
Troughtt The 'About Advanced
saction provides ar cvencen tay securtyy and capasi ties, incluing cocured socrets cotection. This directly adetosins the Github Adieenced Security Giblibb Advenced Secuitty featuree and cap asaiiies p platform. Key features scorning for o001 fource depanoienccss societs deteect screso prosects. and secure socing
Figure 25. Full trajectories of success trajectory 2 in Figure 22 with with task 'Find the Security topic in GitHub Resources and answer the role of GitHub Advanced Security' executed by model LLaVa-7B PAE.
35

---

Proposer-Agent--Valaatoo (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
to find the Security tapic in understand the role of The current webpaag the does not provide direct security offerings To navigatl the Jhould Iualy use functionall by explore the pwct further Type advanced
Thought e The search for "piinus advenced by" relevent horrepage The current
Let's build from here
Let's build from here
cresting accourt
lonly indiiiuua develosent rather understanding their for organ zation. Is find erplore the urces Ci ch [3)
security
appears
to be in overvise of security. testing, and Thare are two main be benotti and cheabl in "What testing section loely
Thought Reasurces tikk the main topic "hi pramis finding information aboir nced securit offering organizations on the Securit ky lick wi tn more dutiallo Ack
Security
Let from here
iucronacion
Bithue's advanced security tesking
Exploring
suctions detail should answer Orthuo kivvances vencee Secur twating"
shitt
Thought:
the (37) not seem to texthox query, luay Just more octions. under stand "sh rt left
Thought The search bar sewnss to he the carrect type "skit left tecting find dictuu about this soltware concapt relation Adiernced ahilt leit
G tri la Advanced Security. ink, to navigate the Action Click (171
Action: type (7)
Thought: You're cornect, elament labeled [7]
Thoughe: after dicking ng the has appeored labeled (0). "chift loft lecling into this ox relevert content darub. teating
search "sh Pt left
will
first need to Sik the '71
search ber. than urue the appeers. Action: Click [?]
Type [0]:
Thought: After going back to the previoue Grtiu "roductt button now button prey oudly took me acverrt-eman eemene. which was into momommaion about Sittu''s Click (2)
Thought The search not
ngh recurn any relevaat resu indicated the "Mo resulo" message this topic. may need to look other areas ot
the
this
Figure 26. Full trajectories of fail trajectory 2 in Figure 22 with task 'Find the Security topic in GitHub Resources and answer the role of
GitHub Advanced Security' executed by model LLaVa- SB SFT.
36

---

Proposer-Agent-vaalutto (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
Thought To hnd the most and the number of players need ceauch for mort "rumber of playero in boogle, actions Type (8) played games steam
Thought: The search result Stea m Charts that provices played games on Steam. and most played games promiiing the Actin: (17)
Thought The "Most played"
most played games on their ramres plnyer counts percertage change
Ovarvan page provddes
CHARTS OVERVIEW
number of playert
top 10 most played ganney
SAATE MARINE
morchs This
ersirers the teak
mortss The
pesk hours
games Steam
each spec
counts
deta their
Action ANSWER
220320
counte may need
Steam right now 669.177 170 Day (60%s inceade from
exlecting differeiz time
action: Screll NWWD
W. kong 128 653 daaeer
playera 11.0% decre seeeeee playars (2%) increase) S...
Figure 27. Extra full trajectories of successful trajectory with task 'Show most played games Steam. And tell me number players in In game at this time' executed by model VLaV PAF.
Thought: The currant
Thought:
homepage wmich does not the starting price for the find this. will Hoeey need to navigate of websito Acter: Click
Mac
thows different Rac moseis including Heac. is not euplicyy ticty ment aned need to click the iMac Actionc Click [16)
Apple Event
iMac
for the most recent litc $1200 with education 5636 of the 1150 gif card on about darting action ANSWER. The starting the most Mae medel educabion diecount
Apdle
Figure Extra full trajectories successful trajectory 2 with task 'Find out the starting price for the most recent model of iMac on the Apple website' executed by model LLaVa-7B PAE.
37

---

Proposer-Agent--valuatoo (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
WOOLY
Contori dgg Oksionary
thuught evamp les
rightt
Thought: papurated with mightt sould may. verbel wantted can for poetisllyy shourd now swarch the Action: Click E171
verze
could may
coule" poesisilty
nead search thoee
sertion on the Cambridty
Easicrs Cuntakáge Didionery
Type 181; might could may
Skg
Thought The search results
"mighh" and "erey" expressing poza Jay have hovever.
Thought: After searches, realze lse may mizi ng informat stacs recessary annd modal like "nigat" "coud" on this Cambrr websfte CoBack
Search suppeedios for riigtt
Search suppestion for riigtt
it seamr are
these
Typm
words, wtric quite
common utage and
Action: a newr search
pronuncaation haps pronuncation union might be
Seacn Sppl
Thought: After going back
there a "Grmmma
right
Thought. With the soarch section move search which should proude for the usage modsl "might" 'may. action: Type 1141;
could may
search to grammar terms
could may
directly for model verbe" "might.. coule": "mayy" and exdenaation need. dittonnry search to Grammnr
desple..
Thought The search results
useld cetission and
model verbo like "might"
port bilyy willinnes
Midit
ability The frst recut
points need about the
including " They express towards what they saying. Passiility obripaionn recenntly and ability ane espressed tenteerree an thown diuerrat modill vo ts. Action...
Figure Extra full trajectories successful trajectory with task 'Look up the use of modal verbs in grammar section for expressing possibility (e.g., 'might', 'could", 'may') and find examples of their usage in sentences on the Cambridge Dictionary' executed by model LLaVa-7B PAE.
38

---

ProposerrAgeent-Bvauttoor (PAE); Autonomous Skill Discovery for Foundation Model Internet Agents
d0
Thought: "o search for plumbers availeble 24 Orando, Maps, Google relvent cegpards 10;: plumoer avallatle now
showr seme plum binging comparees that opan are looking for what avanable spen hours, ws may further
beywords
plum berb avalabtle now open 24
action
alll show some plumbirg 24 hours, snich not what plumber available now but need to furthr refne the diffrrent asproach like hours fisted Action: Click
the "Gpen 24 noursl Than, for plumb ng companieg cortain hours but not This should help us find not open around clock. crui able hours enandd.
opon
Thought: showing plumbing camperies
hours the Orlando area. "The fitwred show hours Minnday to Cenurday. AKSWES The search results show are avarible now but not Orlando, erea. The their open no hours from indicating when they For exampln one mouit ot a PM Monday
Friday, then reapers 2...
Figure 30. Extra full trajectories of successful trajectory with task 'Search for plumbers available but open 24 hours in Orlando, FL, executed by model L.-77 PAE.
39


Reviews:
[{"review_id": "IVVdvaD8Yx", "reviewer": "Reviewer_FPLC", "summary": "The authors propose Proposer-Agent-Evaluator (PAE) as a web agent that generates probable tasks (instructions) by itself and performs RL fine-tuning with binary rewards from VLM-based evaluators. For the generation of tasks for training, they employ a number of different pieces of information about target websites, such as website names and human demonstrations. Using the Set-of-Marks (SoM) annotations for web page screenshots, they fine-tune their VLM with binary rewards that are generated by prompt-based VLM evaluators, which take the last three web page screenshots and the agent's answer as input. The authors suggest that their approach brings performance improvements relative to supervised fine-tuned VLM agents.\n\n## update after rebuttal\n\nThank you for providing the rebuttal.\n\nRegarding the originality of this work, using an autonomous evaluation and fine-tuning based on the filtering was a practice used by \"Autonomous Evaluation and Refinement of Digital Agents\" (Pan et al., 2024, cited by this submission, as well), although the filtered behavior cloning was for their iOS agents, and it was first presented as a preprint on Apr 9, 2024 and as a non-preprint at the Multimodal Algorithmic Reasoning Workshop (CVPR) on Jun 17, 2024. Using foundation models for task proposal in the web navigation domain has been used by WebVoyager (He et al., ACL 2024) or even Mind2Web (Deng et al., NeurIPS 2023). As I mentioned in my original review, I am not suggesting that there are no differences from existing work, and I agree that empirically showing that self-improvement can work could be meaningful. But overall, my view regarding the originality remains similar.\n\nFor the empirical evaluation, I believe demonstrating the effectiveness of the proposed framework with a stronger setup on a more difficult set of tasks would provide better empirical takeaways.\n\nWhile I appreciate the authors for providing the response to my review, I keep my original rating due to my primary concerns about this work.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback on the paper. Just a gentle reminder that by the definition established by the ICML2025 committee, concurrent work includes all papers from within 4 months of the submission deadline. This applies to all other pipelines for web agents [1,2,3] with similar components (task proposers, agents, and evaluators). As such, they should not be considered as an established standard. With that being said, we also provide additional discussions on the differences with these works to address your concerns regarding novelty and also answer your other concerns below. **Please let us know if your concerns are addressed and if so, we would appreciate it if you could re-evaluate our work based on this context. We are happy to discuss further.**  \n\n## Overall, the originality of this work looks somewhat limited to me\nFirst, we would like to emphasize that while other pipelines [1,2,3] with similar components (task proposers, agents, and evaluators) have been applied to web agents because of the effectiveness of these components, they are all made public within 4 months of the submission deadline of ICML. As stated in the reviewer guideline of ICML, they should be considered as concurrent instead of an established standard and “Authors cannot expect to discuss other papers that have only been made publicly available within four months of the submission deadline. “.\n\nAdditionally, a key difference is that [1,2,3] generate instructions and evaluation rewards from stronger proprietary models such as GPT4o to train weaker models, while our PAE system focuses on exploiting the asymmetric capabilities of foundation model web agents to achieve self-improvements, where even weaker models can be used to improve the performance of stronger agents. We propose that this is a meaningful contribution, as it demonstrates a path to advance state of the art capabilities. In our main results, we have also performed extensive experiment analysis seeking to understand when such self-improvements are possible (the use of a reasoning step, the use of sparse outcome-based reward, different choices of the models etc) and how well such self-improvements can generalize (to unseen tasks and unseen websites). This is the novel finding of this paper.\n\n\n[1] Nnetscape navigator: Complex demonstrations for web agents without a demonstrator, 2024.\n[2] Openwebvoyager: Building multimodal web agents via iterative real-world exploration, feedback and optimization, 2024b.\n[3] Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning, 2024.\n\n\n## Using the original WebArena would give better comparisons across different papers.\nIn addition to experiments on WebArena Easy as reported in the main text, in appendix G Table 4 we also reported the performance of our models on the original WebArena using only screenshots as observations, and provided additional explanations in terms of why WebArena Easy is used as opposed to original WebArena. For your convenience, we also include results from Table 4 below. These results in Table 4 would be directly comparable with other papers testing screenshot-only models on WebArena. At the time when the main experiments of this paper were conducted, the best open-source VLM that we were able to train was LLaVa-Next but they were not able to achieve higher than chance performance (around 5%) for most of the websites on WebArena. In these cases, RL would be almost impossible because the base model does not perform meaningful explorations for these hard tasks.\n\n|                |                 | OpenStreetMap | PostMill | OneStopMarket | Average |\n|----------------|-----------------|--------------:|---------:|--------------:|--------:|\n| Proprietary    | Claude 3 Sonnet|           24.3|      10.6|          11.2 |    14.6 |\n| Open-Source    | Qwen2VL-7B     |            0.7|       0.0|           1.3 |     0.7 |\n| Open-Source          | InternVL2.5-8B |            2.6|       0.2|           3.3 |     2.3 |\n| Open-Source          | LLaVa-7B       |            0.0|       0.0|           0.0 |     0.0 |\n| Ours           | LLaVa-7B SFT   |           15.2|       1.4|           5.8 |     7.2 |\n\n\n\n## The use of the term \"skill\" may not be appropriate in this context and may be replaced with a clearer term. \nWe used the term “skill” because most of the tasks that can be completed by open-source VLM web agents were restricted to rather short-horizon and atomic tasks like “find a chicken recipe with more than 4.8 reviews”. To avoid confusion, we will replace them with the term “policy” as suggested in the context.\n\n## The readability and presentation of the manuscript can be improved. Many of the tables and figures have too small fonts to read.\nThank you for the concrete advice on improving the presentation of the paper. We will increase the fonts in the updated version.", "claims_and_evidence": "- The assumption about providing the knowledge and information about desired websites to the task proposer is reasonable, because coming up with tasks that are useful and executable at the same time for different websites can be challenging.\n- On the other hand, the use of demonstrations on the websites comes with its own challenge of being outdated as the websites get updated. This can make the proposed approach less \"automated,\" especially for websites that are not very well-known or get frequently updated. In this sense, the demonstration (screenshot)-guided task generation may not fully solve the issue of anticipating what is possible on each website.", "methods_and_evaluation_criteria": "- The proposed method, Proposer-Agent-Evaluator, makes sense for web agents. As mentioned in the Claims and Evidence section, generating tasks in real-world domains like web navigation should be well-grounded.", "theoretical_claims": "There is not much of theoretical claims from this submission.", "experimental_designs_or_analyses": "- Although the authors provide an explanation for creating the \"easy\" split of WebArena, I believe using the original WebArena (even if it does not include the full list of websites it provides) would be better for comparison across different papers.", "relation_to_broader_scientific_literature": "- The task proposer may be applicable in other domains or environments. In that sense, there is a possibility of broader application of this work.", "essential_references_not_discussed": "I think relevant papers are reasonably cited.", "other_strengths_and_weaknesses": "- Overall, the originality of this work looks somewhat limited to me. My understanding is that using foundation models to generate task instructions, trajectory data collection with agents, and foundation model-based evaluators for evaluation and trajectory filtering (some of the corresponding prior work for these items is already mentioned in this work) are part of the standard practice for training web agents, these days. While some decisions for these components may have difference, I think the overall pipeline is similar to the norm.\n- The use of the term \"skill\" may not be appropriate in this context and may be replaced with a clearer term. For instance, I don't think there would be much difference if \"policy\" was used, instead. At least in the context of agents and reinforcement learning, \"skill\" and \"skill discovery\" sound more appropriate for focusing on learning behaviors at levels smaller/lower than the task level (i.e., \"skill\" level), so that they can be combined or leveraged to solve more complex downstream tasks.", "other_comments_or_suggestions": "- I think the readability and presentation of the manuscript can be improved. For instance, many of the tables and figures have too small fonts and are hard to read.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rwFngZ64er", "reviewer": "Reviewer_D3jd", "summary": "The paper presents a novel framework, Proposer-Agent-Evaluator (PAE), enabling foundation-model internet agents to autonomously discover and learn diverse skills without relying on human-annotated task templates. By leveraging visual-language models (VLMs), PAE autonomously generates tasks (Proposer), executes them using an agent policy guided by chain-of-thought reasoning, and autonomously evaluates task success. Experiments demonstrate significant improvements in zero-shot generalization performance across realistic internet browsing tasks compared to existing baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback on the paper. We have included additional clarifications in the multi-turn nature of WebArena and WebVoyager, and details of computational efficiency. We are running additional experiments comparing different RL algorithms and we will share them here once we have preliminary conclusions. **Please let us know if your concerns are addressed and if so, we would appreciate it if you could re-evaluate our work based on this context. We are happy to discuss further.**  \n\n\n\n## Experiments are only conducted on single-turn interactions on WebVoyager and WebArena. It would be beneficial to experiment with multi-turn interactive tasks or long-term interaction scenarios\n\nTo the best of our understanding, WebVoyager and WebArena environments are multi-turn and interactive environments where the agent needs to interact with the websites for a few steps (~10-20 steps) to be able to complete a task like “Find the Easy Vegetarian Spinach Lasagna recipe on Allrecipes and tell me what the latest review says” as shown in appendix Figure 7. While we agree that these environments are still relatively short-horizon in that most tasks require between 10-20 steps for successful executions, these are already the hardest interactive benchmarks on which existing open-source VLM web agents can achieve non-trivial performance results. We expect that more research in credit assignments over longer horizons would be possible in the future as the capabilities of open-source VLMs improve.\n\n## The paper lacks a detailed analysis of computational efficiency\nOur main experiments of 7B models were conducted using 1 8x40G A100 and 3 8x24G A10G machines where the gradient update steps are performed on A100 only and A10G machines are only used for distributed data collection (i.e. trajectory rollouts). The trajectory collection phase takes around 90% of the training time while gradient updates only take 10%. The trajectory collection speed using all 4 machines is around 1k-2k trajectories per hour, so 7B experiments can be completed within 2 days. Our run of 34B model was conducted using 3 8x40G A100, where gradient update steps are performed on one of the A100 machines and all machines participate in trajectory collections. The training for the 34B model takes around 5 days under this setup. We will include this section in the appendix for the revised version.\n\n\n## The paper lacks comprehensive comparisons with other reinforcement learning algorithms and does not sufficiently explore performance differences with other LLMs.\n\nThe novel finding of this paper is to study whether an entire self-improvement framework of LLMs for web agents is possible and what the necessary design choices are. While additional RL results are good to have, because of the computational constraints discussed in the last response, we were unable to perform systematic tuning for other reinforcement learning algorithms like ppo and grpo with more hyperparameters and more sample complexity. Therefore, we refrained from drawing comparisons between different RL algorithms. With that being said, we are running additional experiments with PPO as the RL algorithm.  We will share the results here once we have preliminary conclusions.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The paper explores an interesting and practical problem—how AI models can automatically discover useful skills without relying on humans manually labeling tasks, which could greatly expand what these models can do in the real world.\n2. The proposed framework (PAE) is straightforward and well-designed. It enables the model to come up with tasks by itself, figure out how to solve them, and evaluate its own performance, without needing human input every step of the way.\n3. The authors ran thorough experiments, clearly showing that their method outperforms existing approaches and achieves strong results.\n\nWeaknesses:\n1. The paper lacks comprehensive comparisons with alternative reinforcement learning algorithms and does not sufficiently explore performance differences with other large language models.\n2. The paper conducts experiments only on WebVoyager and WebArena environments. Although these environments are rich, the tasks tested are predominantly simple, single-turn interactions (e.g., searching products or clicking elements). It would be beneficial to experiment with multi-turn interactive tasks or other long-term interaction scenarios.\n3. The paper lacks a detailed analysis of computational efficiency, including specifics on computational resources, data efficiency, and training costs.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xErS8pSolZ", "reviewer": "Reviewer_T3UV", "summary": "This paper introduces Proposer-Agent-Evaluator (PAE), a learning framework designed to enable foundation model-based Internet agents to autonomously discover and refine new skills without human supervision. PAE consists of three core components: (1) a task proposer, which generates skill acquisition tasks based on website context, (2) an agent policy, which attempts the tasks, and (3) an autonomous evaluator, which assesses success based on visual observations. The evaluation signal is then used to refine the agent’s policy through reinforcement learning. The paper demonstrates PAE’s effectiveness on web navigation benchmarks, particularly WebVoyager and WebArena, where it achieves a 50% relative improvement in success rate.\n\n## update after rebuttal\nI keep my rating since most of my concerns have been addressed. Regarding the long-term agents, I was thinking that some works—such as OpenManus—have started to support this direction. However, since these developments occurred within the past three months, they may not fall within the scope of this review’s evaluation period. Thus, I would keep my rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback on the paper. We have provided additional clarifications to the questions that you have raised on how PAE handles cases where the proposed task is infeasible, how the agent recognizes when to reuse a learned skill, and whether intermediate reward signals can help. **Please let us know if your concerns are addressed and if so, we would appreciate it if you could re-evaluate our work based on this context. We are happy to discuss further.**  \n\n\n## However, failure analysis is limited—cases where PAE-generated tasks lead to incorrect generalizations or inefficient behaviors are not thoroughly discussed.\nWe actually **do have comprehensive analysis in appendix B with manual classifications of the trajectories from each model**. We have also provided qualitative examples of each failure mode in Figures 7-11. We will include them in a revised version of the paper. \n\n\n## How does PAE handle the situations where the proposed task is infeasible or ambiguous?\nAn important factor for PAE to choose to use vanilla REINFORCE without a baseline value function is that this loss function (line 239 in the manuscript) automatically takes care of the cases when the proposed task is infeasible or ambiguous. In these cases, the agent would not be able to finish the task and therefore can only get a reward 0. As shown in the loss function in line 239, if the trajectory reward is 0, it automatically zeros out the loss so it does not affect the training process, except a slight waste of sample efficiency during online trajectory collection.\n\n\n\n\n## Does the agent recognize when to reuse a learned skill versus learning from scratch? How does PAE facilitate long-term skill retention?\n\nA crucial difference in skill discovery considered in this setting of web agents versus skill discovery in traditional hierarchical deep reinforcement learning [1, 2] is that most tasks that current sota open-source VLM web agents can complete are rather atomic and short-horizon (~10 steps), such as “Find the Easy Vegetarian Spinach Lasagna recipe on Allrecipes and tell me what the latest review says” as shown in appendix Figure 7. Under such a setting, skills are essentially equivalent to the tasks that the agents are practicing on so the issues of identifying new skills and long-term skill retention are not major concerns at this stage. However, as open-source models get more capable and are able to handle more complicated tasks, we believe that there will be more interesting research problems related to those concerns and the framework of PAE will serve as a foundation to facilitate such research.\n\n[1] HIQL: Offline Goal-Conditioned RL with Latent States as Actions\n\n[2] Data-Efficient Hierarchical Reinforcement Learning\n\n\n\n## Would intermediate reward signals improve agent learning?\nThe choice of the reward is actually an important ablation choice that we have made to maximize the exploitation of asymmetric capabilities of SOTA VLMs as skill proposers/evaluators and as agents. As shown in Figure 3, we found the use of sparse outcome rewards only achieved the best performance while other choices such as autonomous step rewards and functional evaluators tend to result in lower performances. Through a more careful inspection, we found that in many cases it would be very hard to give an accurate step-level reward because there can be multiple paths to solving the tasks. That said, we also believe that as the tasks get harder and models get more capable, how to exploit autonomous intermediate reward signals would be an important future direction to be explored.", "claims_and_evidence": "The paper claims that:\n\n1. PAE enables autonomous skill discovery \n2. PAE improves generalization\n3. PAE’s improvement is not strictly dependent on stronger models for evaluation and task generation\n\nThese claims are supported by Table 1, 2, and 3.", "methods_and_evaluation_criteria": "The task proposer utilizes contextual information (such as website names or user demonstrations) to generate training tasks. The agent policy is trained using reinforcement learning, incorporating a reasoning step before execution. The evaluator assesses task success through a binary (0/1) reward signal based on final state screenshots. Evaluation is conducted on WebVoyager and WebArena, with success rates compared against SFT-trained models and proprietary VLMs.\n\nThe methodology is well-structured, but long-term evaluation, e.g., whether PAE-discovered skills remain useful over extended training periods is not explored.", "theoretical_claims": "The paper does not proposed theoretical claims.", "experimental_designs_or_analyses": "The experiments effectively test the proposed method by comparing different agent training strategies and task evaluation techniques. The following aspects strengthen the study:\n\n- Comparison with SFT baselines, which highlights PAE’s advantages.\n- Scaling experiments, showing consistent performance improvements across different model sizes.\n- Evaluation on unseen tasks and websites, demonstrating generalization.\n\nHowever, failure analysis is limited—cases where PAE-generated tasks lead to incorrect generalizations or inefficient behaviors are not thoroughly discussed.", "relation_to_broader_scientific_literature": "The paper contribute to self-supervised skill discovery with RL and web-based foundation model agents.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n\n- Addresses scalability issues in web-based agents by autonomously generating skill acquisition tasks.\n- Robust across different models, showing that PAE generalizes well beyond a specific agent architecture.\n- Strong empirical results, demonstrating zero-shot generalization to unseen websites.\n\nWeaknesses:\n\n- The paper does not discuss whether the learned skills persist over time or if they degrade when learning new ones. Understanding skill retention is crucial for real-world deployment.\n- While the agent receives reward feedback, the paper does not explicitly explain how it tracks what skills it has acquired. Without an explicit tracking mechanism, the agent might relearn already mastered skills instead of focusing on truly novel capabilities.\n- Sparse reward signal—the 0/1 evaluation approach might miss intermediate learning signals that could improve policy refinement.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. How does PAE handle situations where the proposed task is infeasible or ambiguous? Would the system benefit from a self-correction mechanism?\n2. Does the agent recognize when to reuse a learned skill versus learning from scratch?\n3. Does PAE facilitate long-term skill retention? Do discovered skills remain useful across multiple training phases, or do they degrade over time due to task shifts?\n4. Would intermediate reward signals (e.g., step-based evaluations) improve agent learning? The paper suggests outcome-based evaluation is more stable, but step-based signals could provide earlier correction if the agent starts diverging from the goal.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes PAE (Propose, Act, Evaluate), a self-improvement framework for training web agents using only open-source vision-language models (VLMs). The system leverages asymmetric model capabilities by using weaker models to propose tasks and evaluate outcomes, thereby improving stronger agents through reinforcement learning without human supervision or proprietary models. The framework is evaluated on WebArena and WebVoyager, two of the most challenging multi-turn web interaction benchmarks, and includes extensive ablations to analyze the effect of different design choices such as reward structure, model selection, and reasoning steps.

The paper is well-motivated and introduces a scalable, practical approach for self-improving web agents. Its main strength lies in the system-level insight that weaker models can guide the improvement of stronger ones, which is a novel and impactful perspective. The experimental analysis is thorough, covering design choices and ablations across challenging benchmarks. However, the work lacks extensive comparison with alternative RL algorithms and baseline models, and long-term skill retention remains unexplored. While some components mirror recent concurrent works, the authors convincingly clarify their contributions and novelty under ICML’s concurrent work guidelines.

In sum, I slightly lean toward accepting this paper, depending on if there is room in the program, and I would not mind if my recommendation was bumped down.
Decision: Accept (poster)