Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale
Authors: Tianyue Ou
Frank F. Xu
Aman Madaan
Jiarui Liu
Robert Lo
Abishek Sridhar
Sudipta Sengupta
Dan Roth
Graham Neubig
Shuyan Zhou
Venue: NeurIPS 2024 poster (2024)
Keywords: AI agents
sythetic data
web navigation
Primary area: natural_language_processing
TL;DR: we introduce a data synthesize approach for computer agents and achieve strong performance

Abstract: LLMs can now act as autonomous agents that interact with digital environments and complete specific objectives (e.g., arranging an online meeting). However, accuracy is still far from satisfactory, partly due to a lack of large-scale, direct demonstrations for digital tasks. Obtaining supervised data from humans is costly, and automatic data collection through exploration or reinforcement learning relies on complex environmental and content setup, resulting in datasets that lack comprehensive coverage of various scenarios. On the other hand, there is abundant knowledge that may indirectly assist task completion, such as online tutorials that were created for human consumption. In this work, we present Synatra, an approach that effectively transforms this indirect knowledge into direct supervision at scale. We define different types of indirect knowledge, and carefully study the available sources to obtain it, methods to encode the structure of direct demonstrations, and finally methods to transform indirect knowledge into direct demonstrations. We use 100k such synthetically-created demonstrations to finetune a 7B CodeLlama, and demonstrate that the resulting agent surpasses all comparably sized models on three web-based task benchmarks Mind2Web, MiniWoB++ and WebArena, as well as surpassing GPT-3.5 on WebArena and Mind2Web. In addition, while synthetic demonstrations prove to be only 3% the cost of human demonstrations (at $0.031 each), we show that the synthetic demonstrations can be more effective than an identical number of human demonstrations collected from limited domains.

Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale
Tianyue Ou*
Frank F. Xu
Aman Madaan .
Jiarui Liu
Robert Lo
Abishek Sridhar
Sudipta Sengupta* * Dan Roth
Graham Neubig
Shuyan Zhou
Carnegie Mellon University + Amazon AWS AI o xAI
ftianyueo, fangzhex, gneubig, shuyanzh)@cs. cmu. edu
Abstract
LLMs can now act as autonomous agents that interact with digital environments and complete specific objectives (e.g., arranging online meeting). However, accuracy is still far from satisfactory, partly due to a lack large-scale, direct demonstrations for digital tasks. Obtaining supervised data from humans is costly, and automatic data collection through exploration or reinforcement learning relies on complex environmental and content setup, resulting in datasets that lack comprehensive coverage various scenarios. On the other hand, there is abundant knowledge that may indirectly assist task completion, such as online tutorials that were created for human consumption. In this work, we present Synatra, an approach that effectively transforms this indirect knowledge into direct supervision scale. We define different types of indirect knowledge, and carefully study the available sources obtain it, methods to encode the structure of direct demonstrations, and finally methods to transform indirect knowledge into direct demonstrations. We use 100k such synthetically-creet -createe demonstrations to finetune a 7B CodeLlama, and demonstrate that the resulting agent surpasses all comparably sized models on three web-based task benchmarks Mind2Web, MiniWoB++ and WebArena, as well as surpassing GPT-3. 5 on WebArena and Mind2Web. In addition, while synthetic demonstrations prove to only 3% cost human demonstrations (at $0.031 each), we show that the synthetic demonstrations can be more effective than an identical number of human demonstrations collected from limited domains."
Introduction
AI agents that operate within digital environment (e.g., browser in computer) intelligently to accomplish complex tasks (e.g., "Create my July online shopping expense report.") have the potential improve the productivity across broad swath of tasks performed by humans every day [2. 33, 6, 58]. However, agents still lack the ability to complete tasks with high degree of reliability, partly due to a paucity of training data for such agentic tasks. Typically, supervised finetuning is a standard way to adapt large language models (LLMs) to tasks such as text generation or classification, large-scale demonstration collections for digital agents are not readily available.
For Al agents, demonstrations typically involve specifying sequence of actions and observations that results in successful task completion, shown the right of Figure 1. Existing works that automatically collect demonstrations (1) set up environments for an agent to interact with, (2) run baseline agent within this environment, and (3) employ scoring functions to remove low quality demonstrations [9] or perform relabeling [1, 26]. All three of these requirements limit applicability variety of practical applications. Setting up an environment that is representative of the actual
Data and code are publically available at https: .ithub.............g. 1o/ aoyyyaaa
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Cancol FayFal paymenn!
Cancel Amazon Prime memhership on Paypal
<IDOCTYPE htnls
[..]]
tosk intentt /
hhtnl long= 'en'> cheach [...]
Log in with your credentials.
Enter the keyword
[..]]
[...]
id=156
</heado
doody> [...]
Select payment
clocc("login")
[..]]
Synthetic agentt
</body> </rtnis
ype ("peerneen "johnéexample.. con")
trajectory
[...]
observation
o0o0 o0oo
M
generation (Synatral
LypeC"search bar", 'AAazoon Prine")
K &click( Amazon Inc. id-156)
Indirect knowledge
action history op.....-1
next action
Figure 1: Our approach aims to synthesize direct demonstrations (right of the arrow) that specify the immediate next actions based previous actions and current observations. The sources comprise only indirect knowledge (left of the arrow), such as tutorials designed for human consumption and randomly sampled observations that lack associated tasks and actions.
environments in which we would like agents act is difficult task, and existing environments for digital agents are generally limited in scope to a few websites or digital apps [58, 7, 45J. Even within these constrained settings, strong LLMs such as GPT-4 struggle on tackling tasks [58], making collecting successful demonstrations with LLMs inefficient. In addition, human annotation is costly and its scope can still be limited [17, 6]. For example, gathering demonstration for canceling a PayPal order requires an actual PayPal account with a legitimate subscription history.
In this work, propose Synatra, data generation approach that synthesizes high-quality tra- jectories for complex digital tasks at scale. This approach is based on the intuition that there is rich trove of existing knowledge that encodes indirect supervision about how perform digital tasks ($2.2). One example of a piece of indirect knowledge is a tutorial that details the sequential breakdown of complex web task, such as "how to cancel recurring payment on Paypal" for human readers (Figure upper left) [53, 57J. While this tutorial provides some procedural guidance, such as "Enter the keyword," it does not specify the executable actions or the tangible observations associated with them. The key idea of Synatra is that, given this indirect knowledge, we can leverage an LLM to re- re-purpose it into more usable form that directly demonstrates the exact action to take given concrete observation ($3). In doing so, we leverage LLMs' ability to paraphrase and code, as well as its general knowledge of how tasks are performed (Figure 1). The main benefit of this approach is that it can scale with the availability of indirect knowledge created for humans, rather than rely on direct human annotations or LLM trajectories.
We carefully study the sources of indirect knowledge, the design of demonstration formats, and the mechanisms for iterative refinement in order synthesize high-quality demonstrations using an LLM ($3, $4). We generate demonstrations of 100k tasks from 21 domains, and finetune 7b Codellama -wllllmmmminntuucec model with this synthetic data. The resulting agent, Synatra-Codellaat surpasses existing open-source models of similar size, excluding those finetuned with human annota- tions, on three web-based task benchmarks: Mind2Web. MiniWoB++,, and WebArena. Moreover, consistently outperforms models that are ten times larger and have been finetuned with interactive data 1)..1). Our findings also indicate that our model on-par or a more accurate option in browser copilot settings. in comparison to GPT-3. 5. Importantly, while each synthetic example incurs only approximately 3% of the cost of a data with good domain coverage can be more effective than identical quantity of limited-domain human demonstrations in real-world web-based tasks in WebArena ($6.2).
demonstration, we demonstrate that synthetic
Problem Formulation
2.1 Controlling Digital Agents through Natural Language
An agent interacts with a computer environment E (S, with state space S, action space A. observation space O and the environment dynamics T: x A + S. While this framework can be applied to different types of task, in this work, we consider a web browser as the unified entry point to access different web applications. We follow WebArena [58] to define the observation space as the contents on screen, represented as in text-hased accessibility tree format. We use the same universal action space in WebArena. This action space resembles the keyboard and mouse operations of

---

computer (e.g,, click, type), and is applicable to arbitrary web applications. Appendix A lists all valid actions. The environment dynamics (e.g. the effect of clicking button) and the states (e.g. the database status) are decided by the implementation of each web application.
Given the natural language intent i, at each time step St. The environment state is updated to 8t41 with new observation Oc+1- This process ends when the agent predicts the stop action. We follow existing works [6, 47, 58, 54] represent 9, with current observation and all previous actions in the form of (é, Q1, 1 0r). A benchmark (e.g. WebArena) supplies a scoring function r(,,s,) S,) that examines the final state and returns the desired goal state is satisfied, and 0 otherwise.
agent issues action at based
2.2 Definition Direct Demonstrations and Indirect Knowledge
We consider the expected action a; given 8; as form of direct demonstration, i.e. (st, at). This allows an agent to directly learn how to predict the next action under given state. On the other hand, indirect knowledge is broadly defined as resources that can benefit the task execution, but is not in the format of state and expected action tuple. We mainly focus on three types of indirect knowledge:
1. Procedural knowledge details the sequence of steps (a/, d22 ...g d',)) required complete specific task i. Unlike an action 0, in trajectories, the steps in procedural knowledge are ungrounded, they lack direct association with any particular observation and are tied specific action spaces. For instance, the tutorial in Figure instructs the user to "login with your credentials" without providing the concrete PayPal login page and the input fields to type.
2. Environment knowledge T that describe the effects of applying different actions in some hy- pothetical states. Example knowledge includes verbal descriptions such as "... after clicking the cancel button, you will see pop up window....
3. Ungrounded observations that are not associated with particular tasks or trajectories. In the context of web-based tasks, an observation could be any random web page with different content and status (e.g., a product page with query in the search field).
3 Scalable Demonstration Synthesis for Digital Agents
In this section, first introduce our design choices on the canonical formalization of trajectories, which account for the structural nature of procedures. Then, we delve into the sources for acquiring indirect knowledge, and the mechanisms for re-purposing this knowledge into direct supervision.
3.1 Trajectories Programs
website "<url>"
Existing works demonstrate that representing task-related procedures as programs beneficial for several reasons, This includes benefits from the structural nature of pro- grams compared free-form text [56, 23], and the flex- ibility of using tools [4, 10, 41]. Inspired by these obser- vations, we represent a Python function that interleaves natural language planning articulated in comments and actions as API calls, as shown on the right. The pink back- ground represents the prompt, while the blue background corresponds to the model's ' response format.
observation "<AXtree page>" objective = "[...]"
past actions
solve(): # sub-task 1: [...]
$ <action NL eap lapattioon
action [...]
sub-task [...]
[...]
The planning process includes both task-level planning, which breaks the task into multiple sub-tasks, and action- level planning, which explains the low-level goals of each executable action. The model's generation consists of chain-op tlain-of-thooe -fituuuhttthh (CoT, [43, 47]) reasoning that analyzes objective, previous actions, and current observations. Since CoT reasoning includes detailed information about the current step that may not relevant for future steps, we further design the model response to include an action summary. This summary serves description of the predicted action that is added
<action CoT reasoning> action action (arg-value) <action summary>

---

to the prompt for future action predictions. A concrete example can be found in Appendix B, We study the empirical effect of program formalization $6.3.
3.2 Synthesizing from Text Procedural Knowledge with Generative Environment
The Internet offers fairly extensive procedural knowledge that describes how to perform high-level tasks by breaking down the task into detailed lower-level steps, such as how-tos and tutorials,
Source We use wikiHow as our main source for these tutorials due its comprehensive coverage of diverse tasks as well as its consistent format. Each article consists of a high-level task description and step-by-step instructions. We performed filtering step and only kept the articles that involve navigation through the graphical user interface (GUI) of computer or a mobile phone. We prompted GPT-3.5 5-turbo with six examples mixing the target articles (e.g.. How to redeem an Amazon gift card online") and non non-target articles (e.g., How to make a pizza) to perform the classification of all wikiHow articles. The prompt shown Appendix C. As a result, we obtained 25k articles that can be used to perform data synthesis.
Synthesis Approach We want to bridge two gaps to re-purpose (ai, ..., a'²,) for task into (al,..., I,-1, or). When re-purposing a sequence of actions (a', "2,,,,,, ..,, dn) for task i into new sequence (a1, al 1, ot), many challenges arise. First, the action descriptions provided in tutorials are not constrained to specific action spaces. Instead, they are presented as free-form natural language (NL) expressions, which can lead to ambiguity. For instance, various verbs such as "enter," "input," " and others may all correspond to the same underlying action, type. Second, NL descriptions are often abstract, omitting concrete actions. For example, the process of "logging in" involves a series of actions, including typing in username and password, but these specific actions may not be explicitly mentioned. Finally, the steps outlined in tutorials are ungrounded, meaning they are not directly associated with observable states or outcomes. Tutorials typically employ generie descriptions to accommodate various instances of conceptually similar tasks. For example, as illustrated in Figure 1, the tutorial merely instructs to "enter the keyword" without addressing any specific scenario.
Based on these findings, we propose an iterative approach that first uses an LLM to rewrite an article into a hypothetical trajectory in the format shown in $3.1, then we leverage a generative model to synthesize the intermediate observation between two consecutive actions. First, in the rewriting step, we ask the assistant LM to perform: (1) propose hypothetical concrete scenario relevant to the task (2) perform basic parsing such as translating "enter the keyword [...]" into type ('search bar", "Amazon Prime"); (3) categorize actions into groups that reflect the sub-task structures outlined by coding blocks. These tasks mainly demand a LLMs's 's creativity, language processing ability, and event understanding respectively. An example of rewriting a how-to article into a trajectory in program format is showed in Appendix E. The prompt for the rewriting step is in Appendix D.
Because the previous procedures still only result in a sequence of ungrounded actions, we next leverage the assistant LM to generate the observations between randomly sampled consecutive actions. We use the consccutive actions of type ("search bar", "Amazon Prime"") and click ("AAmazon Inc", id=156) in Figure 1 as an example. There are mainly two requirements for generated observations. First, the observation must reflect the outcomes of past actions. In the example, this corresponds to a page with a user logged in, and a search input field filled with "Amazon Prime". Second, the observation encodes the necessary elements to perform the next action. In the example, this corresponds to a payment history list with a payment to Amazon. We prompt the assistant LM with the action sequence generate a HTML snippet that fulfills the above requirements. Since the next action requires the concrete element to interact with, we ask the model to insert a tag of dd="'ext- aatiinatttageeeeeeeeeeeee grounding. 5 This step mainly requires a model's coding capabilities, particularly in web development. However, we find that it is not necessary for the LLM to generate HTML with high fidelity and
emmentt in the corresponding HTML node indicate the
"Som existing works also study using program formalization for web-based tasks [11, 46], but focus on action-level interventions without incorporating task-level planning.
bhttps::/www wikihow. com/Main- -Pae http///wwwwwhhwww///o////.
4 'httpa:/ //www. www..wikioo.. wikihow. com/ hom//pply-aaa a-ift----ff-tteo Card-Carr--a Code to Amazon#Online 'The tag is removed through post prot-prn....ing.

---

complexity, which is a open research question [36]. The full prompt is in Appendix D and an example for this step in Figure 5.
Note that in addition to wikiHow, there are other resources that share similar a similar procedural flavor, such the captions of YouTTue how-to videos [25]. Our transformation mechanism is generally applicable to such resources, but we leave concrete examination of them as future work.
3.3 Synthesizing from Random Observations
While the procedure in the previous section results in real procedures, the LLM-based method generates simplified observations. To compensate for this, we also perform data synthesis with real observations, and use synthesis to generate simplified procedures. We show that these two sources can compensate each other by examining the generated data in $4 and comparing the actual web-based task performance in $6.3.
Source We utilize Clue [27] as our data source, which comprises HTML snapshots of more than 10 billion web pages. Our initial analysis indicates that random sampling approach would likely lead to homogeneous distribution dominated by less interactive pages, such as news articles. In contrast, more complex web-based tasks typically require interactions with various web elements to advance the tasks. To diversify the sampled web pages, employed temperature sampling approach to select pages based on their content categories. In general, web pages from higher frequency top-level domains in Clu are typically more interactive, such Amazon and Reddit, while domains with lower frequency are less interactive, such as news article. We use a temperature sampling with
T 0.6 so that the sample probability of choosing page in domain i, P p.ah /Ex P where ps is the original probability of choosing page in domain i, and k is all the available domains. In doing so, we up-sample more interactive sites while maintaining diversity on more rare sites. More details are listed in Appendix J.
Synthesis Approach We treat each sampled web page intermediate observation time step f, aiming to synthesize the specific task i, the previous actions 01,... .... and the subsequent action de consisting of action, corresponding target element in the observation, and natural language summary of the action. We first convert a web page into its corresponding accessibility tree the beginning of each node, and sample a segment present to the assistant L.M. We follow the WebArena convention of assigning a unique ID to each node in the tree to ease the challenge of referencing the nodes. To increase the diversity of the tasks, we first instruct the model brainstorm k task categories relevant to the web domain. Then the model randomly selects three of these categories and develops them into concrete scenarios with past actions leading up to the current observation and the next action to take. The prompt is in Appendix F and an example generation is in Appendix G.
3.4 Data Filtering
To ensure the quality of the training set, we apply two-part filtering pipeline. In the first part, we ensure that data samples are both complete and coherent. For a sample to pass this filter, it must include all required components in the correct format. These components include: (1) an action that falls within the defined action space, (2) valid and meaningful action target element in the corresponding web page, (3) NL texts that are well formed, e.g. without the use of ".."" in the texts as an abbreviation by the generative model, (4) overall comprehensive generation without placeholders from the prompt (e.g.. <brief step description>).
To further eliminate accurately formatted but unresponsive actions, we apply a second filtering step using next state prediction. Here, we use the LLM to predict the next state, o++1, based on the current state of and action a, in our synthesized data. If the model predicts that 0++1 = or, the action is deemed to have no impact, and we filter out the action accordingly.
Data Statistics
We assess the distribution of history lengths, task objectives, and observations the form of accessi- bility trees. The first statistic reflects general task complexity, longer trajectories typically indicate
5

---

0.6
Mind2Web
Mind2Web
Tutorials
0.4
Tutorials
0.4
Random Observation
Observation
0.0
0.0
-0.2-
-0.2
-0.4
-0.2
0.0
0.4
-0.4
0.2
Figure 2: Left: t-SNE task intent embedding. Right: t-SNE plots of accessibility tree embeddings
more complex tasks. The latter two measure the domain diversity of our synthetic data. We also present these statistics for the Mind2Web dataset as an example human collnnccllected trajectories.
As shown on the right, the majority of our synthetic data consists of trajectories with history length of fewer than eight steps, regardless of the source. Fewer trajectories have longer histories, with the longest ex- ceeding steps. Examples with longer histories represent more complex scenarios such as "Create a new YouTube ad cam- paign for the Summer Collection with a focus on the 25-34 age group interested in sports, fitness, fashion, and style". In addition, we analyze the distribution generated intents by projecting their high- dimensional vectors using the embedding model wel--mpnttbbaseeee embeddings with t-SNE [40]. As shown the left of Figure 2, intents synthesized from random observations exhibit broader coverage compared to those from tutorials. This may be because humans often prefer write tutorials for critical domains, while randomly sampled observations offer more objective reflection of the diverse internet. Although our data synthesis process is entirely independent of Mind2Web, the generated intents show substantial coverage Mind2Web tasks, which were created by human annotators from top-use websites. Finally, we apply the same embedding and visualization approach for accessibility trees and display the results on the right of Figure The generated observations from tutorials overlap significantly with real web pages, both from random observations and those in Mind2Web.
Distribution of History Lengths
Tutorials
Random Observation
Mind2Web
Proportion
0.3
0.2
0.1
0.0 12
9.10 13:14 29:2021222 History Length
[30]. We visualize these
Cost The average end-to-end cost per sample is $0.031, out of which $0.028 is used to generate the actual sample, and $0.003 is used by our prediction based filtering pipeline. These values represent the final average cost generate one valid sample, including the additional cost for samples that were generated but later filtered out, which we distributes across non-filin collfeeeeed samples.
Experimental Setup
Agent training We finetune CodeLl ama-7b [31] with 99, 920 web-navigation instruction exam- ples, sourced from WikiHow articles and ClueWeb web pages 6, Training details can be found in Appendix K.
We select three evaluation tasks in the domain of web navigation. (1) the Mind2Web test set [6]. It consists three categories, namely, cross-task, gross-website. and cross- domain- ranked by their degree of deviation from the training data. Since model is not trained
Evaluation tasks
'We select CodeLlama-7b after comparing Llama-2, Llama-3, Llama-3 3--Instruct CodeLlama. CodeLlama- Instruct, DeepSeek Coder, and Mistral by finetuning and evaluating on Mind2Web's train and test set.

---

Table 1: Performance of various models in three web-based task benchmarks. We measure step accuracy (%) for Mind2Web, and task success rate (%) for MiniWoB++ and WebArena. The numbers of FireAct-7b taken from [5]; AutoWebGLM-77 synthetic data in [19]. All other numbers are produced by our work.
represents the model trained with only
Model
Mind2Web
MiniWoB++
WebArena
Short Reference-based Execution-basee
Single step
Long Execution-based
API-based Models
GPT-3.5
12.79
39.57
6.16
GPT-4
29.09
53.04
14.41
Open-souree Instructed Models
Codeilema-innstruct-7
6.62
23.04
0.00
Llama3-chat-8b
11.50
31.74
3.32
Llama3-chat-70b
22.27
48.70
7.02
Open-source Interactive Data Finetumed Models
FireAct-7b [3]
0,25
[gentLM-7b[51]
2.99
15.65
0.86
CodeActAgent-7bb [41]
3.13
9.78
2.34
(SG [19]
2.50
AgentFlan-7b [5]
3.80
20.87
0.62
Lemur - chb [46]
14.28
21.30
2.95
AgentLM-70b [51]
10.61
36.52
3.07
Sypatra-Cooeillam--7b
15.85
38.20
6.28
on the Mind2Web training set, all categories are treated as held-out sets. Therefore, we report an overall step accuracy as the average across all examples. In addition, we simplify the two-stage setup used in Mind2Web, where the model first selects the top 50 candidate elements from the HTML before predicting the action on a selected element. Instead, we input in-viewport accessibility trees that include the ground- truth element. This approach more challenging because it introduces more irrelevant elements, but it streamlines the prediction process by climinating the need for an additional element selection model. (2) MiniWoB++ [21]. It is an interactive benchmark with simplified web pages and low-level tasks (e.g., "Enter BUL to the box"). On average, completing task in Mini WoB++ requires fewer steps than that of WebArena. We tested on a text-only subset of tasks used in [51, 41]. To get the final score, we take the average score of five runs. (3) WebArena. In WebArena, agents are required to navigate in real-world web applications to accomplish high-levels web-based tasks (e.g., "How much spent last month grocery"). Both MiniWoB++ and WebArena offer outcome-bate evaluations, using programmatic checkers to validate key features after execution and assess whether the task has been successfully completed.
Baselines We compare the performance of our model with 12 baseline models, including API-based models such as GPT-4, open-source instructed models like CodeLlama, and models finetuned with interactive data. For instance, Agent-LM [51] is finetuned supervised data for tasks such as web navigation and interactive coding. Since most baseline models are not optimized for code generation, we follow Zhou et al. [58] and prompt the models generate natural language responses instead of programs. The same prompt, providing general web navigation instructions without explanations, is used across three datasets. We show the prompt in Appendix H. Notably, adopt the original prompts AgentLM and CodeActAgent [41] when evaluating them on MiniWoB++ to be consistent with their original evaluations. We further remove the task specifcii in-context examples to ensure a fair comparison with other models in zero-shot settings.
specific
6 Results
6.1
Main Results
Table presents the performance of various models across three web-based task benchmarks. Overall, Laaa achieves the best performance among models of comparable size. Notably,
7

---

Table Error rates (%) on five fine- fine-grained metrics.
Error Type
GoT-4-turbo
L.ama3 Baanttt-cch---ee -aaa-hhtt- 70b
Llama3 Saama3-chat--a5 chat-8b
Codel.lama- Instruct 7b
Durs
Nonexistent element
0.56
0.37
2.40
94,70
0.70
Invalid actions
1.75
2.17
9.74
58.88
4.91
Click non-clickable
5.34
3.40
15.20
100.00
5.77
Type non-lypable
1.01
1.34
11.97
32.62
4.34
Repeated type
26.85
44.72
32.14
46.32
6.79
Synatra-CodeLlann significantly outperforms its base model CodeLl waw.aaaaaanntttt-777 While CodeLlama ama- ietraaainittutttt-ee fails to complete any tasks in WebArena, Synatra-CodeLL suc- cessfully executes 6.28% of them. Furthermore, Synatra-CodeLlamm elevates the performance of CodeLlama- instruct- from 6.62% to 15.85% in Mind2Web (a 139.42% relative improve- ment) and from 23. 33.04% to 38.20% in MiniWoB++. More encouragingly, Synatra- Mematra-CodeLl-e demonstrates superior performance on Mind2Web and WebArena compared to GPT-3.5. It also outperforms Lemur-chas nemur-chat-706 which is finetuned with interactive data and is ten times larger, across all three benchmarks. The results suggest that our data synthesis approach is effective in helping the model predict the next action (as in Mind2Web) and performing simple tasks with a few steps (as in +i) The synthesized data can also guide the model towards executing real-world complex tasks more accurately.
surpassed the performance of all open-source model finetuned with interactive data. Among these models, AgentLM, CodeActAgent and AgentFlan include demonstrations to perform web-based tasks in their instruction finetuning dataset. However, we find that these models may not serve as capable agents to perform in the finetuned models. For instance, AgentLM and CodeActAgent use Regex expression match interactive element on a web page and require carefully selected in-context examples to showcase which are the proper Regex expression for different examples. However, Regex expressions only work for simple web pages with few elements as in MiniWoB++, while it is prohibitive to do pattern matching in complex web pages as in Mind2Web and WebArena. As a result, when we experiment with the more generic action space which is suitable for all three benchmarks without in-context examples, we see these models have a significant performance degradation, On the other hand, aaatraa-o targets the generic web-based tasks and does not encode any dataset artifacts during training. Even though all three benchmarks are completely held-out during data generation, Synatra-Coder ama achieves consistent superior performance all benchmarks.
Synatra-Coden
tasks due to the special design choice encoded
6.2 Analysis
What does the model learn from the synthetic data? Our analysis suggests that the baseline acquires essential knowledge multiple levels from synthetic data. The error rates for different error types are shown in Table 2. At the low level, synthetic data helps the model emit valid actions. For ex- ample, while approximately 95% of actions predicted by the base model CodeLl mmm-innttucct77 involve interacting with non-existent elements on the web page, Synatra reduces this error to just 0.7%. This low eтror rate closer to that of larger models such as Llama3 chat-70b and GPT-4, which are better at following complex instructions with multiple requirements. Additionally, the synthetic data improves the model's ability to understand web pages more accurately. To assess this, we measure the ratio of invalid actions, including both invalid click and type, where the predicted action attempts to interact with an element that is not clickable (e.g., a piece of text) or not typable (e.g., button). We observe that models in the 7b-8b range have high error rates in interpreting basic web elements. For instance, the strong 8b model -llama-chaat-a mistakenly click non-clickable element over 15% of the time, while Synatra reduces this error to under 6% approaching the performance of GPT-4. Finally, at task completion level, Synatra shows a stronger ability to track progress accurately. Specifically, when filling out forms on web pages, Synatra-Code Coaaeeaaoaaaaaammmmaaaaaaaaa is 74% less likely than turbb to repeatedly input the same words into the same field, common error in models. This indicates that our synthetic data enables the model to more precisely recognize completed actions and identify the remaining steps needed to achieve the goal. More qualitative study can be found in Appendix I.
Can Synatra be as effective as human annotations? We compare models trained with 9k human demonstrations (human only) from Mind2Web [6] and 9k demonstrations generated by Synatra. The
8

---

results are shown in Figure 3. Simply training CodeLlama with the Mind2Web human annotations provides modest improvement of 3.96% on MiniWoB++, while failing entirely on WebArena. In contrast, Synatra led to a substantial im- provement of 17.81% on MiniWoB++ and 4.56% on WebArena. The limited performance of the human annotations can be partially at- tributed to their restricted task coverage: notably, Mind2Web lacks information seeking tasks that require string answer. Furthermore, the hu- man trajectories did not specify conditions for terminating task execution. Despite adding such trajectories from Synatra (human + synthesis), the model still under-performed.
Synatra
Human only CodeLlama
Human synthesis
40.87
Task Success Rate
36,09
27.00
2.09
0.00
0.00
We hypothesize the diversity of tasks plays role in this discrepancy, since many tasks cannot be covered without pre-set environment (e.g., return order) (Figure 2). These findings un- derscore the efficacy of Synatra, which also exempts from the complexities of developing recording tools and managing communication with annotators. However, it is important to recognize that the quality of human demonstrations is presumably higher, but they require meticulous design and control during the data collection process.
MiniWoB++
WebArena
Figure 3: the comparison between the models trained with trajectories generated by our approach and the data collected from human.
6.3 Ablations
In this section, we perform an ablation to validate the design choices of our data synthesis approach.
WebArena Success Rate
Model performance improves with more synthetic data To assess the impact of scaling our synthetic data, we trained three CodeLlama ama 7B models with 18k, 50k, and 100k sam- ples respectively, while keeping all other parameters con- stant. We then evaluated the models on WebArena. As shown on the right, the success rates on WebArena steadily increase the synthetic training data scales from 18k to 100k samples. This highlights the potential of scaling syn- thetic data with our approach.
6.0
5.0
4.5
18k
50k Count of Training Samples
100k
Representing trajectories as programs is beneficial To verify if the programs format is helpful, convert 30k trajectories to the NL format similar setting in WebArena [58] and compare its performance with model trained with the exact data, but our program format. The results are shown in Figure 4a. We the NI format. We hypothesize that program, in the form of API calls, is potentially more natural format to represent sequence of actions. Our observation also echo the observations of using program representation for non-programming tasks [23, 29, 41], while our experiments further contributes insights towards finetuning setups for interactive tasks.
that performance drops both Mini Mon+ and WebArena when using
Different sources of indirect knowledge complement each other Our indirect knowledge comes from two sources: tutorials and randomly sampled web pages. In the former source, the procedural knowledge from tutorials are written and verified by human. However, there is no guarantee on the authenticity of the generated observations of web pages. contrast, the observations from the latter sources are completely real, while there is no guarantee of the trajectory accuracy. We hypothesize that the two sources can compensate each other. To test this hypothesis, we trained three models: one using 9k synthetic data mixed from both sources, and two others each using 9k data exclusively from one the sources and the results are shown at Figure 4b. observe a noticeable performance degradation when models are trained with data from only one source. This indicates that utilizing multiple sources yields a more comprehensive dataset by integrating the precise procedural knowledge from tutorials with the realistic observations of web snapshot data.

---

Finetune Synatra RAG-Llama3.1
Program Format
Both sources
Tutorials only
Format
Random observation only
36.52
4.31
Task Success Rate (96
40.87
4.56
Task Success Rate (%)
6.28
Task Surcess Rate
36.25
3.33
26.09
4.20
4.06
4.19
1.48
36.09
WebArena (a) Comparison between differ-
MiniWoB+>
MiniWoB
WebArena
(b) Comparison between different sources indi- rect knowledge.
(c) Different ways to use knowledge.
ent trajectory formats.
Figure ablation on trajectory formats, sources of knowledge, and ways use knowledge.
Models have difficulty learning from indirect sources alone To access if turning knowledge into trajectories helpful, we tested a retrieval- -uggmette approach that directly uses the col- lection of tutorials the knowledge base. We first projected text tutorials to embeddings with all-mpnet-
[30]. Then retriever retrieves the most relevant three tutorials measured by cosine similarity. These tutorials were included as additional context in the prompt. We tested this approach with LLama3, 1 i-instruct- 8B. As shown in Figure 4c, feeding tutorials directly im- proves model performance advantage. This comparison indicates that models have difficulty making use indirect knowledge to solve agent tasks, and trajectories is preferable format for agent learning.
WebArena marginally and finetuning our data shows substantial
7 Related Work
Learning from Indirect Supervision Due the costly nature of human supervision, many digital agent learning works explore learning from existing yet indirect knowledge sources [8, 35, 12, 55], reinforcement learning optimization that learn from environment feedback [37, 21, 32, 39J. More recently, LLM self-improvement during inference time [34, 49, 28, 44, 14, 52] has been applied on creating more complex digital agents in the wild. Our work fills the gap of training with synthetic data generated from existing resources, i.e. indirect supervision, complex web navigation tasks.
Prompting Approaches for AI Agents Existing methods include performing reasoning about the current statues before proceeding to next actions [43, 47, 22], search and planning [48, 13], and serffieriiiittoon [18, 34, 24]. Our focus on instruction tuning data generation without human supervision hopefully enable synthetic data generation recipes for various kinds of instruction prompts for agent tasks.
Data Generation for Interactive Agents and Instruction Finetuning Existing works design ways of generating training data that adapt LLMs agent-specific tasks [11, 19], while our work aim to generate more realistic data without human intervention. Considering more broadly over all instruction finetuning: [59, 50, 38, 16] generate instructions of specific task given few examples. [42, 15] generates task-agnostic large-scale instruction tuning data without given examples. Li al. [20] adopts an iterative instruction data generation and model finetuning approach, While our work generates instruction tuning datasets from readily available, indirect, unstructured knowledge resources.
Conclusion
We propose a data synthesis approach Synatra. Empirically we showcase that finetuning with data generated from our approach can improve existing general-purpose LLMs to perform better on digital agent tasks. Since Synatra can synthesize trajectories given a single, static piece of indirect knowledge, we argue that when equipped with a capable LLM,, a regular tutorial designed for human consumption and random observation, can also be re-purposed trajectory. We show that even considering the relative high cost calling state-of-the-art L.MMs such as GPT-4, synthesizing more cost-effective than collecting human demonstrations of similar quantity for model training.
10

---

Acknowledgements
This research project has benefitted from the Microsoft Accelerate Foundation Models Research (AFMR) grant program, and a grant from Amazon Web Services. We would also like to thank Center of AI Safety (CAIS) and CMU FLAME center for compute support.
References
[1] M. Andrychowicz, F, Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[2] Branavan, H. Chen, L. Zettlemoyer, and Barzilay. Reinforcement learning for mapping instructions to actions. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 82-90, Suntec, Singapore, 2009. Association for Computational Linguistics. URL. htttps://acaatttoll/a wwwp:/gyyyrr//o9-1000000
[3] B. Chen, C. Shu, F. Shareghi, N. Collier, K. Narasimhan, and S. Yao. Fireact: Toward language agent fine-tuning. arXiv preprint 1rX0v:2310.05915, 2023.
[4] W. Chen, X. Ma, X. Wang, and W. W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588 125888 2022.
[5] Z. Chen, K. Liu, Q. Wang, W. Zhang, J. Liu, D. Lin, K. Chen, and F. Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models. arXiv preprint arXiv:2403. 128811 2024.
[6] X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su. Mind2web: Towards a generalist agent for the web, 2023.
[7] A. Drouin, M. Gasse, M. Caccia, I. H. Laradji, M. D. Verme, T. Marty, L. Boisvert, M. Thakkar, Q. Cappart, D. Vazquez, N. Chapados, and A. Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks?, 2024.
[8] L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang, H. Zhu, A. Tang, D.-.. Huang, Y. Zhu, and A. Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge, In Thih Conference on Neural Information P Processing Systems Datasets and Benchmarks Track, 2022. URL https:ttt////. teattts://oppeeee http///wpw..ee...eeeepee.. .twt:/orummidddrroo..oee..
[9] H. Furuta, O. Nachum, K.-H. Lee, Y. Matsuo, S. S. Gu, and I. Gur. Multimodal web navigation with instruction-finetuned foundation models. arXiv preprint arXiv:2305. 2023.
[10] L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. Pal: Program- aided language models. In International Conference on Machine Learning, pages 10764-10799. PMLR, 2023.
[11] Gur. H. Furuta, A. Huang, M. Safdari, Y. Matsuo, D. Eck, and Faust. real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint urXiv:2307.12856 2023.
[12] J. Han, L. Yang. D. Zhang, X. Chang, and X. Liang, Reinforcement cutting -agett learning for video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9080-9088 2018
[13] S. Hao, Y. Gu, Ma, J. J. Hong. 7. Wang, Z. Wang, and Hu. Reasoning with language model is planning with world model. arXiv preprint 2305 14992, 2023.
[14] H. He, W. Yao, K. Ma, W. Yu, Dai, H. Zhang, 7. Lan, and D. Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arX19:2401.13919, 2024.
11

---

[15] O. Honovich, T. Scialom, O. Levy, and T. Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09668) 2022.
[16] O.
Honovich, U. Shaham, S. R. Bowman, and O. Levy. Instruction induction: From few examples to natural language task descriptions. arXiv preprint ar0iv: 1075. 2022.
[17] P. C. Humphreys, D. Raposo, T. Pohlen, G. Thornton, R. Chhaparia, A. Muldal, J. Abramson, Georgiev, A. Santoro, and T. Lillicrap. A data-driven approach for learning to control computers. In International Conference Machine Learning, pages 9466 9182. PMLR, 2022.
[18) G. Kim, P. Baldi, and S. McAleer. Language models solve computer tasks. ArXiv preprint, abs/2303. 17911, 2023. URL httph::t/paaat //tp:://aaxvvooa brrg/bb//ii 2203./2333111 10000..170 17491.
[19] H. Lai, X. Liu, I. L. Iong, S. Yao, Y. Chen, P. Shen, H. Yu, H. Zhang, X. Zhang, Y. Dong, et al. Autowebglm: Bootstrap and reinforce large language model-based web navigating agent. arXiv preprint arXiv:2404.036448 2024.
[20] X. Li, P. Yu, C. Zhou, T. Schick, L. Zettlemoyer, O. Levy, J. Weston, and M. Lewis. Self- alignment with instruction backtranslation. arXiv preprint wrXiv:2308.062560 2023.
[21] E. Z.. Liu, K. Guu, P. Pasupat, Shi, and Liang. Reinforcement learning on web interfaces us- ing workflow-guider exploration. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Truck Proceedings. OpenReview.net, 2018. URL
[22] C.-f. A. Sridhar, H. Zhu, F. F. Xu, and S. Zhou. Hierarchical prompting assists large language model on web navigation. In The 61 Annual Meeting Of The Association For Computational Linguistics, 2023.
[23] A. Madaan, S. Zhou, U. Alon, Y. Yang, and G. Neubig. Language models of code few-shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1384-1480 1403, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. URL https / /alaattooooggyy org/2022. 2022. emnlp-man mainn.pmmi.... 90.
[24] A. Madaan, N. Tandon, P. Gupta,
S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
[25] A. Miech, D. Zhukov, J. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic. Howtol00m: Learning a text-video embedding by watching hundred million narrated video clips. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 20/9, pages https://doi. org/ 10. 1109/ ICCV. 2019. 2019.00272.........0222222222221
IEEE, 2019. doi: 10.1109/CCV22219..0022. URL
[26] S. Murty, C. Manning, P. Shaw, M. Joshi, and K. Lee. Bagel: Bootstrapping agents by guiding exploration with language. arXiv preprint arXiv:2403.08 2024.
[27] A. Overwijk, C. Xiong, X. Liu, C. VandenBerg, and J. Callan. Clueweb22: 10 billion web documents with visual and semantic information. arXiv preprint arXiv:2211.15844 2022.
[28] J. Pan, Y. Zhang, N. Tomlin, Y. Zhou, S. Levine, and A. Suhr. Autonomous evaluation and refinement of digital agents. arXiv preprint arXiv:2404.064774 2024.
[29] H. Puerto, M. Tutek, S. Aditya, X. Zhu, and I. Gurevych. Code prompting elicits conditional reasoning abilities in text+ code 11ms, arXiv preprint arXiv:2401.10065, 2024.
[30] N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese nert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https: /ttp:///axxvii//ii www.r/ii.oo//bb//////e 10084.
[31] B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, al. Code Ilama: Open foundation models for code, arXiv preprint arXiv:2308. 12950, 2023.
12

---

[32] P. Shaw, M. Joshi, J. Cohan, J. Berant, P. Pasupat, H. Hu, U. Khandelwal, K. Lee, and K. N. Toutanova. From pixels to ui actions: Learning follow instructions via graphical user interfaces. Advances in Neural Information Processing Systems, 36, 2024.
[33] T. Shi, A. Karpathy, L. Fan, Hernandez, and P. Liang. World of bits: An open-domain platform for web-based agents. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML. 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3135-3144 1344. PMLR, 2017. URL htattp://proceedingg. www.mml.............. www.:/pressssssiiieeeeee 1hhi17a.. html.
mlr.
[34] N. Shinn, B. Labash, and Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. ArXiv preprint, abs/2303. 1136,, 2023. URL https: org/ abs/ 2303. 11366.
[35] M. Shridhar, X. Yuan, M. Côté, Y. Bisk, A. Trischler, and M. J. Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net,
2021. URI. htwps: //tepss://// /ppprevieneeeeeeeeeeeee net//forum'dd=OIXXYYcGaa
[36] C. Si, Y. Zhang, Z. Yang, R. Liu, and D. Yang. Design2code: How far are we from automating front-end engineering? arXiv preprint arXiv:2403.03163 2024.
[37] D. Silver, A. Huang, C. J. Maddison, A. Guez, I.. Sifre, G. Van Den Driessche, Schrittwieser, I. Antonoglou, V. Panncershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-48) 2016.
[38] C. Singh, J. X. Morris, J. Aneja, A. Rush, and J. Gao. Explaining data patterns in natu- ral language with language models. In Y. Belinkov, S. Hao, J. Jumelet, N. Kim, A. Mc- Carthy, and H. Mohebbi, editors, Proceedings of the 6th BlackboxNLP Workshop: Analyz- ing and Interpreting Neural Networks for NLP, pages 31-55, Singapore, Dec. 2023. As- sociation for Computational Linguistics. doi: 10. 10.18653/v 122/533////22.... h20.6/////22bbbbbcboolllppl 1.3. URL /stps://aclantholloo
org/2023. blackboxnlp-1.3.
[39] Y. Song, D. Yin, X. Yue, J. Huang, S. Li, and B. Y. Lin. Trial and error: Exploration-baser trajectory optimization for Ilm agents. arXiv preprint arXiv:2403.025500 02.50...000,22222 2024.
[40] L. van der Maaten and G. E. Hinton. Visualizing data using t-sne. Journal of Machine Learn- ing Research, 2605, 2008. URL httpt://pppppp.... www.nssccccaaolaanccoommm wwwp:ooooo/Colulooo. 5855042.
[41] X. Wang, Chen, Yuan, Y. Zhang, Li, Peng, and H. Ji. Executable code actions elicit better Ilm agents, 2024.
[42] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint 22x5::22/2.10560 2022.
[43] Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q V. Le, Zhou, al. Chain-of- thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 24837, 2022.
[44] Z. Wu, C. Han, Z. Ding, Z. Weng, Z. Liu, S. Yao, T. Yu, and L. Kong. Os-copilot: Towards generalist computer agents with self-impprvemmnnt arXiv preprint arXiv:2402.07456, 2024.
[45] T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y. Liu, Y. Xu, S. Zhou, S. Savarese, C. Xiong, V. Zhong, and T. Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024.
[46] Y. Xu, H. Su, C. Xing, B. Mi, Q. Liu, W. Shi, B. Hui, F. Zhou, Y. Liu, T. Xie, et al. Lemur: Harmonizing natural language and code for language agents. arXiv preprini 130,06300, 2023.
13

---

[47] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. ArXiv preprint, abs/2210.03629, 2022. URL https: / //arxiv. org/ /a////b///// abs/2210. 2210022200.2222 03629.
[48] S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. ArXiv preprint, 0oss/2355.1000) 2023. URL https: /tttps://arxii org/ 2205/ 10601.
[49] S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
[50] S. Ye, D. Kim, J. Jang. J. Shin, and M. Seo. Guess the instruction! flipped learning makes language models stronger zero-shot learners. arXiv preprint arXiv:2210.02969 2022.
[51] A. Zeng. M. Liu, R. Lu, B. Wang, X. Liu, Y. Dong, and J. Tang. Agenttuning: Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310. 12823, 2023.
[52] C. Zhang, Z. Yang, J. Liu, Y. Han, X. Chen, Z. Huang, B. Fu, and G. Yu. Appagent: Multimodal agents smartphone users, 2023.
[53] L. Zhang, Q. Lyu, and C. Callison-Burch Reasoning about goals, steps, and temporal ordering with wikihow. arXiv preprint arXiv:2009.09 07690, 2020.
[54] B. Zheng, B. Gou, J. Kil, H. Sun, and Y. Su. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614, 2024.
[55] Zhou, Alon, F. F. Xu, Z. Wang, Z. Jiang, and Neubig. Docprompting: Generating code by retrieving the docs. ArXiv preprint, abs/2207.05998 2022. URL /tttps:///rrivv org/ abs/2207 05987.
[56] S. Zhou, P. Yin, and G. Neubig. Hierarchical control situated agents through natural language. In Proceedings the Workshop Structured and Unstructured Knowledge Integration (SUKI), pages 67-84, Seattle, USA, 2022. Association for Computational Linguistics. doi: 10.18653/ v1/2022.suki-1s.. 1.8. URL htt.thtss/acaaathlllllaaa
org/2022. suki-1.8.
[57] S. Zhou, I.. Zhang, Y. Yang, Q. Lyu, P. Yin, C. Callison-Bucce Burch, and G. Neubig. Show me more details: Discovering hierarchies of procedures from semi-structured web data. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 2998-3012, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022...e acl- howg:// URL 214.
org/ acl llngg.
[58) S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations (ICLR), Vienna, Austria, May 2024. URL https: arxiv. org/ htg/aba/2301 13854.
[59] Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba. Large language models human-level prompt engineers. arXiv preprint arXiv:2211.01910 2022.
14

---

Table of Contents in Appendix
Action Space
16
B Trajectory Representation
16
Prompt to Filter wikiHow Articles
16
D Prompt to Generate Demonstrations from Tutorials
17
Example Synthetic Demonstration from Tutorials from wikiHow Article
19
F Prompt to Generate Direct Demonstrations from Random Observations
20
Example Generated Trajectories from Random Observation
22
H Prompt Example
25
I Case Study
26
J Data Selection from Random Observations in ClueWeb
28
Training Settings
28
L Broader Impacts
29
M Limitations
29
15

---

A
Action Space
Table 3: Action Space of WebArena
Action Type
Description
noop
Do nothing
(lick(elen) hover (elem)
Click at an element
Hover on an element
type (elem, text)
Type to element
press (key_comb)
Press key comb
scroll (dir)
Scroll up and down
tab_focus (index)
focus on i-th tab Open a new tab
new_tab
tab_ close
Close current tab
go_back go_forward goto (URL)
Visit the last URL
Undo go back Go URL
B Trajectory Representation
website "<url>"
observation "<AXtree of the page>"
objective "Find and review the estimated value of your property the website."
past actions
def solve():
# sub-task 1: Look up your property on Zillow
# step Search for your address Zillow's homepage search to open the property page. sype(element.id="111
string="Main St")
# atep 2: From the property details page, navigate to the "Zestimate" section. scroll(down)
tub- Begin adjusting the estimated value # step 3: Click on 'Edit home facts' to adjust details that might affect the home's estimated value. id="9945")
C Prompt to Filter wikiHow Articles
Prompt to filter wikiHow articles
Given the title of an article, determine if is about performing task solely with computer or mobile phone's s graphical user interface, and without any physical world configurations.
input: How to Set Up Chromecast WiFi (Using an Android Phone or Tablet) output: Set Up Chromecast WiFi involves both a mobile application and physical interactions with the Chromecast device such plug in the device, the answer is "No"
input: How Change Your Desktop Wallpaper on Linux Mint (Using the Limux Mint Wallpapers) output: Linux Mint is a desktop operating system, and changing the desktop wallpaper is typically done through the system settings or desktop environment's configuration tools, which are desktop applications, so the answer "Yes"
16

---

input: How delete file using command line in Linux output: Command line interface (CLD) in Linux text-based interface not graphical user interface (GUT), so the answer is "No"
imput: How Reboot iPad (Frozen iPads) output: Rebooting an iPad usually involves physical actions like pressing and holding buttons on the iPad, so the answer "No"
input: How Connect the Kindle Fire the Internet (Connecting to an Existing Wi-Fi Network) output: Kindle is neither a computer nor mobile phone, so the answer "No"
imput: How Pair AirPods to an iPhone Your AirPods Won't Connect)
output: Pairing AirPods with an iPhone typically includes physical actions such as opening the AirPods case near the iPhone and possibly pressing button on the AirPods case, so the answer is "No"
input: ((Title of the article) output: (Model prediction]
D Prompt to Generate Demonstrations from Tutorials
Prompt to rewrite an article trajectory program format
# Task overview
You are given an article about performing a task in web browser. Your goal is to make this article as accessible as possible to user who is not familiar with the functionalities of the websites and the task at all.
# Guideline
Read the article carefully and follow the instructions below:
Assume you start with the home page of the web application, skip the initial 'goto' action.
Break down the article into sequence of steps.
every step, provide concrete example that reflects real execution. This example should clearly describe the element you are interacting with, the concrete value of an element you select, the precise content you type and other details. Never use broad descriptions. The example should be creative and realistic, avoid boilerplate text such email @ eeeexxleeecom Make that the example consistent across steps.
Following the concrete example, provide Python API corresponding the example. Group all API calls into multiple sub-scctions, each section corresponds to logical and actionable sub-task.
There are special scenarios and here are the ways deal with them: If the article describes multiple scenarios multiple ways approach the goal, you common one to describe. If there are repeated steps, make sure to unroll the steps and describe each them canonically. Always assume you perform this task using a web browser, if the original article uses a desktop app or mobile phone app, simply assume the corresponding web app exists. Hence, any steps regarding installation or login be skipped.
your judgement to choose the most
# APIs
The APIs are as follows: 'click(element, dese: str) - Click on an element.
"element desc' is the the displayed text the most representative attribute of the HTML element. "hover(element desc: str)' Hover
element.
'elick and type(element desc: str, content: str)' Click an input element and type 'content' into it. "key press(key comb: str)' Press a key combination. 'key comb' is the combination of keys you want to press on. The default OS is MacOS if there is no explicit specification in the article.
'goto(url: str) Navigate 'url"
go po__back)' Go back the previous page.
'go forward() Go forward to the next page.
17

---

"new "ww_tab('' Open new tab.
(lose_tab()) Close the current tab.
'switch_ Switch_tabbttab.o taa___aabttbbeeeeeeeee int) Switch to the tab with index 'tab index", starting from 0.
# Response format
Your response should follow the following format.
"python
sub-task <index>; <sub-task description>
# step <index>; <the real execution with concrete values for each argument> <API, do not skip the keys in the API calls>
# step <index>; <the real execution with concrete values for each argument>
<API, do not skip the keys API calls>
<repeat for all sub tasks>
task: <task command given smart assistant, only the necessary details expectation needed.>
...
# Article
([Article here]]
Prompt generate observation for consecutive actions
HTML Background Knowledge
Commonly used interactable elements HTML:
['a', 'button', 'input', 'textarea', 'select', 'option', 'label', 'form', 'details', 'summary', 'map', 'area', 'iframe", "embed", 'object', 'dialog'. 'menu", 'fieldset', 'legend', 'datalist', "output', 'progress' 'meter', "keygen'J
# Task Overview
You given:
browser-based task
seuqnece of past actions to perform the task and
The next action perform the task.
Your goal is to recover the HTML the dynamic of web application with the following requirements: The web page embodies same level of content richness as advanced web applications on the internet. That is, the web page should have around 80 elements and at least 20 interactable elements. The depth of the DOM tree should be around 7. The length is least 3000 tokens.
Analyze the past actions and determine which of these actions have visible functional impacts the web page you design. Reflect the effects these past actions your HTML code. This may involve updating text, adding new elements, or modifying the layout or styles to represent the state of the web page after these actions.
Design the interactable element that enables the next action. Make sure the choice of element type, attributes, and other essential characteristics correct. For example, text field is interactable. Once the element designed, assign the attribute
telaaacciion-aageeellmmmnt to this interactable element. Please focus on making the static HTML visually rich. Ignore CSS animations style and JavaScript functionality in HTML code. Provide the concrete reason perform the next action.
# Response format
"'html
<HTML that fullfils the requirements, make 'next-action-targer target-elemeenttt always included>
<Summarize progress by analyzing past actions. Provide brief reason performing the next action. Keep it short. Use imperative sentences.>
18

---

# Provided information task: [task description) )
past actions:
"* "python
([past actions)
next action:
ppython
[next action)
E
Example Synthetic Demonstration from Tutorials from a wikiHow Article
wikiHow article
How Use Google Chat iPhone (Enabling Google Chat in Gmail)
Open Gmail on your iPhone or iPad. This app icon looks like white and red envelope. If you don't have it, you download the Gmail app for free from the App Store.
Tap and tap Settings. The three-line menu icon is the top left corner of your sereen, and the option is generally at the bottom of the menu next to gear icon. If you have multiple Gmail accounts, select the one want to use Google Chats with. You can repeat these steps if you want to use Google Chats with multiple Gmail accounts.
Tap to check the box next to "Chat. A checked box indicates that Google Chat is active your Gmail account and you'll 'll see the Chat and Spaces tabs near the bottom of your screen. If Google Chat is enabled, you can tap these tabs to move between conversations in Chat and your emails in Gmail. If this is the Google Chats app see your conversations in Chat.
turned off, you'll 'Il need
The output from the rewriting step
# sub-task 1: Navigate Settings activate Google Chat in another account step 1: Click menu icon to reveal options
click(element="Met Icon")
# step 2: Go into Settings by clicking it clickelemeett="Sttiine "Setinngss))
# step 3: Select another Gmail account, this time for jane. jame.doo@gmail.com gmail.com click(element="jance @ gmail.com"
# sub-task 2: Activate Google Chat in the chosen account # step 4: Click enable Google Chat by checking the box dickfelement="Cheed box next to Chat")
# step 5: Ensure the Chat feature is activated
stopfanswer="Goodt Chat activated for jane gmail.com")
# task: Enable Google Chat web version of Gmail for jane.doe @ @gmail.com
19

---

Console
Elements
Recorder
Sources
Network
Action space Seairch mail
<1DOCTIPE
<htnl lang="en's
Jane Doe
ccheads io </head>
<body>
Inbox
container
Starred
ts-"grail-contaieer
<I- Top savigat in
Snoazed
* >div id-"nav-bar's </divs
Important
<how Side seru with setting options mix
Sent
sd="sleemennu </aside>
section
Drafts
<div sd=*msin-ontent's
Categories
<hisSet-ings//tt
Settings
atlis
ide"settings-tabe
<section ide"ssttings-co
Settings
<h2-Chat and Nectv/h>>
<!- Settings for Chat and Neet -
* <div ide"chat-settinng"
*fieldieet
<legens-Setect Account Google Chats/logend> kuls
* <+1> </1i>
Inbox
Felio
Chall Meel
target clesent:
account for chat
Advanced
v=label=
<inout rapeo""adio" "rre"""caaaccountt value="] ane. dod.doorlllcc.. aeP"oecttactit-taarg target-elarr
Chat and Meet
jame.donggwil..ce
Google
</1i>
aiputthext-aciin--trrge
<li> in </1l>>
jane doc gmail.com
ddseto
example uecr:prrir@@mml..commmm
//section
</esu>
</div>
</bodyo
The rendered generated HTML between step 2 and step 3, where step is goto("Googl) Chat') and step 3 is click click(" ('jjane. doe@gmail. "oomgmmii.ccom'.. The concrete element to interact with is tagged with ad="next-action www.iaattttiinta--aee-eee-
Figure 5:
F
Prompt to Generate Direct Demonstrations from Random Observations
Prompt to Generate Direct Demonstrations from Random Observations
## Task overview
Given the accessibility tree of a web page, your goal is to propose creative and diverse browser-basse based tasks that involves interacting with this page, along with the previous actions that lead to the current state and the next action needed to be taken to accomplish the task.
Here are the allowed actions that you can take to interact with the web page: "elickeeemment str, element id: int=0)' Click an element. 'element' is the displayed text or the most representative attribute of the HTMI. element. 'element_id id' is the index of the element the beginning of the node.
alickkand__n aand_typ(eelemett str, content: str, element_ik id: int=0y- Click and type 'content' into an 'element'. "key press(key comb: str)* Press a key combination. 'key comb' is the combination of keys you want to press on. The default OS is MacOS if there is no explicit specification in the article.
"goto(url: str)* Navigate
'url'
"go back()* Go back the previous page. go_forward()' Go forward to the next page.
"new_tab(" Open new tab.
'close_table Close the current tab.
'switch tab(tah index: int)* Switch the tab with index 'tab index', starting from "scroll(upldown)" Scroll the page up or down.
"stop(answer: str=") - The task completed. If the task is to seek information, include the answer as string. Otherwise, leave it empty.
20

---

## Guidelines
You will follow the guidelines below to perform the task:
1. Examine the web page understand the the domain of the web page.
Brainstorm & task categories that could be performed the wehsite. Be creative.
For each task category, propose a concrete task that has this weh page as one of its steps. You want concrete task to be unambiguous and clear so that no further clarification is needed to perform the task.
4. Given concrete task, you ask to come up with the past actions that leads to the current page, well the next action.
Requirement past actions: You should write down each past action the details. You want group all actions into multiple sub-sections, each section corresponds to a logical and actionable sub-task. The next action could start with new sub-task. You can omit the 'elemement idd if they are not in the current page. There should only be one action at each step. DO NOT give goto() new tab() as first step.
* Requirement for next action: Provide the reasoning behind your past actions and the progress in completing the task. Also, describe your understanding of the current page and the concrete reason to execute the next action. If the action takes an element as the argument, is important that you understand the role and the attributes of that element so that the action can be appropriately applied. Make sure to always include the 'element id' in your next action if there is any. Any 'element id' must from the given Accessibility Tree.
## Format of the response
You are asked to provide the action sequence for task #1 with roughly 7 past actions; task #2 with roughly 0 past actions; task #4 with roughly 6 past actions; task #5 with roughly 4 past actions; task #6 with roughly 10 past actions. Your answer should follow the following format:
<Analysis and understanding about the domain and the concrete content of the web page> <The list of 8 creative task categories> <The concrete tasks for task category #1 #2 #4 #5 #6. Remember, concrete task needs include concrete details so that further clarification required when performing Use imperative sentences.>>
"*.
"python
# task: <repeat concrete task #1>
# past actions (history)
# sub-task I: <sub-task description>
# step I: <step deseription>
<action>
step 2: <step description>
<action>>
# sub-task 2: <sub-task description>
# step 3: <step description>
<action>
# step 4: <step description>
<action>
# sub-task 3: <sub-task description>
step 5: <step description>
<action>
# step 6: <step description>
<action>
# sub-tase 4: <sub-task deseription>
step 7: <step description>
<action>>
#
next action
# step <index>: <summarize the progress so far and analyze the current state of the web page. Provide the concrete reason to perform the next action> <action>
# step summary: <brief step description>>
...
"python
# task: <repeat concrete task #2>
21

---

# past actions (history)
# sub-task <index>: <sub-task description>
# next action
# step <index>: <summarize the progress so far and analyze the current state of the web page. Provide the concrete reason to perform the next action>
<action>
# step summary: <brief step description>
...
## The Accessibility Tree
[4812] link 'Shopbop Designer Fashion Brands'
[4815] LineBreak
[4819] LineBreak
[1742] LayourTableCelll
[1743] LayouTTableCell "Amazon Warchouse Great Deals on Quality Used Products'
[4822] link 'Amazon Warehouse Great Deals on Quality Used Products'
[4825] LineBreak
[4829] LineBreak
G Example Generated Trajectories from Random Observation
Generated trajectories from random observation
Model Brainstorming
The domain appears be that of major online retail website, likely Amazon, considering the references to "Prime." "Best Sellers," and "Amazon Basics," as well as services like "Magazine Subscriptions" and "Prime Video." The structure of the page includes variety of product categories, search texthox, and links to other services such account sign-in and shopping cart. This suggests that this page shop for wide range of items, manage their account, engage with support services.
Creative Task Categories: Below eight creative task categories that could be performed on this type of website:
Product Searching: Looking up products in specific category or using a search query.
Account Management: Signing into account, creating account, managing existing account details.
Navigational Inquiry: Using the site's navigation to find different departments or special features.
Customer Support: Accessing customer service features or disability support services.
Deal Hunting: Searching for best sellers, deals of the day, exclusive prime deals.
Media Consumption: Navigating to and consuming digital media such as cBooks, Prime Vidco, or music.
Educational Browsing: Exploring educational materials or subscribing to learning re- sources.
Wishlist Management: Adding items wishlist gift registry for future purchase.
Concrete Tasks The concrete tasks for task categorics #1, #2, #6, #7, #8 are as follows:
Concrete Task #1: Product Searching
Find and search for popular fitness trackers within the "Sports & Outdoors" category.
22

---

task: Find and search for popular fitness trackers within "Sports & Outdoors" category.
past actions (history)
sub-task Open the site and navigate to the product categories. step 1: The user opens the main page of the retail website. go forward ()
the SSports & Outdoors" category. step The user accesses the menu view the category list. click (clement= 'Open Menu')
sub-task 2: Navigate
step 3: The user browses through the categories. scroll (down)
step 4: The selects the "Sports & Outdoors" category, click (element= Spportst='Sports Sports Outdoors'. element_id=7964)
next action
step 5: The
now needs search for fitness trackers within the selected category.
click and type ( element= 'Sercchh' Search', content= fitness trackers', @eement_id=76333
step summary: Initiate search for fitness trackers in the Sports & Outdoors category.
Concrete Task #2: Account Management
Sign into your personal account to check on active subscriptions.
task: Sign into your personal account to check active subscriptions.
past actions (history)
sub-task Sign into the account.
next action
# step 1: The user is already a page where they can initiate the sign-in process.
click_ande cllck_and_yppeeeleeent 'Sarchh' content= myusername exxemaalmmeeeameeccmcmmmmmmm com't clement_id=7657
step summary: Try to sign in by using the search bar to enter the username.
Concrete Task #6: Media Consumption
Navigate to Prime Video to browse the latest movie releases.
23

---

task: Navigate to Prime Video to browse the latest movie releases.
past actions (history) sub-task I: Access the main page.
step 1: The user starts by reaching the home page, where they can find various services.
go ())
step The user considers different media options available on the site.
(eroll(pp)
sub-task 2: Go to Prime Video.
step 3: The clicks the "Prime Video" " link. click (element= Prime Video', element_id=7961
next action
step 4: To search for the latest movie releases within Prime Video, the user needs to use the search function. click and eypeeelement "element==Searrch. Search', content='latedt movie releases', element_id id=7657)
step summary: Search for the latest movie releases Prime Vidco.
Concrete Task #7: Educational Browsing
Subscribe magazine technology within the "Magazine Subscriptions" category.
task: Subseribe to magazine about technology within the "Magazine Subscriptions" category.
past actions (history)
# sub-task 1: Enter the site and locate the magazine section step 1: The products.
starts the home page where they browse
go forward
step 2: The user opens the categories menu to look for subscriptions.
click (elemmeettt Open Menu')
OOen
step 3: The user scrolls through the menu to find the magazine subscriptions.
scroll (down)
sub-task Select the "Magazine Subscriptions" category. step The user selects the "Magazine Subscriptions option from the categories.
click ( element= Magazine Subseriptions'. step 5: The user is now presented with different types of magazines but wants find technology-related ones. scroll (down)
mimnett_id79553
action
step With the magazine subseriptions displayed, it is time search for technology magazines specifically. click click_and_it tas_aann_ttteeeeenneee (element==Seeatiee Search', content= "technotcc magazine', 3lement_id=76577
step summary: Search for technology magazines in the Magazine Subscriptions section.
24

---

Concrete Task #8: Wishlist Management
Add popular sci-fi novel your wishlist for future purchasing.
task: Add popular sci-fi novel to your wishlist for future purchasing.
past actions (history)
sub-task I: Access the website 's book section. step 1: The user has started the homepage and is looking for books.
click (element= Books', element id=ent))d==007)
next action
step The user wants to find and add sci-fi novel their wishlist.
e click click_ande taock__nn_ttppeeeleee___ (elementt "Saarchh Search', content= content='popular popular sci fi novels ', element_ momeent_dd7657)
step summary: Begin the search for popular sci-fi novels to add to the wishlist.
H
Prompt Example
Example prompt used in baseline
System
are an autonomous intelligent agent tasked with navigating web browser.
You will be given web-based tasks.
These tasks will be accomplished through the use of specific actions you can issue.
Here's the information you'll 'Il have:
user's objective: This is task you're trying complete.
current web page's accessibility tree: This is simplified representation wehpage, providing key information.
The current web page's ss website: This is the page you're currently navigating. The open tabs: These the tabs you have open.
previous action: These are the actions you have performed. may helpful track your progress.
The actions can perform fall into several categories:
Page Operation Actions:
click [id]: This action clicks on element with a specific id the webpage.
type [id] [content] Spress_ented afte-=0r O 1]: Use this to type the content into the field
with id. By default, the "Enter" key is pressed after typing unless press_enter enter after is set to hover [id]: Hover over element with id.
To be successful, it is very important follow the following rules: You should only issue an action that valid given the current observation.
2. You should only issue one action at a time.
3. You should follow the examples to reason step by step and then issue the next action.
4. Generate the action in the correct format. Start with "In summary, the next action I will perform
is" phrase, followed by action inside .... For example, "In summary, the action will perform "elick [1234]nM*
25

---

User
OBSERVATION:
[1744] link 'HP CB782A#ABA 640 Inkjet Fax Machine (Renewed)
[1749] StaticTexxt '$279.49'
[1757] button Add to Cart"
[1760] button Add to Wish List"
[1761] button 'Add to Compare"
WEBSITE: onestopmarket
OBIECTIVE: Add HP Inkjet Fax Machine to cart.
PREVIOUS ACTION: None
Assistant
Let's think step-by-step. This page lists the information of HP Inkjet Fax Machine, which the product identified in the objective.
To add it to the cart, will click the "Add to Cart" button. In summary, the next action I will perform is click [1757].
User
OBSERVATION:
Tab 1 (current):
Curry Brand Shoes & Gear Under Armour
[4976] RootWebArea 'Curry Brand Shoes & Gear Under Armour' focused: True
[5059] link 'Skip to main content"
[5060] StaticText 'Skip to main content'
[5400] link 'FREE U.S. Shipping Orders S99+ & FREE Returns'
[5066] link 'Need Help?"
[5067] button 'US' hasPopup: menu expanded: False controls: country-dropdon
[5422] button 'Welcome James' hasPopup: menu
[5073] navigation 'Main Navigation'
[5660] menuitem 'Men' hasPopup: menu expanded: False controls: men
[5668] menuitem 'Curry' expanded: False
[5672] textbox 'Search by Keyword or Item hasPopup: listbox required: False controls:
search-results
[5099] button 'add to shopping bag"
[5030] main
WERSITE: underarmour
OBJECTIVE: Search the cheapest Curry brand unisex athletic shoes number 5.5, add to cart and checkout.
PREVIOUS ACTION: step 1: Click menuitem Curry on the main navigation bar, # step 2: click on link Shoes
I Case Study
We conducted detailed examination of instances where Cypatra-CodeLLa successfully com- pletes tasks that GPT-4-turbo fails to accomplish on WebArena. Two key patterns emerged from these cases, which we outline here. Our analysis focuses on hard-level tasks requiring multiple steps to complete.
identifies details more effectively In several scenarios, Synatra-Code Caraaa-oooealaamm outperforms GPT-4-turbo by detecting and utilizing detailed infor- mation such as hidden links and buttons on the page, whereas GPT-4-turbe focuses only on the most prominent components. In the scenario illustrated in Figure 6, both agents are tasked with displaying issues labeled as "bug." GPT-4-turt immediately attempts use the search box, but lacks the knowledge of how to search effectively in this case and guesses the keyword, which does not yield results. In contrast, Synatra-Codel.. the page and accurately identifies a link that lists all issues labeled as "bug."
Synatra-CodeLla
pays closer attention to content displayed on
26

---

Tab 0 (cureent): Issues
Dashboard
GitLab
GPT-4-turbo
Synatra-CodeLlaam
Dushboard
GitLab' focused: True
Let's think slep-by-slep. This page has textbox with ID [3669] to filter results. To get all the opened issues that report bugs, for 'type bug'. The textbox can filter issues bused on labels, GitLab, the label for bugs "type: bug'. The pressing the 'Enter' key necessary after typing to submit the search. In summary, the next action will perform type [3669] [rype: bug] [1]
slep Since alreudy the Issues section, need to filter the issues by the 'Bug' label list only relevant issues.
[2374) link 'Skip to content'
[2496] link 'Dashboard'
[2548] button Pospp: menu
search
expanded: False
click(element id-"2682") slep summary: Filter issues by 'Bug' label.
[3669] texibox 'Search GitLab'
required: False
[3576] generic 'Use the shortcut
key/ to start seurch!
(a lot of links and texts here)
'Administrator'
[2680] Static 'xa0
[2682] link 'type: bug
[3436] link 'Assigged to
Byle Blaze!
Figure 6: example task where Synatra-CodeLl ama is successful by paying more attention to the displayed web page while GPT-4-turbe is not.
Similarly, in the scenario shown in Figure 7, both agents are tasked with "finding the number of commits Killian made on 3/5/2023." At this stage, both agents have already navigated to the page displaying Killian's commits, and they can now provide the answer by simply counting. However, turbb fails to register the displayed information: in its reasoning, it attempts to "view all the commits," even though they are already visible. In contrast, Synatra-Codele ama pays closer attention to the information on the page and accurately identifies the relevant link showing Killian's s commits.
Tab 0 (current): Commits main The AllY Project/ allyproject.com GitLab
GPT-4-turbo
Cynatra-CodeLLane
Let's think step-by-step. need to look for the commits made by Kilian on 3/5/2023. First, I need to find way to view all the commits. The search box has an id of [2892], which can use to find the commits. can type "kilian commits on 3/5/2023" into this box and press enter. In summary, the next action will perform is ""type [2892] [kilian commits 3/5/2023] [1]"
step 3: Identify the commit made on
[2892] textbox 'Scarch GitLab'
3/5/2023 by Kilian Valkhof and count the number of commits.
roquired: False
[5720] generie 'Use the shortcut
click(element id-"5964") step summary: Click on the commit message to see the details of the commit.
key/ scarch'
(a lot of links and lexts here)
[4758) StaticText
commit'
[5963] img "Kilian Valkhof's avatar"
[5964] link 'Fix typos in
the code examples (H1524)
[5968] link Kilian
Valkhot
Figure 7: An example task where Manatra--odeLLln ama successful paying more attention to the displayed web page while GPT-4-turbo is not.
Cynatra-CodeLLee demonstrates clearer logical flow between steps In the example shown in Figure 8, both agents are filling out a form and have correctly entered the start and end dates. However, GPT-4-turbo incorrectly predicts that the next step is to re-enter the starting date, while Cernatra-Codellann
accurately interprets the status of the web page ("with the date range set to include the entirety of QI 2023"), plans accordingly ("we are ready to generate the refund report"), and executes the correct action. This example illustrates that data generated by Synatra enhances
27

---

the model's capacity to accurately interpret and respond to context, leading to more efficient and reliable task execution in dynamic settings.
[1504] button 'Show Report'
GPT-4-turbo
Synatra-CodeL
[2278] StaticTex: Date Used'
Let's think step-by-step. The task is to generate refund report for QI, and the end date is 3/3 1/2023,
il slep 5: With the date range include the entirety Q1 2023, ready generate the refund report. The action click the 'Show Report' button display the data.
[2279] combobox 'Date Used
hosPopup: menu expunded: False
which has been correclly entered in the "To" textbox, The "From" textbox with [1281] should contain the date which marks the beginning of which 01/01/2023, will input this date into the "From" textbox. After filling in both the "From" dates, will need to click the 'Show Report' button with [1504] generate the report.
[2283] combobux 'Period' hpsp:
menu expanded: False
[2324] Static Text 'From'
[1281] textbox 'From roquired: True
click(element, id-"1504")
[2346] Static 01/01/2023'
# step summary: Generate the refund report Q1 2023.
[2286] button 'undefined 'uc627''
[2333] StaticText 'Too
"To"
[1284] textbox 'To focused: True
required: True
[2497] Text 03/31/2023"
[2289] button 'undefined 'uc627'
summary, the next action perform is "type [1281]
[2292] StaticText 'Order Status'
[2293] combobox 'Order Status'
[01/01/2023] [1]
hasPopup: menu expanded: False
Figure 8: An example task where Synatra-CodeLlama is successful while GPT-4-turbo is not.
J Data Selection from Random Observations in Clue Web
inspect a random sample of Web on its domain distribution. As shown in Figure 9, the majority of domains appear only once, making up 69.1% of all the web pages.
50-100
25.50
12-25
6-12
4
>100
3
Dategories
64.2%
2
7.9%
3.7%
4 2.2%
1.3%
6-12 3.6%
12-25 -4.5%
25.50 2.9%
50-100 3.5%
>100 6.1%
Figure 9: Frequencies of domains and proportion of each frequency in ClueWet
K
Training Settings
The CodeLlama checkpoints are fine-tuned with 100 GPUs with deepspeed acceleration frame- work. We set the context length 4096 tokens. To train on 100k dataset, we train with 6 A100
28

---

80G GPUs for about 40 hours. We use batch size of 48, and learning rate of 4e-5. We use cosine annealing and a warm-up ratio of 0.03.
L Broader Impacts
The broader impacts of this work extend across several domains. Firstly, the approach has the potential to democratize the development of digital agents by making the training process more affordable and accessible. Organizations with limited resourees utilize existing public data to train competent agents without the need for expensive data collection efforts. This could lead to more widespread adoption and innovation in Al applications, particularly in regions sectors that previously could not afford such technology.
Secondly, by enabling digital agents to perform more complex tasks effectively, this work can significantly enhance productivity and efficiency various industries. For example, customer service, online troubleshooting. and data management tasks could be accelerated, allowing human workers to focus on more creative or complex problem-solye solving tasks.
Furthermore, the technology has implications for accessibility, as it could help develop more intuitive and user-friendly interfaces for people with disabilities or those who are not tech-savvy. Agents trained with diverse range of demonstrations can offer more personalized and context-aware assistance, improving user experience across digital platforms.
Lastly, the ethical and societal implications of this technology also constitute a critical area impact. While the technology can lead significant efficiencies and capabilities, it also raises questions about the potential for job displacement, and the need for robust guidelines to ensure that the deployment of such agents aligns with ethical standards. These broader impacts underscore the importance of interdisciplinary approaches to the development and governance of AI technologies, ensuring they contribute positively to society.
M Limitations
One major limitation to the work is the potential variability in the quality and relevance of the indirect knowledge sources. We attempted to mitigate this through use high-quality sources such as WikiHow and broad sources such Clue Web with intelligent sampling strategies, but still the danger of unrepresentative data remains.
Another concern is the generalizability of the synthesized demonstrations. While the approach allows for the generation of large volume of training data, the synthetic nature of these demonstrations may not fully capture the complexity and nuances of real human interactions with digital environments. As result, agents trained on this data may still struggle with unexpected or less typical scenarios not covered in the training data.
Furthermore, there is the risk of overfitting to the specific formats and tasks represented in the indirect knowledge sources, If the diversity of these sources is limited, the agents may not develop the flexibility needed to handle a broad range of tasks across different platforms environments.
Lastly, the reliance on large language models and complex synthesis processes might introduce significant computational costs and environmental impacts. The energy consumption and carbon footprint associated with training such large models are concerns that need to be addressed to ensure sustainable development in AI technologies. These limitations highlight the need for ongoing research, improved data curation methods, and the development of more robust models that can better generalize from synthetic training environments real-world applications.
29

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We empirically show the claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations in the analysis of empirical results, as well as in Appendix M.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
30

---

Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide hyperparameters, experimental settings, and data generation prompts for LLMs throughout the paper.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [Yes]
Justification: All the code and data is submitted through Supplementary Material. We will publicly release all the data and code on GitHub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: guwgg//des///// /Cto//ooodisssseeeeeee www.iisssospoiiiioiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We detail experimental settings in our experiment section.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to costly nature of LLMs (expensive pricing for competitive API-based models and GPU requirements for open-weight models), we do not perform multiple runs for the same experiment setting.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
32

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We detail compute resource usage in Appendix K.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //te:r//s.....///eee
Answer: [Yes]
Justification:
Our research conform with NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix L.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We existing models (CodeLlama,, OpenAI GPT) and datasets Web22, WikiHow) that have been either safeguarded or processed with moderation when released. We consider the issue not applicable to work as we all work with existing safeguarded datasets and models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly linked and cited the base model and source data corpus used in our experiments. They all have permissive licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswi thcode. paperswithcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
34

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: There are generated synthetic data as part of our proposed data synthesis method, and finetuned models using these data.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects involved.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
35


Reviews:
[{"review_id": "oFe1MVNQpe", "reviewer": "Reviewer_5MC7", "summary": "This paper studies the problem of dataset generation for the digital task in the context of LLM. The motivation is that the current data collection process often requires human annotation, which is very costly and not scalable. As a result, the paper proposes using large language models (LLMs) to generate a synthetic dataset based on indirect knowledge. The indirect knowledge often refers to tutorials designed for human consumption and observation trajectories that lack associated tasks and actions. In contrast, \"direction\" knowledge consists of complete states and actions. To generate synthetic data based on indirect knowledge, the paper proposes several methods: (1) writing a prompt to let LLMs imagine their missing actions or states given partial trajectories, and (2) randomly sampling the context from the external large-scale dataset such as webpages. Table 1 shows the results that the proposed approach achieves the best performance among the baselines.", "questions": "1. I am curious to learn more about the author's thoughts on perpetual data machines, and the comparison to self-refine papers.\n2. Please answer the weakness section", "limitations": "See the weakness section.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy to follow and read.\n2. The method seems to solve the common lack of demonstration data issue in the current LLM fine-tuned paradigm.\n3. The method seems to be novel.", "weaknesses": "1. The method falls into the category of the perpetual data machine, in which one uses LLMs to generate its own training dataset. In addition, the proposed method is similar to self-refine paper (https://arxiv.org/pdf/2303.17651), in which LLMs iteratively fine-tune itself based on the generated dataset. \n2. Table 1 does not compare the proposed method to the existing methods that use their own dataset to fine-tune such as self-refine paper. For instance, the paper could add a baseline in which the model is fine-tuned based on the other iterative fine-tuned procedure on the same dataset used in the paper. This will further verify the proposed method", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper! We are very happy to hear the encouraging words about the work being novel and addressing the lack of demonstrations in LLM finetuning for agentic tasks. We believe the comments are resolvable during the rebuttal period. Please see our response below. We would be very happy to address additional questions during the discussion period if anything is unclear.\n\n> ### The comparison with self-refine\n\n\nThe core difference between our approach and self-refine is that we apply LLM refinement *for different purpose*. Our proposal is a *data synthesize approach* that leverage LLM to transfer non-demonstrations into direct demonstrations and further finetune a model on the synthesized data. Our goal is to enhance a model’s performance on computer tasks. On the other hand, self-refine focuses on refining a model's *response* to a given prompt, it is a prompting mechanism. Self-refine does not involve generating training data and finetuning. \n\n> ### Comparison with models finetuned on their own dataset\n\n*Also included in global rebuttal*\n\nDirect demonstrations for web-based tasks are scarce, resulting in a limited number of LLMs fine-tuned with such data. We made our best effort to survey or implement approaches that are fine-tuned with direct demonstrations. We include three additional baselines and the results are shown below:\n| Model                        | Mind2Web| MiniWoB++ | WebArena |\n|-----------------------------|----------------------|-----------------|-------------------------|\n| AutoWebGLM-7b-42k | - | - | 2.50 |\n| Mind2Web-CodeLlama-7b | **39.33** (held-in) | 27.00 | 0.00 |\n| RAG-Llama3.1-Instruct-8b | - | - | 4.20 |\n| Synatra-CodeLlama-7b                 | 17.26    | **39.57**  | **4.80**           |\n\nHere, `AutoWebGLM-7b-42k` is finetuned with their 42$k$ synthetic direct demonstrations. The result is taken from [1]. `Mind2Web-CodeLLama-7b` is finetuned with the Mind2Web training data of 9$k$ direct demonstrations. The finetuning is done by us with the same configurations as Synatra. For `RAG-Llama3.1-Instruct-8b`, we used the same collection of tutorials as the retrieval pool and employed cosine similarity of semantic embeddings of task descriptions to find the three closest tutorials. These tutorials were then included as additional context when prompting LLama3.1-instruct8B to complete the tasks. We aims to compare different ways of using existing knowledge with `RAG-Llama3.1-Instruct-8b` \nWe can see Synatra achieves the best performance on all *held-out* datasets. \nWe also note that finetuning datasets used in AgentLM, CodeActAgent include a reasonable amount of out-of-box web-based demonstrations. In fact, Mind2Web training set made up 66% of the examples in AgentLM’s training set. We will incorporate the additional experiments to our updated draft. We are happy to investigate other approaches if we missed any. \n[1] Lai et al. Autowebglm: Bootstrap and reinforce a large language model-based web navigating agent.\n\n\n\n> ### Discussion on perpetual data machine\n\nCreating perpetual data machine is an exciting and open research question. Within the scope of our work, we propose leveraging existing resources. Currently, massive text corpora are primarily used for next-token prediction during pretraining, yet each piece of text can embed more diverse information. As models continue to improve through iterative training, they may be able to extract additional information or transform knowledge to enhance their capabilities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7iDVZTIfsM", "reviewer": "Reviewer_Gani", "summary": "This paper investigates turning indirect knowledge from the internet (such tutorial) to the direct knowledge (in the form of textual demonstrations) for LLMs. The algorithm design and the experiments are based on the web browser domain, which is a general interface to various web applications. The proposed method, Synatra, handles two types of knowledge source: tutorial articles and HTML snapshots, and presents corresponding demonstration synthesis approach. Such knowledge is then transferred to the LLMs by fine-tuning the LLMs on the synthesis data. Experiments show that Synatra can improve performance of CodeLlama-instruct-7b, outperforming open-sourced/fine-tuned models.", "questions": "1. How is success rate calculated in these web tasks?\n2. In Synatra, how is the LLM fine-tuned on the synthesis data? \n3. What is the source of performance improvement? Based on the results in Figure 4 (right), Synatra clearly outperforms human annotation method. I do not think this is simply attributed to the coverage of the annotation data. Maybe it is because GPT-4 has been trained on these testing tasks?", "limitations": "Authors have discussed the limitations in the conclusion section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, I think the paper is well-motivated. There are many tutorials available on the internet. It is interesting to explore how to effectively utilizing such knowledge. \n\nThe method is simple and effective, while the writing is easy to follow.", "weaknesses": "I have some concerns about the experiments. \n\n1. The comparison experiment is a bit unfair to me, for several reasons:\n- There are some RAG based methods to be considered, as this can be regarded as a knowledge retrieval problem. \n- Much of the baselines are not fine-tuned. Although fine-tuned with interactive data, the data in AgentLM, CodeActAgent is not out-of-the-box like Synatra. Maybe the authors can consider baselines that fine-tune LLMs with more direct demonstrations.\n- In Synatra, the demonstration is processed and generated by GPT-4. These high quality data further improves the performance of  Synatra.\n\n2. There lacks some important details about the method, e.g., how is the LLM fine-tuned and the training hyper-parameters. Besides, the configurations about baselines and evaluation dataset should be detailed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper! We are very happy to hear encouraging words about our motivation and effectiveness of our method. We believe the comments are resolvable during the rebuttal period. Please see our response below. We would be very happy to address additional questions during the discussion period if anything is unclear.\n\n> ### Comparison with RAG-based approach \n\nThanks for pointing out this. We agree that the RAG-based approach is an alternative method of utilizing existing knowledge. Below is our comparison with the RAG-based approach. Specifically, we used the same collection of tutorials as the retrieval pool and employed cosine similarity of semantic embeddings of task descriptions to find the three closest tutorials. These tutorials were then included as additional context when prompting LLama3.1-instruct-8B to complete the tasks. Note that many RAG methods rely on *benchmark specific in-context examples*, we *do not* use any in this case. The results on WebArena are shown below:\n| Model                       | WebArena |\n|----------------------------|-------------------------|\n| RAG-Llama3.1-Instruct-8b  | 4.20 |\n| Synatra-CodeLlama-7b                 |  **4.80**           |\nWe found that adding these additional tutorials fail to match the performance of Synatra. Since these tutorials were retrieved from wikihow articles, it is likely that they have been part of the large pre-training data already. However, converting them into direct demonstrations supplies additional information on the actual task completion, and hence can further improve performance.\n\n> ### Comparison with LLMs finetuned with more direct demonstrations \n\n***Please Refer to the same section in global rebuttal due to character limit.***\n\n\n> ### Details about the method, e.g., how is the LLM fine-tuned and the training hyper-parameters; the configurations about baselines and evaluation dataset\n\nWe apologize for the missing details. The details of the training are listed in Appendix I. The CodeLlama checkpoints are fine-tuned with A100 GPUs with deepspeed acceleration framework. We set the context length to 4096 tokens. To train on 50k dataset, we train with 6 x A100 80G GPUs for about 20 hours. We use a batch size of 48, and learning rate of 5e-5. We use cosine annealing and a warm-up ratio of 0.03. We use supervised fine-tune with full-parameters. We decided on this rather than lora from early experiment comparing the two’s performance. We only calculate loss on the response of the agent, while the observation, task title, past histories etc. are masked in loss calculation.  We will update section 5 to properly refer to the corresponding section in appendix I. We will update the details of baseline and evaluation setup in section 5 as well. For WebArena, we evaluated with a temperature of 0 and default set of parameters. For MiniWoB++, we evaluated on a subset that is the same as what's used in AgentLM/CodeActAgen. We take the five run-average success rate. We re-ran all the baseline models with the same setting for fair comparison. We will update these details in section 5 and appendix. And we will also release our training/evaluation code. \n\n> ### How is success rate calculated in these web tasks?\n\nBoth benchmarks provide validators to programmatically check if a task is completed or not inside their corresponding environments. For instance, if a task is to put a certain item into the shopping cart, a program check would be to verify if the said item is in the cart by the end of the trajectory. \n\n\n> ### What is the source of performance improvement? \n\nOur analysis shows that synthetic data teaches the model on multiple levels.\nFirst, it provides an understanding of basic task formats. Initially, the base model had issues such as repeating actions, interacting with nonexistent elements, or performing invalid actions not specified in the prompt. After training, these errors significantly decreased, approaching GPT-4 levels. The detailed breakdown is shown below:\n| Model | Click on Non-functional Elements (\\%) | Repeated Typing Actions (\\%) | Click on Non-existent Elements (\\%) | \n|-----------------------------|---------------------------------------------------|--------------------------------------|-------------------------------------------------| \n| GPT-4-Turbo | 6.4 | 26.85 | 1.66 |\n| Synatra-CodeLlama-7b | 4.9 | 6.8 | 0.7 |\n\nSecond, synthetic data enhances the model’s understanding of web pages. For instance, the base model correctly typed into a typable element (e.g, a input box) less than 75% of the time. Post fine-tuning, this accuracy improved to over 93%.\nLastly, we qualitatively examined model’s planning on the task before and after training, we found that synthetic data yields more accurate task planning and hence improve task performance.\n\n \n> ### Does the improvement come from GPT-4 trained on the test tasks?\n\nIt is highly unlikely that GPT-4 was trained on the exact test tasks. Even if GPT-4 encountered some of the test data during training, our approach does not rely on its prior knowledge of them. Our data synthesis process is entirely independent of the evaluation datasets, including WebArena, MiniWob, and Mind2web. We did not utilize any specific test tasks or testing environments, such as the websites in WebArena, during data generation. Instead, we leveraged GPT-4 to transform randomly sampled wikiHow articles, targeting general web-based tasks, and randomly sampled web pages. Given the vast volume of web content, it is improbable that the sampled pages would overlap with those seen during the test.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zdFICWw3JH", "reviewer": "Reviewer_JodV", "summary": "This paper proposes a demonstration generation approach for learning UI control by mixing two different sources. The first one uses GPT4 to generate scenarios, grounded actions, and observations using WikiHow plans which is initially filtered by GPT-3.5. Another data source is ClueWeb, form which the authors extract web pages, sample a segment for any page, and ask GPT4 to come up with scenarios and next actions. Both data sources are combined into a single dataset of 50K traces where a program format is used. A CodeLlama model is fine-tuned with the resulting dataset. Results show that it achieves comparably when compared to other baselines, including a basic GPT-based prompting method. Ablation analysis shows that program format is important, both data mixtures add value, and proposed dataset transfer to other benchmarks better than Mind2Web traces.", "questions": "Please see my above concerns for particular questions.", "limitations": "Limitations of the work are addressed.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper proposes a data synthesis approach for training UI agents. It combines WikiHow plans and actual observations with the strength of GPT-based models to fill in the gaps. the approach is agnostic to any domain and can serve as a good initialization.", "weaknesses": "My main concerns are weak baselines and lack of some others.\n\n1. There are many baselines missing from the comparison and available baselines are weak. For example, HTML-T5 [11] achieves 49.7 and MindAct [6] achieves 30.9 on Mind2Web dataset. Performance on Miniwob++ reaches up to 99.2% [A]. There are many other baselines including prompting and fine-tuning based ones with different pre-training that should be included.\n\n2. Can you further fine-tune on demonstrations from available benchmarks? While a semi-supervised approach is helpful, how much does it improve when human demonstrations added?\n\n3. You also mentioned that a unique \"id\" is assigned to target elements in your training demonstrations. Given that the actions condition on the current state, what prevents the model from overfitting to this unique id when detecting salient elements?\n\n4. You mention that you use viewports in Mind2Web benchmark to fit into the context window. But to extract viewports, you need to know where the ground truth element resides in the DOM (or accessibility tree) or you assume you have access to an oracle scrolling agent. I think both cases are different from your baselines and suggests leakage of test data.\n\n5. When you use NL format rather than programs, how do you ground generated actions to executable environment actions?\n\n6. What is the performance of the model trained with Mind2Web demonstrations on Mind2Web test sets? When tested on other benchmarks, how do you ground the action on the particular benchmark?\n\n7. It is not clear if the claim that collected demonstrations have a similar distribution as Mind2Web as they have fatter tails.\n\n8. Text and legend in Figure-2 are not readable.\n\nA. Longtao Zheng, Rundong Wang, and Bo An. Synapse: Leveraging few-shot exemplars for human level computer control. arXiv preprint arXiv:2306.07863, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper! We are very happy to hear encouraging words on our domain agnostic data synthesis approach that fills in the current gap. We believe the comments are resolvable during the rebuttal period. Please see our response below. And we would be very happy to address additional questions during the discussion period if anything is unclear.\n\n> ## Comparison with baselines\n\n> ### Comparison with LLMs finetuned with direct demonstrations (human annotated)\n\n***Please Refer to the same section in global rebuttal due to character limit.***\n\n> ### Comparison with approaches that achieves high performance on MiniWoB++ or Mind2Web \n\nThank you for bringing this up, we are aware of the many models and methods, we did not include them as baselines because of the following reasons:\n* Many approaches incorporate dataset-specific designs, making them challenging to apply to generic web-based tasks like WebArena. For instance, MindAct trained only with Mind2Web training data, as we can see with `Mind2Web-CodeLlama-7b`, while the model achieves strong performance on Mind2Web test set, it has limited performance on all held-out datasets. Similarly, Synapse uses trajectories as examples in the context; its prompting schema heavily relies on in-domain examples for MiniWoB++ and Mind2Web. In these datasets, the trajectories are short and can fit within the context window, and they offer relatively large training data on similar tasks or websites. However, in WebArena, trajectories can easily exceed the context length, and there are no provided training trajectories. \n* Many approaches are orthogonal to our data synthesis approach. The key component in MindAct is the ranking model that reduces a full HTML to the top-k potentially relevant elements, to remove irrelevant context. It is possible to apply a similar two-stage pipeline to our approach, instead of feeding the full AXtree into the model, we can first perform a reranking to select a subset of nodes in the AXtree instead. Alternatively, we can simply generate key elements rather than the full HTML as the observation. \n* Some approaches do not have public access. At the time we wrote the response, HTML-T5 is not open-sourced yet.\nWe will incorporate the additional experiments to our updated draft. We are happy to investigate other approaches if we missed any. \n\n\n> ## Can you further fine-tune on demonstrations from available benchmarks? How much does it improve when human demonstrations added?\n\nFinetuning with both human demonstrations and our synthetic data is straightforward because our data design is general for web-based tasks. Further finetuning the model on high-quality human demonstrations can presumably bring significant gain. However, such data is scarce, and we hope our work can encourage the creation of more high-quality data.  In addition, our experiment shows that adding human demonstrations that are collected without meticulous control may not improve the model’s performance on general web-based tasks. As shown in the table in global rebuttal, simply training the model on human demonstrations collected in Mind2Web results in very limited performance on the two held-out sets. We also experiment with adding Mind2Web data into our synthetic data, and we also see a similar overfitting trend. \n\n> ## Could assigning unique “id” result in overfitting when detecting salient elements?\n\nWe peformed a post processing to replace the unique ID with a **random** integer number in the final data. Hence, the model will not overfit to any specific ID.\n> ## You mention that you use viewports in Mind2Web benchmark to fit into the context window. But to extract viewports, you need to know where the ground truth element resides in the DOM (or accessibility tree) or you assume you have access to an oracle scrolling agent. I think both cases are different from your baselines and suggests leakage of test data.\n\nThe main reason of doing this processing is to make sure the web page observation can fit into the context window of a model. We use the same processed web page to all baselines and our approach. Hence it is a fair comparison. \n\n> ## When you use NL format rather than programs, how do you ground generated actions to executable environment actions?\n\nThe NL format we uses still requires the generated action to be in set formats. Here is an example:\n\nPast actions:\n1. click [4904] where 4904 is img calendar wheel with marker \n\nFor actions to be parsed accurately, they still need to be in the format such as click [xxx]\n\n> ## Performance of the model trained with Mind2Web demonstrations on Mind2Web test sets? \n\nThe performance of the model trained with Mind2Web demonstrations on Mind2Web test sets is as followed:\n| Set | Step Accuracy | \n|--------------|-------------------------------|\n | M2W-cross task | 0.47 |\n | M2W-cross website | 0.34 |\n | M2W-cross domain | 0.37 |\n\n> ## When tested on other benchmarks, how do you ground the action on the particular benchmark?\n\nWe follow WebArena to represent a web page as an accessibility tree with ID at the beginning of each tree node. This can be done by processing the HTML in the three benchmarks. In this way, we can refer to any element with its corresponding ID such as `click(element_id=123)`. \n\n> ## It is not clear if the claim that collected demonstrations have a similar distribution as Mind2Web as they have fatter tails.\n\nSorry for the confusion. Our goal was to demonstrate that our approach allows us to freely fit our generated trajectories to arbitrary distributions, such as Mind2Web. We will update Section 4 to clarify this.\n\n> ## Text and legend in Figure-2 are not readable.\n\nWe will make sure to update the figures in the camera ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uAvAsChg4F", "reviewer": "Reviewer_UgDQ", "summary": "The paper proposes a method for synthetically creating a dataset of agent trajectories for navigating websites. They propose using annotated programs, specifically python code interleaved with comments, as a representation of task and motion planning trajectories, where tasks are described in natural language in the comments, and actions are python API calls to the environment. A dataset of trajectories is created by applying GPT-4 to two sources of data. The first source is wikiHow, which contains ungrounded high-level tutorials. GPT-4 is used to generate plausible trajectories consistent with the tutorial. The second source is ClueWeb, which contains HTML snapshots of real websites. GPT-4 is again used to generate plausible trajectories consistent with the website. The paper finds that the resultant dataset of trajectories, when used to finetune CodeLlama-instruct-7b, outperforms open-weight models of similar sizes on Mind2Web, MiniWoB++, and WebArena. This result demonstrates that finetuning on the dataset transfers well to tasks.", "questions": "- open-source vs open-weight. Be careful with the distinction! (e.g. L54)\n- How would GPT-4 perform if trained on synatra? Is synatra limited to distilling agent-capability from stronger models to weaker models? What stopped you from finetuning GPT-4 or GPT-3.5 on synatra?\n- In 6.3 you write that you \"conducted a detailed examination\" comparing synatra with GPT-4, but I can't seem to find the analysis.\n- Given that the comparison with human annotation in 6.2 doesn't seem very fair, I'm not sure if you can claim in the abstract that \"we show that the synthetic demonstrations can be more effective than a similar number of human demonstrations.\"\n- To help answer the quality/quantity trade-off, one might look at how finetuning codellama does when varying the size of the dataset. As prior work [0] suggested, maybe only a small number of examples is needed, which pushes the trade-off towards obtaining smaller quantities of high-quality human-annotated data instead of large quantities of synthetic data.\n\n[0] Zhou, C., et al. \"LIMA: Less Is More for Alignment. arXiv, Article.\" arXiv preprint arXiv:2305.11206 (2023).", "limitations": "- Broader impacts adequately discusses potential unethical usage of LLM agents enabled by this work.\n- Needs discussion of robustness of LLM agents and potential security risks posed by deploying them.\n- Limitations of using synthetic data adequately discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-motivated, and for some sections clearly written\n- Creative usage of two different data sources, tutorials and snapshots, to complement each other\n- Makes progress towards building competent LLM-agents\n- Demonstrates that synthetic trajectories improve benchmark performances.", "weaknesses": "- Requires the use of GPT-4 to generate the synthetic data. While the paper demonstrates that using GPT-4 to generate training data can improve smaller open weight models, it's not clear if this can help push the SOTA. \n- Unequal comparison with human annotation: Sec. 6.2 compares with the human annotated trajectories in Mind2Web. As acknowledged by the authors, Mind2Web tasks are very restricted and simple compared with the tasks in synatra. A fairer comparison would be to collect a smaller set of human annotations on roughly the same task distribution as synatra. \n- 6.3 rather ad hoc", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper! We are very happy to hear the encouraging words on our motivation, creative usage of data sources, and performance improvement. We believe the comments are resolvable during the rebuttal period. Please see our response below. We would be very happy to address additional questions during the discussion period if anything is unclear.\n\n\n> ### How could Synatra possibly help push SOTA? \n> ### Is synatra limited to distilling agent-capability from stronger models to weaker models?\n> ###  Varying the size of the dataset.\n\nThis is a great question, we believe Synatra holds the potential of pushing SOTA when scaling up. We additionally train a CodeLLama-7b on 100$k$ synthetic data and the results on different size of synthetic data is below:\n\n\n| Training Data Size | Success Rate on WebArena| \n|--------------------|--------------------------------------------------| \n| GPT-3.5| 6.16 |\n| 9k | 4.56 |\n| 50k | 4.80 | \n| 100k | 6.28 |\n\nWe can see that training on more synthetic data provides 1.48% improvement on WebArena, surpassing GPT-3.5. In addition, among the 51 tasks `Synatra-CodeLLama-7b` got correct, we found `GPT-4-turbo` failed on 57% of them. For instance, while `GPT-4` suffers from severe errors such as repeatedly typing the same content to the same input box (26.85%), this only happens 6.9% of the time for `Synatra-CodeLLama-7b`. This indicates our approach could assist models in performing tasks the data generation model (e.g., `GPT-4-turbo`) cannot accomplish. \n\n> ### Can we compare the model trained on smaller set of human annotations on roughly the same task distribution as synatra? \n\nWe agree it will be interesting to have such experiment, however, as discussed in the introduction section, collecting human annotations of similar task distribution can be challenging. The main difficulty is on configuring the task prerequisites. For example, in Synatra, we have synthetic trajectories to cancel Amazon order. However, it requires an Amazon account with qualified order for cancellation for human trajectory collection. Such setup is *instance specific*. Hence, it is prohibitive to collect human annotations at scale as in Synatra. \nIn the paper, we also note that “it is important to recognize that the quality of human demonstrations is presumably higher, but they require meticulous design and control during the data collection process.” Hopefully this could further clarify the challenges in human data collection, and highlight the simplicity of our proposed approach.  \n\n> ### I'm not sure if you can claim in the abstract that \"we show that the synthetic demonstrations can be more effective than a similar number of human demonstrations.\"\n\nWe will soften the claim to only compare with human demonstration *collected in a dataset-agnostic way*, meaning that the data collection process is not meticulous design and control. \n> ### What stopped you from finetuning GPT-4 or GPT-3.5 on synatra?\n\nOn the one hand, the cost of finetuning GPT-3.5 or GPT-4 on **89 millon** tokens for a few epochs is prohibitive for us. On the other hand, we hope to push the frontier of open-weight models. We have spent more efforts on scaling up the data size as discussed above, and we are working on scaling up the model to stronger open-weight models.\n\n> ### 6.3 rather ad hoc\n\nThank you for pointing this out. We will add more quantitative study in this section and move the case study to Appendix in our camera ready version.\n\n> ### Open-souce -> Open-weight\n\nThank you for the suggestion. We will update the paper to reflect the accurate wording.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
> ### Comparison with LLMs finetuned with direct demonstrations

Direct demonstrations for web-based tasks are scarce, resulting in a limited number of LLMs fine-tuned with such data. We made our best effort to survey or implement approaches that are fine-tuned with direct demonstrations. We include three additional baselines and the results are shown below:
| Model                        | Mind2Web| MiniWoB++ | WebArena |
|-----------------------------|----------------------|-----------------|-------------------------|
| AutoWebGLM-7b-42k | - | - | 2.50 |
| Mind2Web-CodeLlama-7b | **39.33** (held-in) | 27.00 | 0.00 |
| RAG-Llama3.1-Instruct-8b | - | - | 4.20 |
| Synatra-CodeLlama-7b                 | 17.26    | **39.57**  | **4.80**           |

Here, `AutoWebGLM-7b-42k` is finetuned with their 42$k$ synthetic direct demonstrations. The result is taken from [1]. `Mind2Web-CodeLLama-7b` is finetuned with the Mind2Web training data of 9$k$ direct demonstrations. The finetuning is done by us with the same configurations as Synatra. For `RAG-Llama3.1-Instruct-8b`, we used the same collection of tutorials as the retrieval pool and employed cosine similarity of semantic embeddings of task descriptions to find the three closest tutorials. These tutorials were then included as additional context when prompting LLama3.1-instruct-8B to complete the tasks. We aims to compare different ways of using existing knowledge with `RAG-Llama3.1-Instruct-8b` 
We can see Synatra achieves the best performance on all *held-out* datasets. 
We also note that finetuning datasets used in AgentLM, CodeActAgent include a reasonable amount of out-of-box web-based demonstrations. In fact, Mind2Web training set made up 66% of the examples in AgentLM’s training set.
[1] Lai et al. Autowebglm: Bootstrap and reinforce a large language model-based web navigating agent.

Meta-review: The paper received acceptance ratings from all reviewers. The reviewers initially had some concerns such as the fairness of the comparisons, lack of some important details, missing baselines, and unequal comparisons with human annotations. The rebuttal addressed some of the concerns. However, the reviewers still have some concerns regarding the quality of synthetic data vs human data and lack of error estimates. Despite the concerns, they still maintained a positive rating. The AC checked the paper, the reviews, and the rebuttal and believes the paper proposes a valuable contribution to the community. Hence, the AC follows the recommendation of the reviewers and recommends acceptance.
Decision: Accept (poster)