Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale

Authors: tianyue  ou
frank f xu
aman  madaan
jiarui  liu
robert  lo
abishek  sridhar
sudipta  sengupta
dan  roth
graham  neubig
shuyan  zhou
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 21:43:26

Abstract: LLMs can now act as autonomous agents that interact with digital environments and complete specific objectives (e.g., arranging an online meeting). However, accuracy is still far from satisfactory, partly due to a lack of large-scale, direct demonstrations for digital tasks. Obtaining supervised data from humans is costly, and automatic data collection through exploration or reinforcement learning relies on complex environmental and content setup, resulting in datasets that lack comprehensive coverage of various scenarios. On the other hand, there is abundant knowledge that may indirectly assist task completion, such as online tutorials that were created for human consumption. In this work, we present Synatra, an approach that effectively transforms this indirect knowledge into direct supervision at scale. We define different types of indirect knowledge, and carefully study the available sources to obtain it, methods to encode the structure of direct demonstrations, and finally methods to transform indirect knowledge into direct demonstrations. We use 100k such synthetically-created demonstrations to finetune a 7B CodeLlama, and demonstrate that the resulting agent surpasses all comparably sized models on three web-based task benchmarks Mind2Web, MiniWoB++ and WebArena, as well as surpassing GPT-3.5 on WebArena and Mind2Web. In addition, while synthetic demonstrations prove to be only 3% the cost of human demonstrations (at $0.031 each), we show that the synthetic demonstrations can be more effective than an identical number of human demonstrations collected from limited domains.

Main paper: 
 introduction: 
 AI agents that operate within a digital environment (e.g., a browser in a computer) intelligently to accomplish complex tasks (e.g., "Create my July online shopping expense report.") have the potential to improve the productivity across a broad swath of tasks performed by humans every day [2, 33, 6, 58] . However, agents still lack the ability to complete tasks with a high degree of reliability, partly due to a paucity of training data for such agentic tasks. Typically, supervised finetuning is a standard way to adapt large language models (LLMs) to tasks such as text generation or classification, large-scale demonstration collections for digital agents are not readily available.For AI agents, demonstrations typically involve specifying a sequence of actions and observations that results in successful task completion, as shown on the right of Figure 1 . Existing works that automatically collect demonstrations (1) set up environments for an agent to interact with, (2) run a baseline agent within this environment, and (3) employ scoring functions to remove low quality demonstrations [9] or perform relabeling [1, 26] . All three of these requirements limit applicability to a variety of practical applications. Setting up an environment that is representative of the actual environments in which we would like agents to act is a difficult task, and existing environments for digital agents are generally limited in scope to a few websites or digital apps [58, 7, 45] . Even within these constrained settings, strong LLMs such as GPT-4 struggle on tackling tasks [58] , making collecting successful demonstrations with LLMs inefficient. In addition, human annotation is costly and its scope can still be limited [17, 6] . For example, gathering a demonstration for canceling a PayPal order requires an actual PayPal account with a legitimate subscription history.In this work, we propose Synatra, a data generation approach that synthesizes high-quality trajectories for complex digital tasks at scale. This approach is based on the intuition that there is a rich trove of existing knowledge that encodes indirect supervision about how to perform digital tasks ( §2.2). One example of a piece of indirect knowledge is a tutorial that details the sequential breakdown of a complex web task, such as "how to cancel a recurring payment on Paypal" for human readers (Figure 1 upper left) [53, 57] . While this tutorial provides some procedural guidance, such as "Enter the keyword," it does not specify the executable actions or the tangible observations associated with them. The key idea of Synatra is that, given this indirect knowledge, we can leverage an LLM to re-purpose it into more usable form that directly demonstrates the exact action to take given a concrete observation ( §3). In doing so, we leverage LLMs' ability to paraphrase and code, as well as its general knowledge of how tasks are performed (Figure 1 ). The main benefit of this approach is that it can scale with the availability of indirect knowledge created for humans, rather than rely on direct human annotations or LLM trajectories.We carefully study the sources of indirect knowledge, the design of demonstration formats, and the mechanisms for iterative refinement in order to synthesize high-quality demonstrations using an LLM ( §3, §4). We generate demonstrations of 100k tasks from 21 domains, and finetune a 7b Codellama-instruct model with this synthetic data. The resulting agent, Synatra-CodeLlama, surpasses existing open-source models of similar size, excluding those finetuned with human annotations, on three web-based task benchmarks: Mind2Web, MiniWoB++, and WebArena. Moreover, it consistently outperforms models that are ten times larger and have been finetuned with interactive data ( §6.1). Our findings also indicate that our model is on-par or a more accurate option in browser copilot settings, in comparison to GPT-3.5. Importantly, while each synthetic example incurs only approximately 3% of the cost of a human-annotated demonstration, we demonstrate that synthetic data with good domain coverage can be more effective than an identical quantity of limited-domain human demonstrations in real-world web-based tasks in WebArena ( §6.2). 
problem formulation 2.1 controlling digital agents through natural language: 
 An agent interacts with a computer environment E = ⟨S, A, O, T ⟩ with state space S, action space A, observation space O and the environment dynamics T : S × A -→ S. While this framework can be applied to different types of task, in this work, we consider a web browser as the unified entry point to access different web applications. We follow WebArena [58] to define the observation space as the contents on screen, represented as in text-based accessibility tree format. We use the same universal action space as in WebArena. This action space resembles the keyboard and mouse operations of a computer (e.g., click, type), and is applicable to arbitrary web applications. Appendix A lists all valid actions. The environment dynamics (e.g., the effect of clicking a button) and the states (e.g., the database status) are decided by the implementation of each web application.Given the natural language intent i, at each time step t, an agent issues an action a t ∈ A based on s t . The environment state is updated to s t+1 with new observation o t+1 . This process ends when the agent predicts the stop action. We follow existing works [6, 47, 58, 54] to represent s t with current observation and all previous actions in the form of (i, a 1 , ..., a t-1 , o t ). A benchmark (e.g., WebArena) supplies a scoring function r(i, s n ) that examines the final state and returns 1 if the desired goal state is satisfied, and 0 otherwise.We consider the expected action a t given s t as a form of direct demonstration, i.e., (s t , a t ). This allows an agent to directly learn how to predict the next action under a given state. On the other hand, indirect knowledge is broadly defined as resources that can benefit the task execution, but is not in the format of state and expected action tuple. We mainly focus on three types of indirect knowledge:1. Procedural knowledge details the sequence of steps ⟨a ′ 1 , a ′ 2 , ..., a ′ n ⟩ required to complete a specific task i. Unlike an action a t in trajectories, the steps in procedural knowledge are ungrounded, they lack a direct association with any particular observation and are not tied to specific action spaces. For instance, the tutorial in Figure 1 instructs the user to "login with your credentials" without providing the concrete PayPal login page and the input fields to type.Environment knowledge T that describe the effects of applying different actions in some hypothetical states. Example knowledge includes verbal descriptions such as "... after clicking the cancel button, you will see a pop up window ...".Ungrounded observations o that are not associated with particular tasks or trajectories. In the context of web-based tasks, an observation could be any random web page with different content and status (e.g., a product page with a query in the search field). 
scalable demonstration synthesis for digital agents: 
 In this section, we first introduce our design choices on the canonical formalization of trajectories, which account for the structural nature of procedures. Then, we delve into the sources for acquiring indirect knowledge, and the mechanisms for re-purposing this knowledge into direct supervision. Existing works demonstrate that representing task-related procedures as programs is beneficial for several reasons. This includes benefits from the structural nature of programs compared to free-form text [56, 23] , and the flexibility of using tools [4, 10, 41] . Inspired by these observations, we represent a Python function that interleaves natural language planning articulated in comments and actions as API calls, as shown on the right. The pink background represents the prompt, while the blue background corresponds to the model's response format.The planning process includes both task-level planning, which breaks the task into multiple sub-tasks, and actionlevel planning, which explains the low-level goals of each executable action. The model's generation consists of chain-of-thought (CoT, [43, 47] ) reasoning that analyzes the objective, previous actions, and current observations. Since CoT reasoning includes detailed information about the current step that may not be relevant for future steps, we further design the model response to include an action summary. This summary serves as a description of the predicted action that is added to the prompt for future action predictions. A concrete example can be found in Appendix B. 2 We study the empirical effect of program formalization in §6.3.The Internet offers fairly extensive procedural knowledge that describes how to perform high-level tasks by breaking down the task into detailed lower-level steps, such as how-tos and tutorials.Source We use wikiHow3 as our main source for these tutorials due to its comprehensive coverage of diverse tasks as well as its consistent format. Each article consists of a high-level task description and step-by-step instructions. We performed a filtering step and only kept the articles that involve navigation through the graphical user interface (GUI) of a computer or a mobile phone. We prompted GPT-3.5-turbo with six examples mixing the target articles (e.g., How to redeem an Amazon gift card online 4 ) and non-target articles (e.g., How to make a pizza) to perform the classification of all wikiHow articles. The prompt is shown in Appendix C. As a result, we obtained 25k articles that can be used to perform data synthesis.We want to bridge two gaps to re-purpose ⟨a ′ 1 , a ′ 2 , ..., a ′ n ⟩ for task i into ⟨a 1 , ..., a t-1 , o t ⟩. When re-purposing a sequence of actions ⟨a ′ 1 , a ′ 2 , ..., a ′ n ⟩ for task i into a new sequence ⟨a 1 , ..., at -1, o t ⟩, many challenges arise. First, the action descriptions provided in tutorials are not constrained to specific action spaces. Instead, they are presented as free-form natural language (NL) expressions, which can lead to ambiguity. For instance, various verbs such as "enter," "input," and others may all correspond to the same underlying action, type. Second, NL descriptions are often abstract, omitting concrete actions. For example, the process of "logging in" involves a series of actions, including typing in a username and password, but these specific actions may not be explicitly mentioned. Finally, the steps outlined in tutorials are ungrounded, meaning they are not directly associated with observable states or outcomes. Tutorials typically employ generic descriptions to accommodate various instances of conceptually similar tasks. For example, as illustrated in Figure 1 , the tutorial merely instructs to "enter the keyword" without addressing any specific scenario.Based on these findings, we propose an iterative approach that first uses an LLM to rewrite an article into a hypothetical trajectory in the format shown in §3.1, then we leverage a generative model to synthesize the intermediate observation between two consecutive actions. First, in the rewriting step, we ask the assistant LM to perform: (1) propose a hypothetical concrete scenario relevant to the task (2) perform basic parsing such as translating "enter the keyword [...]" into type("search bar", "Amazon Prime"); (3) categorize actions into groups that reflect the sub-task structures outlined by coding blocks. These tasks mainly demand a LLMs's creativity, language processing ability, and event understanding respectively. An example of rewriting a how-to article into a trajectory in program format is showed in Appendix E. The prompt for the rewriting step is in Appendix D.Because the previous procedures still only result in a sequence of ungrounded actions, we next leverage the assistant LM to generate the observations between randomly sampled consecutive actions. We use the consecutive actions of type("search bar", "Amazon Prime") and click("Amazon Inc", id=156) in Figure 1 as an example. There are mainly two requirements for generated observations. First, the observation must reflect the outcomes of past actions. In the example, this corresponds to a page with a user logged in, and a search input field filled with "Amazon Prime". Second, the observation encodes the necessary elements to perform the next action. In the example, this corresponds to a payment history list with a payment to Amazon. We prompt the assistant LM with the action sequence to generate a HTML snippet that fulfills the above requirements. Since the next action requires the concrete element to interact with, we ask the model to insert a tag of id="next-action-target-element" in the corresponding HTML node to indicate the grounding. 5 This step mainly requires a model's coding capabilities, particularly in web development. However, we find that it is not necessary for the LLM to generate HTML with high fidelity and complexity, which is a open research question [36] . The full prompt is in Appendix D and an example for this step is in Figure 5 .Note that in addition to wikiHow, there are other resources that share similar a similar procedural flavor, such as the captions of YouTube how-to videos [25] . Our transformation mechanism is generally applicable to such resources, but we leave concrete examination of them as future work.While the procedure in the previous section results in real procedures, the LLM-based method generates simplified observations. To compensate for this, we also perform data synthesis with real observations, and use synthesis to generate simplified procedures. We show that these two sources can compensate each other by examining the generated data in §4 and comparing the actual web-based task performance in §6.3.Source We utilize ClueWeb [27] as our data source, which comprises HTML snapshots of more than 10 billion web pages. Our initial analysis indicates that a random sampling approach would likely lead to a homogeneous distribution dominated by less interactive pages, such as news articles. In contrast, more complex web-based tasks typically require interactions with various web elements to advance the tasks. To diversify the sampled web pages, we employed a temperature sampling approach to select pages based on their content categories. In general, web pages from higher frequency top-level domains in ClueWeb are typically more interactive, such as Amazon and Reddit, while domains with lower frequency are less interactive, such as news article. We use a temperature sampling with T = 0.6 so that the sample probability of choosing a page in domain i,P ′ i = p 1 T i / k p 1 Tk , where p i is the original probability of choosing a page in domain i, and k is all the available domains. In doing so, we up-sample more interactive sites while maintaining diversity on more rare sites. More details are listed in Appendix J.We treat each sampled web page as an intermediate observation at time step t, aiming to synthesize the specific task i, the previous actions a 1 , ..., a t-1 and the subsequent action a t consisting of an action, a corresponding target element in the observation, and a natural language summary of the action. We first convert a web page into its corresponding accessibility tree at the beginning of each node, and sample a segment to present to the assistant LM. We follow the WebArena convention of assigning a unique ID to each node in the tree to ease the challenge of referencing the nodes. To increase the diversity of the tasks, we first instruct the model to brainstorm k task categories relevant to the web domain. Then the model randomly selects three of these categories and develops them into concrete scenarios with past actions leading up to the current observation and the next action to take. The prompt is in Appendix F and an example generation is in Appendix G.To ensure the quality of the training set, we apply a two-part filtering pipeline. In the first part, we ensure that data samples are both complete and coherent. For a sample to pass this filter, it must include all required components in the correct format. These components include: (1) an action that falls within the defined action space, (2) a valid and meaningful action target element in the corresponding web page, (3) NL texts that are well formed, e.g. without the use of "..." in the texts as an abbreviation by the generative model, (4) overall comprehensive generation without placeholders from the prompt (e.g., <brief step description>).To further eliminate accurately formatted but unresponsive actions, we apply a second filtering step using next state prediction. Here, we use the LLM to predict the next state, o t+1 , based on the current state o t and action a t in our synthesized data. If the model predicts that o t+1 = o t , the action is deemed to have no impact, and we filter out the action accordingly. 
data statistics: 
 We assess the distribution of history lengths, task objectives, and observations in the form of accessibility trees. The first statistic reflects general task complexity, as longer trajectories typically indicate Figure 2 : Left: t-SNE of task intent embedding. Right: t-SNE plots of accessibility tree embeddings more complex tasks. The latter two measure the domain diversity of our synthetic data. We also present these statistics for the Mind2Web dataset as an example of human-collected trajectories. As shown on the right, the majority of our synthetic data consists of trajectories with a history length of fewer than eight steps, regardless of the source. Fewer trajectories have longer histories, with the longest exceeding 20 steps. Examples with longer histories represent more complex scenarios such as "Create a new YouTube ad campaign for the Summer Collection with a focus on the 25-34 age group interested in sports, fitness, fashion, and style". In addition, we analyze the distribution of generated intents by projecting their highdimensional vectors using the embedding model all-mpnet-base-v2 [30] . We visualize these embeddings with t-SNE [40] . As shown on the left of Figure 2 , intents synthesized from random observations exhibit broader coverage compared to those from tutorials. This may be because humans often prefer to write tutorials for critical domains, while randomly sampled observations offer a more objective reflection of the diverse internet. Although our data synthesis process is entirely independent of Mind2Web, the generated intents show substantial coverage of Mind2Web tasks, which were created by human annotators from top-use websites. Finally, we apply the same embedding and visualization approach for accessibility trees and display the results on the right of Figure 2 . The generated observations from tutorials overlap significantly with real web pages, both from random observations and those in Mind2Web.Cost The average end-to-end cost per sample is $0.031, out of which $0.028 is used to generate the actual sample, and $0.003 is used by our prediction-based filtering pipeline. These values represent the final average cost to generate one valid sample, including the additional cost for samples that were generated but later filtered out, which we distributes across all non-filtered samples. 
experimental setup: 
 Agent training We finetune CodeLlama-7b [31] with 99, 920 web-navigation instruction examples, sourced from WikiHow articles and ClueWeb web pages 6 . Training details can be found in Appendix K.Evaluation tasks We select three evaluation tasks in the domain of web navigation. (1) the Mind2Web test set [6] . It consists of three categories, namely, cross-task, cross-website, and crossdomain-ranked by their degree of deviation from the training data. Since our model is not trained Table 1 : Performance of various models in three web-based task benchmarks. We measure step accuracy (%) for Mind2Web, and task success rate (%) for MiniWoB++ and WebArena. The numbers of FireAct-7b is taken from [5] ; AutoWebGLM-7b(S1) represents the model trained with only synthetic data in [19] . All other numbers are produced by our work. on the Mind2Web training set, all categories are treated as held-out sets. Therefore, we report an overall step accuracy as the average across all examples. In addition, we simplify the two-stage setup used in Mind2Web, where the model first selects the top 50 candidate elements from the HTML before predicting the action on a selected element. Instead, we input in-viewport accessibility trees that include the ground-truth element. This approach is more challenging because it introduces more irrelevant elements, but it streamlines the prediction process by eliminating the need for an additional element selection model. ( 2) MiniWoB++ [21] . It is an interactive benchmark with simplified web pages and low-level tasks (e.g., "Enter BUL to the box"). On average, completing a task in MiniWoB++ requires fewer steps than that of WebArena. We tested on a text-only subset of tasks used in [51, 41] . To get the final score, we take the average score of five runs. (3) WebArena. In WebArena, agents are required to navigate in real-world web applications to accomplish high-levels web-based tasks (e.g., "How much I spent last month on grocery"). Both MiniWoB++ and WebArena offer outcome-based evaluations, using programmatic checkers to validate key features after execution and assess whether the task has been successfully completed.Baselines We compare the performance of our model with 12 baseline models, including API-based models such as GPT-4, open-source instructed models like CodeLlama, and models finetuned with interactive data. For instance, Agent-LM [51] is finetuned on supervised data for tasks such as web navigation and interactive coding. Since most baseline models are not optimized for code generation, we follow Zhou et al. [58] and prompt the models to generate natural language responses instead of programs. The same prompt, providing general web navigation instructions without dataset-specific explanations, is used across three datasets. We show the prompt in Appendix H. Notably, we adopt the original prompts of AgentLM and CodeActAgent [41] when evaluating them on MiniWoB++ to be consistent with their original evaluations. We further remove the task-specific in-context examples to ensure a fair comparison with other models in zero-shot settings.6 Results 
main results: 
 Table 1 presents the performance of various models across three web-based task benchmarks. Overall, Synatra-CodeLlama achieves the best performance among models of comparable size. Notably, pages with a few elements as in MiniWoB++, while it is prohibitive to do pattern matching in complex web pages as in Mind2Web and WebArena. As a result, when we experiment with the more generic action space which is suitable for all three benchmarks without in-context examples, we see these models have a significant performance degradation. On the other hand, Synatra-CodeLlama targets at the generic web-based tasks and does not encode any dataset artifacts during training. Even though all three benchmarks are completely held-out during data generation, Synatra-CodeLlama achieves consistent superior performance on all benchmarks.What does the model learn from the synthetic data? Our analysis suggests that the baseline acquires essential knowledge at multiple levels from synthetic data. The error rates for different error types are shown in Table 2 . At the low level, synthetic data helps the model emit valid actions. For example, while approximately 95% of actions predicted by the base model CodeLlama-instruct-7b involve interacting with non-existent elements on the web page, Synatra reduces this error to just 0.7%. This low error rate is closer to that of larger models such as Llama3-chat-70b and GPT-4, which are better at following complex instructions with multiple requirements. Additionally, the synthetic data improves the model's ability to understand web pages more accurately. To assess this, we measure the ratio of invalid actions, including both invalid click and type, where the predicted action attempts to interact with an element that is not clickable (e.g., a piece of text) or not typable (e.g., a button). We observe that models in the 7b-8b range have high error rates in interpreting basic web elements. For instance, the strong 8b model Llama3-chat-8b mistakenly click a non-clickable element over 15% of the time, while Synatra reduces this error to under 6% approaching the performance of GPT-4. Finally, at task completion level, Synatra shows a stronger ability to track progress accurately. Specifically, when filling out forms on web pages, Synatra-CodeLlama is 74% less likely than GPT-4-turbo to repeatedly input the same words into the same field, a common error in models. This indicates that our synthetic data enables the model to more precisely recognize completed actions and identify the remaining steps needed to achieve the goal. More qualitative study can be found in Appendix I.Can Synatra be as effective as human annotations? We compare models trained with 9k human demonstrations (human only) from Mind2Web [6] Figure 3 : the comparison between the models trained with trajectories generated by our approach and the data collected from human.In contrast, Synatra led to a substantial improvement of 17.81% on MiniWoB++ and 4.56% on WebArena. The limited performance of the human annotations can be partially attributed to their restricted task coverage: notably, Mind2Web lacks information seeking tasks that require a string answer. Furthermore, the human trajectories did not specify conditions for terminating task execution. Despite adding such trajectories from Synatra (human + synthesis), the model still under-performed.We hypothesize the diversity of tasks plays a role in this discrepancy, since many tasks cannot be covered without pre-set environment (e.g., return an order) (Figure 2 ). These findings underscore the efficacy of Synatra, which also exempts from the complexities of developing recording tools and managing communication with annotators. However, it is important to recognize that the quality of human demonstrations is presumably higher, but they require meticulous design and control during the data collection process.In this section, we perform an ablation to validate the design choices of our data synthesis approach. Model performance improves with more synthetic data To assess the impact of scaling our synthetic data, we trained three CodeLlama 7B models with 18k, 50k, and 100k samples respectively, while keeping all other parameters constant. We then evaluated the models on WebArena. As shown on the right, the success rates on WebArena steadily increase as the synthetic training data scales from 18k to 100k samples. This highlights the potential of scaling synthetic data with our approach.Representing trajectories as programs is beneficial To verify if the programs format is helpful, we convert 30k trajectories to the NL format similar to setting in WebArena [58] and compare its performance with model trained with the exact data, but in our program format. The results are shown in Figure 4a . We can see that performance drops on both MiniWoB++ and WebArena when using the NL format. We hypothesize that program, in the form of API calls, is potentially a more natural format to represent sequence of actions. Our observation also echo the observations of using program representation for non-programming tasks [23, 29, 41] , while our experiments further contributes insights towards finetuning setups for interactive tasks.Different sources of indirect knowledge complement each other Our indirect knowledge comes from two sources: tutorials and randomly sampled web pages. In the former source, the procedural knowledge from tutorials are written and verified by human. However, there is no guarantee on the authenticity of the generated observations of web pages. In contrast, the observations from the latter sources are completely real, while there is no guarantee of the trajectory accuracy. We hypothesize that the two sources can compensate each other. To test this hypothesis, we trained three models: one using 9k synthetic data mixed from both sources, and two others each using 9k data exclusively from one of the sources and the results are shown at Figure 4b . We observe a noticeable performance degradation when models are trained with data from only one source. This indicates that utilizing multiple sources yields a more comprehensive dataset by integrating the precise procedural knowledge from tutorials with the realistic observations of web snapshot data.Task Success Rate (%) To access if turning knowledge into trajectories is helpful, we tested a retrieval-augmented approach that directly uses the collection of tutorials as the knowledge base. We first projected text tutorials to embeddings with all-mpnet-base-v2 [30] . Then a retriever retrieves the most relevant three tutorials measured by cosine similarity. These tutorials were included as additional context in the prompt. We tested this approach with LLama3.1-instruct-8B. As shown in Figure 4c , feeding in tutorials directly improves model performance on WebArena marginally , and finetuning on our data shows a substantial advantage. This comparison indicates that models have difficulty making use of indirect knowledge to solve agent tasks, and trajectories is a preferable format for agent learning. 
related work: 
 Learning from Indirect Supervision Due to the costly nature of human supervision, many digital agent learning works explore learning from existing yet indirect knowledge sources [8, 35, 12, 55] , reinforcement learning optimization that learn from environment feedback [37, 21, 32, 39] . More recently, LLM self-improvement during inference time [34, 49, 28, 44, 14, 52] has been applied on creating more complex digital agents in the wild. Our work fills the gap of training with synthetic data generated from existing resources, i.e., indirect supervision, on complex web navigation tasks.Prompting Approaches for AI Agents Existing methods include performing reasoning about the current statues before proceeding to next actions [43, 47, 22] , search and planning [48, 13] , and self-verification [18, 34, 24] . Our focus on instruction tuning data generation without human supervision can hopefully enable synthetic data generation recipes for various kinds of instruction prompts for agent tasks.Data Generation for Interactive Agents and Instruction Finetuning Existing works design ways of generating training data that adapt LLMs to agent-specific tasks [11, 19] , while our work aim to generate more realistic data without human intervention. Considering more broadly over all instruction finetuning: [59, 50, 38, 16] generate instructions of a specific task given a few examples. [42, 15] generates task-agnostic large-scale instruction tuning data without given examples. Li et al. [20] adopts an iterative instruction data generation and model finetuning approach, While our work generates instruction tuning datasets from readily available, indirect, unstructured knowledge resources. 
conclusion: 
 We propose a data synthesis approach Synatra. Empirically we showcase that finetuning with data generated from our approach can improve existing general-purpose LLMs to perform better on digital agent tasks. Since Synatra can synthesize trajectories given a single, static piece of indirect knowledge, we argue that when equipped with a capable LLM, a regular tutorial designed for human consumption and a random observation, can also be re-purposed a trajectory. We show that even considering the relative high cost of calling state-of-the-art LLMs such as GPT-4, synthesizing is more cost-effective than collecting human demonstrations of similar quantity for model training. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We detail compute resource usage in Appendix K.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In Appendix L.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We use existing models (CodeLlama, OpenAI GPT) and datasets (ClueWeb22, WikiHow) that have been either safeguarded or processed with moderation when released. We consider the issue not applicable to our work as we all work with existing safeguarded datasets and models. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We properly linked and cited the base model and source data corpus used in our experiments. They all have permissive licenses. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Given the title of an article, determine if it is about performing a task solely with computer or mobile phone's graphical user interface, and without any physical world configurations.input: How to Set Up Chromecast WiFi (Using an Android Phone or Tablet) output: Set Up Chromecast WiFi involves both a mobile application and physical interactions with the Chromecast device such as plug in the device, so the answer is "No" input: How to Change Your Desktop Wallpaper on Linux Mint (Using the Linux Mint Wallpapers) output: Linux Mint is a desktop operating system, and changing the desktop wallpaper is typically done through the system settings or desktop environment's configuration tools, which are desktop applications, so the answer is "Yes" input: How to delete a file using command line in Linux output: Command line interface (CLI) in Linux is a text-based interface not a graphical user interface (GUI), so the answer is "No" input: How to Reboot an iPad (Frozen iPads) output: Rebooting an iPad usually involves physical actions like pressing and holding buttons on the iPad, so the answer is "No" input: How to Connect the Kindle Fire to the Internet (Connecting to an Existing Wi-Fi Network) output: Kindle is neither a computer nor a mobile phone, so the answer is "No" input: How to Pair AirPods to an iPhone (If Your AirPods Won't Connect) output: Pairing AirPods with an iPhone typically includes physical actions such as opening the AirPods case near the iPhone and possibly pressing a button on the AirPods case, so the answer is "No" input: {{Title of the article}} output: {{Model prediction}}Prompt to rewrite an article to a trajectory in program format # Task overview You are given an article about performing a task in a web browser. Your goal is to make this article as accessible as possible to a user who is not familiar with the functionalities of the websites and the task at all.-Assume you start with the home page of the web application, skip the initial 'goto' action.-Break down the article into a sequence of steps.-In every step, provide a concrete example that reflects a real execution. This example should clearly describe the element you are interacting with, the concrete value of an element you select, the precise content you type and other details. Never use broad descriptions. The example should be creative and realistic, avoid boilerplate text such as email@example.com. Make sure that the example is consistent across steps. -Following the concrete example, provide the Python API call corresponding to the example.-Group all API calls into multiple sub-sections, each section corresponds to a logical and actionable sub-task.There are special scenarios and here are the ways to deal with them: -If the article describes multiple scenarios or multiple ways to approach the same goal, you can use your own judgement to choose the most common one to describe. -If there are repeated steps, make sure to unroll the steps and describe each of them canonically. -Always assume you perform this task using a web browser, if the original article uses a desktop app or mobile phone app, simply assume the corresponding web app exists. Hence, any steps regarding installation or login can be skipped.The Your goal is to recover the HTML and the dynamic of a web application with the following requirements:-The web page embodies a same level of content richness as advanced web applications on the internet. That is, the web page should have around 80 elements and at least 20 interactable elements. The depth of the DOM tree should be around 7. The length is at least 3000 tokens.-Analyze the past actions and determine which of these actions have visible or functional impacts on the web page you design. Reflect the effects of these past actions in your HTML code. This may involve updating text, adding new elements, or modifying the layout or styles to represent the state of the web page after these actions.-Design the interactable element that enables the next action. Make sure the choice of element type, attributes, and other essential characteristics are correct. For example, a text field is not interactable. Once the element is designed, assign the attribute id="next-action-target-element" to this interactable element.- The rendered generated HTML between step 2 and step 3, where step 2 is goto("Google Chat") and step 3 is click("jane.doe@gmail.com"). The concrete element to interact with is tagged with id="next-action-target-element".Prompt to Generate Direct Demonstrations from Random Observations ## Task overview Given the accessibility tree of a web page, your goal is to propose creative and diverse browser-based tasks that involves interacting with this page, along with the previous actions that lead to the current state and the next action needed to be taken to accomplish the task.Here are the allowed actions that you can take to interact with the web page: 'click(element: str, element_id: int=0)' -Click on an element. 'element' is the displayed text or the most representative attribute of the HTML element. 'element_id' is the index of the element at the beginning of the node. 'click_and_type(element: str, content: str, element_id: int=0)' -Click and type 'content' into an 'element'. 'key_press(key_comb: str)' -Press a key combination. 'key_comb' is the combination of keys you want to press on. The default OS is MacOS if there is no explicit specification in the article. 'goto(url: str)' -Navigate to 'url' 'go_back()' -Go back to the previous page. 'go_forward()' -Go forward to the next page. 'new_tab()' -Open a new tab. 'close_tab()' -Close the current tab. 'switch_tab(tab_index: int)' -Switch to the tab with index 'tab_index', starting from 0. 'scroll(up|down)' -Scroll the page up or down.'stop(answer: str=")' -The task is completed. If the task is to seek information, include the answer as a string. Otherwise, leave it empty.## Guidelines You will follow the guidelines below to perform the task: 1. Examine the web page to understand the the domain of the web page. 2. Brainstorm 8 task categories that could be performed the website. Be creative.3. For each task category, propose a concrete task that has this web page as one of its steps. You want the concrete task to be unambiguous and clear so that no further clarification is needed to perform the task. 4. Given a concrete task, you are ask to come up with the past actions that leads to the current page, as well as the next action. * Requirement for past actions: You should write down each past action in the details. You want to group all actions into multiple sub-sections, each section corresponds to a logical and actionable sub-task. The next action could start with a new sub-task. You can omit the 'elemement_id' if they are not in the current page.There should only be one action at each step. DO NOT give goto() or new_tab() as first step. * Requirement for next action: Provide the reasoning behind your past actions and the progress in completing the task. Also, describe your understanding of the current page and the concrete reason to execute the next action. If the action takes an element as the argument, it is important that you understand the role and the attributes of that element so that the action can be appropriately applied. Make sure to always include the 'element_id' in your next action if there is any. Any 'element_id' must come from the given Accessibility Tree.## Format of the response You are asked to provide the action sequence for task #1 with roughly 7 past actions; task #2 with roughly 0 past actions; task #4 with roughly 6 past actions; task #5 with roughly 4 past actions; task #6 with roughly 10 past actions. Your answer should follow the following format:<Analysis and understanding about the domain and the concrete content of the web page> <The list of 8 creative task categories> <The concrete tasks for task category #1 #2 #4 #5 #6. Remember, a concrete task needs to include concrete details so that no further clarification is required when performing the task. Use imperative sentences.> "'python # task: <repeat concrete task #1> # -------# past actions (history) # sub-task Generated trajectories from random observationThe domain appears to be that of a major online retail website, likely Amazon, considering the references to "Prime," "Best Sellers," and "Amazon Basics," as well as services like "Magazine Subscriptions" and "Prime Video." The structure of the page includes a variety of product categories, a search textbox, and links to other services such as account sign-in and shopping cart. This suggests that users on this page can shop for a wide range of items, manage their account, or engage with support services.Creative Task Categories: Below are eight creative task categories that could be performed on this type of website:• Product Searching: Looking up products in a specific category or using a search query.• Account Management: Signing into an account, creating a new account, or managing existing account details.• Navigational Inquiry: Using the site's navigation to find different departments or special features.• Customer Support: Accessing customer service features or disability support services.• Deal Hunting: Searching for best sellers, deals of the day, or exclusive prime deals.• Media Consumption: Navigating to and consuming digital media such as eBooks, Prime Video, or music.• Educational Browsing: Exploring educational materials or subscribing to learning resources.• Wishlist Management: Adding items to a wishlist or gift registry for future purchase.Concrete Tasks The concrete tasks for task categories #1, #2, #6, #7, #8 are as follows:Concrete Task #1: Product Searching # --------------------# p a s t a c t i o n s ( h i s t o r y )# --------------------# sub -t a s k 1 : S i g n i n t o t h e a c c o u n t .# n e x t a c t i o n # s t e p 1 : The u s e r i s a l r e a d y on a p a g e where t h e y c a n i n i t i a t e t h e s i g n -i n p r o c e s s . c l i c k _ a n d _ t y p e ( e l e m e n t = ' S e a r c h ' , c o n t e n t = ' myusername@example . com ' , e l e m e n t _ i d =7 65 7) # s t e p summary : Try t o s i g n i n by u s i n g t h e s e a r c h b a r t o e n t e r t h e u s e r n a m e . ------------------- # n e x t a c t i o n # s t e p 4 : To s e a r c h f o r t h e l a t e s t movie r e l e a s e s w i t h i n P r i m e Video , t h e u s e r n e e d s t o u s e t h e s e a r c h f u n c t i o n . c l i c k _ a n d _ t y p e ( e l e m e n t = ' S e a r c h ' , c o n t e n t = ' l a t e s t movie r e l e a s e s ' , e l e m e n t _ i d =7 65 7) # s t e p summary : S e a r c h f o r t h e l a t e s t movie r e l e a s e s on P r i m e Video .Subscribe to a magazine about technology within the "Magazine Subscriptions" category. ------------------- # n e x t a c t i o n # s t e p 2 : The u s e r w a n t s t o f i n d and add a s c i -f i n o v e l t o t h e i r w i s h l i s t . c l i c k _ a n d _ t y p e ( e l e m e n t = ' S e a r c h ' , c o n t e n t = ' p o p u l a r s c i -f i n o v e l s ' , e l e m e n t _ i d =7 65 7) # s t e p summary : B e g i n t h e s e a r c h f o r p o p u l a r s c i -f i n o v e l s t o add t o t h e w i s h l i s t .Example prompt used in baselineYou are an autonomous intelligent agent tasked with navigating a web browser. You will be given web-based tasks. These tasks will be accomplished through the use of specific actions you can issue.Here's the information you'll have: The user's objective: This is the task you're trying to complete. The current web page's accessibility tree: This is a simplified representation of the webpage, providing key information. We inspect a random sample of ClueWeb on its domain distribution. As shown in Figure 9 , the majority of domains appear only once, making up 69.1% of all the web pages. The CodeLlama checkpoints are fine-tuned with A100 GPUs with deepspeed 7 acceleration framework. We set the context length to 4096 tokens. To train on 100k dataset, we train with 6 x A100 80G GPUs for about 40 hours. We use a batch size of 48, and learning rate of 4e-5. We use cosine annealing and a warm-up ratio of 0.03.The broader impacts of this work extend across several domains. Firstly, the approach has the potential to democratize the development of digital agents by making the training process more affordable and accessible. Organizations with limited resources can utilize existing public data to train competent agents without the need for expensive data collection efforts. This could lead to a more widespread adoption and innovation in AI applications, particularly in regions or sectors that previously could not afford such technology.Secondly, by enabling digital agents to perform more complex tasks effectively, this work can significantly enhance productivity and efficiency in various industries. For example, customer service, online troubleshooting, and data management tasks could be accelerated, allowing human workers to focus on more creative or complex problem-solving tasks. Furthermore, the technology has implications for accessibility, as it could help develop more intuitive and user-friendly interfaces for people with disabilities or those who are not tech-savvy. Agents trained with a diverse range of demonstrations can offer more personalized and context-aware assistance, improving user experience across digital platforms.Lastly, the ethical and societal implications of this technology also constitute a critical area of impact. While the technology can lead to significant efficiencies and capabilities, it also raises questions about the potential for job displacement, and the need for robust guidelines to ensure that the deployment of such agents aligns with ethical standards. These broader impacts underscore the importance of interdisciplinary approaches to the development and governance of AI technologies, ensuring they contribute positively to society.One major limitation to the work is the potential variability in the quality and relevance of the indirect knowledge sources. We attempted to mitigate this through use of high-quality sources such as WikiHow and broad sources such as ClueWeb with intelligent sampling strategies, but still the danger of unrepresentative data remains.Another concern is the generalizability of the synthesized demonstrations. While the approach allows for the generation of a large volume of training data, the synthetic nature of these demonstrations may not fully capture the complexity and nuances of real human interactions with digital environments. As a result, agents trained on this data may still struggle with unexpected or less typical scenarios not covered in the training data. Furthermore, there is the risk of overfitting to the specific formats and tasks represented in the indirect knowledge sources. If the diversity of these sources is limited, the agents may not develop the flexibility needed to handle a broad range of tasks across different platforms or environments.Lastly, the reliance on large language models and complex synthesis processes might introduce significant computational costs and environmental impacts. The energy consumption and carbon footprint associated with training such large models are concerns that need to be addressed to ensure sustainable development in AI technologies. These limitations highlight the need for ongoing research, improved data curation methods, and the development of more robust models that can better generalize from synthetic training environments to real-world applications.• The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Justification: Our research conform with NeurIPS Code of Ethics Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Data and code are publically available at https://oootttyyy.github.io/synatra 38th Conference on Neural Information Processing Systems (NeurIPS2024).Some existing works also study using program formalization for web-based tasks[11,46], but focus on action-level interventions without incorporating task-level planning.https://www.wikihow.com/Main-Pagehttps://www.wikihow.com/Apply-a-Gift-Card-Code-to-Amazon#OnlineThe tag is removed through post-processing.We select CodeLlama-7b after comparing Llama-2, Llama-3, Llama-3-Instruct, CodeLlama, CodeLlama-Instruct, DeepSeek Coder, and Mistral by finetuning and evaluating on Mind2Web's train and test set.https://github.com/microsoft/DeepSpeed 


Decision: The paper received acceptance ratings from all reviewers. The reviewers initially had some concerns such as the fairness of the comparisons, lack of some important details, missing baselines, and unequal comparisons with human annotations. The rebuttal addressed some of the concerns. However, the reviewers still have some concerns regarding the quality of synthetic data vs human data and lack of error estimates. Despite the concerns, they still maintained a positive rating. The AC checked the paper, the reviews, and the rebuttal and believes the paper proposes a valuable contribution to the community. Hence, the AC follows the recommendation of the reviewers and recommends acceptance.

Reviews: review: paper_summary: paper_summary: This paper studies the problem of dataset generation for the digital task in the context of LLM. The motivation is that the current data collection process often requires human annotation, which is very costly and not scalable. As a result, the paper proposes using large language models (LLMs) to generate a synthetic dataset based on indirect knowledge. The indirect knowledge often refers to tutorials designed for human consumption and observation trajectories that lack associated tasks and actions. In contrast, "direction" knowledge consists of complete states and actions. To generate synthetic data based on indirect knowledge, the paper proposes several methods: (1) writing a prompt to let LLMs imagine their missing actions or states given partial trajectories, and (2) randomly sampling the context from the external large-scale dataset such as webpages. Table 1 shows the results that the proposed approach achieves the best performance among the baselines.
strength_weakness: strength_weakness: 1. The method falls into the category of the perpetual data machine, in which one uses LLMs to generate its own training dataset. In addition, the proposed method is similar to self-refine paper (https://arxiv.org/pdf/2303.17651), in which LLMs iteratively fine-tune itself based on the generated dataset. 
2. Table 1 does not compare the proposed method to the existing methods that use their own dataset to fine-tune such as self-refine paper. For instance, the paper could add a baseline in which the model is fine-tuned based on the other iterative fine-tuned procedure on the same dataset used in the paper. This will further verify the proposed method
questions: questions: 1. I am curious to learn more about the author's thoughts on perpetual data machines, and the comparison to self-refine papers.
2. Please answer the weakness section
limitations: limitations: See the weakness section.
paper_summary: paper_summary: This paper investigates turning indirect knowledge from the internet (such tutorial) to the direct knowledge (in the form of textual demonstrations) for LLMs. The algorithm design and the experiments are based on the web browser domain, which is a general interface to various web applications. The proposed method, Synatra, handles two types of knowledge source: tutorial articles and HTML snapshots, and presents corresponding demonstration synthesis approach. Such knowledge is then transferred to the LLMs by fine-tuning the LLMs on the synthesis data. Experiments show that Synatra can improve performance of CodeLlama-instruct-7b, outperforming open-sourced/fine-tuned models.
strength_weakness: strength_weakness: I have some concerns about the experiments. 

1. The comparison experiment is a bit unfair to me, for several reasons:
- There are some RAG based methods to be considered, as this can be regarded as a knowledge retrieval problem. 
- Much of the baselines are not fine-tuned. Although fine-tuned with interactive data, the data in AgentLM, CodeActAgent is not out-of-the-box like Synatra. Maybe the authors can consider baselines that fine-tune LLMs with more direct demonstrations.
- In Synatra, the demonstration is processed and generated by GPT-4. These high quality data further improves the performance of  Synatra.

2. There lacks some important details about the method, e.g., how is the LLM fine-tuned and the training hyper-parameters. Besides, the configurations about baselines and evaluation dataset should be detailed.
questions: questions: 1. How is success rate calculated in these web tasks?
2. In Synatra, how is the LLM fine-tuned on the synthesis data? 
3. What is the source of performance improvement? Based on the results in Figure 4 (right), Synatra clearly outperforms human annotation method. I do not think this is simply attributed to the coverage of the annotation data. Maybe it is because GPT-4 has been trained on these testing tasks?
limitations: limitations: Authors have discussed the limitations in the conclusion section.
paper_summary: paper_summary: This paper proposes a demonstration generation approach for learning UI control by mixing two different sources. The first one uses GPT4 to generate scenarios, grounded actions, and observations using WikiHow plans which is initially filtered by GPT-3.5. Another data source is ClueWeb, form which the authors extract web pages, sample a segment for any page, and ask GPT4 to come up with scenarios and next actions. Both data sources are combined into a single dataset of 50K traces where a program format is used. A CodeLlama model is fine-tuned with the resulting dataset. Results show that it achieves comparably when compared to other baselines, including a basic GPT-based prompting method. Ablation analysis shows that program format is important, both data mixtures add value, and proposed dataset transfer to other benchmarks better than Mind2Web traces.
strength_weakness: strength_weakness: My main concerns are weak baselines and lack of some others.

1. There are many baselines missing from the comparison and available baselines are weak. For example, HTML-T5 [11] achieves 49.7 and MindAct [6] achieves 30.9 on Mind2Web dataset. Performance on Miniwob++ reaches up to 99.2% [A]. There are many other baselines including prompting and fine-tuning based ones with different pre-training that should be included.

2. Can you further fine-tune on demonstrations from available benchmarks? While a semi-supervised approach is helpful, how much does it improve when human demonstrations added?

3. You also mentioned that a unique "id" is assigned to target elements in your training demonstrations. Given that the actions condition on the current state, what prevents the model from overfitting to this unique id when detecting salient elements?

4. You mention that you use viewports in Mind2Web benchmark to fit into the context window. But to extract viewports, you need to know where the ground truth element resides in the DOM (or accessibility tree) or you assume you have access to an oracle scrolling agent. I think both cases are different from your baselines and suggests leakage of test data.

5. When you use NL format rather than programs, how do you ground generated actions to executable environment actions?

6. What is the performance of the model trained with Mind2Web demonstrations on Mind2Web test sets? When tested on other benchmarks, how do you ground the action on the particular benchmark?

7. It is not clear if the claim that collected demonstrations have a similar distribution as Mind2Web as they have fatter tails.

8. Text and legend in Figure-2 are not readable.

A. Longtao Zheng, Rundong Wang, and Bo An. Synapse: Leveraging few-shot exemplars for human level computer control. arXiv preprint arXiv:2306.07863, 2023.
questions: questions: Please see my above concerns for particular questions.
limitations: limitations: Limitations of the work are addressed.
paper_summary: paper_summary: The paper proposes a method for synthetically creating a dataset of agent trajectories for navigating websites. They propose using annotated programs, specifically python code interleaved with comments, as a representation of task and motion planning trajectories, where tasks are described in natural language in the comments, and actions are python API calls to the environment. A dataset of trajectories is created by applying GPT-4 to two sources of data. The first source is wikiHow, which contains ungrounded high-level tutorials. GPT-4 is used to generate plausible trajectories consistent with the tutorial. The second source is ClueWeb, which contains HTML snapshots of real websites. GPT-4 is again used to generate plausible trajectories consistent with the website. The paper finds that the resultant dataset of trajectories, when used to finetune CodeLlama-instruct-7b, outperforms open-weight models of similar sizes on Mind2Web, MiniWoB++, and WebArena. This result demonstrates that finetuning on the dataset transfers well to tasks.
strength_weakness: strength_weakness: - Requires the use of GPT-4 to generate the synthetic data. While the paper demonstrates that using GPT-4 to generate training data can improve smaller open weight models, it's not clear if this can help push the SOTA. 
- Unequal comparison with human annotation: Sec. 6.2 compares with the human annotated trajectories in Mind2Web. As acknowledged by the authors, Mind2Web tasks are very restricted and simple compared with the tasks in synatra. A fairer comparison would be to collect a smaller set of human annotations on roughly the same task distribution as synatra. 
- 6.3 rather ad hoc
questions: questions: - open-source vs open-weight. Be careful with the distinction! (e.g. L54)
- How would GPT-4 perform if trained on synatra? Is synatra limited to distilling agent-capability from stronger models to weaker models? What stopped you from finetuning GPT-4 or GPT-3.5 on synatra?
- In 6.3 you write that you "conducted a detailed examination" comparing synatra with GPT-4, but I can't seem to find the analysis.
- Given that the comparison with human annotation in 6.2 doesn't seem very fair, I'm not sure if you can claim in the abstract that "we show that the synthetic demonstrations can be more effective than a similar number of human demonstrations."
- To help answer the quality/quantity trade-off, one might look at how finetuning codellama does when varying the size of the dataset. As prior work [0] suggested, maybe only a small number of examples is needed, which pushes the trade-off towards obtaining smaller quantities of high-quality human-annotated data instead of large quantities of synthetic data.

[0] Zhou, C., et al. "LIMA: Less Is More for Alignment. arXiv, Article." arXiv preprint arXiv:2305.11206 (2023).
limitations: limitations: - Broader impacts adequately discusses potential unethical usage of LLM agents enabled by this work.
- Needs discussion of robustness of LLM agents and potential security risks posed by deploying them.
- Limitations of using synthetic data adequately discussed.

Comments: title: Response to author response
comment: Thanks for your reply. I hold my opinion that the synthetic data quality can not outperform human demonstration. However, that is not the point of this paper. I believe how to leverage LLM knowledge is a promising direction. Thus I increase my score.
Tthanks for the clarification. I still believe error estimates are important particularly when the figures seem so close together. Nevertheless, I believe the work to be essentially sound, and have updated my score.
Thanks! The variance between runs is as deterministic as possible with the current compute, as we have set the temperature to zero in LLM inference and fixed random seed during training. To find out the variance due to different random subset of training data in the 50k sample case, we need to sample different sets of data and do multiple training runs, which is out of our compute budget.
Thanks for the follow-up! First, we hope that by showing models trained on Synatra is better than GPT-4-turbo in terms of repeating actions, interacting with nonexistent elements, and performing invalid actions help address the concern that Synatra is only learning from GPT-4-turbo and not teaching models new information GPT-4-turbo doesn't know. 

Regarding the comparison with human annotation (Mind2Web in this case), we agree that this could be more clear. Here is our finding from our experiments: we trained our two models with the exact same setting, one with Mind2Web's training set and the other with Synatra. As shown here and in global rebuttal:

| Model                        | Mind2Web| MiniWoB++ | WebArena |
|-----------------------------|----------------------|-----------------|-------------------------|
| Mind2Web-CodeLlama-7b | **39.33** (held-in) | 27.00 | 0.00 |
| Synatra-CodeLlama-7b                 | 17.26    | **39.57**  | **4.80**           |

we found that the model trained with Mind2Web's training set actually **outperforms** the others on Mind2Web's test set. But it is also true that it is not as good as a Synatra trained model in MiniWoB++ and WebArena. The reason for this difference is relatively intuitive: by using synthetic data pipeline, we are able to generate all of the actions in the action space, including stop(), go_back(), go_forward(), press(), goto(), etc, while the action space of the Mind2Web data set only has click(), and type(). The limited and biased action space played a key part in the low performance of Mind2Web-CodeLlama on WebArena. Additionally, websites' domain distribution could play a role too -- Mind2Web covers a somewhat limited number of sites, while Synatra has broader coverage. One other possible suggestion is potential format biases, but we have parsed the format of Mind2Web's training set to be exactly the same as Synatra, so we can rule this out as a factor. So the remaining plausible explanations are the differences in website domains and the types of actions represented in the dataset.

We hope this helps clarify, and we will further improve the discussion in the final version of the paper!
Thanks for the response. I am surprised by the large increase in WebArena success rate from 50k to 100k. Do you have error estimates for these figures?
Thanks for your detailed response. It seems that other reviewers also concern about the comparison and the reliance on GPT-4. I appreciate authors' additional results, addressing some of concerns regarding the comparison.

In addition, I still think the source of performance improvement over human demonstration is unclear. Specifically, I am not convinced that demonstrations generated by GPT-4 are more effective than that of human. You mentioned 'basic task formats'. However, human can also provide demonstrations that conform to the task format. The advantage of proposed method is that it does not require human cost, but requiring a high-quality LLM for annotation.