Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control
Authors: Longtao Zheng
Rundong Wang
Xinrun Wang
Bo An
Venue: ICLR 2024 poster (2024)
Keywords: AI Agents
Large Language Models
Prompting
Primary area: general machine learning (i.e., none of the above)
TL;DR: We introduce Synapse, a computer agent featuring state abstraction, trajectory-as-exemplar prompting, and exemplar memory.

Abstract: Building agents with large language models (LLMs) for computer control is a burgeoning research area, where the agent receives computer states and performs actions to complete complex tasks. Previous computer agents have demonstrated the benefits of in-context learning (ICL); however, their performance is hindered by several issues. First, the limited context length of LLMs and complex computer states restrict the number of exemplars, as a single webpage can consume the entire context. Second, the exemplars in current methods, such as high-level plans and multi-choice questions, cannot represent complete trajectories, leading to suboptimal performance in long-horizon tasks. Third, existing computer agents rely on task-specific exemplars and overlook the similarity among tasks, resulting in poor generalization to novel tasks. To address these challenges, we introduce Synapse, a computer agent featuring three key components: i) state abstraction, which filters out task-irrelevant information from raw states, allowing more exemplars within the limited context, ii) trajectory-as-exemplar prompting, which prompts the LLM with complete trajectories of the abstracted states and actions to improve multi-step decision-making, and iii) exemplar memory, which stores the embeddings of exemplars and retrieves them via similarity search for generalization to novel tasks. We evaluate Synapse on MiniWoB++, a standard task suite, and Mind2Web, a real-world website benchmark. In MiniWoB++, Synapse achieves a 99.2% average success rate (a 10% relative improvement) across 64 tasks using demonstrations from only 48 tasks. Notably, Synapse is the first ICL method to solve the book-flight task in MiniWoB++. Synapse also exhibits a 56% relative improvement in average step success rate over the previous state-of-the-art prompting scheme in Mind2Web.

Published as a conference paper at ICLR 2024
SYNAPSE: SAREECTOOY-SSEEEEEMEE PROMPTING WITH MEMORY FOR COMPUTER CONTROL
Longtao Zheng'
Rundong Wang' Nanyang Technological University, Singapore
Xinrun Wang
Bo An 1.2
Skkworrk AI, Singapore
https: github. io/ Synapae
ABSTRACT
Building agents with large language models (LLMs) for computer control is burgeoning research area, where the agent receives computer states and performs actions to complete complex tasks. Previous computer agents have demonstrated the benefits of in-context learning (ICL); however, their performance hindered by several issues. First, the limited context length of LLMs and complex com- puter states restrict the number of exemplars, as a single webpage can consume the entire context. Second, the exemplars in current methods, such high-level plans and multi-choice questions, cannot represent complete trajectories, lead- ing suboptimal performance in long-horizon tasks. Third, existing computer agents rely on spsk-sp exemplars and overlook the similarity among tasks, resulting poor generalization novel tasks. To address these challenges, we introduce SYNAPSE, a computer agent featuring three key components: i) state abstraction, which filters out task-irrelevant information from raw states, allowing exemplars within the limited context, ii) trajectory-as as-exeplla prompting. which prompts the LLM with complete trajectories of the abstracted states and actions to improve multi-step decision-making and iii) exemplar memory, which stores the embeddings of exemplars and retrieves them via similarity search for generalization novel tasks. We evaluate SYNAPSE MiniWoB++, standard task suite, and Mind2Web, a real-world website benchmark. In MiniWoB+++ SYNAPSE achieves 99.2% average success rate (a 10% relative improvement) across 64 tasks using demonstrations from only 48 tasks. Notably, SYNAPSE is the first ICL method solve the book-flight task in MiniWoB++. SYNAPSE also exhibits 56% relative improvement in average step success rate over the previous fate--oftthe-nnd prompting scheme in Mind2Web.
INTRODUCTION
Creating agents capable of performing complex tasks reduce human effort in routine operations is a long-standing goal in artificial intelligence (DeepMind Interactive Agents Team 2021). critical and pragmatic step towards this ambitious vision to develop agents that are proficient in computer control (Shi et al. 2017 Liu et al.) 2018). Such an agent perceives states of the computer, c.g., screenshots or webpage HTML, and performs actions via keyboard and mouse to complete tasks specified in natural language, such as flight booking and email management. Different from training- based methods (Humphreys et al.] [202) that rely on large-scall behavioral cloning (BC) (Pomer- leau 1989) and deep reinforcement learning (RL) (Sutton and Barto 2018), recent studies highlight the advantages in-context learning (ICL) (Brown et al.
2020, Wei et al.) 2022a b) achiev-
ing general and data-efficient computer control (Yao et al. 2022a Kim et al. 2023 Deng et al.)
2003). Specifically, these computer agents utilize few-shot demonstrations, also termed exemplars, prompt large language models (LLMs) (Brown al., 2020 Chowdhery al.) 2022 OpenAl 2023 inter alia) to generate actions. For example, RCT (Kim et al. 2023) prompts LLMs to gener- high-level plan and grounds each action based on the plan. Concurrent with our work, Deng et al. (2023) introduce Mind2Web as a benchmark for real-world web navigation and computer agent, MindAct. It prompts LLMs predict the next action using multi-choice questions (MCQ), with candidate choices generated by an dement-ranking model.
"Corresponding authors

---

Published as a conference paper at ICLR 2024
State Abstraction
Trajectory-as-Exeme
Environment
Raw
Clean State
History
Typlo Books fights Ioom
Taok Bock fight from
Coservation (TV siae
"wet's
Next Action Actors apertope-
Large Language Models
IlI. Exemplar Memory
State Abstraction Prompts
Exemplary Trajectories
Task
Embedding Models
Similarity
book book-flight flight
email-inbox
Metadata
Search
[1.1, 2.8, -]]
[2.4, 0.6, ..]
Figure I: SYNAPSE consists of three main components. The process begins with state abstraction, where raw computer states (e.g., the HTML of webpages) are processed into concise task-relevant observations via few-shot learning of the LLM. This step reduces the number of tokens needed for each state, prerequisite for the second component: TaE prompting, the LLM prompted with exemplary trajectories (a sequence of abstracted states and actions) and the current history to determine the next action. These prompts are retrieved from the exemplar memory using similarity search. The retrieval process utilizes the embeddings of task metadata, with each metadata mapped to the corresponding exemplars. Fig, Fig.2 further shows an illustrative comparison of TaE prompting with the prompting schemes of other computer agents.
enijectory-as-exemppe
(TaE) prompting. In
Despite the promising results of previous LLM-based computer agents, they still fail to solve some tasks within MiniWoB+ (Shi et al. 2017 Liu et al. 2018), a relatively simplified computer control task suite, mainly due to the following issues. First, the limited context length of LLMs, combined with complex computer states, poses challenges for few-shot learning. For example, the book-flight task in MiniWoB++ is particularly challenging due the extensive length of the webpage dis- playing flight details. Also, the task- ireeevnteeevant information in raw states can distract LLMs from generating correct actions. Second, the exemplar structures used by existing computer agents can lead to error accumulation in multi-round I.LM queries and struggle to capture the complete tra- jectory information. Specifically, existing computer agents query LLLMs each step, and neither high-level plans nor MCQs can represent full trajectories. For example, even when prompted with correct step-by-step plan and previous actions, RCI may still incorrectly predict the next action. As result, existing ICL methods struggle to solve long-horizon tasks. Even for simplified tasks, they still rely on self-correction. Third, current computer agents require task-specific exemplars within predefined scope. For example, in MiniWoB++, both email-inbox and email-inbox handle email-related tasks. Similarly, multi-orderings and multi-layouts share similar interfaces. However, existing methods overlook this task similarity and hard-code the mapping from tasks exemplars.
To address these challenges, we present SYNAPSE, an L.LM-powered computer agent featuring a novel prompting scheme and a memory mechanism. As illustrated in Fig. [ SYNAPSE has three core components: i) state abstraction, which extracts task-relevant observations from raw states to reduce the token length per trajectory, ii) jijjectory-asexemppe successful trajectories as few-shot exemplars to prompt the LLM for enhanced multi-step decision- making, and iii) exemplar memory. which stores successful trajectories and retrieves relevant ones via similarity search for generalization to novel tasks.
(TaE) prompting, which utilizes full
We thoroughly evaluate SYNAPSE on two benehmarks: MiniWoB++ (Shi et al. 2017 Liu et al,, 2018), standard research task suite, and Mind2Web (Deng et al. 2023). dataset across diverse domains of real-world web navigation. In MiniWoB++, SYNAPSE the first ICL method that achieves human-level performance (Fig. 3.. Specifically, S SYNAPSE solves 64 tasks with a success rate of 99.2% using demonstrations from only 48 tasks without relying on self-correction. In com- parison, the current ICL state-of-the-art (SOTA) approach (Kim et al. (023) is limited to handling 54 tasks, requiring task-specific demonstrations for each. Compared to BC+RL and fine-tuning SOTA baselines, SYNAPSE achieves superior performance while being simpler, more flexible, and
2

---

Published as a conference paper at ICLR 2024
more sample-efficient. A notable achievement is its success on the book-flight task, which requires long-horizon secision-making under intricate states. We further demonstrate the effectiveness of our approach in Mind2Web by incrementally augmenting the LLM with three components of SYNAPSE, i.., state abstraction, TaE prompting, and exemplar memory. When the underlying L.LM is GPT-3.55 achieve improvements of 32%, 50%, and 56% in the step success rate over MindAct, the SOTA ICL method in this domain. When we use CodeLlama-7B (Roziere et al.] 2023), SYNAPSE also achieves an average 2.5x step success rate compared MindAct. These experiments showcase the superiority of S SYNAPSE over existing computer agents in both standard and
benchmarks.
RELATED WORK
There has been increasing attention on building autonomous agents with LLMs (Yang et al., 2023 Mialon et al.] 2023 Xi et al., 2023 Park et al. (023). Huang et al.) (202)) and [Brohan et al. (2023) combined the LLM-generated plan with an embedding and a value function to determine robot policies, respectively. Inner Monologue (Huang et al. 2023) incorpo- rated environment feedback into a closed-loop system. These methods focused on prompting for better high-level semantic plans and relied on BC or RL for low-level motor control policies. Unlike robotics, low-level actions for computer control are highly semantic, and the effective prompting scheme for directly grounding actions remains unexplored. To address this, SYNAPSE proposes vrajectory-as-eeemplae prompting for directly grounding these actions. Recent efforts to improve the capabilities of reasoning, planning, and coding of LLMs, such as chain-of of-uuhughtttt (Wei al., 2022a; Kojima et al. (2022) least-to-most (Zhou et al., (022). ReAct (Yao et al., 2022aa). tree-of- thoughts (Yao et al.] 2023). 3) Reflexion (Shinn et al.] a (023), and self-debugging (Chen et al. 2023).
Building Agents with L.Mss.
can be combined with SYNAPSE to improve decision- decision-making. The methods that leveraged external
tools (Wu et al., 2023a Shen et al., 2023) Schick et al.] 2023: Lu al., 2023 Xu et al.) 2023) are also orthogonal to SYNAPSE. Code Policies (Liang et al. 2023) and ProgPrompt (Singh et al.) 2023) formulated agent policies as code generation. Similarly, SYNAPSE can be strengthened using code for both state abstraction and action generation. Voyager (Wang et al. (023a) used a skill library for life-long skill acquisition, which also complements our framework. Wu et al. (2023b) also identified the issue of limited context, using a pretrained question-answering language model to filter states, similar to MindAct. In comparison, our state-abstraction principle can leverage both LLMs (few-shot learning) and pretrained state abstraction models.
Agents for Computer Control. In the pursuit of creating agents capable of human-like com- puter interactions, MiniWoB ++ task suite (Shi et al. 2017] Liu et al. 2018) has become a standard benchmark. Early attempts to solve MiniWoBi
primarily used BC and RL (Shi al., 2017 Liu
et al. 2018 Gur et al. 2018 Jia et al. 2018 Gur et al. (021). which were not sufficient to achieve notable breakthrough came with CC-Net (Humphreys et al. 2022).
human-level performance.
which achieved human-level performance but required an extensive dataset of 2.4 million demon- strations, equivalent to 6,300 hours of human effort. However, it is challenging to generalize this approach for new tasks and user customization. WebN-T5 (Gur et al. 2023a), a fine-tuned variant of the pretrained T5 model (Raffel et al. (2020), was trained on 12,000 demonstrations across 56 tasks. Similarly, WebGUM used 346,827 demonstrations to solve them. However, these fine-tuned LLMs still demanded a large amount of data and fell short of human performance in MiniWoB++. WebGPT (Naaano al. ) and WebShop (Yao et al., 2022b), which operated in web- web-browsing environments, focused more on refining search queries rather than on general computer control. Recently, RCI (Kim et al.
2023) employed recursive self-correction solve MiniWoB++ with success rate of 90.6% in 54 tasks. However, it relied on task-specific exemplars, which limited its generalization novel scenarios. In contrast, SYNAPSE achieves better performance in MiniWoB+ without relying on self-correction, using demonstrations from fewer tasks. There is also a lot of con- current work in building computer agents and benchmarks. AdaPlanner (Sun et al. 2023) utilized environment feedback for self-correction and achieved a success rate of 92.9% in 53 tasks, but it had similar issues to RCI. Pix2Act (Shaw et al. 2023) solved 59 MiniWoB+ tasks with tree search and BC on 1.3 million demonstrations. WebAgent (Gur et al. 2023b) employed small element- ranking model for HTML state filtering and an LLM for few-shot action generation, but did not investigate the exemplar structure and memory. Among many sophisticated benchmarks, such as Mind2 Deng et al. 2003)) AgentBench (Liu et al. 2023a), WebArena (Zhou et al., 2023). and Android in the Wild (Rawles al. 2023), we choose Mind2Web for the real-world evaluation.
3

---

Published as a conference paper at ICLR 2024
3 SYNAPSE
As illustrated in Fig. 1 SYNAPSE consists of three main components: state abstraction, trajectory- as-exemplar prompting, and exemplar memory. Here is the general pipeline of SYNAPSE: when task occurs, SYNAPSE first retrieves relevant few-shot exemplars from memory by similarity search over the embedding space of task metadata, e.g., task descriptions and initial states. At each step, SYNAPSE first prompts the LLM to convert raw states into clean, task observations. Exem- plary trajectories and the current trajectory (a task description and a sequence observation-aation pairs) are subsequently fed into the L.M to generate the next action.
3.1 PROBLEM SETTING
In this work, we focus on computer control tasks, where an agent interacts with the environment to accomplish a task specified by natural language. The state and action space for computer agent are consistent with how humans interact with computers. At each step, it receives a computer state, such as HTML of webpages actions can be either code or natural language, depending on the benchmarks. As the decision- making engine, LLMs are fed with few-shot exemplars to process raw states into clean observations and generate actions, When the task is completed, the trajectory is considered successful.
screenshots, and performs actions via keyboard and mouse. The
3.2 STATE ABSTRACTION
The number of exemplars in few-shot learning can drastically influence the performance of LLM- powered computer agents (Wang et al.) 2023b; Kim et al. 2023). However, the context limits of LLMs, combined with the intricacy of computer states, often restrict the number of exemplars. To illustrate, directly processing long HTML documents of real-world websites can be prohibitively costly because of many task insk-irrelevant elements. As result, the performance of existing computer agents is hindered by the limited number of exemplars, and they often incorporate self-correction as remedy. Moreover, many task-irrelevant elements in raw HTMI states can potentially distract LLMs from accurate action generation. Even though some LLMs can handle longer inputs, their performance becomes worse the input context grows longer (Liu et al. 2023b).
To reduce the length of each state, SYNAPSE takes advantage of the few-shot learning ability of LLMs to extract task-relevant information from raw states and form clean observations for subse- quent action generation, as shown in Fig. [ Specifically, we propose few-shot state abstraction prompts in both explicit and implicit forms. For scenarios where the context can handle multiple states, such as email-inbox in MiniWoB++, we apply explicit abstraction through state-observation pairs, denoted as (state, observation). In this case, the L.MM is prompted with these pairs as few- shot exemplars, followed by the current raw state, to generate the clean observation. Alternatively, in scenarios with complex states where explicit abstraction not applicable, such book-flight in MiniWoB++, we adopt implicit abstraction by pairing task descriptions and state- sate-pprsnng code, denoted as (task, code). We feed these pairs as few-shot exemplars to the LLM, along with the cur- rent task, to produce the code. This code takes the task and the raw state as parameters and returns clean observation. the code execution raises an error, we ask the LLM to perform zero-shot state abstraction. Within Mind2Web, we further demonstrate that the state abstraction principle can aid effectively utilizing existing state abstraction tools. Specifically, Mind2Web has a pretrained element-ranking model that ranks and filters HTML elements to obtain a clean observation com- prising the top-k elements. As a simplified implementation 5 for the previous and current observations, respectively. In comparison, the SOTA method set k to 50. Although this reduces the recall from 86% to 53%, i.e., only 53% of the clean observations contain the target element, it achieves higher step success rate (Sec. 4.4). State abstraction shrinks long-win winded raw states into concise and shorter ones, enabling trajectory-as- axeexepla prompting (Sec.) 3.3). which is impossible for previous methods due context limits.
state abstraction, we set l to 3 and
3.3 TRAJECTORY-AS -EEESPAA PROMPTING
The prompts for action generation in previous computer agents often overlook the sequential nature of decision-mmking For example, each RCI exemplar consists of a high-level plan, while Min- dAct formulates exemplars step-by-step MCQs, as illustrated in Fig. 2 Without considering the
4

---

Published as a conference paper at ICLR 2024
complete trajectories, these methods struggle capture previous environment interactions. Fur- thermore, these methods query LLMs for one action at each step, leading to error accumulation over time. Therefore, these methods require self- ceff-correction even for simplified scenarios in MiniWoB+ especially for tasks with many steps and repeated actions, such as use-autocomplete and use-spinner.
RCI
MindAct
Trajectory-as-Eeepllaa
Use tarminal below to deleta fiee ending with the extension .gif
Use terminal to
Task: Use the terminal below to delete a file ending with the extension -gff
file ending with the exterssoo .if
HTML state: <htmii userS is
Observation: <htmi> userS </html>
Plan:
script.zip shark.gff ... -ohtmb
Actione
1. Type "Is" command to list all files in the terminsl.
Previous actions:
agent.typollss ls)
apent. agent,pressf eentt""
agent, press| enter))
2. Type "mm [*iename]* command to delete the file ending with "ggi".
What should be the naxt action?
Observation: <htmbs ... users
A. None of the above
script.zip shark ... </htmlo
3. Press "enter" key typing each command to execute the command.
seent,reesslentte eeentrr))
Action:
C. spent..ype((mm shark.gif"
speett.ypo//mm shark.gif)
Answer:
presst anter))
Figure 2: Comparison of sejjector--a-exemmppe prompting with other prompting schemes. The illustration is based on the terminal task in MiniWoB++ +++ where the agent is asked to delete a file ending with specific extension. To solve this task, RCI (Kim et al. (023) prompts the LLM with step-by-step plan combined with the current state and the previous actions to generate each action, while MindAct (Deng et al.. 2023) prompts the LLM with MCQ-formatted exemplars each step. In contrast, SYNAPSE uses a straightforward prompting scheme based on trajectory-level exemplars. This exemplar structure offers a consistent and interactive format, is more informative, and enables the LLM to produce temporally abstracted actions until new state is required. As shown above, the LLM generates two consecutive actions: type(ls) and press(enter) without querying the new state. After executing these actions, it pauses to receive the new state for subsequent actions. We provide several complete trajectories in Appendix C for illustration.
To address this challenge, we introduce trajectory-as-cxemllaa (TaE) prompting, utilizing complete trajectories to prompt the LLM for action generation, formatted as (task, observation, action, observation, action). The first step of TaF prompting to feed the L.M with successful trajectories. Subsequently, the LLM is prompted with the current trajectory (the task description and previous clean observations and actions) to produce the next action. Take Fig. 2 as an example, where the agent is asked to delete an item using the terminal. We pass few successful trajectories to the LLM to illustrate how the problem is solved, followed by the current task description "Delete a file ending with the extension .gif" and the clean observation representing "the terminal is empty". We then execute the action returned by the LLM Sugent.type(44 prompt by appending the action and the next clean observation to the prompt. This process iterates until the agent completes the task or reaches the maximum number of steps. Similarly, in Mind2 Web, we iteratively append states filtered by the dementtranking model and target actions to the prompt since it is a static dataset. This straightforward prompting scheme enhances the decision-making ability of the LLM for several reasons. First, the consistent format of interleaved observations and actions is well-suited for grounding actions, which allows conveniently parsing actions and setting stop tokens for LLM responses. Also, it provides more information for prciisionmmaingg More importantly, it implicitly prompts the LLM to generate temporally abstracted actions and query new states only when necessary. Temporal abstraction also results lower cost and latency. Com- pared to human-ccaftedd plans and MCQs, TaE exemplars can also be directly converted from human demonstrations. Therefore, TaE prompting improves the action accuracy in long- long-horizon tasks.
and construct the next
3.4 EXEMPLAR MEMORY
Previous computer agents rely on task- exemplars. For example, the mapping from tasks to few-shot exemplars is hard-coded in the RCI agent. Therefore, it requires exemplars for each of the 54 tasks tested, despite analogous tasks and interfaces, such as email-inbox and email-inbox- -l-turk,, fhick-hheckboxes-ble large and chick-checkboxes traanseeettrrrsferr etc. In other words, existing computer agents lack a mechanism that matches relevant exemplars and tasks automatically. As a result, these agents cannot leverage task similarity and struggle to generalize to tasks.
To exploit task similarity and enable generalization, SYNAPSE introduces exemplar memory D = (K, V), where K is fixed-sized array of embedding vectors of task metadata and V is the cor-
5

---

Published as a conference paper at ICLR 2024
responding state abstraction prompts and exemplary trajectories. This memory is constructed by encoding all the metadata using an embedding model and storing their corresponding exemplars in vector database. As illustrated in Fig. [ for given task, SYNAPSE first encodes the task metadata and performs similarity search in the vector database, which retrieves relevant trajectories as few- shot exemplars. Formally, the retrieval process can be denoted as arg top-Rde sim(q. d) where q is the query metadata and sim is the Euclidean distance in the embedding space. In MiniWoB+ +++ the metadata consists of the task description concatenated with the initial state from five random seeds of the selected 48 tasks. We retrieve the top three exemplars from memory and use the most common one to retrieve its exemplars, for there is clear task division. For example, if the retrieved exemplars belong to enter-date, enter-date, and click-button, we retrieve all exemplars of enter-date. In Mind2Web, we encode the website name, domain, and task description as metadata for trajecto- ries in the training set, and retrieve the corresponding exemplars of matched metadata for few-shot learning. Using the exemplar memory, SYNAPSE can automatically determine which exemplars to prompt, which lays the foundation for general-purpose and adaptive computer agent.
4 EVALUATION
We evaluate our approach on two benchmarks, MiniWoB++ and Mind2Web. The MiniWoB++ task suite integrates diverse tasks that mirror the intricacies of real-world human-computer interac- tions (Shi et al., 2017 [iu et al., 2018). Although is a relatively simplified environment, current computer agents struggle to solve several challenging tasks, such as book-fighh,, click-checkboxes- soft, and use- aus-autooooplee.. Mind2Web (Deng et al., 2023) is a realistic dataset containing human demonstrations of open- open-domain tasks from various real-world websites, such as Airbnb and Twitter. It mcasures in-the-wild generalization across different tasks, websites, and domains.
4.1 EXPERIMENTAL SETUP
To ensure fair comparisons, we query the same APIs of LLMs as in the prior work. In the MiniWoB++ experiments, we query 3.t-3.5-tt turboo------o 0301 and run 50 episodes to produce the results for each task. For Mind2Web, the default LLM is gpt-3. --tturbc-6kkk0613 We configure the temperature to O, i.c., greedy decoding. We use text -ee.t--mbeding-dde-eeeee the embedding model. For efficient retrieval in large-scale memory, we use Faiss (Johnson et al., 2019) to store embeddings and perform similarity search.
as
For the environment setup in MiniWoB++, we follow the configurations of RCI (Kim et al. 2023) The state space is the raw HTML code, while the action space includes click-xpath, move-mouse type, press, and click-options. The main metric here is the success rate of the agent accomplishing tasks. In Mind2Web, the observation space is the HTML provided in the dataset, and the action space includes click, type, and select. The metrics we measure include element selection accuracy (Ele. Acc), the success rate for each step in task (Step SR), and the success rate for the whole task (SR) against the human-annotated ground truth. The Mind2Web dataset is split into a training set and three test sets: Cross-Task, Cross-Was Website, and over tasks from the same websites, unseen websites from similar domains, and completely unseen domains in the training set, respectively. We only store the training set in the exemplar memory.
evaluate generalizability
4.2 BASELINES
We conduct extensive experiments to evaluate the performance of SYNAPSE in comparison to the SOTA approaches on +iniWoB++. For BC+RL baselines, we use CC-Net (Humphreys et al., 2022) and Pix2Act (Shaw et al., 2023). both of which combine large-scale BC and RL. In terms of fine- tuning baselines, we compare with WebGUM (Furuta et al. 2023) and WebN-TS (Gur et al. 2023a), two language models fine-tuned on a large number of demonstrations, As for ICL methods, the baselines are RCI (Kim et al. 2023) and AdaPlanner (Sun et al. 2023), both of which incorporate self-correction solve 54 and 53 MiniWoB++ tasks, respectively. We also include human scores
sourced from Humphreys et al. (2022) for additional benchmarking. In Mind2Web, we compare with MindAct (Deng al. 2023), the current SOTA ICI. method in this benchmark.

---

Published as a conference paper at ICLR 2024
MiniWoB++ success rate across tasks
MiniWoB++ performance
Synapse
Human
Fine-tuning
1.0
oC
Average success rale
1.0
0.992
0,962
0.935
0.935
0,929
0,906
0.803
0,665
0,184
0.305
Average role
0.5
P0
WatGLL
0.8
I
9.6
0.6
Xwazh
0.4
cc Net BO1
0.2
0.0
Syлаpаe
RizzAo!
CG-Net
Human AdaPlanng""
RCI
VeeGuM
PizAAc (BC)'
WobN-TS CC-Nei (BC)
10"
10.
15
157
10.
10a
Amount of ex exemplars per task
Figure 3: SYNAPSE is the first ICL method that achieves human-level performance in MiniWoB+ It outperforms previous self-correctiog methods, including RCI and AdaPlanner. A comprehensive task-wise evaluation is shown in Appendix [A] *Pix2Act and AdaPlanner are concurrent with our work. We excluded them from the right-side figure due to their overlap with CC-Net and RCI. The outlier tasks are determined with an interquartile range of 1.5.
4.3 ANALYSIS ON MINIWOB++
Fig. 3 showcases the average performance of the various methods across tasks. With mean suc- cess rate of 99.2%, SYNAPSE achieves human-level performance and outperforms all baselines on MiniWoB++. As shown in Fig. 4a SYNAPSE outperforms BC+RL SOTA, especially on text- processing tasks, e.g., terminal and text-transform, while being more flexible. Compared to ICL (Fig. 4b) and fine-tuning SOTA (Fig. 4c). SYNAPSE achieves better performance on all tasks. is particularly noteworthy that its superiority does not rely self-correction. In comparison, RCI and AdaPlanner, although equipped with self-correction, fail to achieve comparable performance.
Net
PCI
WAGCUM
(a) SYNAPSE VS CC-Net.
(b) SYNAPSE VS RCL.
(c) SYNAPSE Vs WchGUM.
Figure 4: Task-wise comparisons between SYNAPSE and various SOTA methods. For clarity, the bars in each figure are arranged in ascending order based the success rate differences. Tasks not reported by other methods and those with differences less than 0.05 are not included.
Compared to previous ICL methods, SYNAPSE overcomes limitations related to context length, complicated states, and multi-step error accumulation (Fig. 4b).. First, SYNAPSE outperforms them on tasks requiring detailed state understanding, such as click-collapsible-2 click-tab-2-hard, and count-shape. These results highlight the effectiveness of SYNAPSE in accurately capturing complex HTML states, while RCI consistently performs worse. Second, SYNAPSE significantly improves performance on tasks involving many steps or repeated actions by reducing the error accumulation in multi-round LLM queries, such as use-autocomplett, use-spinner, etc. Third, SYNAPSE can solve tasks that these approaches cannot handle due the limited context length, such as book-flight and click-pie (Fig. 5). Moreover, it generalize from cxisting exemplars to unseen tasks because it does not rely on task-specific exemplars. Therefore, SYNAPSE can solve 64 tasks, more than all previous ICL methods, using exemplars from only 48 tasks.
Most the observed failure cases are due to incorrect reasoning by LLMs. For example, in some cases of the count-shape task, the LLM miscounts the number of target items, leading the selection of an incorrect answer. Similarly, in the text-transee sot-trranfoomm task, the LLM occasionally produces actions with characters similar to, but different from, those in the HTML (e.g., recognizing jrpf as jrfp).

---

Published as a conference paper at ICLR 2024
4.4 EVALUATION ON REALISTIC WEBSITES
We also validate the effectiiveness of the state abstraction principle, TaE prompting, and exemplar memory in the Mind2Web experiments. The current SOTA agent in this benchmark, MindAct, leverages clement-ranking model filter the top-50 relevant elements the clean observation. uses MCQ to recursively query the LLM to select an action from five candidates until one action chosen all options incorrect. Although MCQ-formatted exemplars outperform direct gener- ation, MindAct often struggles to choose the correct element (Deng et al. 2023). In contrast, in our experiments, both SYNAPSE and its variants use direct generation, more straightforward prompt- ing scheme. As shown in Tab. I we incrementally equip direct generation with state abstraction, trajectory prompting and exemplar memory, which achieves improvements of 32%, 50%, and 56% in average Step SR across three levels of generalization based on GPT-3.5, respectívely. SYNAPSE also achieves 2.5x Step SR compared MindAct on average based on CodeL.lama-7B.
Table 1: Mind2Web results and ablations with CodeLlama-7B (top) and GPT-3.5 (bottom). We gradually add state abstraction, TaE prompting, and memory to demonstrate the effectiveness of each component. In SYNAPSE top-ranked elements in clean observations, outperforming the MCQ- MindAct. SYNAPSE w/ state abstraction TaE further shows the benefits of using trajectories as exemplars. In both these variants, we use static few-shot exemplars. Finally, we encode the training set as memory and retrieve exemplars via similarity search, which further improves performance.
state abstraction, we simply use direct generation with fewer
prompting used in
Method
Cross Task
Cross Website
Cross-oomain
Ele. Acc
Step SR
SR
Ele. Acc
Step SR
SR
Ele. Acc
Step SR
SR
MindAct (CodeLJamma-B))
11.2
7.7
0.4
12.4
9.0
0.6
13.8
9.9
0.2
SYNAPSU w/ abstraction
21.4
15.5
0.8
13.5
15.6
SYNAPSE w/ state abstraction TaE
27.0
24.5
1.2
21.4
17.9
0.0
22.1
19.9
0.7
SYNAPSU w/ abstraction TaE memory
29.1
3.2
19.7
MindAct (GPT:3.5)
20.3
17.4
0.8
19.3
16.2
0.6
21.6
18.6
1.0
SYNAPSE state abstraction
29.8
SYNAPSE w// state abstraction TaE
32.8
29.2
2.0
28.0
22.7
0.6
20.0
26.2
1.8
memory
24.2
4.5 ABLATION STUDIES
In this section, perform ablation studies evaluate the effectiveness of three design choices of S SYNAPSE and provide insights into the contributions of each. These findings validate the effec- tiveness of i) state abstraction in handling complex states and providing more exemplars within the limited context, ii) TaE prompting in enhancing the capability of multi-step decision-making, and iii) exemplar memory facilitating generalization across tasks.
Ablating State Abstraction. In Fig. we illustrate the consistent performance im- provement achieved by state abstraction in MiniWoB++.. Due to the limited context, LLMs without state abstraction struggle to solve tasks with complex states, e.g., book-flight, while SYNAPSE effectively overcomes this. This technique also improves performance on tasks with ambiguous descriptions, such as email- mabox-nl-turke increasing the success rate from 52% 100%. In Mind2Weh, direct generation with the top five elements improves the success rate by 32% compared to MCQ with the top-50 elements (Tab. []. although reducing A leads to a drop in recall from 86% to 53%. These results emphasize the effectiveness of state abstraction in helping LLMs generate actions under com- plex states and vague task descriptions.
state abstraction
W/ state abstraction
Suiooss
Ded th clae leght thitog inpias
Amm aertoo 2
Clok lonk meaas
State abstraction enables solving Wo tasks with complex states (e.g., book-flight). also improves performance by providing more exemplars (e.g., email-inbox).
Figure
8

---

Published as a conference paper at ICLR 2024
Ablating Trajectory-as-Eeemplaa Prompting. TaE prompting significantly benefits LLMs in tasks with long horizons and repeated actions such as guess-number, use-spinner, and use- autocomplete, which are particularly challenging for SOTA ICL methods. For example, RCI has success rates of 20%, 88%, and 58% for these tasks. In contrast, SYNAPSE solves them with success rates of 100%, 100%, and 98%. This improvement is attributed to the comprehensive history and temporal action abstraction introduced by TaE prompting. In Mind2Web, TaE prompting further improves the average step success rate by 19% compared SYNAPSE with state abstraction only. also outperforms MindAct by 50% across three levels of generalization, as shown in Tab,
Exemplar Memory Enables Generalization. Exem- plar memory is crucial for generalization, especially for similar computer control tasks. Rather than previous methods that rely on task-specific exemplars. SYNAPSE's memory can automatically find relevant exemplars for the task hand. It enables the agent to identify and utilize exemplars to generalize from email-inbox-nl-- -nltuukk and
The average distance of re- trieval and Step SR improvements of memory (A) in Mebd2Web.
Table 2:
Test set
Distance
Agrr
Acidetaama
Cross-Task
17.0
1.4
1.9
WebsstWebsite
24.3
L.0
related tasks such as email-inbox. It also adapts to different layouts, e.g., from multi-layouts to ondii-orderings. Consequently, SYNAPSE solves 64 tasks in MiniWoB++ with demonstrations of only 48 tasks. It achieves better performance not only for the predefined tasks but also for new tasks outside the scope. For the 16 unseen tasks, the average success rate is almost 100%. For tasks within the scope, the memory mechanism also accurately matches relevant exem- plars. SYNAPSE only encounters a 2% failure rate in click-tab-2-haat due to exemplar mismatches. In Mind2Web, the memory further improves SYNAPSE with state abstraction and TaE prompting, leading to a 6% improvement in Step SR for cross-task and cross-website generalization. However, the improvement is relatively marginal in cross- generalization, which might be attributed to the LLM being influenced by exemplars from unrelated domains. The average distance (similarity scores) between retrieved exemplars and the target task in three Mind2Web test sets are shown in Tab. _ A greater distance implies less similarity. This accounts for why memory does not aid in cross-domain generalization, as these domains are entirely unseen.
Cross-Domain
32,9
0.2
-0.2
DISCUSSION, LIMITATIONS & FUTURE WORK
In this paper, we introduce SYNAPSE, a computer agent with prompting technique that is broadly applicable to any decision-making task. SYNAPSE addresses three main challenges faced by current computer agents: the limited context length, the unexplored exemplar structure, and task-specific exemplars. has three key components: i) state abstraction, which converts raw states to clean ob- servations, reducing the tokens of each state to allow for more few-shot exemplars, ii) trajectory-as- exemplar prompting, which prompts the LLM with trajectory-level exemplars to improve multi-step decision-making and iii) exemplar memory, which stores and retrieves exemplars to enable general- ization. Our method outperforms existing methods on 64 tasks in the MiniWoB + benchmark with 99.2% average success rate without self-correction. It is particularly noteworthy that our method successfully solves the challenging book-flight task in MiniWoB++ SYNAPSE also achieves a 56% relative improvement over SOTA ICL methods in Mind2Web, a real-world website benchmark, at three levels of generalization (cross-task, cross-website, and cross-domain).
We acknowledge that there exist limitations in our framework. High inference latency is a major concern due to the use of LLMs. Using our prompting scheme to distill a more responsive, task- specific agent from existing LLMs could be remedy. Another concern is our dependence on the quality of exemplars. Therefore, training a zero-shot computer agent with instruction tuning (Wei) al. 2021, Chung et al., 2022) based on our prompting techniques is a promising research di- rection. Also, the memory module can be seamlessly combined with human intervention for user customization and task adaptation, which opens up an avenue for future research. In addition, the memory structure and retrieval process can be further investigated for better generalization. It is also worthwhile to combine our methods with compositional generalization methods (Zhou et al. 2022] Wang et al. 2023a) to solve more complicated tasks. Finally, while SYNAPSE currently operates on text, it would be interesting to explore multi-modal and video understanding capabilities to tackle more challenging tasks, e.g., pixel-based Android control (Li et al. 2020: Toyama et al. 2021)

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We would like thank the OpenAl Researcher Access Program for granting us API access for this project. This research is supported by the National Research Foundation Singapore and DSO Na- tional Laboratories under AI Singapore Programme (AISGAward No: AISG2-GC-2023-009)) This research is also supported by the National Research Foundation, Singapore under its Industry Aligmment Fund - Pre-poote pre-tooiii (IAF-PP) Funding Initiative. Any opinions, findings and conclu- sions recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, all resources such as code, prompts, and agent trajectories have been made publicly available bttpps://itzhen github. 1el...oo//yye L///yynape Additionally, details on LLMs and environments are available in Sec. 4.1 and Appendix B
ETHIC STATEMENT
Autonomous agents with computer control capabilities have inherent risks, such performing dan- gerous operations and producing spam. To mitigate these risks, our experiments restrict agent inter- actions controlled environments (MiniWoB+ +)) and static dataset (Mind2Web). Recognition and vigilance of these risks are crucial, especially when conducting real-worke experiments or deploying these agents in practical settings.
REFERENCES
DeepMind Interactive Agents Team. Creating multimodal interactive agents with imitation and self- supervised learning. arXiv preprint /2.0222//2.03763, 2021.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang, World of Bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, pages
PMLR, 2017.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang, Reinforcement learning on web interfaces using workflow-guides exploration. In International Conference on Learning Representations, 2018.
Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam Santoro. and Timothy Lillicrap. A data-driven approach for learning to control computers. International Conference on Machine Learning, pages 1466-94822 PMLR, 2022.
Dean A Pomerleau. Alvinn:
An autonomous land vehicle in a neural network. In Advances in Neural Information Processing Systems, pages 305-313, 1989.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, 2020.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thougge prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 21824-24837, 2022a.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsumori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions Machine Learning Research, 2022b. ISSN 2835-8856.
10

---

Published as a conference paper at ICLR 2024
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022a.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks, In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards generalist agent for the web. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:220..023111 2022.
OpenAL. GPT-4 technical report. arXiv preprint arXiv: 2023.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 0e0.3308./299 /2900, 2023.
Sherry Yang, Ofir Nachum, Yilun Du Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foun- dation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.04 129 2023.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, et al. Augmented language models: survey. arXiv preprint arXiv:2302 07842,, 2023.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint 23X::2309.008644 2023.
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris. Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. Proceed- ings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1 1-22, 2023.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118-9147. PMLR, 2022.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et Do as I can, not as say: Grounding language in robotic affordances. In Conference on Robot Learning, pages 287-318. PMLR, 2023.
Wenlong Huang. Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng. Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning, pages 1769-1782. PMLR, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages
2022.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuur- mans. Claire Cui, Olivier Bousquet, Quoc V Le, et al. Least-to-most most prompting enables complex reasoning in large language models. The Eleventh International Conference on Learning Rep- resentations, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
11

---

Published as a conference paper at ICLR 2024
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Re- flexion: Language agents with verbal reinforcement learning. Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In The Twelfth International Conference Learning Representations, 2023.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Vi- sual ChatGPT: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671 2023a.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hug- gingGPT: Solving Al tasks with ChatGPT and its friends in Hugging Face. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu. Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai- Aei Chang, Ying Nian Wu, Song-Chum Zhu, and Jianfeng Gao. Chamelcon: Plug-and-play compositional reasoning with large language mod- els. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, and Dongkuan Xu. Re- WOO: Decoupling reasoning from observations for efficient augmented language models. arXiv preprint arxiv:2305. 18323333...... 2023.
Jacky Liang. Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages
IEEE, 2023.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. ProgPrompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11523 1550.. IEEE, 2023.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. In NewrlPS 2023 Foundation Models for Decision Making Workshop. 2023a.
Yue Wu, So Yeon Min, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Yuanzhi Li, Tom Mitchell, and Shrimai Prabhumoye. Plan, eliminate, and track languaa models are good teachers for embodied agents. arXiv preprini arXiv:2305.0244 2023b.
Izzeddin Gur, Ulrich Rueckert, Aleksandra Faust, and Dilek Hakkani- Tur. Learning to navigate the web. In International Conference Learning Representations, 2018.
Sheng Jia, Jamie Ryan Kiros, and Jimmy Ba. DOM-Q-NET: Grounded RL on structured language.
In International Conference on Learning Representations, 2018.
Izzeddin Gur, Natasha Jaques, Yingjie Miao, Jongwook Choi, Manoj Tiwari, Honglak Lee, and
Aleksandra Faust. Environment generation for zero-shot compositional reinforcement learning. In Advances Neural Information Processing Systems, volume 34, 2021.
Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin V Huang, Aakanksha Chowd- hery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. Understanding HTML with large language models. In ICLR 2023 Workshop Mathematical and Empirical Understanding of Foundation Models, 2023a.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):548--551, 2020.
12

---

Published as a conference paper at ICLR 2024
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. WebGPT: Browser-assisted question-answerte with human feedback. arXiv preprint w1X1:22112.09332,200
Shunyu Yao, Howard Chen, John Yang, and Karthik R Narasimhan. WebShop: Towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems, 2022b.
Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. AdaPlanner: Adaptive plan- ning from feedback with language models. In Teirth-sevnnth Conference on Neural Information Processing Systems, 2023.
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova. From pixels to UI actions: Learning to follow instructions via graphical user interfaces. In Thirly-seventh Conference on Neural Information Processing Systems, 2023.
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world WebAgent with planning, long context understanding, and pro- gram synthesis. In The Twelfth International Conference on Learning Representations, 2023b.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. AgentBench: Evaluating LLMs agents. In The Twelfth International Conference on Learning Representations, 2023a.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, al. WebArena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 2023.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for Android device control. In seirth- Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Bryan Wang, Gang Li, and Yang Li. Enabling conversational interaction with mobile UI using large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1-17, 2023b.
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03 2023b.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. JEEE Transactions Big Data, 7(3):535- 547, 2019.
Hiroki Furuta, Ofir Nachum, Kuang-Huei Lee, Yutaka Matsuo, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instruction-finetune foundation models. In The Twelfth International Conference on Learning Representations, 2023.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representutions, 2021.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling ciiruction-finttnne language mod- els. arXiv preprint arXiv:2210. 11111116,1 2022.
Yang Li. Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile UI action sequences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8198 8210, 2020.
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. AndroidEnv: A reinforcement learning platform for Android. arXiv preprint
2021.
13

---

Published as a conference paper at ICLR 2024
A ADDITIONAL RESULTS
Table 3: Per-task mean success rate comparison between SYNAPSE, humans, and baselines. The data for human participants are obtained from CC-Net (Humphreys et al. 2022). The aggregated performance of the baselines prior to CC-Net is represented as others (Gur et al. 2018: Jia et al., 2018: Liu al, 2018 Shi al. 2017). We attach the average success rate and the number of solved tasks for each method at the bottom of the table.
Tisk
AdaPlanner
PixtAct (BC)
CC-Net
CC-Net (BC))
WiKIUM
WiKN-
Others
hisect-angle
0.80
books flight
1.00
0.87
191a
10fa
0.87
0,00
0.98
0.00
1.00
circe-iicle
1V9.
0.82
m/a
m/a
11/a
0.93
0.80
7V5.
2013
1.00
chouse-date
1,00
0.97
0.79
0.06
0.97
0.12
0.13
0.00
1.00
choose eas
chouse Jate medium
0.98
m/a
pla
ma
0.99
0.26
0.60
0.00
m/a
chouse-lit
1.00
0.98
1.00
1.00
máa
0.99
0.19
0.24
0.26
0.26
cizcle center
0.52
0.97
0.36
0.98
click buttin
clack buteooeee---eee
1.00
0.94
1.00
1.00
0.99
1,00
1,00
0.44
1.000
1.00
1.00
click checkboses
1.00
0.97
1.00
1.00
L.00
0.99
0.98
0.32
1.00
0.96
1.00
click cosck-ceckbboe--e hang
1,00
0.87
0.91
0.99
1,00
0.71
0.00
0.99
0.22
0.84
clack ceeckcchcbbooesssttt
0.72
0.80
0.95
0.988
0.54
0.94
chockcheckboeess transfer
1.00
0.98
1.00
0.98
1.00
0.76
0.99
0.36
0.99
0.63
0.64
click-collapede collageible
1,00
0.99
1.00
L.00
0.94
0,80
1,00
0.81
0.98
0.00
1.00
clickcolllpsibbe collspsbble-2
0.17
0.99
click
click-dlalug
1.00
1.00
1.00
1.00
1.00
0.12
1,00
0.95
1.00
1.00
1.00
2lickciialog22
1.00
0.99
1.00
1.00
1.00
0.75
1,00
0.88
0.43
0.24
1.00
click-link link
0.86
0.99
0.59
1.00
1.00
clack-ttem
0.78
0.37
0.37
0.13
click mack-menn-12
4/6
0.98
m/
mla
0.83
0.52
1V5.
0.16
optionn
1.00
0.99
1.00
L.00
L.00
0,00
0.99
0.21
1.00
0.87
1.00
0.51
1.00
click
0.01
0.00
0.07
shicksshades
1.00
0.91
1.00
L.00
0.99
0,76
1,00
0.04
0.00
0.00
0.99
clicksshape
0.98
0.38
0.98
0.75
0.94
0.19
0.95
0.11
0.77
0.53
0.64
click
0.95
0.74
1.00
click-tab-2
0.97
0.85
0.98
0.98
0.27
0.95
0.18
1.00
clickcab---caary
0.99
mla
0.99
0.77
0.99
0.61
IVa.
a/a
m/a
click 2--dard
0.98
0.96
0.76
0.78
0.97
0,00
0.98
0.19
0.95
0.12
sick-rab-2-mediun
click sest
1.00
1,00
1.00
100
click 2est
1.00
0.99
1.00
1.00
1.00
1,00
1,00
0.95
1,00
1.00
100
click nest-rest-randeerr
0.99
1.00
1,00
1.00
0.94
click
100
copy-paste
0.04
min
0.79
8.04
IVA
000
pay-epase-2
1.00
0.94
nda
0.66
0.01
IVA
a/a
0000
count shape
0.50
0.82
0.40
0.50
0.70
0,00
0.85
0.21
0.68
0.41
0.76
drng-box
0.99
0.99
1,00
9.61
IVA
1:0
drag cabe
a/a
0.59
0.79
0.23
IV1.
8110
dma--ttm
0.98
1.00
0.85
1.00
0.61
IVA
drag-items
0.41
deg--nems-grid
0.87
0.89
0.60
0.98
8.05
000
dmg-shiges
W/
0L06
0.98
0.96
0.99
0.26
IVA
0.92
drag sori-ammbers
0.92
0.95
0.08
1.97
0.11
IVL
0.66
awail-inhoo
inill-nnbox-delete
0.99
1.00
0.99
IVA
smail-inhox-fee Jnboo-forward
n/a
0.96
1,00
9.01
m/a
email-inhot- seail-innot----r-ta forwand-nl
1.00
0.91
1.00
1.00
1,00
0.00
1.00
0.60
smail-nnbox-foxe www.-nooo--ooad----auu.
amail-inbox- impostant
0.99
0.99
IVA
aia
amail-inhor-nl-aade
1.00
0.09
0.98
0.90
1.00
0.05
0.98
0.23
0.93
amail-inhot noseroll
0.96
1.00
0.13
IVA
email-inh mmail-nnbox--re reply
m/a
0.95
8.11
enter-tate
1.00
0.97
0.96
1.00
1.00
0.50
1.00
9,00
1,00
0.00
1.00
enter-jussweel
1.00
0.96
1.00
0.98
1,00
0.02
1.00
0.97
1.00
cater-text-2
0.97
0.98
8.04
A/a
0.00
cnter--tst--lymaaee -lymami
1.00
0.97
1.00
0.96
1.00
0.39
1.00
0.98
1.00
enter-lime
1.00
0.58
1.00
0.96
1.00
0.78
0.97
0.04
0.00
0.00
0.90
0.31
find-word
1.00
0.8R
8.05
1V/a
0.00
Tocns-test
1.00
1.00
1.00
1.00
1,00
0.99
1.00
1.00
1.00
focus-text-2
1.00
0.99
0.94
1,00
0.96
1,00
1.00
1.00
grid-coardisate
goess-mumber
0.21
0.11
0.20
highlight-eel
1/a.
0.97
mia
nla
n/u
1,00
0.51
TV/a.
mla
0.90
highlighi-est--2
0.97
1,00
0.40
1/11
0.13
identify shape
login-user
1.00
0.96
1.00
1,00
0.00
1.00
0.82
1.00
1.00
0.04
0.68
0.98
1.00
0.02
0.99
0.72
e/a
moving items
0.88
0.13
0.78
multi- butis
14

---

Published as a conference paper at ICLR 2024
multi-ordeiing
0388
148
navigate tree
0.98
0.98
0.36
0.82
0.99
0.00
0.99
0.33
1.00
090
110
number number-checkbaee checkboues
199.
0.96
0.80
0.00
000
010
21
resice sexiarea
185
0.94
0.90
100
1011
0:20
right angle
scroll
0.57
1.1.
0.23
0.04
00
scvoll-text-2
1/00
INR.
0%
search engine
1.00
0.97
100
1.00
190
10
1.00
0.15
0.96
100
simon says
0.62
0,00
0,02
IVG.
0.00
simple-nlgehra
simple anttmmetic
social media
1.00
0.96
0.98
0.82
0.90
0.00
1,00
0.21
100
social-medis-all
1,00
0.89
1.00
1,00
I1775
$1000
0.31
0.00
100
media some
0.98
0.00
lest next-ecinor
6/6
0.88
0.95
0.11
IVA
1/1
0.0
test transform
0.98
0.56
0.80
0.92
0.91
0,50
0.19
2012
0.00
unicode-test
6ia
use autocompplete
0.98
0.98
0.58
0.88
0.99
0.95
1,00
0,00
0.98
0.22
0.98
uas-colerwhied
0.50
0.97
0.98
0.95
01.88
IVA
181
1.00
uso-ilides22
./6
0.97
63a
1.00
0.09
0.95
0100
IVA
0.15
use-spinner
1,00
0.98
0.88
0.90
191a
1,00
0.44
0.11
0.07
0.17
visual-adftiion
Average
0,992
0.935
0.906
0,929
0.962
0.665
0,935
0.305
0.802
0.484
0.646
sulved
Human
AdaPlannor
shape
Metteernn..rrgg. Sarge.
Po2Ac
WeEN-TS
Syларe
Figure 6: Task-wise comparisons between SYNAPSE, human, two concurrent methods (AdaPlanner and Pix2Act), and WebN-T5, a fine-tuned LLM. Similar to the result Sec. [ the bars in each figure are sorted in ascending order based the success rate differences. Tasks not reported by other methods and those with differences less than 0.05 are not included.
B ENVIRONMENT DETAILS
B. MINIWOB+
To ensure fair evaluation, we use legacy branchlof MiniWoB ++ as in previous work. Of 64 tasks that we test (see Tab. 3). 48 are provided with exemplars, and 16 are unseen tasks. The unseen tasks are choose-list. click-checkboxes- sareckkboee-trrnsffrr. click-test-2, click-test, copy copy-paste, login-uset. click-checkboxes. click-option, click-tab-2-haad hhard,, fomboxxfrrwwrd-nl... nl, email-inbox. enter-text, multi-one muuti-orderings, simple-arihmmetic, and unicode-test. The average num- ber of exemplars per task for RCI and SYNAPSE 1.32 and 3.45, respectively. However, the number
email-inbox-forre
15

---

Published as a conference paper at ICLR 2024
Mind2Web recall with different
Mind2Web performance with different &
100
Cross- Posss-Task Task
Cross-Website
Step success rate (89)
Cross-Domain
Recall (8)
15-
10
Pross-Task
20-
Cross-Wer Pross-Websitte
Cross- Cross-Domain Domain
0-
0
30
The number of elements after state abstraction (4)
The number of elements after state abstraction (k)
(a) @ecall@k:
(b) Step success rate
Figure 7: Ablations of the number of elements after state abstraction in Mind2Web. The backbone language model here is CodeLlama-Instrruce
Book to arehhapent createay light Book bu chaupuaa or uoway lign: Book the diseppent susss light
180002116
1041/0011
1000020015
One-Way Hight
Book One-Way Highh
Whdt Oet 2016
From:
Hartford, CT (BDL)
Book Fight for $616
Traverse City, Mll (T)
Oet 2016 (8OL)
10:05/2018
Search
Search
05 2016
(a) MiniWoB++
(b) Mind2Web
Figure 8: An illustration of MiniWoB + and Mind2Web.
of exemplars for each task varies. For simple tasks, use similar number of exemplars as in RCL To solve complex tasks (e.g., book-flight, we use more exemplars, which lowers the average data efficiency. For example, in book-flight, we include exemplars for better robustness. Moreover, RCI relies on self-correction (task, state, and agent grounding) while SYNAPSE does not. The suc- cess rate RCI will decrease by around 50% when one of them is missing. Also, one of the benefits of our method is that we can leverage more exemplars in context because of state abstraction.
We use the raw HTML code to represent the agent's state space. We provide the action space of the computer agent in the system prompt. The action space consists of five keyboard and mouse operations, following the prior work (Kim et al. al, [202). We use code generation to ground the actions generated by the LLM, the same as the concurrent work (Sun al. (023). We exemplars that specify to click on the first element of all matching ones to avoid clicking on multiple items specified by the xpath. The action space described in the system prompt:
Role: System
You large Language model trained navigate the web. To accemplish the task, methods in the lowing Agent claas to generate act until you need the new state to
proceed.
class Agent:
def _init____e_ (self, arga):
Action: type a string via the keyboazd def type(self, charecters: stz) Nones
Actions click HTML element with valid xpath def elick xosth (self, xpeth: strl:
Actions:
press key on keyboard, including: enter, space, arrowleft, arrewright, backspace, arrowup, arroиdown, command+a, commandec commancrv
16

---

Published as a conference paper at ICLR 2024
def press (self, kypetype: ste) -> None:
Action: click option ITML element in List with valid xpath def olick_ootion xpath: str):
Action: move mouse curser on an HIML element with walid xpath def mevemouse (self, xpath: atr):
In MiniWoB+++, there are two types of state abstraction prompts. For scenarios where the context can handle multiple states, we utilize state-observation pairs. These pairs are collected alongside the trajectories. The states are obtained from the environment, while the cleaned observations provided by humans. To further reduce human efforts, we can also automatically infer cleaned obser- vations given trajectories using LLMs.. In scenarios with complex states where explicit abstraction not applicable, we implicitly abstract states by pairing task descriptions and state- sarte-pasiig code. Similarly, the task descriptions are already given by the environment. The code here is collected via zero-shot sampling from LLMs (GPT-4). The code is tested by executing it with states derived from the environment and having its results reviewed by humans. The state abstraction prompts are retrieved from the memory, together with trajectories.
B.2 MIND2WEB
Mind2Web offers a comprehensive collection of over 2,000 tasks from 137 websites in 31 different domains. The task descriptions in Mind2Web only provide high-level goals. It intentionally avoids detailed, step-by-step instructions, encouraging the agents to autonomously understand and perform tasks, rather than simply following instructions. With Mind2Web, agents can interact with wehpages in way that goes beyond the basic operations of searching or reading. They can click, select, and type into website elements. Such interactions mimic typical human activities on websites, thus expanding the range of tasks the agent can perform.
Mind2Web provides sets for three levels of generalization: cross-task, cross-websites, and cross- domain. To evaluate cross-task generalization, Mind2Web provides 252 tasks from 69 websites, where the agent has possibly interacted with these websites perform similar tasks. The cross- website test set consists of 177 tasks from 10 unique websites not seen in the training set but belong- ing to domains familiar the agent. This setup tests the agent's ability to navigate new websites within known domains and tasks. The cross-domain test set, containing 912 tasks from 73 websites, requires the agent to adapt to completely unseen domains in the training set. The remaining data, consisting of 1,009 tasks from 73 websites, are used for the training set.
C PROMPTS
In this section, provide several examples of our prompting and LLM responses for illustration. We also open source all trajectories in our project website
C.I TERMINAL
To finish the terminal task in MiniWoB++, the agent needs first list the files and then delete one with a certain extension. By using TaE prompting, SYNAPSE completed this task in four steps: two for state abstraction and two for TaE prompting.
C.I.1
STEP 1: STATE ABSTRACTION (EXPLICIT)
Input:
> Role: User
State:
<div id="wrap" dat:-wab_ rof="22 dato-wab_epa- opa=""0*>>
<div id="quezy">Use terminal below delete file ending with extension <apen
"lass="bodd>>.ppe patppppp////pp////// <div ida"area" data-wob_refte retu"3" "ota-xoo_ppaa""g">eet data-wob_ratwaar eata-wab_epa=*eee
17

---

Published as a conference paper at ICLR 2024
id="cerminai" data-wab_ref="g" data-wob_ep=""d""
<div "d="terminal-hoade aata-wch_rofs"" datt==we__ppp=""""==eriiri
<div "d""terminal-coneeni rat=-woo_ret=rj= data-wob "ata==oo_epp="e00" epa=*e0*>
<div class="Lemminal output" data rob_sef==8" data vob_eps=""0">
<span "output" data-wch_ref=*9" data-wob_eps=* "tt==oocccp"""""""cccccmmm type help for list of
available cormanda.</span
</divo<atv "taas"tttmllllouupppp data-woo ret=r10" data-woo_ret="10
data-wob dots
<epan clasg=*outpon ouuput" srro-wcbrrof=[[ data-woh "otp login:
Kay 2023</agar>
</oiv><dvv "isas="terminal-1
data-woh_"Pfff "of="112" ssta-kob_ops="add
capan class="user" data wob_re[="13" data "obeeeeeeee"""""assssssssmm
<apen id-"sctive-input
"assmadodmdd""///gg <apan =d="sppttflicker" data-wot_ref="1e ctt=xob_epo="ed>>
</div>
</div>
</div>
<input type="text" jd="temminll-trget "ata="oo_ret="13 sata-wob_eps="e0"
</diva</div>
</div>
observation:
terminal below delete end' the extenston .png
State:
id-"wzap" seta-wob_ref-t22 bata-wob_eps="eD
<div id="query*>0ls the terminal below to delete file ending with the extension
class="bold">, "orool/dd..rr//pao////ii
<div id="area" asta-rob_ref="]]
data-wob_rof="d "ata-wob_eper"e0>>
<div id="terminal" aata-wob_rof="5 data-woh_opa="ade
<div id""termina-hhede: data-web_
<div id="Lesminal--oom contents" data woa_ref=17 date wob_eps=*e0*,
class""teriiaaoootppi data-wcb_refete data-wcb "er
<span "lass="output" data web_re[=*9* data wed_epa=te0">Xelcom Type help for 1ist of available cormands.</span>>
</div><div cleep="herminolooourpp
snt--oob_rff=100 aata-wob_spa==a0""
<span clags=*outpin data-wch_ref=""] data-woh, ops="""">statt data-woh_opss""00
login: Toe May 12 2023</apar>
</divesdiv class="Lecmine!" line" wob_:e[==122"
"pp"""00>>
<apan class="user" date-wob_ref="21
vata-web_epare "eb.
<apan id="active-inpur" class=* "comand"><<appnn
<epan -lickkee wob wob_seobbee"""1"" sbta=voobepp"""e">>/pppp
</div> </div>
</div>
<input Lype="Lext" id="terminal target wob_ref="15" vob_eps="=0">
</div></div>
</div>
Observation:
Jse the teminaal below to delete a file anding with the extension .py
<div id="uzap" data-wob_red rob_ret="2" data data-wob_est eeb_
<div id="quezy*hUse
terminal below to delete extensions/span>..// </diis/
<apan class="bold"5to
id="area" ssta-wob_refang" caca-rob_opas"s(M
data-wob_rafe*at data-wob_opa=-e0"
<div id="cerminai" data-wob_ref-"5" data-wob_opsm"e0""
<div id="teemiall-sdde"" data-wch_ "erp"""___"""""" data-wch "pd"p=""""""""""r"iiii""
<div ide"terminal-contee: data-woa_ret=t1)
sitaawoob_epa-tee
<div clazs="Lezning output" data wob wob_:ef="g" data wob veb_eps=*e0">
<span "lang="outputs data-wcb_ref=*9> data-web Meoeemmmme Type help list
available <ormanda.</span>>
</div><div claas=*terminll-ootto data-woa_rera11rr data-wob
data-=ot_eps--600"> ops""ad">>
<epan class="output! dota-wch_rof="ll "11" data-wob "oppvoooopss"d">aass login: Kay 2023</apam>
viviv cleas="terminal-
data-woh_rafaa112
data-woh, refa"12" data-wah, aora-wah_ops="dd"
<apan class-"user" wob_ret0t11 data *tgl
<apan sd""active-ingue class= "oomm"""mm/""/pppp
<apan "d="tupttfllicke"" rata-wob_refs"LE deta-wooo deta-wab_epa="ed
</div> </div>
</div>
<input type="text" jde"terminll-aarget aeta-woarret="199 sata-wob_epss=tpp
</divs</div>
</div>
Observation:
Use the terninal below to delete file that has na file extension.
State:
<div id="scap" robe-wob_ data-vob_eps="edt
<div "s= the terminal below to delete a file ending with the extension <apan class="bold", httehooodd..gg//paareeeee wppgs//pamm<//ivv
<div id="area" data-wob_ref="3 data-wob "et==st data-wob_rafs*4q "ota= id="cermina!" data-kob_roffan <div "antterminl--hhdee
data-woh_
bata-wob_set-rae bata==/o_epp-rreoseemmee
<div id="terminal contents" data woa_ref-r? date wob_eps-*
<div class=*essmiaalooopprr data-wcb_reds""" "5
<span class="oulput" data seb_ref= data *eo"====""""ecomm Type help for 1ist of
commands. samaadd.<</pan>>
18

---

Published as a conference paper at ICLR 2024
</div><div clee=="temminl-oootppp" Beaa--ooa_ree____a_ data-wob_opee"e"""
<apan "aaag="outpur" fata-wch_rofa*lin data-woh, data-voh_oppe""en oppe_"""==""""""t"" login: Sun Sep 24 2023</span>
s/dive<div claas="terminll-Iie"
aata-wob_sef="pp data-wob "ap===ab_eps""d""
<epen class="user" date wob_ref="13" date "eopppp=
<span id="active-input" classe "oommcddsm///pppii
<apan id="sxput-llicker" saaawoo_rrff=Le @ata-vab_apa="dd">0
</div>
</div>
</div>
<input Lype="Lext" id="terminal Larget" data woo_ret="15" data wol__eps="e0">
</div></divs
</div>
Observation:
Output:
Role: Assistant
the terminal below delete file ending with the extension .pp.
C.1.2 STEP TAE PROMPTING
Input:
System Prompt (as ment lonee In Sec.
Exemplary Trajectory
Task:
Use the terminal helow to delete file ending with the extension
Trajectory:
Role:
User
Observation:
the below delete ending the extensios .ng
Action:
Role:
sgent..yye('"1.
sgent.press (1 ('eater')
Observation:
the
onding with extension. .png
userl$ 18
alloy.pag script.zip shark.html sudo.gpg
Action:
> Role: Assistant
Note that it is not allowed to use <.xxx here. alloy. .nng is the file ending with the extenston .png- Type complete filenama, agent, Cype (" rm alloy (ngg))
agent.press. ('enter")
Exemplary Trajectory 2
Role: User
Task: Use the terminal below to delete file ending with the extension .py
Trajectory:
Observation:
ending extension. -py
Action:
Role: Assistant
egent.Lyper ('1)')
sgent.press ('enttrr)
Observation:
Use the terninal below te delete file ending with the extension .py 1s
bash.gpg imago.gif nintendo.py
Action:
Role: Assistant
19

---

Published as a conference paper at ICLR 2024
loce it is net allowed use +-xxx # nintando. is the file ending with the extenston Type the complete filenome. agent, cype (" rm nintendo-py")
egent preas ('enter")
Exemplary Trajectory 3
Role: User
Task:
Use the terminal below to delete a file that has no file extension.
Trajectory:
> RRole: User
Observation:
Jae the terminal below to delete file that has no file extension.
Action:
Role: Assistant
agent.type ('1s')
agent.press ( ('enter') 111
Role: User
Observation:
Use the terminal below Lo delete file that has no file extension.
user18 1s
encrypted.sh chrame.txt search
Action:
Rote that it is not allowed to use x.xxx here. search is the file that has file extension while encrypted.sh and chrome. not. Type the complete filename. agent.type smarch") agent.press ("enter") *..
Current Trajectory
> Role: User
the terminal below to delece file ending extension .9pg
Trajectory:
User
Observation:
Use the terninal below to delete a file ending with the extension 9p9
Action:
Output:
Role: Assistant
... agent. .tpttttyypee ("
('etter))
C.1.3 STEP 3: STATE ABSTRACTION (EXPLICIT)
Input:
> Role:
User
<div id="uzap" Mate-wob_ref-12 baaa-wob_eps=eeb
<div id="quory"s0se terminal belaw to delete anding extension <apen
<div id="area" data wob wob_reI="g" data wob rob_eps=""("><div data wob_ref="4" data wob_
<div id="cerminsi" dat==wab_ref="5" "sta==ob_eps=">>>>
<div id="Lerminal-heed <div "ie"temmmnal-ootten:"
data
wob_---2 data-wob
<div "hasss
data-wcb__ett ree="" data-wcb_epa-ff Type help
<span elaag=* "output" data-wch_rafa-ge data-woh_epa=* available <//pand
</divo<div cless="Lerminal output" dats wob_ref==10" data wob sph_ep"""""">
<span class="output" data-wch_ref="]1 stta-voh_eps=""d">ie login: Kay 2023</span>
</div><div elemn="t
data-wch_ wotsooo_ooss'"" data-woh_opo="dd0t
<span clasg="used "use=" " data-wab "twmbbeea==0000ssc//o <apan claas=*commans saaa-wob_rof="_ </dlv><ddv claas="torminalloontt hata-woa_ref-*ee data-woh_eps-"d0""
20

---

Published as a conference paper at ICLR 2024
<epan Mlass=*outputs data-wcb_refs* data--ob__oprr""""""aall script.zip
sido-gpg</spont
s/divesdiv cleas="termmnal data-wob_ sef="12" data-wob "taa=wob_eps"ee""
<epen class="user" date wob_ref="13" date web.
cless="command" "commadd" rata-wob_ref="E" data- <apan id="active-input" classe
<apan id""ssput-flicker" saa-woo_rrf==14 aaaa-vob_ppa="00 "laas="hide"><//""""""
</div>
</div>
</div>
<Input Lype="Lext" id="Lerninal Larget" data roo_ret=""5" data wob_eps=="e0">
</div></divs
</div>
Observation:
Use the terninal below to delete a file ending with the extension .png
useris
alloy.png script.zip shärk.atml suco.qpg
State:
<div id="weap" fate-wob_ref=tt data-wob_eps=ted
<div "Uue the terminal below to dalete a file anding with the extension <apan class="bold"> htppp///...p///ppaa//ttt
<div id="area" data wob "el="""""" data wob "bb_eps="e"*eedi dats wob_rel=*g* data wob_eps=*e0">
id="cerminai" data-wab_ref="5" data-wob_ep=""""">
<div id="Lerminal-hees
data-web_sef-- data wotpepp=pp-s""00 be..pppoto....emiiial////di
<div db""terminal-contenn
data-wob_ref-r?? wote=wob_eps-"e0""
<div clags="terminal-oupppt
aata-wcb_reef=" data-uch_eptr "p
<span "lapg="output" data-wch_rafargh tata=woa_opa=="00> come! Type help for Hat of
cormands.</span>
<ddiveediv cless="Lecciinl outpssiilll-ouuuut" dats wob rob_ref==10" data wob "e"_""""""">>>
<epan "asppu" "oupput" data-wcb_reé=*H
atta-wcb_eps="e0"
login: Kay 2023</apan>
</divoediy class="Lecminel line" data wob_ree="11"" datd rob_eps="ed">
<span class="user" "at"="""_re=="33" data-wob data- "pereppppeee <epan claas=*=omman
data-sob_refs"_5" dats-wab_epe http:/bbeaatee"/l////pppp
</clv><ddv "tras="ttrminlllouuaa dats-woa_ r0*=*20" data-woh, gata-wob_ops-"ad0 ops"""0">>
<span class=*outyy *oupuut" span>
wob_re[="211
image.gif ninleado
</dive<divv clea=="teminalllline" data-wcb "ee=-ccbreef"12" data-rob_epe="e0">
<epan class="user" wote=""t_re data date-wob_epe "ob
<span id="active-input" classn"command" data- ref="lE"" reta-seb_reft"le" d
<apan idd"spptt-llekke" data-woh_ refe"1"" dsta-wob " claas=""><<span>
</div>
</div>
</div>
<input type="text" td="temminlltarrge"" tata=woa_rof==15! "ata==oh__pp==:00>>
</div></div>
</div>
Observation:
Use the Lecninal below to delete a file ending with the extension
user\$
bash..gg image.gif nintendo.py
State:
<div id="yrap" date-wab_raf="2" data-wab_eps==edl
<div id="query*>use the terminal below to dalete a file that has <apan clasa="bold">ro file oxtenatons/span>.
<div id="area" deta wob wob_ref="3" data wob "pdt==c data wob_rel=*yn data wob_eps=*e0"> id="cerminai" data-wob_ref-"gn "st"
<div id="teminal-hoader data-wch__se zof="6* data-wch_epsster http.//pppsto/oo////////////
<div cde"terminll-content data-woa_ refurye data-wob, opa-too__p_
<div clapsi"temminll-oupprr <span clagg=*output" hattawoh_raffrge tata=woa_epa= come! Type help for Hat
aata-wcb_rafsaee "e0
available cormands.</spp </diveediv
detta-obb_re==11 data-wob- data-wob_ "o0"sp""""""
kepan class=*output" "oupput" data-wch_refa"
deta-wob_eps==ee0t
login: Kay 2023</span>
</civ><ddv cleas="terminalllle data-wch_ refa*in" certa-wob__epee ops=""d*>
"etra=xo__ep==edd""
<span slass="user* data-wols "ot=woo_rre="11" eata-wob
<span cooma=""ommand" vot_-eobbee data wob_egg=te0"5ier//paa
</divo<div "outtttott data-woa_ret=*20 ret=*20" data-wob "ot_--woo_pps="d""
<apan class=* "output"" span>
wcb_ref=*21 web_.ees"""""ecccyyeell
chzome, searche/
</div><divv cleas="tominal-linne data-wch, "00==" "p==="a
<apan class="user" bata-wob_reravn data-woh_eps-ee "aroo_spss"0002sssrrss///0
<apan id="active input" class=*oommand" "commadd" data wob_reff"11 data wob wob_esg-teStt<//pan
<span id-"input-flicken data-wob_ref="10 "ot===" class=*"S</ppan>
</div>
</div>
</div>
<input type="text" td="tommial-torgat data-wch_refst data-woh_aps="dd0
</divs</div>
</divo
Observation:
Use the Lecninal below te delete file that has no file extension.
user
21

---

</div> Published as a conference paper at ICLR 2024
enceypted.sh
search
State:
<div id="uzap" aita-wob_ret-"e data-wob, 6eta-wob_epa--ep>>
<div id="query*>Use the terminal below delete file ending with the extension
class="Eoid">... "egpddd".gggg/p//m/////a/
<div id="area" deta-tot_ref=""" data-sob_opa="ec4
rata-wob_raf=r4 dbta-wob_opa="ed
<div id="terminai" tata-wob_ref="g data-wob tata-wob_eps-"gd""
<div id="teeminal-haade" data-wch_sefs"G drr.-wih..pre
aata-woo_refaryy aatt-wob_eppa-eep
<div class="Cerminal output" wotaswob_ssef-gg data web eps-*e0">
<apan clasg="output" data-wcb_eef-t9" data "ebeeeeee
Type help for list of
available <ormand.<</spa>>
</div><éiv claag="tarminal
data-woa_rete
ref="10 oops-ooh_pps=a00
<apan class- data-wob_ref="ll data-wob
tata-wob_eps-"dd""t eps-""d">ttst login: Sun Sep 24 2023</apar>
</div><div cleas="temmmalllline aatt-woh_refatjl data-wab
class="user" sardwoo_rre""1d" cata-wob_ "oroooope"""""aaarrr"""
cspan clase="commant" date wob_ref="11" data wob tebbeeesee
</div><div "less""eemiial-ounn <span class= "output" data-wch_raf==22 span>
data-wob 20 data-wob_ops"""0"> asta-vob_epa="d00
media.html window.gpgs/
</divecdiv cless="Lerminal line" data wob__ef- wob_ref==12" data wob <ps="""*>>>
<apan class="user" tata-wob_ref-113* data "ob_eppe"""">"use"""
<apan "de"actiie--ngue" comma="" data-wob_re reb"_ref=16" data "et rebieereeee
capan -iiippt--iiikke" data wob_res=bee""" data- =0 class=""sc/ppan>
</div>
</div>
<laput type="text" id="terminal target" data wob_rel="15" data wob_eps="=0">
</div></div>
</dive
Observation:
Output:
Role: Assistant
Jse the Lecninal below Le delete rile ending with the extensios, lon .p9 user 1s
index.rb medic.html windew.gpg
C.1.4 STEP 4: TAE PROMPTING
Input:
System Prompt (same as step Exemplary Trajectory Isame step Exemplary Trajectory Isame as step
Exemplary Trajectory
step
Current Trajectory
Role: User
Task: the terminal below to delete file ending with the extension .9pg
Trajectory:
Role:
Observation:
Use the terninal below to delete endtnng with the extenston .gpg
Action:
> Role: Assistant
agent. (' ls')
sgent.press ('enterr)
Role: User
Observation:
Jse the terminal below to delete a file ending with the extension .9p 1s
index. rb media. html window.gpg
Action:
Output:
> Role: Assistant
10
Koto that it is not allowed to use s,xxx here.
window. gog the fi endting of extenator gpp-- Type the compl Tename.
22

---

Published as a conference paper at ICLR 2024
agent. type rm windew-gpg') agent preas (enterr)
C.2 BOOK-FLIGHT
To finish the book-flight task in MiniWoB++, the agent needs first search the flight and then book one with certain condition. SYNAPSE completed this task in four steps: two for state abstraction and two for TaE prompting. In step 3, due the long HTML state, we implicit state abstraction with code. Temporal abstraction here reduces the number of LLM queries, leading to lower latency and higher success rate in Long-horizon tasks.
C.2.1
STEP 1: STATE ABSTRACTION (EXPLICIT)
Input:
Role:
User
State:
<div id="urap" data-wob_rett "or""oo___""""
epg=*e0">>
<div id="query"ABook the <span
one way flight from: cspan class ""ood"baartford, cr</span> to: <epan class="oodds<ree//ppppp on <apan class="bold"
-10/05/201ss/apamn </dli/ </div>
<div id="area" deta-rot_ref-ll data-wob_epe""d"5
<div id="menu" data-wob_rof="4" dsta-woh_opp=""00"
<h2 Id="header-book" dsta-wob data-wab_reff"s ref="s" cata-wob data-xob_epa-"g00 opa=*""">Rook Your One-Way Flight</h2>>
class="ingut container*
wob_re(="e"
wob_epa=te">>cing id-*flight from"
"aag=="lighttiappp ui -uuttcopplete-inpute type="tex:" placeholder="PPom: sutocomplete=*
dsta-tot_reff="" data-kob "tt==voo__pp==s""=///
<div allas="/npur-oontsiine data-woa_refs ref=*" dats-wob "es_abb_pp=*=00><<>i id=*flight-to"
"figh="iigtt-iaput t--uuscooplete-iippr type="tex:" placcholder="io: autozampleten*
off" data-sob_refaaa
<div aerdeepatuur--onttalie
asta-wob_ref="1
data-wob_ed data-wob_epa-"ep>>
<div cless-ede header" wcb_ref="11"
"bbeepaate Dates/div>
<div "asst=iippttcoonte
data-wob_ref-"12"
tata-wcb_epa="e00s "eo==""e0><iipps"" "d="datppickes
class="flight-ingu hasDstepicker" type="text" readonly="" cata-wob_rof-tne deta- "___eps="""><</1v> </div>
<61v saaas="aarrhoottttalnee data-wob_ refa"14" tats "80*>
<bution 1d="search* data web_set="15" data
</div>
</div>
rd="results" class="hlde*></div>
</div>
</div>
Observation:
Type the flight from: 'olaztSord, CT' Lo: (IVC)' (Baattford, CT is nos airport code. (TVC) sirport code.),
10/05/2015.
State:
<div id="urap" aata-wob_rer-re data-wab, "00"""0"""
<div id="query"oBeok the
flight Trom: cspan class
"holld"smn<<sppmm bo: <apan class="bold "s NY</span> on <apan "laas="boldt
>11/05/20164 >t2/00//211<//ppan>..kk/
<div id="area" feta-wob_ree--2 daca-wob_ed eps="""">>
id="menu" aata-wob_refs""" dsta-wob_eps="e0
<h2 hd="headerbbook" aata-wob_ref=55
"at==oob_p===_"">000 Your One-Way Flight</h2>>
data-woa_re(=*s aotta-wob_epata0020 id="fLight from" <div "apass==iiptt-cntainnen
class="flight input ui autocomplete inputt type="tex:" piaceholde:="From:: sutocomplete=*
off" datt-wobbree="" data-wob_epas "ps==="s</""
cless="input
wob_re
wob_egg=tetthcinput id-"fLight
Mlags="llght-iappe ui si-autscomplete-ingue type="tex:" piateholder="!!! autozompletes"
dsta-wob_ref-e data-woh_aps= "sa"o___p=""""""di"""
<atv "aaas="dpartuuro-on ner" stta-wob_refat1g jata-wob_epa-ree
<div deparuueee header" data wob_res==11" data wob
Date</dive
<div watn-iippccooostiaae
1o= data-web_eps "ep"=""""""inn"""""""""""""""""" 1d=*aatepicker
claas="flight input hasDatepioxer" type="Lext" readonly=th data-wob_red wob_ref="13" data"
vob_eps-"e3><</di>
</div>
cleas=""carchh-cnntttie" data-woh_ refa"14" data-wab, htas-wob_opa="pt0 "60==
<button 1d="search" wota-wob_set="13
</div>
</div>
<div id="results" class="Hide">//dk
</div>
</div>
23

---

Published as a conference paper at ICLR 2024
type flight from: "IENT) to: "ialip, Ne" (I8NT) sirport code. islip, is airport code.), and select the date 11/05/2016.
State:
<div id="ysap" date rob_ref=""" data vob_ees=*e0*>
<diy "u=" the cspan "aaa""""oo"""caprre///// one-way flight from: <span class ""bold">Anvik, AK</span> sepan "ood""""oo" <apan claas="hold" /12/2//2015<//pam..co
id="area" deta-mot_ tata-rob_apa="e0"
<atv id="menu" cata-kob_refatae data-woh_ops_""d""
<h2 id="header book" deta wob_ref-""" data wob_ epoo Your One Ray Flight</h22 <div "lagh-ttiihtt-ipptt ud-=utacompiete-imppu type="tex:" placcholder="From: cutocompletes
Ld="flight frightt-feen
off" bata-sob_refa"" pata-sob_opa="00t00 <div "iaas="inpuu-contalll
data-woa_reds data-wob_epa-tee tpo=_eppa id="fLight-to!
clazs==flighttinppt -t-autacomplate--mmpi type="tex:" placcholders*to: autocompletest
ofr" tata-wob_ref="g
<div cless="departure centainer" data wob_ref="1j" date wob_eps="e0">
<div class=tapprtuure-heede data-wob_refe"l1 abz==oot_gge"""""">>atttt Date</dive
<div class=" "anaa==inptttcoottta
date-xob_raf="12 aata-wch_eps=*ddd ">0">=<ppppppt= td="datopccker
clags="rlight- haapsteploxer" type="text" readonly="h tata-woh_refa"1 data- wob wob_epsee <ps="""">></div>>>>>>
</div>
<div claas="search container" wota-woo_eee"] dela-wob wob_wow-wweeeeeeeeee "0">""0">
<button id="search" data-wcb_sete wcb_sef="15" data "ota
</div>
</div>
<div id-"resulta" class=*aide"></div
</div>
</div>
Observation:
Output:
Role: Assistant
Type the flight from: 'Anvik, AKT (MOT)" (MOT code. Anvik, AX is not airport codo.), and select the date 12/23/2016.
C.2.2 STEP 2: TAE PROMPTING
Input:
Syatem Prompt (same as the terminal task)
Exemplary Trajectory
Role:
Task:
Book the chespest one way flight from: ford, CT to: TVC on 10/05/2016.
Trajectory:
Role: User
Observation:
Type the flight Irom: CoartCord, CT' (DVC) (Baztford, CT is airport code. (TVC)
is an sirport code. and se lect. the date 10/05/2016.
Action:
> -Role: Assistant
..1
Type the flight from: "arrtford, cT tos (TVe)
(eett.cllot_xxatthh///-BBB flight-fron! agent.type ('llertford, CI*)
agent.preas ( ("arrowdwwn")
agent.ppess ("entez")
cgent.cllick_xpathh" (tspathh"////iiaaa fligatttiitttt
agens.ttye ("VV)" (TVC)")
the airport code, you must parenchesas
agent. ("srrowdewn")
agent.press wwwessppessssseeeeeee ("eatez")
Select the date 10/05/2016
date: 12/31/2016 (December October=2), Click "Brev"
for in (2) agent.alict "egpattallokk_pxchh("////// (etthhh"/////ttttteeeeeeo
click the date kg'
phantttclickkppattt///aaaaa () ()*"3("")
Observation:
The pro ce the cheapest one-way flihh $92
24

---

Published as a conference paper at ICLR 2024
Action:
Role: Assistant
egent oegel..iii...aattr///ttttttttttd Book Elight For )321)"1
Exemplary Trajectory 2
Role:
User
Task: Rook the shorteat cnn-way flight froms Abf ene, Patarshurr//lleerater 10/15/2016.
Trajectory:
Role: User
Type the Elight from: 'Abilene, TX bos rtt Fecersburg/Cleerwater PL (Abilene, TX is not airport code. St Delersbuug//Clearwateer FL not an sisport code.), , and select the date
10/15/2016.
Action:
Assistant
Type the flight from: rabilene, TX' bot Petersburg//leerrattt
(tt.hlickkkahhh///aaaaaaaa riidgtrigggtteeeeee..eiii Flight-Trort (gent.typpe"Abblene IX")
agent.preas ("arrowdown")
agent.press (ww.:/eessseeeteeee (*enter*)
agges.clicc..paatthh/ flight-to"
agent.type (*St eetersbbrg//leawwaeer. FL?)
agent.pprass ("arooodoo""
egent.preas
Select the date 10/15/2016
Curcent date: 12/31/2016 (ectmber--Ottoeerra)) click on "Frev' twice
in (2):
agent.al 1ok http.llhll__xxphhhh_111111 ("s-th""///ttttll
Click the date <15'
agent. eteet.elick_ppath""" ("///t/xx//////////a/ ()=1)/ ()=(15(]")
>
Role:
User
The price of the shertest one-way flight is $680
Action:
Role: Assistant
agent.oliot http://lllooppaahhh////tttttteee Rook f1 $680'
Exemplary Trajectory 3
> Role:
User
cne-way flight
islip, on 11/05/2016.
Trajectory:
User
Observation:
Type the flight from: ' (MNT)' to: "Talip, we (INN) 1s an at rport code. Tel ip, NY is not ar
airport code.), select date 11/05/2016.
Action:
-Rooe:
Type the flight from: (MNT) to: 'Islip, NY
sgent.elick_xpahh""// (// (fid-fflightt [light-fron']] from!]")
For the airport code, you must use parenchesea agent..pees ("aIcowdown")
agent.type (* (MIT)")
agent.ppres (arresss""etteee ("enter*)
agent.ttye (*Tsl NY*)
sgent.press ("azzoxxoo""
agent.press ("eater*)
Select the date 11/05/2016
spedt.clik..pathh///11eede (tte///e///""ddrattetteeeee"""""
Current date: 12/31/2016
(December-Novemmaerr) click (prev" cnce
htt.:allckk.pathht//1ffeefe Click the date kgt cpentt.eicck_ppattt"//aaaaaaatt spea:..liel..pathhh"/oaaaaaaa
Observation:
The pro ce the shortest one-way flihh
25

---

Published as a conference paper at ICLR 2024
Action:
Role: Assistant
egent heg.eeellllk..patt///btttttttttr Book Elight for (341"1*)
Exemplary Trajectory
Role: User
Task: Rook the cheapeat cnn-way flight froms LRE Kanhattan, 12/12/2016.
Trajectory:
Observation:
Type the flight from: "ILKE) to: "Monhattan, (LKEE) on airport code. Manhattan, not airport code.), and select the date 12/12/2016.
Action:
-Role: Assistant
Type flight from: (IXE)
'Hanhattan,
(talicckppathhh//-lll_ the atrport code, you most use parenchesea agent.tyye (* (IRE)") (*
fligat-from""")
agent.preas ("azzowdown")
sgent.press ('e.tteeess""eee......... ("enter")
agent.ellok_xpathheW// (1 [liggtttttt
Lyee Seeent..ype""Saniiie ("Menhattan, K3*)
agens.press ("erroxdexn")
agent-preas (*entar*)
Select the date 12/12/2016 egent_ (taphhh""///1iidddaaeeeeeee
Curcent date: 12/31/2016 (Dezember-Decemberre) Ne need to click on "Frev! click the date '12'
bpeent.:lic_.ppathp. /t/ft///////////////// ()= aggns.click_xppthhoe [8id= (ta
Role: User
Observation:
The price of the cheapest one-way flight is $152
Action:
> Role: Assistant
() Book tiight for $1821
Exemplary Trajectory 5
Task:
Book the chespest 2light froms MMH to: BTR 11/16/2016.
Trajectory:
Role:
Observation:
Type flight (WBB)) to: (BTR) ((MH) (BTR) airport codes.), select data 11/16/2016.
Action:
> -Role: Assistant
Type (t//pptthh///11ie airport code, you must parencheses
Irom: (MMI))
(BTR)'
<peet.cliic_xpptth(////
from! 1
agens.ttyp (WMH)
agent.preas (rontt.peess"asrseeet ("srrovdewn")
agent.preas ("enter")
flight to
("
For the airport code, you must use parenchesea
agent.ppess ("arsoxdww"")
agent.press (twp:eesssseeeteee (*enter*)
Select data
cke
Curcent dale: 12/31/2016
(December November=1), Click on "Prevr once "iiick_ppathh"///ffittttl (reiicceee"Beeeee [")
click on the date <18'
speett.click.pppthhh//aaaaaeee () (/a-
...
Role: User
Observation:
price of the cheapest one-way flight is $36
Action:
Assistant
agent.ollet htt.k/kpthh//////tttooooo Book flight $36* ]
26

---

Published as a conference paper at ICLR 2024
Current Trajectory
> Role:
User
shespeat ene-way flight Anvik, toz on 12/23/2016.
Trajectory:
Role:
User
Observation:
Type the flight from: 'Anvik, AK to: (MOT)" (MET is an nirport code. Anvik, AX in not an 12/23/2016.
airport code.),
Action:
Output:
Type the Elight Trom: rAnvik, AK bos (MOT)
<gent.elick_xpath""" ("/-[idh" fiidgf flight-frortt agent.type ("pnnikkkk Anvik, AX*)
agent. ("szeowdcwa")
agent..press ("entteerr"" (*enter*)
agpms.click_xppahh"" (/
agent.typo (MCT))
For the airport code, most use parencheses
agent.press ("eate=*)
Select the data 12/23/2016
Current date: 12/31/2016 [pecomber-Dacemmeerie No need to click 'Frev!
agent @gentttoloo__ppthh::: ("/aattxx ()-223)) ()-2333") agent. http::elllccktahht//tt//a//a search"
C.2.3 STEP STATE ABSTRACTION (IMPLICIT)
Input:
Role: User
Fille: the following raw HTML state into clean observation via code based on the task.
<div id="wrap" hats-wab_ref="a data-wab_ega=*er opa="""0>
id="query*>Rook <epan class="hol d "esccl""pssss///ppa one-way flight from: <epan class ""bold"sAnvik, AK</span> to: <spar. "oll=="bo"""MOTTsC on <span class="bold" (12/23/2015<//pamm </div>>
<div id="area"
ref="3" woob_eps=""">>
(a very long webpage containing flight information)
</div>
</div>
Brite code (between Uhzee backticks) Lo extract the price of the wanced Licket. Firat check IC we want shortest or cheapest fl ight. Parse the HTM. vaing faautifulSon Toop through available flights, Check Lime en whether we want the shortest the chaapest, Ke have searched with airports end the Keef integer price wariable "obs" in this format: 'The prica of the (preference) one-way flight is siprice J.', The string of the raw state is already in the variable 'state" so do not repeat the state in the code.
(split "I" (x")
price based
date,
need
ticket
string
are examples:
Write code within three backticks to Book the cheapest cne-way flight from: Hartford, eT 10/05/2016.
to:
Code:
"pyython
frem b54 impert BesutifulSoup soup = fueuutifulSoupptt..
'html.parser')
preference 'cheapest' flost (' ('inf")
bast_flight_ppst
Loop through tha find_all("aiv" clasa_="(lightt);
flights
for flight in the preference is cheapess, compare flight prices price int (iiighh..iddduuutttt
class_""Eligtt ("sata-price"))
if prica < bast_ bast__flight_rries best__fightt price fest_flight_pree price
price
oba I'The prica the (preference) ane-way flight "(prica)
27

---

Published as a conference paper at ICLR 2024
...
Write oode within three backticks to Book the shorteat one-way flight from: Abilene, TX lo: Rele:sburg/Clessveeeer FL 10/15/2016.
Code:
"''ython
from 284 import Resuti Soup
soup feastifulsoup (state, 'html-parsar') preference 'shortest'
best_ILieht_duratie
(loat('in2")
best_flight None
# Loop through all the availsble flights
for flight in soup. findalindall ("div", class_="flightt)
the preference la shortest, compare fLight durations
duration flight.ffndd ("div" closs_=ttemo-burttior cttparrttttr"........o.. .text.aplit
hours int (duration [=-1]] if "h' in curation[0] else
minutes int [du]ation[]]]:: 11) 'a' in duration[1] else
total_minutes hours 60 minutes
if total_minutes
best_fligitt_duee
best_fifgnt_duratt best_flight flight
total_mimost
price (beet__light.deeee Einight.iidd/uutttet! class==fEightppiize"" ("data-p:ice"]) obs I'The price of the (preference)
flight de $(price)
Now, write code within three backticks to Book the chespest one-wаy flight from: Anvik, 12/23/2016.
Code:
Output:
** "ppythor
from 284 import DesutiCulSoup
'html.parser') preference 'chaapest'
soup BesutifulSoup
beat_f feat_flight_pra
flost ('
Loog Lhrough all the availsble flights
flight soup. (" class_="flightt);
the preference la cheapest, compare riight prices
price int (flight...idd" (inghhtttEddd"/utt"" class_-"Elight "ish=="iettttpcce ("data price"))
best_flight_prrie: if nest_flight__riie = price
best_tlignt_prion
obs T'e price of the (preference) one way flight is $(pzice),' *.A
C.2.4 STEP 4: TAE PROMPTING
Input:
System Prompt (same as stee
Exemplary Trajectory Isane
Exemplary Trajectory Isame as step
Exemplary Trajectory
step
Exemplary Trajectory Isase as stop 2)
Exemplary Trajectory Isame as step
Current Trajectory
> Role: User
chespest ene-way flight Anvik, toz on 12/23/2016.
Trajectory:
Role:
User
Observation:
Type the flight Irom: "Anvik, AKT Lo: (MOT)* (MOT is an airpors code. Anvik, AX is not an 12/23/2016.
airport code.), select
Action:
Rolee:
Type the flight from: 'Anvik AKT cos (MOT)'
sgent.elick_xpath""" (1/pathh///-Iiir (fiiidifiighteeeee flight-fron!1
agent.type sxype("Anvike ("Anvik, AX*)
agent.press ("arrowdcwn")
agent.prees ("entar")
Srpnt.olick_xpattee (//o//xpatthht///1itre http://ittaarfllggttoooooo fiight-trr1")
28

---

Published as a conference paper at ICLR 2024
agent.type (MCT)
the airport code, you must parencheses
agent. preas ("arrowdewn")
agent. (*enter")
Select the date 12/23/2016
Current date: 12/31/2016 (Dcomber-Decombar-l nead click 'Frev' # click on the date '23'
agent. agent.elick_wee wtthh ()=1331))
httpp:lllll_ppthhh///1aaae searchh""))))
Role: User
observation:
price of the cheapest one-way flight is $65.
Action:
Output:
>
Role: Assistant
.*.
agent wttp:..cllkk_ptth//////c [tex: () Book flight for 3"1
C.3 TEXT-TRANSFORM (FAILURE CASE)
We also demonstrate failure case in MiniWoB++ where the LLM incorrectly recognizes jrpf as jrfp when typing the transformed text into the text field.
C.3.1
STEP 1: TAE PROMPTING
Input:
System Prompt (same as
taak)
Exemplary Trajectory
Role: User
Observation:
id-"wzap" date-wob_ref=-21 data-wob_egs-
<div id="scl* class="rezp" data-vob_refie data-wob_ "ps===//pp text below
id="query*>Type
taxt and preas Submi:.</div>
<div id="area"
data-kob_eps="et""
<div id="captcha" data wob wob_ref""" data wot_-wob_epas
wewp:oeeeeee""r///eppe stasf=mt:::ssorm: (kewX((dep)
skewy (-26deg);* data-wcb_ref=""
styles"tranafomm skewx
{ 4deg) skewf( 21deg); " data wob_re(="7= data wob "ak_epp="""">er aaapa//a//aaaapppaaaeee style="Lzand form (-eee)-
(30deg);" date-wob_ref-"91 tronaform: skowx 1-8deg] skewy (-25deg);" dacc-wab_ data-wab_ "opp""""pp//ppppp /ppp////c//////>> <atv id""forn" style="margin- maw.::oo
style=*
dasa-wob_ refu"m dat data-wob_ae epa="
eD*>
kinput type="tex:" id-"tt" lata-wcb_sef=_1: wita-wob_eps="e""
<button id="subbtn" alaa=""scondry-actttii styles tnrrgnniat
data-rob_rof="12-
"essvoot </div>
</div>
</div>
Action:
> Role: Assistant
agent.click, httr.cccllckxpattt////1eeee 1") locate the input box by elicking en it
agent.type rezp') Type the string input
wttp://llo__xpahht////c [81der subben" [
Exemplary Trajectory 2
Observation:
id="urop" hotz-wab_ref=ty
<div id="so1" claas="aoy" bata-wob_set="s data-woo_epa-te <pa=<=0"></div>
<div id="query*SType the text below into the text field and press Submis.</div>
<div id="area" deta-wob_red rer="""""""""" daca-xob_epae"et"
<div id="taptcha" acta-wob_rofswe cata-
(-24deg); dat==w tett atyls="transform: (kewx)
style="transform
(12dag) (-1 lata-woh_refste " data-woh_epsi "os==""""c</ppa>>coe "transsmr
(1dex skewY (-23deg); data-wab_ret- sat-wobbreftgr
29

---

string into the element with the id. Published as a conference paper at ICLR 2024
id="form"
dasa-wob_ref=ng) data-kob_eps="e)
Kirput type="text" Id="tt" data-wob_ sef="100 stta-wob_eps="ed"
<button id="suabin" secos=yey action" style="margin tep:17par" data vob_ref-"11" "ospooooops"""""sssssssssss </div>
</div>
</div>
Action:
Role: Assistant
cltcck.patthi///-lliarrar [") Locate the input box by clicking en it
agent. (Maoy") Type string in the input hox
Current Trajectory
> Role: User
Observation:
<div id-"wzap" date wob_ref=*2 data wob vob_eps=*e0">
<div id="scl" class="irpÉ" data-wob_ref="""
<div id="query*>Typ the text below into the text field and preas Submis.</div>
<div id="area" ceta-wob_ret="a
cata-sob_epa-"ett
<div id="captcha" data vob_ref=*g" data vob_epp="e(""<ee atyse=" skewX( 13deg)
skewy (21deg) data-wch_ref-"6
style style="transfoom skewx
(Edeg) skewf (17de)) 1"
vob_re[="7"
style="Lcanstoon
skewX(-13den) skewY (32deg) " data-wob retf"=" data-wob "at__=/=""ed""////ppaaaaa
style="
cransform: skewx (2deg) skewy (19deg); data-wcb_reff data-wob_ "0"
<div id="form" style="aargin-- -enn--effii :f wt 3px/margin-top:1 9px;" hate-wab_bfftta_e data-wob_ epa=** e0*>
<input type="text" id-"tt" data data-wcb_ wcb_see="111
data wob_-wob_pps"e0"
<button id="subbtn" clas="sstoodary-actiio" mtgle=mmarginntoo:llee 16xx1 data-wob_refaty
vob_eps=""d">Subee httpsssbbiit/uuuutttt t/button>>
</div>
</div>
</div>
Action:
Output:
> Role: Assistant
...
Locate the input hox by clicking on it sgent.type j=Ep") ) # Type the string in the input box
agent. wwtpkccpccc.k_phhh.... [/-[///dd/uubbtttt
C.4 MIND2WEB
We only demonstrate one step in Mind2 because real-world tasks and websites are complicated. The observations are obtained using their clement-ranking model. We k to 3 for the trajectories in memory and previous observations in current history and k = 5 for the current observation.
Input:
> Role: System
trained navigate for the next observaticn. Bere is the sction space:
large language
Cutput the action
1. "CLICK [id] ": Click on an HTM. element with its id. TTYPE [id] [value] ":
"Ss_eC" (id) [velue] "; Select a value for an HIML element by its id.
Exemplary Trajectory
> Role: User
Remova the on my cart
Trajectory:
Observation:
'chtml> <divo <a id=131 shopping cart> <i icon of shopping cast /> cdivs <apan> 3 </span> isens </div> <div> 952.97 </div> </a> <ul> id=10012 computer peripherals> div> Corputer Peripherala </div> </a> sa id=10523 electronics> <div> electronics </div> </ul> </div> </atmls>
Role: Assistant
Action:
'CHTCK [131)* ([1nk] Shopping cart CMTCI
30

---

Published as a conference paper at ICLR 2024
Observation: '<html> <div> <div> <img 16=18447 kingspec internal sclid state drive /> <hutton 1d-18445 buttonz <i trash Remove </button> </div> <aivs sa view details> SAMSUNG PRC M.2 2230 2T3 PCIe Gen 4,0 x4, NVX 1.3c Samsung V KAND Internal Selid State Drive (
BSU) N0-V822106/AR </>> <a id#274E1> internel saps </a> </div> </div> </html>'
Role: Assistant
Action: 'CLTCK (18445)* ([button] trash REMOVE -> CLTCK)
Exemplary Trajectory 2
Role: User
Task: Add Wastern Digital internal 885 with iTs atorage and 8CB 8am DOR to the cart.
Trajectory:
Observation: 'chtml> edive <input id-145 search search site /> cul> ea id-10059 computer peripheralaz <div> computer perrpts </div> </a> 15-1 electrontcs>
Electronica </div> </a> </ul> </divs </htal>
Role: Assistant
Action:
[145]* ([searchbox] Search Site CLICK)
> Role: User
Observation:
'<html> <dive <input id-36913 search search site
id-46898 computer
peripherals> <diys Computer Pecipherals </divs </a> <a id-17409 electronice> <divs
Electronica </div> </a> </ul> </div> </html>*
> Role: Assistant
Action: "YPE [36915] [Nestern Digital internel SSD TB) ([ssarchbox) Search Site >> TYP:: Digital internal 3SD > Role: User
Observation: 'chLml> <dive ediv> <input 1d-81831 search search western digital internal sad 1 /> <button 1d-73846> <span> Search </span> </button> </diy> <a western digital> < img western digital
</html>
Exemplary Trajectory 3
Role:
Task: Upgrade the count of the current SSD in my cart to 10
Trajectory:
Observation: '<heml> <divs id=130 shopping cart> <i icon of shopping /> <div> <epan> </span> Itens </div> <div> $85.97 </div> </a> <uls <a id=10016 computer peripherala> <
Corputer Peripherals </div> </a> 1d-10527 electronics> <aiv> Electronics </div>
</a> </ul> </div> </atml>
Action: 'CLICK [130]* * ([link] shopping Cast +> CLICK!
Role:
observation: '<html> <divs <dive id=26807> Add </a> <a> Edit </a> </a> </div> <div <oiv> <input 1d-18580 text gty 1 /> <divo timit 20 </div> </div> <div> <a 1d=26918> Add </a> </dive eldive </div> </atml>*
</a> Edit </a>
> Role: Assistant
Action:
[18560] [10]* [[textbox] gyy TYEE: 10)
Role: User
Observation: '<html> <dive <divs 1d-46342> Add <a> Edit </a> <as Done </a> </divs <alv <div> <input id=38114 text qty 10 /> <dive nimit 20 </div> </div> <dive id=46473>
Add </a> <a> Edit </a> <a> Bone </a> </div> </dive </div> </htmi>'
Action:
*CLICK [38114] ([textbox) qty >> ESTER)
Current Trajectory
Role: User
Taak: Add Lhe cheapest SBD to my cart
Trajectory:
Observation: '<html> <div> <divs <i id=104 site zenu /> <input id=99 search search site /> </
1dw91.9 compater systema>
computer Syst </div> </a> 1d=10010
compucer periphezale> <div> Computez Peripherals </dive </a> sa id-10521 electronice: div> Electronies </divs </a> </uls </divs </html>
Output:
Role: Assistant
Action: 'CLICK (9139] [[link]
Computer Systems CLICK)
31


Reviews:
[{"review_id": "NM6dRxDVIt", "reviewer": "Reviewer_z7sh", "summary": "This paper presents a prompting-based method (SYNAPSE) for computer control that takes in states and outputs actions in a discrete, human-computer-interaction-like action space. The paper is motivated by gaps in existing computer control methods: in-context learning, which struggles with long-horizon tasks and generalization and often needs post-hoc correction; and trained/fine-tuned methods, which are data- and compute-inefficient.\n\nSYNAPSE consists of three components: a state abstraction procedure in which an LLM is used to extract important information from the raw state, thereby significantly reducing token length; \"trajectories as exemplars\" (TaE) prompting, in which full, relevant trajectories (sequences of clean states and actions) are added to the prompt for generating each new action; and exemplar memory in which trajectories and their associated tasks are encoded and stored, to be used as exemplar trajectories when a related task is being planned. \n\nExperiments are conducted on MiniWoB++ and Mind2Web. The MiniWoB++ experiments show SYNAPSE's major data efficiency (and moderate performance) gains over BC+RL and finetuning baselines, as well as performance and simplicity gains over ICL baselines. The Mind2Web experiments demonstrate the same, along with its generalization capability across tasks, websites, and domains. SYNAPSE beats SOTA on various tasks from MiniWoB++, including notably difficult ones for which human-level performance is not achieved by prior work.", "questions": "- What exactly is step success rate vs. success rate?\n- It is inherently interesting that this approach does not need self-correction unlike other prompting-based approaches, but does that offer any significant performance/efficiency gain?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "#### Quality\n- Method is intuitive and effective\n- Experimental setup is useful. It is useful to have one standard benchmark with extensive comparison to baselines and one realistic benchmark on which more qualitative advantages like generalization are shown. \n- Results are promising - the similarity to human performance on MiniWoB++ and the significant increases on generalization show the power of SYNAPSE's TaE and exemplar memory components. \n- Ablations are useful as well to concretely break down each component of the approach. \n- Overall a well-constructed, well-executed paper. \n\n#### Clarity\n- Very well written!\n- Figures are especially useful compared to a lot of papers. Tables as well. Overall, results are not only promising but well-communicated. \n\n#### Originality\nThe paper positions itself well in prior work. As far as I am aware, the specific three components used here have not been combined elsewhere. \n\n#### Significance\nThe performance on existing computer control benchmarks is impressive, especially when comparing to human baselines. Figure 4 particularly tells a story of a significant improvement in capability. The results on MiniWoB++ show only some improvement, so it's unclear how significantly SYNAPSE distinguishes itself, but the results on Mind2Web show a lot of improvement margin.", "weaknesses": "#### Quality\n- The same ideas (gaps in prior work, the three components and their descriptions, the benchmarks being used) are repeated several times throughout the paper. While I very much agree that some amount of repetition is crucial to get a reader to understand what you are saying, in this case it's not just two or three times but four or five, or even if it's three, there's lots of detail every time. It feels as though the repetition is there to fill up space, even if that's not really the case. It gets tiring to read the same high-level concepts over and over. \n\n#### Clarity\n- I don't quite understand what \"step success rate\" is as opposed to \"success rate\" - it's brought up several times as the metric for Mind2Act, but not defined as far as I'm aware.\n- It doesn't seem to be made clear what exactly the state abstraction entails. In figure 1, the only difference between \"raw state\" and \"clean state\" is `<body><h1>…</h1></body>” --> <input id=”where” type=”text”>`. There aren't clear examples in the main text, but intuition about this would be very useful. \n- Not clear how SYNAPSE's history-in-prompt approach is significantly different from Mind2Act's (or lots of other works') to the point of being novel. \n- From both Fig 1 and the text, it's unclear when new trajectory exemplars and example state abstractions are generated - is it per task (that's what I'd think) or per step? \n- Fig 3: might help to have clearer visual distinction of SYNAPSE and the human baseline, especially since the colors aren't always visible. Lines/shading/callout boxes could help. \n\n#### Originality\nThough the approach is technically unique and clearly effective, there are some originality concerns in terms of what the paper claims. For example, the paper positions itself against works like Inner Monologues and SayCan by saying that they focus on high-level planning; however, though RCI uses *even higher* level planning, SYNAPSE is still working with a discrete, abstract action space. Meanwhile, the low-level BC/RL/motion planning+control used for the robotics papers cited are operating in a much more granular, potentially continuous, action space. For another example, the paper spends considerable space discussing how it is unique compared to other prompting-based methods in helping the LLM understand the current state. However, MindAct gives trajectory-so-far back to the LLM when prompting for the next step. SYNAPSE gives observation as well, thereby informing the LLM in more detail about the current state, this just isn't a very original idea. It's an important step past MindAct, but not a big one, especially relative to how much it's discussed as a unique contribution. \n\n\nNits:\n- Page 5, last paragraph: `Obseravtion` is a misspelling", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ElJSw0BJhH", "reviewer": "Reviewer_mGYB", "summary": "This paper proposes a new technique to prompt language models with example trajectories for computer control tasks. It utilizes language models to abstract clean states for long-context conditioning, and design a retrieval module to improve generalization.", "questions": "1. SYNAPSE uses state abstraction to make sure LLM can learn from compact information while ensuring the context length is well utilized. However, in the experiments, SYNAPSE sets k=5 for retrieval. How does the performance change if k grows? And why longer context is not helping here?\n\n2. Table 1 seems confusing. It's non-standard to use \"SYNAPSE w/ state abstraction, SYNAPSE w/ trajectory-as-exemplar, SYNAPSE w/ training set as memory\" to indicate gradually adding \"state abstraction, trajectory-as-exemplar, training set as memory\" 3 components. If you want to demonstrate a component *A* is useful, better to write it in \"SYNAPSE w/o *A*\", as SYNAPSE is a method that combines all components\n\n3. For Table 1 training set as memory experiments, do you add all the training data for 3 scenarios (\"cross-task, cross-web, and cross-domain)? Or only the corresponding training data (e.g. cross-task but same web when evaluated on cross-task setting).\n\n4. For tasks that need code to abstract state information, how can you examine if a code can successfully extract desired information? if examined from task success, it can be because of the suboptimal execution of policy.\n\n5. For the cross-domain tasks, retrieval doesn't seem to help, can you give a better explanation about this result? The retrieval number is set to 5, which is a small number. Is it because the retriever doesn't return the desired examples? Maybe showing some retrieved examples can be more intuitive.\n\n6. \"The action generation starts from prompting LLMs with successful trajectories to warm up LLMs\nwith the dynamics of the current environment\". Is it describing prompting with retrieved trajectories? Or that's some other warmup procedure? If it's cross-task/cross-web/cross-domain setting, how's current environment defined?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing and presentation is clear in general\n2. The proposed method surpassed baselines like CC-Net, RCI on various benchmark like MiniWoB++ and Mind2Web", "weaknesses": "1. Some details are not well explained, see questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8zycPyL01T", "reviewer": "Reviewer_bu8M", "summary": "This paper studies the problem of data driven computer control using large language models (LLM's). It adds three components to improve capabilities of current approaches. It abstracts from the state of the environment by parsing it via an LLM such that only information necessary for the task is kept. It conditions on complete trajectories of similar tasks. It keeps a memory from which similar example trajectories are sampled.  The method is compared against strong baselines on two current benchmarks in the field of data-driven computer control. It shows improved task success rate on both, while often needing less data. Ablations on each component of the method show that the state abstraction procedure is the main reason for the improved performance.", "questions": "- Have you considered building up the memory from scratch by adding successful trajectories over the course of training?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The method is mostly explained well and it is easy to follow the explanation. Experiments and results are presented well.\n\nImproved task success rate on two benchmarks in the field of data driven computer control.\n\nCompetitive baselines from a variety of recent approaches in the domain of data driven computer control.\n\nAblations investigate the effect of each component of the method. \n\nThe state abstraction approach allows the method to be tested on tasks that were infeasible for prior methods.", "weaknesses": "- From the results in the MindAct task it seems that adding any trajectories helps but it does not matter which ones.\n\n- Have you studied the effect on performance on the MiniWob++ environment for different amount of tasks in the memory. The comparison to RCI does not seem fair at the moment (Figure 3)?\n\n- It would be helpful to have an illustrating example of the result of the state abstraction procedure (i.e. a figure of a (state,observation)-pair) as well as an example prompt (maybe in the Appendix). Figure 1 only shows that some <body> and <h1> tag got removed but it's not yet clear to me what the result of the state abstraction procedure looks like. Its also not clear to me where the examples from the state abstraction prompt come from. Are they designed a-priori? If yes how and how difficult is that? Are the same examples reused for all states or also taken from the memory?\n\n- The text states (last paragraph of 4.3) that failed trajectories are displayed in Appendix C, but Appendix C only refers to the supplementary material?\n\n- In the original paper of MindAct, they used 3 in-context exemplars for GPT3.5  but Synapse uses 5, so the comparison favours Synapse. In general I would maybe add a sentence that clarifies that results are SOTA, conditioned on the fact that GPT3.5 was used as the underlying LLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)