Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Web
Authors: Hiroki Furuta
Yutaka Matsuo
Aleksandra Faust
Izzeddin Gur
Venue: Accepted by TMLR ()

Abstract: Language model agents (LMA) recently emerged as a promising paradigm on muti-step decision making tasks, often outperforming humans and other reinforcement learning agents. Despite the promise, their performance on real-world applications that often involve combinations of tasks is still underexplored. In this work, we introduce a new benchmark, called CompWoB -- 50 new compositional web automation tasks reflecting more realistic assumptions. We show that while existing prompted LMAs (gpt-3.5-turbo or gpt-4) achieve 94.0% average success rate on base tasks, their performance degrades to 24.9% success rate on compositional tasks. On the other hand, transferred LMAs (finetuned only on base tasks) show less generalization gap, dropping from 85.4% to 54.8%. By balancing data distribution across tasks, we train a new model, HTML-T5++, that surpasses human-level performance (95.2%) on MiniWoB, and achieves the best zero-shot performance on CompWoB (61.5%). While these highlight the promise of small-scale finetuned and transferred models for task compositionality, their performance further degrades under different instruction compositions changing combinational order. In contrast to the recent remarkable success of LMA, our benchmark and detailed analysis emphasize the necessity of building LMAs that are robust and generalizable to task compositionality for real-world deployment.

Published in Transactions on Machine Learning Research (12/2024)
Exposing Limitations of Language Model Sequential-Task Compositions on the Web
Agents in
Hiroki Furuta
Yutaka Matsuo²
Aleksandra Faust!
Izzeddin Gur
matsut, } u.telab...uttokooooooo w.t.kyo..oe.jpp
fsandrofouss, izzeddii @google.com 'The University of Tokyo
Google DeepMind
Reviewed on OpenReview:
Abstract
Language model agents (LMA) recently emerged as a promising paradigm on muti-step decision making tasks, often outperforming humans and other reinforcement learning agents. Despite the promise, their performance on real-world applications that often involve combi- nations of tasks is still underexplored. In this work, we introduce new benchmark, called Comp WoB 50 new compositional web automation tasks reflecting more rcalistic assump- tions. We show that while existing prompted LMAs (gpt-3.5-ture 5-turbo gpt-4) achieve 94.0% average success rate on base tasks, their performance degrades to 24.9% success rate on compositional tasks. On the other hand, transferred LMAs (finetuned only on base tasks) show less generalization gap, dropping from 85.4% to 54.8%. By balancing data distribution across tasks, we train a new model, HTMI-T5+++, that surpasses human-level performance (95.2%) on MiniWoB, and achieves the best zero-shot performance on CompWoB (61.5%). While these highlight the promise of small-scale finetuned and transferred models for task compositionality, their performance further degrades under different instruction compositions changing combinational order. In contrast to the recent remarkable success of LMA, our benchmark and detailed analysis emphasize the necessity of building LMAs that are robust and generalizable to task compositionalil for real-world deployment.
1 Introduction
Based the exceptional capability of large language models (LLMs)) (OpenAT, 2023; Anil et al., 2023; Touvron et al., 2023) in commonsense understanding (Brown et al., 2020; Chowdhery et al., 2022), multi- step reasoning (Wei et al., 2022; Kojima et al., 2022), program synthesis (Chen et al., 2021) and self- improvement (Shinn et al., 2023; Madaan et al., 2023; To et al., 2023), language model agents (LMAs) have recently emerged to tackle various decision making problems, such as robotics (Huang et al., 2022a; Ahn et al., 2022), information retrieval (Nakano et al., 2021; Yao al., 2022b), and external tool use (Wu el al., 2023; Shen et al., 2023). Especially, web automation (Shi al., 2017) has attracted attention a lot as an promising application of LMAs, because LMAs with prompting (Kim al., 2023; Sun et al., 2023; Zheng et al., 2023) outperform humans and other learning-basse based agents, such as imitation (Furuta al., 2023) reinforcement learning (Humphreys et al., 2022).
Despite their human-level proficiency in MiniWoB (Shi et al., 2017), a standard web automation benchmark, it is still unclear whether LMAs could deal with challenges in the real world; such as complex observation (Gur et al., 2023), domain generalization (Deng et al., 2023), and ambiguity of instructions (Zhou et al., 2023b). These challenges are exacerbated due to the open-ended nature of real-world tasks, making it infcasible for the agent system to prepare exemplars and prompts for any unseen task in advance. Moreover, it is reasonable and practically important to assume the zero-shot evaluation of the off-the-shelf agents at deployment. This is an orthogonal assumption to the recent progress in LMAs learnable from exccution coclioo///s/ii fsedbffoooo (Shinn
* Work done as Student Researcher Google DeepMind.

---

Published in Transactions on Machine Learning Research (12/2024)
Base MiniNod
ticnal XiniWoB
paweaedd Lildee dlunlin
Upcote modelsprompls
Online
Leamable LM Agents
Marike (Tris See)
Sublt
Wispomon
Deployment
Revens-
Zaro-shot Evaluation
Login
Logh
Off-the-Shelf LM Agents
CompWoB
Bazn Tank Instructions: (Soive task A]
(Test Set)
(Solve task A), (soive taak R), and then (solve task C) heyerse-Ordaz Instructions: (Bolve task B), and (soive task C) after (soiving task A)
Corpositional 7ask Instructions:
Figure 1: (Left) We design Comp WoB, as novel decision making benchmark for LMAs, by leveraging the high composability of simulated web environments. We first select base tasks from the original MiniWoB (Shi et al., 2017) based on the brute-force task complexity averaged among existing LMAs. While considering the feasibility and reality, we systematically combine them into a sequential single task (o.g, click-cheeckooos-trreneee click-dialog> click- wett---eccoxxssstreeeeee
+ enter-password + mater--assword_c.ie click-dialog). The instructions of base tasks are stitched with "and then". LMAs are asked to satisfy the given instructions sequentially (e.g. satisfying the success criteria of task A + B + C). We also implement reverse-order instruction settings, where the instructions are provided upside down (e.g. solve task B, and solve task C, after solving task A). These complex yet controllable strategies make the analysis of LMA's behaviors easy, while maintaining complex and ambiguous aspects of real- world web enviromments. (Right) In this paper, we assume the zero-shot evaluation of the off- agents at deployment, which is practically important because (1) it is infeasible to cover all the possible
sumppps/demousttationss for user requests in advance and (2) iterative trial-and-error to adjust system prompts hurts user experiences. CompWoB works hold-out test environment accompanying with MiniWoB as a train environment.
et al., 2023; Ma et al., 2023) during the deployment. Considering the agents are deployed as a service, (1) it for user requests in advance and (2) iterative
is infeasible to cover all the possible trial-and-error to adjust system prompls hurts user experiences.
In this work, extensively study the transferability of LMAs to more realistic sequential task compositions. We first design a new controlled test bed, called CompWoB, with 50 compositional tasks by combining a set of base lasks based on their difficulty (Figure 1, right). CompWoB works as a hold-oul test environment accompanying with MiniWoB as a train environment (Figure 1, left). Each compositional task is implemented from to 8 base tasks in a single-page or multi-page environment with instructions linked together using simple connectors such as "and then". Only providing the knowledge about base tasks, we investigate the generalization performance of existing SoTA prompted LMAs (Kim et al., 2023; Sun et al., 2023; Zheng et al., 2023) with planning, self-improvement, program synthesis, and structured prompts that are supported and gpt-4. Our findings indicate that their performance drops significantly, from 94.0%
by 1ypt-3.5-trre success on base tasks to 24.9% success on compositional tasks. In contrast, small-scale LMAs finetuned only on base tasks and zero-shot-transferee to compositional scttings (i.c. transferred LMAs), deal with unknown task compositionality better, achieving 54.8% success rate on average. By rebalancing the data distribution, we train a new model, HTMI--55+++ that achieves human-level performance on MiniWoB and performs the best among all the LMAS on compositional tasks. In contrast, small-scale LMAs finetuned only on base to compositional settings (i.e. transferred LMAs), deal with unknown task
tasks and compositionality better, achieving 54.8% success rate on average. By rebalancing the data dist distribu, we also train a new model, HTML-T5++, that achieves human-level performance on MiniWoB and performs the best among all the LMAs on compositional tasks. We further point out that LMAs struggle to handle complex instruction compositions permuting the order of sub-instructions, where prompted agents are more robust to the difference in the order of compositions compared to transferred agents (6.9% vs 23.8% drop in performance). Finally, we illustrate that instruction length and observation complexity are useful indicators of compositional task performance.
In contrast to the recent notable success of LMAs, our benchmark and detailed analysis highlight building robust and generalizable LMAs to be safely deployed in the real world. In summary, our key contributions are:
We empirically show that (1) prompted LMAs even with gpt-4 suffer from generalizing to compositional web automation tasks much more than transferred LMAs, and (2) LMAs are highly sensitive to the order of instructions.
2

---

Published in Transactions on Machine Learning Research (12/2024)
We develop Comp WoB simulated web environments for LMAs to measure the generalization to the realistic task compasitionality and complex instructions.
We explore a new data mixture heuristic for finetuning LMAs, where we find that it is effective to add demonstrations of the tasks that suffer from data shortage and then gradually reduce the ratio of easier tasks to focus more on challenging lasks. HTMI-T5++, trained on our novel heuristic, achieves human-level performance on MiniWoB (95.2%) and the best zero-shot transfer to CompWoB (61.5%)
Our experiments reveal that while more capable models, such as gpt-4, lead to better performance, they still struggle with task compositionality or complex instructions.
2 Related Works
Web Automation Although prior works have worked on imitation learning and reinforcement learning (Liu et al., 2018; Gur et al., 2019; Jia et al., 2019; Humphreys et al., 2022), web automation has become a popular domain as application of LMAs (Gur et al., 2022; Kim et al., 2023). In carlier work, finetuned LMAs, based on at most 3-billion parameters, amortize the training costs with the strong prior knowledge on web environments (Gur et al., 2022; Furuta et al., 2023; Shaw et al., 2023), but they often result in sub-optimal performances due to the insufficient data coverage. Recently, by leveraging capable private LLMs (Brown et al., 2020; Ouyang et al., 2022) with reff-mefine (Kim et al., 2023), program synthesis (Sun et al., 2023), structured instruction-state translation (Zheng et al., 2023), or hierarchical prompts (Sridhar et al., 2023; Ma et al., 2023), prompted LMAs with few-shot exemplars have outperformed finetuned LMAs and shown superior performance to humans and RI-finetuned agents. In contrast, our work discusses the transferability and robustness of those LMAs in zero-shot web automation with set of compositional tasks, and resolves the sub-optimality of finetuned LMAs via data-rebalancing.
In addition to MiniWoB (Shi et al., 2017), a representative web simulator, several works have conducted world evaluation (Gur et al., 2023) and proposed novel benchmarks reflecting real-world assumptions, such as a simulated o-commerce site (Yao al., 2022a), sand-boxed real-world websites (Zhou et al., 2023b; Drouin et al., 2024), an adaptive sim-to-real bridge with unsupervised auto-curricula (Gur al., 2021), and large-scale web interaction dataset curated by human annotators (Deng et al., 2023). However, roal-world web automation may make the analysis challenging because it often faces miscellaneous obstacles, such as complex HTML observations, domain gaps between websites, and ambiguous instructions. In this work, we design CompWoB under realistic assumptions while controlling task difficulty and ambiguity of instructions, and investigate what may prevent the genoralization capability of LMAs in compositional tasks.
Language Model Agents Beyond the common NLP tasks, LLMs could act as autonomous agents (Wang et al., 2023b; Qin al., 2023a) to solve the given instruction-folloving tasks, by considering the context in the prompt states (Ahn et al., 2022; Yao et al., 2022b; Hsich et al., 2023) and sequentially planning and manipulating external "vools" or "actuators", such as calculators (Parisi el al., 2022), retrievers (Schick et al., 2023; Hao et al., 2023), APIs (Qin et al., 2023b; Tang et al., 2023a), programs (Gao et al., 2023; Wang et al., 2023a; Liang et al., 2023; Song et al., 2023; Cai et al., 2023), robotic commands (Huang ct al., 2022a;b; Tang et al., 2023b), computer game (Nottingham et al., 2023; Wang et al., 2023c), or other foundation models (T.u et al., 2023; Hsieh et al., 2023; Wu et al., 2023; Shen et al., 2023; Yang et al., 2023). Those prior works have worked on proposing novel benchmarks (Li et al., 2023; Xu et al., 2023; Patil et al., 2023) and comparing backbone LLMs (e.g. open-sourced v.s. private) (Ruan et al., 2023; Liu et al., 2023b;a). Despite their success, it is still unclear how such LMAs designed for specific tasks can generalize out-of-domain problems, which should be an important perspective since we may not prepare prompts and exemplars for all the possible problems in the real world. In this work, we measure the transferability and robustness to unseen compositionality and instructions in a realistic web automation scenario.
Large Language Models for Compositional Tasks Several works have investigated compositional natural language problems with LLMs, such as semantic parsing (Furrer et al., 2021; Shaw et al., 2021; Zhou et al., 2023a), logic grid puzzles (Dziri et al., 2023), mathematical reasoning (Chen et al., 2023), programming (Zelikman et al,, 2023), and planning (Brahman et al., 2023), which shows that dynamical
3

---

Published in Transactions on Machine Learning Research (12/2024)
selection of exemplars for decomposed sub-problems (Drozdov et al., 2023) model scaling (Qiu et al., 2022) could help generalization. While those are focused static tasks, our paper deals with task compasitionally in interactive decision making, especially, in web automation where the task may have more explicitly dccomposable structures (Gur et al., 2021) than natural language tasks.
3 Preliminaries
Web automation could be described as a deterministic sequential decision making problem, which consists of state space S, action space A, deterministic transition function T ; S x A + S, a set of instructions g, a set of contextual information (i.c. prompts for LLM) C, and episodic reward function (i.c. success criteria) r S x 9 + 10. At each time slep I, the language model agent a infers the action conditioned on the prompt, instruction, current state, and previous actions π S Ax* xCxg A, and moves the next state: St+1 = Ts,,, When the agent reaches the terminal state (e.g. Login button is clicked) or the max time step is exceeded, the episode is marked as success if the instruction is satisfied (i.e. r(st, a.) 1). The state 8 E S is a raw HTML, and we assume the programmatic action space: function (selector, text). function is either click, move or type, selector is an integer index or XPath that can uniquely specify the element, and text is a text imput for type function.
Web automation tasks can be decomposed into a set of primitive base tasks. For instance, (1) clicking several checkboxes, (2) fulfilling the password form, and (3) closing the dialog window. Such combination could be open-ended and have some dependencies. In this work, we assumee that the task & E V is characterized by corresponding subtree of HTML (Sy S) and instructions (G G), and can be dependently combined each other as long as the compositional task is executable.
Task Compositionality
4 Language Model Agents for Web Automation
To be self-contained, here review the representative LMAs, which are selected based on their superior performance and novelty in using LLMs for web automation; RCT (Section 4.1), AdaPlanner (Section 4.2), Synapsc (Section 4.3). These are characterized with different base LLMs, prompting and pipeline. To clarify their methodological difference, we provide the pseudo code in Appendix B. Moreover, we resolve the sub-optimal performance of transferred LMAs (Section 4.4) with data-rebalanced finetuning (Section 4.5).
4.1 RCI
The agent with Recursive Criticism and Improvement (RCT) prompting (Kim et al., 2023) first generates an open-loop plan to follow a given instruction using few-shot demonstrations. Next, it uses a prompt-based critic to identify the errors in the plan and improves it by reflecting on self-criticized outputs, which is referred as an explicit RCI (ERCI) loop. After ERCI, the agent follows the self-improved plan step-by-step. Before exccuting the action at cach step, the agent grounds the action the current state (i.c. HTML, open-loop plan, and previous actions) and refines its formatting to be parsable, which increases the feasibility and reduces ballucinations. These final steps referred All of those play complementary roles to achieve human-level proficiency. While ERCI and 3 IRCI loops are recommended, we observe that the optimal number of self-improvement iterations may differ across the tasks (see Appendix C.1).
implicit RCI (IRCI) loop without the relf-criticism.
4.2 AdaPlanner
In contrast to other LMAs, AdaPlanner (Sun et al., 2023) leverages the capability of program synthesis in LLMs to mitigate the hallucination in plan. Conditioning on the instruction, the description of permissible actions in the web environments, and few-shot demonstrations, the agent first generates an open-loop plan in a Python function, where each snippet corresponds to the action. Once the agent receives environmental foedback at each step, such as assertion errors in the code, other functional errors, or "ask" action to LLMs, it adaptively re-generates the plan for the remaining steps closed-loop manner. LLMs more capable of code have performed better, such as text-davinci-002 than gpt-3.5--e 5-turbo.
4

---

Published in Transactions on Machine Learning Research (12/2024)
4.3 Synapse
Synapse (Zheng et al., 2023) argues that LMAs perform better if well-designed structured prompts are provided, even without selfimmrovement program synthesis. The structured prompt- ing is formed by two pre-processing strategies: state filtering and task reformulation. State filtering gradually transforms raw HTML into simple formatted text, such as a Pythonic list or dictionary, in a multi-step manner, which may improve the state understanding of LMAs. Task reformulation translates given instructions or raw HTML into decomposed queries: for instance, translating "select 12/08/201 as the date and hit submätt R into "select the datepicker al step 1, click 'Prev' 7 times at step 2-8 (May is 7 months before December), click the date '12' at step 9, and finally submit at step 10" (translated instruction), or mapping proper noun into corresponding XPath (translated HTML). While detailed structured prompts have led to strong performances, those should be specialized for each primitive task in MiniWoB by leveraging 7 different types of reformulation. See Appendix C.3 for further details.
Models
Dataset Size
Success Rate
HTML-T5
3:7K episodee
85.6%
424K (+77K)
94.1%
HTMI-T8++
351K (+77K (33)
94.6%
(ours)
922K (+77K 102K)
91.8%
282K (+77K 142K)
95.2%
211K (+77 183K)
95.0%
RCI (Kim al. 2023) AdaPlanmer (Sun et al., 2023)
90.6%
92.9%
Human
99.3%
Net (Huraphreys RCI (gpt-4) (Kim et al,, 2023)
2022)
94.0%
Synapse (Zheng al. 2023)
98.5%
Table 1: Average success rate of finetuned LMAs in 56 tasks on MiniWoB. Adding 77K episodes and reducing redundant thousands of episodes, HTML-T5++ achieves competitive performance to prompted LMAs, RL-finetuned agents, and humans, while improving the suc- cess rale from 85.6% to 95.2%.
4.4 Finetuned and Transferred Language Model Agents
In addition the prompted LMAs, LMAs finetuned on base tasks have also been developed (Gur et al., 2022; Furuta et al., 2023; Shaw et al., 2023), which are built on pre-trained language models, such as T5 (Raffel et al., 2020), Flan-T5 (Chung et al., 2022), HTML-T5 (Gur et al., 2023), or Pix2Struct (Lee et al., 2023), with web automation demonstrations. Those LMAs take HTML (or screenshots) and previous actions as inputs and predict the text-format next actions in closed-loop manner. Since pre-trained language models have a sufficient inductive bias for web environments and instruction-folloinng, finetuned LMAs can dfataeefficienlly achieve competitive performance to the RL-finetuned agents trained from scratch with domain-specific architectures (Humphreys et al., 2022; Liu et al., 2018). Compared to the prompted LMAs relying on private LLM API, it is possible to build on-premise agents based on tractable-size models (at most 3 billion parameters), which may reduce inference time and costs. However, prior works have pointed out that finetuned LMAs struggle to the sub-optimal performance (Zheng et al., 2023) and they require demonstrations on the order of hundreds of thousands while prompted LMAs just need hundreds of episodes (Kim et al., 2023). In this paper, we extensively evaluate such finetuned LMAs in zero-shot transfer settings; LMAs are finetuned only with base task demonstrations and should deal with unseen compositional tasks. We call those tronsferred LMAs in the later sections.
4.5 Data- Rebalancin Improves Finetuned Language Model Agents
Furuta et al. (2023) utilized agent-driven data collection instead of humans to improve the performance of finetuned LMAS further. For each task, 10k demonstrations are collected and filtered based on task success, which resulted in challenging tasks having much less than 10k demonstrations due to the sub-optimal performance of LMA on these tasks. We identify that by fixing the data-imbalance problem, the performance of finetuned LMAs can be significantly improved, achieving super-human performance on MiniWoB. We first run Synapse (Zheng et al., 2023) on MiniWoB and collect 77K additional demonstrations across 16 tasks on top of 347K demonstrations (Furuta et al., 2023) to compensate for the lack of data in specific tasks. We then estimate the "brute-force" task difficulty averaging suecess rates for representative web automation agents. Based on those proximal measures, we classify 65 base tasks into three categories, such as easy (0.8 1.0), medium (0.6 0.8), and hard (0.0 - 0.6) (see Appendix D). We then balance the mumber of episodes based on the task difficulty, where we gradually reduce the ratio of casier tasks focus more on challenging tasks. For instance, we remove X% episodes from top-k tasks in easy group. We heuristically design the following data-mixing strategies:
5

---

Published in Transactions on Machine Learning Research (12/2024)
# Episode Steps
# of Instruction Tokens
Benchmark
Average
50th
90th
95th
Max
Average
50th
90th
95th
Max
MiniWoB (Shi et al., 2017)
3.6
4.3
5.5
5.8
7.0
8.5
14.8
30.5
12.9
23.2
Mind2Web (Deng et al., 2023)
7.7
6.0
14.0
17.0
37.0
19.2
16.0
33.0
42.0
84.0
CompWoB WoB (Ours)
6.8
12.8
14.2
17.8
37.3
34.2
55.2
57.9
100.0
Table 2: The statistics (Mean, Max, 50/90/ 95th percentiles) of episode steps and instruction tokens from MiniWoB (Shi et al., 2017), Mind2Web (Deng et al., 2023), and CompWWB. CompWoBB successfully increases task complexity from MiniWoB, and has sulficient difficulties the same as Mind2Web, a representative benchmark from real websites.
/9/th/95th
Original dataset released by Furuta et al. (2023) (347K episodes).
Adding 77K episodes from 16 tasks suffering the lack of data (424K episodes).
Removing 50% episodes from top-10 easy tasks (424K- 73K 351K episodes)
Removing 80% episodes from top-10 easy lasks (424K- 102K 322K episodes)
Removing 50% episodes from easy lasks (424K -142K = 282K episodes)
Removing 80% episodes from top-15 15 easy tasks and removing 50% episodes from other 11 easy tasks (424K 183K 241K episodes)
See Appendix E for further details. The aulomation of data mixture design (Xie et al., 2023) would be an important future direction.
We finetune HTMI-T5-XL (Gur et al., 2023), a pre-trained language model with local and global attention in the encoder and mixture of long-span denoising, on these rebalanced datasets. Table shows that all the data-rebalance strategies improve the success rate, and reducing 50% episodes from easy tasks (finally 282K episodes in total) is the most effective rebalancing strategy. This suggests that finetuned LMAs can be as capable as prompted LMAs decision making tasks. Moreover, due to the inherent compositional nature in challenging web automation tasks, data rebalancing may improve the capability of dealing with unseen task compositions. We include HTML-T5++ bascline in the later sections.
5 Designing CompWoB Controlling Task Complexity and Compositionality
Even though MiniWoB includes spectrum of simulated enviromments, they have still focused on narrow and single-task instances. We need more advanced environments to measure the generalization to the various functional challenges in real-world web automation, such as complex observation (Deng et al., 2023), instruc- tion (Zhou et al., 2023b), and task compositionality (Gur et al., 2021). We design Comp WoB (i.c. compositional MiniWoB), as a general lest bed for LMAs, by leveraging the high composability of simulated web environments (Figure 1). In Comp we systematically combine several base tasks (from 2 to 8) in the original MiniWoB, weweeecbxxxesttrrsseee + enter-password which LMAs can already solve, into a single task (c.g. httpppasswor_cccccd-iaaaa-eeeeee click-dialog)). This allows us con- wtw.oooossstttttseeeeeeeeeeeeeee
click-dialog + trol the complexity of HTML and instructions, ensuring novel tasks are solvable to some extent. Moreover, CoB can work as hold-out test environment accompanying with MiniWoB as train environment. While tasks are visually simplified, CompWoB reflects broader spectrum of functionalities in real-world websites without sacrificing controllability. For instance, the task in Figure sketches the structure login form with agrecment checkboxes and a pop-up window, like Internet banking. We also designed tasks with page transitions, such as from the login form to the email browser.
5.1 Details of Task Design
As mentioned in Section 4.5, we first calculate the average success rates among representative web automation agents as brute-force task complexity, and classify 65 primitive tasks in MiniWoB into categories (easy, medium, hard) based on those scores. The details of classification are described in Appendix D. We randomly select base tasks from easy group, filter those combinations by their feasibility and reality, and make 50 compositional tasks. We divide those into five categories: two-way tasks (20), three-way tasks (10), n-way

---

Published in Transactions on Machine Learning Research (12/2024)
tasks (5), transition tasks (5), and easy-medium two-way tasks (10). In n-way tasks, we combine from 4 to 8 tasks sequentially, and in transition lasks, we implement explicit page transition, for instance, transiting from the login form to the email browser. We also sample several tasks from medium group to construct casy-medium two-way tasks. You can find the full list of tasks in Appendix H. For the implementation, we stitch the instructions with "and then" " and put each HTML on the same depth. LMAs should satisly the given instructions sequentially, such as from task A, B, to C. Moreover, to test whether LMAs can deal with complex and ambiguous instructions, we propose reverse-order instruction settings, where the instruction is provided upside down while its task order is semantically the same (e.g. solve task B and C, after solving A). These simple yet controllable strategies make the analysis of LMA's 's behaviors tractable while reflecting compositional aspects of real-world tasks.
5.2 Number of Tasks
MiniWoB has around 104 base tasks in total; all the two-way and threc-way combinations of these tasks would give 185,460 compositional tasks. It is infeasible to manually curate all the two-way two-way/three-ww wayr combinations. Unfortunately, it is also nontrivial to automate this process due to the locality of the environment implementations in MiniWoB. Each environment is mostly self-contained, making it nontrivial to modularize the whole benchmark/ baanmarkkcodebasse /codebase so that every combination can be automatically generated. Alternatively, we outline a set of design principles solvability, feasibility, and reality, which we follow to manually curate 50 compositional tasks and 50 reverse-order instruction tasks that could cover wide variety of difficulty, and compositionality. We believe these guiding principles would be applicable to other compositional generalization problems such as robotic navigation. Based these design principles, our compositional benchmark can casily be extended in the future to study even more challenging and compositional web automation problems. We would also like to highlight that, because previous works were lested around 50 60 MiniWoB tasks (Kim et al., 2023; Sun et al., 2023; Zheng et al., 2023; Gur et al., 2022; Furuta et al., 2023), 50 compositional tasks is a decent number of tasks to evaluate the agents. Our addition of a hold-out. compositional test set doubles the number of tasks for the community to study.
5.3 Inherent Compositionality and Task Dependency in Web Automation
While many of the real-world tasks have inherent compositionality to some degree, these tasks are not explicitly designed for compositionality, making it difficult to systematically investigate the generalization gap. This can be analyzed with our CompWoB by separating the difficulty of the base tasks themselves and the diffieulty of the task compositions.
Real-world websites have dependencies between sub-tasks. For instance, an email client requires a successful login to proceed to writing an email or reading social media posts of a specific user requires navigating to the personal page of that user. In CompWoB, we have implemented these kinds of dependencies in the reward function using logical AND operations. For instance, while we allowed agents to continue to "write an email" sub-task without successful login, the episode flagged as a failure (i.c. zero reward) even if the "write an email" sub-task is successful. We inform agents of these dependencies using connectors in instructions such as "solve B after A" or "solve A then B Our analysis in Section 6.4 also suggests that, in addition to task compositionality, long instructions and deep HTML sub-tree can lead to challenging tasks.
5.4 Comparison to Other Web Automation Benchmarks
We design CompWoB on top of MiniWoB, which enables us to test the human-level LMAs (Kim et al., 2023; Sun et al., 2023; Zheng et al., 2023; Gur et al., 2023) as strong baselines. Because the success of in-domain tasks is ensured, we could focus on the analysis of generalization to unknown task compositions and their functionality in a controllable way. In contrast, other benchmarks copying real websites fail to obtain decent agents (for instance, the episode success rate of Mind2Web (Deng et al., 2023) is around 5-10% at most; around 15% success at most in WebArena (Zhou et al., 2023b)) and it seems to be harder to identify the attribution of crrors due to the complex nature of real websites. Morcover, while realistic benchmarks (Deng et al., 2023; Zhou et al., 2023b) often provide deterministic instructions and observations, CompWoB has randomized instructions generated from set of templates, as well as elementtrrandomizel HTML. For the
7

---

Published in Transactions on Machine Learning Research (12/2024)
Avg. Success Rate (96
100
95.2
90.6
92.9
98.5
94.0
85.6
85.4
80
75.5
61.5
60
56.6
54.8
46.3
40
17.9 17.9
22.5
28.8
20.6
17.8 15.4
25.4
24.9
20
WebGUM HTML-TS HTML-T5,
(zero](1s)) (2ndXcomb) AdaPlanner Synapsel (stpsss(sss)
Transferred (Ave.)
Prompted (Ave.)
MiniWeB
CompWoB
LMAs in 50 CompWoB tasks. The light color represents the performance the original MiniWoB, and the dark color for CompWoB. We use gpt-3.6-turie as the backbone LLM for prompted LMAs (RCI (Kim et al., 2023), AdaPlanner (Sun et al., 2023), Synapse (Zheng et al., 2023)), and transferred LMAs with 3-billion parameters. Transferred LMA, especially HTML-15+++ achieves the best generalization in compositional tasks, suppressing the performance degradation (from 95.2% to 61.5%). On contrary, prompted LMAs drop their performance significantly; even the best RCI that uses combined task prompts in the composition just achieves 28.77 28.7% success (from 90.6% base tasks). This indicates, in contrast to the base task performances, prompted LMAs are more vulnerable and transferred LMAs can deal with unknown task compositionality better than expected.
Figure Average success
task complexity compared other benchmarks, we provide the statistics (mean, max, percentiles) of episode steps and instruction tokens from MiniWoB (Shi et al., 2017), Mind2Web, and CompWoB in Table 2. In terms of episode length and instruction length, CompWoB successfully increases task complexity from MiniWoB, and has sufficient difficulties the same Mind2Web, representative benchmark from real websites.
6 Results
Evaluation Methodology We evaluate both transferred and prompted LMAs with base MiniWoB demon- strations on the unseen compositional tasks in a "zero-shot" manner; iee. we do not provide any demonstrations on the compositional tasks for the training corpus and exemplars to measure the generalization. We test. 50 compositional tasks and run 100 episodes per task. We adopt 1pt-3..5-tubbl backbone LLM, unless otherwise mentioned. We assume the optimal exemplar retriever throughout experiments and always provide the pre-defined prompts to LMAs. We borrow hyper-parameters and prompt templates from respective papers with minimal change to respect our zero-shot transfer setting. As a sanity check for the quality of environments and baseline agents, we also test LMAs with oracle demonstrations in Appendix
We test prompting strategies: (1) zero-shot (without any exemplars), few-shot with (2) first-task exemplars, (3) second-task exemplars, and (4) combination exemplars (i.e. both first and second tasks). For consistency and limited context length, always consider the first two tasks even if the mumber of primitive lasks is more than two, and fix the number of sell-improvement iterations to 1 explicit RCI and 3 implicit RCI as recommended in the original paper. The exemplars we use are provided by Kim et al. (2023).
RCI
AdaPlanner Following the original code, we use the same exemplars provided by Sun et al. (2023) for tasks where those base lasks are included, such as enter-text and click-widget (see Appendix C.2). Otherwise, the agents are prompted in a zero-shot manner.
Synapse We test 3 prompting strategics: few-shot with (1) first-task, (2) second-task exemplars, and (3) best exemplars (i.e. maximum score between (1) and (2)). Because prompls and modules are quite different. among tasks, we do not merge the prompts and use proper param-paraetters corresponding to the given exemplars designed by Zheng et al. (2023).
6.1 Language Model Agents Struggle to Handle Task Compositionality
Figure 2 shows that, in CompWoB, all the LMAs face performance degradation. Among those, transferred LMAs achieve better success rate (54.8%) than prompted LMAs (24.9%) on average. In particular, HTML- T5++ achieves the best generalization, suppressing the performance drop from 95.2% 61.5%. This might be because finetuned LLMs are better at natural language compositional tasks prompted L.MSs in general.
8

---

Published in Transactions on Machine Learning Research (12/2024)
Avg. Success Rate (%)
100
80
61.5
60
56.6
54.8
46.3
40-
29.1
29.7
31.0
C122
17.915.72.551..0
2..
19.7
28.7
20.618.9
25.
16.0 (3.4)
24. 24.918.0
20-
-2.21
(-0.5)
17.8 9.8
15.4 [-3.3]
WebGuM HTML-TS HIMLTS.:
RCI (zero) (1st)
(2nd)
(comb)
AdaPlanner Synapse(lst) (2nd)
(best)
Transferred (Ave.)
Prompted (Ave..)
CompWoB
Reverse-Drder Instructions
Figure 3: Average success rate of language model agents in reverse-order instruction settings. We use gpt-3.b-turbo as the backbone LLM for prompted LMAs (RCI, AdaPlanner, Synapse), and transferred LMAs with 3-billion parameters. Notably, most LMAs significantly degrade the success rate when reverse-order instructions are provided (the performance gap highlighted with red number). This trend more remarkable transferred LMA (54.8% + 31.0% average) than prompted LMA (24.9% 18.0% on average), which suggests that any kind of LMAs are susceptible to the order of compositional instructions. capability as general language models might be important to parse semantically complex instructions into the correct plan.
See Appendix K for further details. In contrast, prompted LMAs degrade their performance drastically; even the best RCI with few-shot combination exemplars (comb) just degrades the success rate to 28.7% from 90.6% in base MiniWoB. These results indicate that LMAs suffer from generalization to lask compositionaliin ionality, and transferred LMAs can relatively deal with that better than prompted LMAs, which is an opposite trend to base MiniWoB performance. Among prompted LMAs, RCI performs better than AdaPlanner and Synapse, which suggests that multiple iterations of self-criticism and improvement might be more robust to out-of-domain decision making from the exemplars than program synthesis with feedback structured prompting with state translations.
In the failure episodes (Table 3), LMAs often miss necessary steps, common to all the prompted LMAs. Since the instructions get long in compositional settings, LMAs may skip important intermediate steps to satisfy the instructions. In addition, they prodict incorrect action types and XPath: for instance, hallucination in XPath (RCI) and mixing up click and type action (Synapse). Without oracle exemplars, LMAs often struggle parse compositional task instructions and understand complex HTML. Appendix also provides average success rates per category.
6.2 Reverse-Orde Instructions Degrade Language Model Agents
As shown in Figure 3, all the LMAs significantly degrade the success rate when reverso-order instructions are provided. This trend is more remarkable in transferred LMAs dropping from 54.8% to 31.0% than prompted LMAs from 24.9% to 18.0% compositional instructions and that transferred LMAs may not generalize well to diverse instructions beyond the dataset distribution. As opposed to Section 6.1, the performance differences among prompted LMAs are marginal, which, however, implies that existing prompting methods, even with self-improvement, may not handle complex task instructions enough. The stronger capability as general-purpose language models or other prompting methods might be important to parse semantically complex instructions into the executable sequential plan.
average, which suggests that any kind of LMMS is susceptible to the order of
Compared to Section 6.1, RCI and Synapse cannot parse reverso-order instructions into plans correctly (Table 4). This can be because base-task exemplars in the prompt just have "lelt-to-right" instructions
RC1 (Mine al, 2000))
Addllsmner (Ien rt al, 3000)
Synapen ('trng al, 2011])
10N7 Sahaet
sobet Rigulli Sabent
Salonl
1. tppe Mottonnttte "ubteee
3. <ill: poosepod
1. clils 1/"lont (n Virpo
2htk /buttoo ond- salkmn?
2. dbck (Mooo waX Ipappt
2. tppe USKR
1. type UBSR
1. dids (nida//
dide //iapal (iddo''"" type yf
11 Poger
3. del [""[tt//rveett]
//aail Vappu
4 10 Hiapal
4. spe //"/a//="ss
2. type UBSR
type [oodo
1/10 suldes
Table 3: Failure examples in CoBp The left columns have correct plans and the right have failure plans. LMAs often ignore necessary intermediate steps predict incorrect action types and XPath.
9

---

Published in Transactions on Machine Learning Research (12/2024)
Avg. Success Rate (96
100
90.6
92.9
98.5
80
60
56.0
41.4
43.5
40
20
20.6
29.5
17.8
29.5
15.4
22.5
25.4
19.2
RCI
(comb) w/ 9pt-4
AdaPlanner w/ text davinci 003
Synapse (1st) w/ 9pt-4
(2nd)
(best)
RCI (comb, reverse) w/ gpt
MinWooB
CampWoB
gpt-4/txxedaainnc-o02
Figure 4: Average success rate large language models with advanced LLMs (gpt-4 for RCI and Synapse, and text-davinci-003 for AdaPlanner).. The lighter color represents the performance in MiniWoB, the medium color does in CompWoB with gpt-3. 5-turbo, and the darker color does with gpt-4 or sext-davinci-003, The more capable models (gpt-4 generalist and text text-davinci-00s still struggle to generalize to compositional tasks (e-g. 56.0% by RCT) or to deal with reverse-order instructions (e.g. 43.5% by RCT). This may indicate that we need much better LLMs to realize deployable agents in the complex real world.
coder) can improve the success rate of prompted LMAs but
from base MiniWoB tasks; strongly conditioning LMAs process instructions in linear order while compositionaliit could imply non-linear processing of instructions. The mismatch between the prompt and tasks could cause the performance drops. LMAs still fail to select correct action types (AdaPlanner) or XPath (Synapse), and they also predict unnecessary actions (RCI).
6.3 Do Advanced LLMs Solve Compositional Tasks?
Figure 4 presents the results when we adopt other advanced LLMs, than 1ptt3..-turrble as a backbone of cach LMA. The more capable models, such as gpt-4 in a generalist aspect and text-dawinci-003 a code generation, can improve the success rate all the prompted LMAs. However, even gpt-4 still far from the generalization in compositional tasks (from 28.7% to 56.0% by RCI) or from dealing with reverso-order instructions (from 19.2% to 43.5% by RCT). This indicates that need much better LLMs to realize deployable systems in complex real-world pecision-making tasks. We provide failure examples in Appendix G.
6.4
What Determines Task Complexity on the Web?
Figure 5 visualizes the correlation between the success rate averaged across WebGUM, HTML-T5, RCL, AdaPlanner, and Synapse (y-axis) and each statistic of compositional tasks (x-axis), such as synthesized success rate a product of base task success rates among composit ional tasks the number of instruction tokens, and max depth of HTML subtrees. Synthesized success rate positively correlates with an average success rate (R 0.691), indicating that compositional task difficulty takes over base task difficulties. addition, the number of instruction tokens (R = -.5579 and the max depth of HTML subtrees (R -0.433) show negative correlations. All those statistically significant in paired t-test with p < 0.01. In contrast, other task statistics, such synthesized success rate with human performance, the number of HTML tokens, and elements in HTML, just show relatively weaker correlations (see Appendix F for the details). This analysis suggests that HTML with larger depth and long instructions make generalizing compositional tasks challenging. The complexity of HTML is determined by its depth rather than its length or the number of
RET (Kit al, 1913)
AdaPlaansr (Bin at al, (011)
Synapen "Shrng et 2011))
fuil
X
1. ctok /laettotett)) 2. clok bostttin/"//a Ialppp
dlok (elpett/ wofl
dlok 2/"ppatt) """peatt"""erroon" [porrond]
1. dlck pe"test// [gredore")
clack (7""0typees"aaooeeee [pansoss]]]]
1. elds (""oont) "Mdl Pippo
2. didk (o byper]
2. dldk 7/ "eatl SodtLr Capel
2. typ Oldd1
bype Ro4bi ricck (""pett/"wwwttyy
2. cise (filtyoen (pannsed" tspe (/"Glge-- "paoncedd"
dek 1//(00) (id. salkns)
3 dek (." Tonkin]]]
clck / (i"tte
dok Middd Sallow]
lype
Table 4: Failure examples in Comp WoB with reverse-order instructions. LMAs often fail to parse the instruction into the correct-order plan, and hallucinate unnecessary actions (e.g. type).
10

---

Published in Transactions on Machine Learning Research (12/2024)
R 0.691. p 0.01
R= - 0.579. 0.001
R 00.433,p=: 0.01
Average Success Ratte
0.8
0.0
0.5
0.8
0.6
0.5
0.4
0.50
0.55
0.60 0.65
0.70
0.75
0.80 0.85
0.30
20
30
50
60
70
30
50
100
3.0
3.5
4.0 4.5
5.0
5.5
6.0 6.5 7.0
Synthesized Success Rate
Instruction Tokers
Max HTML Subtree Depth
Figure 2D-scatter plots between success rate averaged among LMAs (y-axis) and each statistic of compositional task (x-axis), such as success rate synthesized with a product of base task success rate, the mumber of instruction tokens, and max depth of HTML subtrees. Synthesized success rate positively correlates with an average success rate (R = 0.691, statistically significant in paired t-test with < 0.01), indicating that base task difficulty may determine compositional task difficulty. In addition, the number of instruction tokens (R = 0.579; p < 0.01) and the max depth of HTML subtrees (R = 00433; p 0.01) show negative correlations, which suggests the high complexity of observation and long instructions make the compositional tasks hard to resolve.
elements. This might come from the hierarchical nature of HTML; in deeper HTML subtrees, the elements near the root tend to be distant from each other after the traversal. Such sparsity may cause confusion during planning. We also report the individual characteristics of each agent in Appendix J.
7 Discussion
Generalizable Prompting Methods The results of Synapse and RCI in Figure 2 imply that those prompted LMAs have "over-fitting" trends to the base MiniWoB tasks. While the robustnoss across the prompts has been investigated in natural language tasks (Wei et al., 2022; Kojima et al., 2022), it is not well understood in decision making. Because we will not be able prepare the optimal self- self-improvement iterations decomposed prompts for all the possible instructions and task compositions, even if using optimal exemplar retrievers, we should care more about the generalization of prompting methods for the agent systems.
Agent-Specialized Large Language Models As shown in Figure 4, the more capable LLMs, such as gpt-4, can improve the performance of LMAs in CompWoB. However, it has not reached the base MimiWolB yet (c.g. from 90.6% to 56.0% in RCI, and from 98.5% to 41.4% in Synapse). Similarly, as described in Section 4.4, transferred LMAs can perform better if the training datasel has a good balance and coverage, but it is far from sufficient generalization to compositionality and instructions. The current pre-trained LLMs may still not be sufficient to generalize to complex decision making tasks, and then, in addition to prompting methods, the development of agent-specializer LLMs with enhanced reasoning and generalization through the instruction-tuning (Zeng et al., 2023) or RLHF/ /AIF (Furuta et al., 2024) would be expected.
Parsing Complex Instructions to Executable Plan Section 6.2 highlights that LMAs are fragile when we increase the complexity of instruction even by the most straightforward reverse-order instructions. This may not be preferable for the real-world application since the instructions might not be casy-to-parse and the users should carefully and concisely tell what they would like to do, which hinders the user's experience. It would be an interesting future direction to investigate better planning modules that could parse complex instructions to correct and executable plans.
8 Conclusion
The robustness and generalization of LMAs are important aspects for real-world deployment. We extensively examine how much existing LMAs, via transferring and prompting, can deal with a set of compositional web automation environments, CompWoB, that consists of resilyyre base primitive tasks. Our evaluation implics the contrary conclusion to the prior works (Table 5); the prompted LMAs are strong solver for primitive web automation lasks but significantly drop their performance in unknown task compositionalily, The transferred LMAs often show sub-optimal performance in basic tasks but can deal with compositional problems much better. Our detailed analysis also highlights that LMAs also face catastrophic degradation when they receive complex, even in the simplest reversed-order instructions, and that the challenges in compositional tasks might come from instruction length and the depth of HTML. subtree. We hope this
11

---

Published in Transactions on Machine Learning Research (12/2024)
inspires the community to build robust and generalizable LMAs to task compositionality toward real-world application.
Base MiniWoB
Reverse- Order Instructions
Advanced Models
Comp WoB
Prompted LMA
94.0% / 98.5%
24.9%
28.7% 61.5%
18.0% / 19.2%
42.3% 56.0%
Transferred LMA
85.4% 95.2%
54.8%
31.0% 34.3%
Table 5: Summary of average
max success rate in web automation.
Ethics Statements
This paper systematically evaluates the performance existing language model agents for web automation in realistic compositional tasks, and unveils remaining challenges towards broader generalization and real-world deployments. This technique could realize capable AT assistant tools digital devices (e.g. computers smartphones), and improve productivity and accessibility for society.
While we anticipate the positive aspects of autonomous agents, for responsible development, we should also consider the potential harmful applications and unintended consequences. The misuse of web automation would threaten cyber security, and the users may get scammed. To reduce these risks, it is essential for researchers, policymakers, and industry to discuss concrete guidelines and regulations.
Acknowledgments
We thank Mustafa Safdari, Yingjie Miao, Yusuke Twasawa, Heiga Zen, and Doina Precup for the help on this work. HF was supported by JSPS KAKENHI Grant Number JP22J21582.
References
Michael Ahn, Authony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Tbarz, Brian Ichter, Alex Trpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey. Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambaa, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do can, not as i say: Grounding language in robolic affordances. arxiv preprint arriv:2204. 01691, 2022.
Rohan Auil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Fric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barhan, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Sroqueetee-hhoo Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Rence Shelby, Ambrose Slone, Danicl Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang. Pidong Wang. Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven
12

---

Published in Transactions on Machine Learning Research (12/2024)
Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report. arxiv preprint arXix:205. 10408, 2023.
Faeze Brahman, Chandra Bhagavatula, Valentina Pyatkin, Jena D. Hwang, Xiang Lorraine Li, Hirona J. Arai, Sounya Sanyal, Keisuke Sakaguchi, Xiang Ren, and Yejin Choi. Plasma: Making small language models better procedural knowledge models for (counterfactual) planning, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Matcusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, nla Sutskever, and Dario Amodei. Language models are few-shot, learners. arwiu preprint ar arXiv:2005.1 465, 2020.
Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. arXiv preprint arXiv:2905. 17126, 2023.
Jiaao Chen, Xiaoman Pan, Dian Yu, Kaigiang Song, Xiaoyang Wang, Dong Yu, and Jianshu Chen. Skills-in- context prompting: Unlocking compositionality in large language models. arXiv preprint ar arXiv:2308.003: 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Flizabeth Barnes, Ariel Herbert-Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Πya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. arv preprint or arXiv:2107.03377 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, ian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyasshcenkoo, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meicr-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. arXiv preprin! 3rXiv:2204.02311 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling instruction-finetund language models. arXiv preprint 2022.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijic Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. arXiv preprint ar arxi::2366.000 2023.
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste.
13

---

Published in Transactions on Machine Learning Research (12/2024)
Workarena: How capable are web agents at solving common knowledge work tasks? 2rxi::2403.07718, 2024.
arXiv preprint
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. Compositional semantic parsing with large language models. In International Conference Learning Representations, 2023.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Vejin Choi. Faith and fate: Limits of transformers on compositionality. arXiv preprint arzio:2305.18654 2023.
Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Schärli. Compositional generalization in semantic parsing: Pre-training vs. specialized architectures. arXiv preprinl arXi::2007.0897 2021.
Hiroki Furuta, Tatsuya Matsushima, Tadashi Kozuno, Yutaka Matsuo, Sergey Levine, Ofir Nachum, and Shixiang Shane Gu. Policy information capacity: Information--hheorete measure for task complexity in deep reinforcement learning. In International Conference on Machine Learning, 2021.
Hiroki Furuta, Ofir Nachum, Kuang-Huu Huei Lee, Yutaka Matsuo, Shixiang Shane Gu, and Tzzeddin Gur. Multimodal web navigation with Inttruction-inettune foundation models. ar preprint arriv:2305. 2023.
Hiroki Furuta, Kuang-Hued Lee, Shixiang Shane Gu, Yutaka Matsuo, Aleksandra Faust, Heiga Zen, and Izzeddin Gur. Geometric-averaged preference optimization for soft preference labels. arXiv preprint arnio:2409.0000 2024.
Layu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. arXiv preprint
2023.
Izzoddin Gur, Ulrich Rucckert, Aleksandra Faust, and Dilck Hakkani-Tur- Learning to navigate the web. In International Conference on Learning Representations, 2019.
Izzeddin Gur, Natasha Jaques, Yingjie Miao, Jongwook Choi, Manoj Tiwari, Honglak Lee, and Aleksandra Faust. Environment generation for zero-shot compositional reinforcement learning. In Advances in neural information processing systems, 2021.
Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa. Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. Understanding html with large language models. arXiv preprint aroiv:2210.039. 2022.
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint 0rzi::2307.12855 2023.
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. arXiv preprint arriv:2305. (ito:2205.1154,, 11554, 2023.
Cheng-Yo Yu Hsich, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. Tool documentation enables zero-shot tool-usage with large language models. arXiv preprint arXiv:2308. 00675, 2023.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as vero-shol planners: Extracting actionable knowledge for embodied agents. arXiv preprint w=Xiv:2201.07200, 2022a.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Tgor Mordatch, Yevgen Chebotar, Pierre Sermanct, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. Xrxiv preprint 0v2002207..05008 2022b.
14

---

Published in Transactions on Machine Learning Research (12/2024)
Peter C Humphreys, David Raposo, Toby Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, and Timothy Lillicrap. A data-driven approach for learning to control computers. In International Conference on Machine Learning, 2022.
Sheng Jia, Jamie Ryan Kiros, and Jimmy Ba. DOM-Q-NET: Grounded RL on structured language. In International Conference on Learning Representations, 2019.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. arXiv preprint arriv:2303.17440 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances In Neural Information Processing Systems, 2022.
Kenton Lee, Mandar Joshi, Iulia Ture, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming- Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. arXiv preprint arpiv:2210. 03347, 2023.
Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A benchmark for tool-augmented lmms. arXiv preprint arXiu:2304.0820 082444 2023.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprinl ar X7i::2200.07758, 2023.
Evan Zheran Liu, Kelvin Gun, Panupong Pasupat, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Represenlations, 2018.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating 11ms as agents. arXiv preprint 2308::2308. 03688, 2023a.
Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh Murthy, Yihao Feng, Zeyuan Chen, Juan Carlos Niebles, Devansh Arpit, Ran Xu, Phil Mui, Huan Wang, Caiming Xiong, and Silvio Savarese. Bolaa: Benchmarking and orchestrating lm-aungentedd autonomous agents. arXiv preprint arXiv:230s 055600, 2023b.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chus Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv preprint ar XrXiv:2304.098./ 2023.
Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, and Dong Yu. Laser: Lim agent with state-space exploration for web navigation. Xiv preprint arxiv:2309.081 2023.
Aman Madaan, Niket Tandon, Prakhar Gupla, Skyler Hallinan, Luyu Gao, Sarah Wiegrelle, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with
arXiv preprint
arziv:2303. 17611, 2023.
Reichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Floundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feodback. arXiv preprint arXiv: 21::::11220000 2rX.v:2112.09332, 2021.
Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannanch Hajishirzi, Sameer Singh, and Roy Fox. Do embodied agents dream of pixelated sheep: Embodied decision making using language guided world modelling. Tn International Conference on Machine Learning, 2023.
OpenAL. Gpt-4 technical report. Xixiv preprint or Nii::203300877,, 2023.
15

---

Published in Transactions on Machine Learning Research (12/2024)
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishknnn Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint arric:2203. 203::220 2022.
Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arXiv:2205.122: 2022.
arv preprint
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connectod with massive apis. arviv preprint or Xrv::2305. 15394, 2023.
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Yc, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji, Zhiyuan Liu, and Maosong Sun. Tool 2023a.
learning with foundation models. arXiv preprint ar Xrx:::304
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis. arx preprinl arXiv:2307.16789 2023b.
Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, and Kristina Toutanova. Evaluating the impact of model scale for compositional generalization in semantic parsing. arXiv preprint Xrx:::200. 12253, 2022.
Colin Raffel, Noam Shazcer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 (140))1-672 2020.
Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Guoqing Du, Shiwei Shi, Hangyu Mao, Xingyu Zeng, and Rui Zhao. Tptu: Task planning and tool usage of large language model-based ai agents. arXiv preprint vrXi::230800327, 2023.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprinl Xiv:2200. 761, 2023.
Peter Shaw, Ming- Wei Chang, Panupong Pasupat, and Kristina Toutanova. Compositional generalization and natural language variation: Can semantic parsing approach handle both? arXiv preprint rrXtvv2010.12725, 2021.
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova. From pixels to ui actions: Learning to follow instructions via graphical user interfaces. arXie preprint Xrxi2:2306. 00245, 2023.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends hugging face. arXiv preprint arXiv:2303. 17580, 2023.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, 2017.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, ar preprint 2tth::2333.113666 2023.
Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, Ye Tian, and Sujian Li. Restgpt: Connecting large language models with real-world restful apis. Xixiv preprint or 2iv:2306. 066.00...6..6. 2023.
16

---

Published in Transactions on Machine Learning Research (12/2024)
Abishek Sridhar, Robert Lo, Frank F. Xu, Hao Zhu, and Shuyan Zhou. Hierarchical prompting assists large language model on web navigation. orXiv preprinl arzin:2305. 14257, 2023.
Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language modcls. arXiv preprint ar 23v: 16653, 2023.
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint X33.6:2306.0 053001 2023a.
Yujin Tang, Wenhao Yu, Jie Tan, Heiga Zen, Aleksandra Faust, and Tatsuya Harada. Saytap: Language to quadrupodal locomotion. In Conference on Robot Learning, 2023b.
Hung Quoe To, Nghi D. Q. Bui, Jin Guo, and Tien N. Nguyen. Better language models of code through self-improvement. arXiv preprint 0rziv:2344.01222 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and elficient foundation language models. arXiv preprinl arziv:2302. 13971, 2023.
Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint WxXi::235.16291. 2023a.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. A survey on large language model based autonomous agents. arXiv preprint ar arx:o:2008.11 1 11432, 2023b.
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. In International Conference on Machine Learning, 2023c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201111903 2022.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint ar arXii:2303.0461 2023.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Qiantong Xu, Fenglu Hong, Bo Lı, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool manipulation capability of open-source large language models. arXiv preprint ar Xix:2:200..0 16554440.6 2023.
Rui Yang, Lin Song, Yanwei Li, Sijic Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction. arXiv preprint arXiv:2305.1852. 2023.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasinhan. Webshop: Towards scalable real-world web interaction with grounded language agents. arXiv preprint arzio:2207.01206 2022a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint SXie:2210003622 2022b.
Eric Zelikman, Qian Huang, Gabriel Poesia, Noah D. Goodman, and Nick Haber. Parsel: Algorithmic reasoning with language models by composing decompositions. arXiv preprinl arzin:2212 10561, 2023.
17

---

Published in Transactions on Machine Learning Research (12/2024)
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. arXiv preprint arsic:2310. 1 12823, 2023.
Longtao Zheng, Rundong Wang, and Bo An. Synapse: Leveraging few-shot exemplars for numan-leve computer control. Xix preprint orXo:2306.078663, 2023.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In International Conference on Learning Representations, 2023a.
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: realistic web environment for building autonomous agents. arXiv preprint 0rziv:3377.1854, 2023b.
18

---

Published in Transactions on Machine Learning Research (12/2024)
Appendix
A Details of LLM API
We used OpenAI API to call LLM inference our experiments. Table shows the API used for each method.
We did most of our experiments from 2023/07 /07 2023/09. We use the official implementations and prompts released by the authors 231, We spent about $3.6K for the experiments in total.
Methods
API
Cost (input/output; /K tokens)
Context Length
RCI (Kim et al., 2023)
gpt-3.5-turbe gpt-4
$0.0015 / 80.002
4K tokens
$0.03 80.06
8K tokens
AdaPlanner (Sun al., 2023)
gpt-3.5-turis
$0.0015 / 80.002
4K tokens
text-davinci-005
$0.02 / 80.02
4K tokens
Synapse (Zheng et al., 2023)
opt-b.b-turbo
$0.0015 / $0.002
4K tokens
gpt-4
$0.03 80.06
8K tokens
Table 6: List of LLM API used in this paper. We did those experiments from 2023/07 /0 to 2023/09.
B Pseudo Code for Prompted Language Model Agents
We explicitly distinguish LLM itself and LMA (combination of LLM, prompting and pipclinc) in this paper. We provide the pseudo code for prompted LMAs to clarify their methodological difference. RCI (Kim et al., 2023) is the first to use prompting in self-refinement loop, outperformigg imitation- reinforeement-lerrnel agents on MiniWoB benchmark that requires millions of demonstrations to work. AdaPlanner (Sun et al., 2023) and Synapse (Zheng et al., 2023) are the follow-up works outperforming RCI via code generation from environmental feedback or via well-designed decomposed prompts with retrieval.
Algorithm Prompted Language Model Agents: RCI, AdaPlanner, Synapse
Input: prompt P, LMA æ, task v environment Env, large language model LLM, number of ERCI NERCI,
number of IRCI NIRCI
1: 5,g + Dav.reset. (1)) 2: 8.9 + LLM(.IPayn,$8.
Task Reformulation (Synapse)
history + ()
while Env is terminated do
{a1,..,,7)) - ( (, 5,
Planning
do 7: criticism +- LLM(. Prei,
for i in range(Nenee
(T)
D Criticism (RCT)
8: {aI...,aT! aT) + 7-11 8, {a a criticism)
> Improvement (RCI)
9: end for
10: for a in fai,..., 0T} do
11: for j in range (Nmcc)) do
12:
a a [--(11P=,> 8, ta, ..., ot history)
Improvement (RCI)
13:
end for 8,r, inf, t- Env Stv.step(a)
14:
15:
(a. ]r + ( 8, 9, fa, a history, info)
> Replanning (AdaPlanner)
16: history + history U {a}
17: end for
18: end while
hhttp:://gituub.com//o//////////
19

---

Published in Transactions on Machine Learning Research (12/2024)
C Details of Hyper-parameter
C.1
RCI
As we described in Section 4.1, RCI has two important syper-parameters to control the number of self- improvement iterations. In Explicit RCI (ERCT) loop, LLMs criticize their own generated plans to identify the problem and then improve it, reflecting self-criticism. In Implicit RCI (IRCI) loop, LLMs ground the action to the current state (i.e. HTML) and refine its formatting to be parsable without self-criticimm, which may reduce hallucinations tiny errors. We here test how many self-improverentt loops RCI requires (IRCI: 1-4, ERCI: 0-2). Table 7 shows that the optimal number of inference loops is different among tasks, while the recommendations are ERCI = 1 and IRCI 3. These two each task.
might need to be adjusted for
(ERCI, IRCI)
Tasks
(0,1)
(0,2)
(0,3)
(0,4)
(1,1)
(1,2)
(1,3)
(1,4)
(2.1)
(2,2)
(2.3)
(2,4)
click-buttan
1.00
1.00
1.00
1.00
0.92
0.84
0.87
0.88
0.93
0.86
0.87
0.87
cl1ck-checckores
0.90
0.94
0.87
0.91
0.96
0.94
0.97
1.00
0.89
0.91
0.94
0.99
click-dialog
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
click-link
0.96
0.95
0.98
0.99
0.91
0.91
0.89
0.88
0.95
0.91
0.91
0.87
click-option
0.87
0.56
0.52
0.83
0.82
0.73
click-scroll-liet
0.88
0.81
0.78
0.79
0.76
0.85
0.74
Table 7: The success rate of RCI with different lyperrparammeeess, optimal parameters differ in each task, while the recommended one is (1,3).
C.2 AdaPlanner
We use the remonsttationn of these tasks where they are included the task composition:
enter-text
click-widget
navigate-tree
login-user-popun
mail--inbor-forrrre-n-tttx
cllck-chhedkoozeellaree
click-tab-2-hard
click-dlalog-2
search-engine
click-cheekkoxxes-soft
use-autocomplete
enter-date
click-dlalog-2
C.3 Synapse
As we explained in Section 4.3, Synapse has several hyper-parameters to construct optimal structured prompts per task to specify whether LLMs translate the instruction or HTML.
Table 8 summarizes the type of reformulation into 7 categories and clarifies which transformed inputs are used for predicting open-loop plans. For instance, Task only requires translated instructions (and few-shot planning exemplars), although Obs takes raw instruction, HTML, and translated HTML as inputs. For the tasks that require temporal abstraction, it also employs stale-conditional decomposition, which factorizes demonstrations into set of exemplars conditioned the environmental states, and can reduce error accumulation over the time step.
Table 9 provides the detailed values for state-filtering and task reformulation, which is quite different across the tasks. These well-designed structured prompts could be the source of the best performance in base MiniWoB. However, compositional scttings, challenging modify them for any combinations. Instead, we assume the optimal retriever always picks up the exemplars for one of the base tasks, and we compute the maximum among the results with given prompis.
20

---

Published in Transactions on Machine Learning Research (12/2024)
Reformmlation Strategles
Inputs
Tbe_Task
Taak
Taakk
Mone Filtee
Instruction
Translated
Haw
Translated
Translated
Raw
Raw
Raw
HTML
X
Raw Translated
Raw Translated
Translated
X
Raw
Translated
Table 8: Summary of task reformulation for structured prompting used in Synapse (Zheng al., 2023). Structured prompts are finely designed per lask.
Tasks
State Filtering
Exemplar Decomposition
Raw Task Only
Reformulation
hook-flight
Troe
True
False
Nome
chocee-date
Faleo
chocee-list
Faler
False
True
Noge
dick-button
False
False
False
None
click-buttoon mook-bbtton-seqqeeee
False
False
False
None
eliok-cheekhoses
False
Falso:
True
None
click-cleekkooeer lange
Faleer
False
True
Nome
click- checkboxes soft
False
False
False
Obe
checkhoeekssses cramsfer
cliek-collppshhle
False
Fallon
Truc
Nome
clickcollppsibee22
False
False
Obs
click-color
False
False
True
Noue
click-disloy
False
False
True
None
click-dislog-2
Falser
False
False
Nome
dink-link
False
False
True
Nome
click-menu
elick-option
False
Fallso
True
None
click-pie
Falser
False
True
Nome
cick-scroll-lie
False
False
True
None
click-shades
False
False
True
Task
click-shape
Tree
Fallo::
False
Ohs Task
click-tab
Faloe
False
True
Nome
2lick-tab-2
False
False
Obe
click-tab-2.
Tree
False
False
Obs Task
click-test
False
click-test-2
Fabse
Follso
False
None
click-widget
False
Falls
True
None
copy-paste
Falso
False
Falso
None
copy-parte-2
False
False
False
Nome
count-shape
email--nhox
False
False
False
Task
emaiil-inboe wowee-ooooo---tuuoooo
Faleer
Falle
False
Task
emnail aaaiil--nooe--lll nl-turk
False
False
False
Task
cuter-date
False
False
True
None
enter-pasewwrd
False
Falls:
True
Nome
Faduer
False
Nome
enter-tert deder-texxt-yyyamnie
False
False
True
Nome
find-word
True
Falso
False
Nome
focus texct
False
False
True
None
focuusteext-2
False
False
True
None
grid-coordinate
False
Falso
True
Nome
guese- ounber
Falser
False
False
Nome
edentify-shape
False
False
False
Nome
login-user
login-nsor--pen
False
Fullso
Truc
Nome
mull mulli-layoodd li-layouta
False
False
Fabee
Nome
malti orderings
False
False
False
None
mavigate-tre
False
False
False
None
read-table
Falso
Fallo:
False
None
werrcheenginee
Fadoe
False
True
Nome
simple algebra
False
False
Noue
simple arithmetic
False
False
False
Nome
False
social media-all
False
False
True
None
pocial-media-some
False
False
True
None
terminal
False
True
Falso
None
text- tracsform
False
False
False
Nome
tic-tac-tor toe mnicode-ten test
Obs
False
False
False
Nome
ne--antoroomplte
Faluo
use-spinner
Faber
Falss
Falee
Task
Table 9: Hyperparameters for Synapse (Zheng et al., 2023). Raw Task Only specified with Task as Reformation flag, and Reformulation specified with Reformat Input flag in the original imprementation.
21

---

Published in Transactions on Machine Learning Research (12/2024)
D Ranking Base MiniWoB Tasks
To ensure the solvability of CompWoB to some extent and to identify the data-rodundant tasks for finctuned LMAs, we estimate the brute-force task difficulty (Furuta et al., 2021) (Table 10). We compute the average success rate for each task across representative previous web automation agents, such as CC-Net (SL, SL+RL) (Humphreys et al., 2022), WGE (Liu et al., 2018), WebN-T5 (Gur et al., 2022), WebGUM (Furuta et al., 2023), HTML-175 (Gur et al., 2023), RCI (Kim et al., 2023), AdaPlanner (Sun et al., 2023), Pix2Act (BC, RL) (Shaw et al., 2023), and Synapse (Zheng el. al., 2023). Based on those proxy difficulty measures, we classify 65 tasks into three categories (Kim et al., 2023): easy (from 0.8 to 1.0), medium (from 0.6 to 0.8), and hard (from 0.0 to 0.6).
Category
Tosk
Task Difficulty
click-button
0.925
click button-ecqueeoe
0.954
elickchhrkkoses
0.335
stcl-cheokboee-tre trannsrr
click collapstble
0.878
clickddisoog
0.925
click
039
elicbeeption
0.330
clicktab
0.888
click sest
1,000
click
wllek-widged
0.945
easy (0.8 1.00
er rrmiiliiduosoorwwddlll
forward
0.844
cater- paae--rd
0.85
cater text
0.922
cufer-test-dymmin
0.333
focms-est
0.900
focus text
0.900
grid-corrdincte
0.320
identify shape
0.918
login-user
0.881
Iogin-n-r--ouup
0.818
layouts navigste-tree unicode cest
0.825
0.900
showw-datt--eeey
0.740
click checkhenee large
0.784
0.754
click collapstibe
0
elickeoddor
0.742
ciickedislog-2
0.790
click mecn
0.607
elick-pie
0.700
click-shape
0.884
elick-ial-2
0.736
Bicketab----hnrd
aediun (0.6 0.8)
copy paste
0.610
emacikisbox
(L778
emall labox turk
0.773
entersdate
0.714
milti-onderings
0.793
read. table
0.600
engine simple-nlgelan
(.755
sin Smmplerritheee
0
media
0.753
teac: transform
0.737
attocompltte
0.782
booke-lightt
0.510
choose-ditte
0.531
choose date mediium
0.497
chocosslid
0.520
elick-scroll-jat
0.401
clickeshorhos
0.501
copy parte-2 2
0.547
hard (0.0- 0.6)
couss-slape
entor-lime
0.52:
Bud-woor
(U,20)
guoss-mmmber
0.363
social media all
0.432
social media-some
(L.322
terminal
0.592
Lio-tacstoe
0.508
nso-spinner
0.457
Table 10: Brute-force lask complexity and difficulty classification of MiniWoB. We split 65 tasks into the three category based the task complexity: easy (0.8 1.0), medium (0.6 0.8), and hard (0.0 0.6).
22

---

Published in Transactions on Machine Learning Research (12/2024)
E Details of MiniWoB Dataset
To resolve the data-imbalance problem, we first run Synapse (Zheng et al., 2023) on MiniWoB and collect 77K additional demonstrations across 16 tasks on top of 347K demonstrations (Furuta et al., 2023) to compensate for the lack of data in specific tasks (Strategy A). We use PaLM 2-L, (Anil et al., 2023) as a backbone LLM for Synapse. We then reduce the number of demonstrations for the tasks the agents can solve to focus on more challenging tasks, Based on brute-force task complexity (Appendix D), we consider the following four strategies shown in Table 11:
Removing 50% episodes from top-10 easy tasks (Strategy B; -733)
Removing 80% episodes from top-10 easy tasks (Strategy C; -102K)
Removing 50% episodes from easy tasks (Strategy D; -142K)
Removing 80% episodes from top-15 easy tasks and removing 50% episodes from other 11 easy tasks (Strategy E; 183K)
We hold out 21K episodes (5% of 347K 77K 424K) validation split, and after the convergence, choose the top-5 checkpoints that achieve higher offline validation accuracy, run those checkpoints online MiniWoB benchmarks, and then report the best success rate. Through the empirical evaluations (Section 4.4), we find that Strategy D realizes a well-balanced dataset to improve the performance. Except for these dataset mixtures, we follow Gur et al. (2023) for other training hyper-parameters of HTML-T5++. We have used cloud TPU-v3, which has a 32 GiB HBM memory space, with 128 cores. Each finetuning experiment. takes about 1-2 days.
23

---

Published in Transactions on Machine Learning Research (12/2024)
Task
Original (347K)
Strat. (424K)
Strat. (351K)
Strat. (322K)
Strat. (282K)
Strat. (241K)
book-flight
2.88%
2.19%
2.84%
3.11%
3.54%
4.15%
chorec-date
0.11%%
1.28%
1.42%
1.55%
1.77%
2.07%
choose-dato-aayy
0.97%
0.84%
0.95%
1.04%
1.19%
1.39%
chocee-date-mediu
0.61%
0.55%
0.63%
0.69%
0.79%
0.92%
chorse-list
0.54%
1.24%
1.42%
1.05%
1.77%
2.07%
click-button
2.82%
2.44%
1.39%
0.61%
1.73%
0.81%
blek--hutton-sqquee
2.88%
2.40%
1.42%
0.62%
1.77%
0.83%
click cheekkoossss
2.81%
2.43%
1.36%
0.55%
1.69%%
0.71%
chesk-ceebbbeeeebe large
0.57%
2.15%
2.16%
2. 19%
3. 3.06%
3.58%
clec--cheekoosee-ooff
2.00%
30%
2.63%
2. 87%
3.27%
3.83%
click-cheekhooess checkhooors-trrasseee
2.88%
2.40%
2.85%
3.1 11
1.77%
2.07%
click- collaaailee
1.71%
1.18%
1.69%%
1885%
1. 1.06%
1.24%
colaa-oollapaiblee
0.63%
0.55%
0.63%
0 0.68%
0.78%
0.91%
click-color
0.74%
1. 23.
1.40%
5.5%
1.74%
2.01%
clic--dialog
2 585
2. 49%
2.85%
3.11%
1.77%
0.83%
2lick-dialog:2
0.95%
1.366%
1.58%
1.70 70%
1.90%
2.28%
click-links
2.87%
2. 98%
1.42%
0.6 6.62%
1.76%
0.83%
elick-ment
0.53%
5%
1.42%
1.55%
1.77%
2.07%
opick-option
2.88%
2. 19%
2.841%
3.11%
1.77%
2.07%
click-pie
1.07%
2. 312%
2.66%
2.90%
3.30%
3. 8.86
click-seroll-list
0.00%
1. 1%
1.16%
1.26%
1.44%
1.69%
shades
0.00%
1. 95%
1.42%
1.55%
1.77%
2.07%
click-shape
1.76%
1. 80%
2.05%
2.24%
2.56%
2.99%
click-tab
2.88%
99%
2.84%
3.10%
1.77%
2.07%
click-tab-2
0.53%
0. 16%
0.52%
0.57%
0. 0..3%
0.76%
2lick-tah-2-harl
0.45%
1. 1.77%
1.91%
2.00%
2.38%
2.78%
elick-testt
2.88%
99%
1.42%
0.62%
1.77%
0.83%
click-teet-2
2 3.88%
2. 19%
1.12%
0.62%
1.77%
0.83%
wlick-widget
2.87%
2. 2.48%
1.41%
0.62%
1.76%
0.82%
count-slapp
1.69%
1.10%
1.26%
1.37%
1.54%
1.83%
email-inhox
1.40%
1. 1.20%
1.47%
1. 1.00%
1.83%
2. 1.14%
email- inbox- forward-nl
2.88%
2.49%
2.84%
3.11%
1.77%
2.07%
email- inbbx---b--eeee forwacd-nl-turk
1.11%
.22%
1.39%
1.52%
0.86%
1.01%
inbox-nl-turk
1.25%
1.08%
1.24%%
1.30%
1.54%
1.80%
enter-date
2.88%
2.40%
2.85%
3.11%
3.54%
4.15%
enter-pessword
2.88%
2.4 2.40%
2.84%
3.10%
1.77%
2.07%
enter-seact
2.88%
2.49%
2.85%
3.11%
1.77%
0.81%
eater-Sext-dyaantie
2.88%
2. 19%
1.12%
0.6 0.62%
1.77%
0.83%
enter-sime
0.00%
1.08%
1.23%
1.30%
1.54%%
1.80%
focus-text
2.88%
2.40%
1.42%
0.62%
1.77%
0.81%
locus-text22
2,88%
2. 19%
1.12%
0.6 6.62%
1.77%
0.83%
grid-coordinute
2.41%
2. 98%
2.38%
60%
1.41%
0.55%
nueee
0. 29%
1. 25%
1.42%
1.55%
1.77%
2.07%
ddettify-shape
2. 6000
2. 2.24%
2.56%
80%
1.00%
0.78%
login-user
2.82%
2.44%
2.79%
3.05%
1.73%
2.03%
login- pper-popup
2 82%
2. 94%
2.78%
3.04%
1.73%
2,03%
multi-layoute
2. 8888%
2. 19%
2.85%
3.11%
1.77%
2.07%
multi- drerings
2 888
2. 19%
2.85%
3,11%
1.5
4.15%
nav igaate-tree
2 2.84%
2. 46%
2.81%
3.07%
1.73%
2 2.02%
scarch-engine
2.56%
2.21%
2.52%
2.76%
3.14%
3.68%
social media
0.76&
0.68%
0.75%
0.82%
0.93%
1.09%
social-nee midi---edi--lll
0.03%
0.02%
0.03%
0.03%
0.03%
0.04%
medil--edia-oome
0.098
0.08%
0.09%
0.10%
0.11%
0.13%
Uo-tac-toe
1.14%
1.43%
1.63%
1.78%
2.03%
2.37%
uoc-mutoooplett
1.00%
0.86%
0.99%
1.08%
1.23%
1.44%
use-spinner
0.15%
1.19%
1.36%
1.48%
1.69%
1.98%
Table 11: Task ratio in rebalanced dataset for HTML-T3++
24

---

Published in Transactions on Machine Learning Research (12/2024)
F
Task Complexity Analysis in Web Automation
R = 0.691, 0.01
R - 0.579, x
R= - -0.433,p
=0000
Aver Average Success Rate
0.88
0.6
0.5-
0.6
0.4
0.4
0.4
0.50
0.55
0.60
0.65 0.70 0.75
0.80
0.85
0.90
20
30
40
50
60
70
80
100
3.0
3.5
4.0
4.5
5.0
5.5
6.0
6.5
7.0
Synthesized Success Rate
Instruction Tokens
Max HTML Subtree Depth
R = p..00
R 0.8.8, p < 0.01
== - 0. 0.342. 0.05
Peerage Success Rate
0.8
0.6
0.6
0.6
0.4
0.4-
0.4
Average
0.60
0.65 0.70 0.75 6.80
0.65
0.50 0.55
1.00
10
20
30
50
50
30
80
90 5.00
1000
1500
2000
Synthesized Success Rate (Human)
Number of Elements HTML
HTML Tokens
Figure 2D-scatter plots belween success rate averaged among LMAs (y-axis) and each statistic of compositional task (x-axis), such success rate synthesized with product of base task success rate, the number of instruction tokens, max depth of HTML subtrees, success rate synthesized with a product base task human performances, the number of elements in HTML, and the number of HTML tokens. Syuthesized success rate positively correlates with an average success rate (H 0.691, statistically significant in paired t-test with < 0.01), indicating that base task difficulty may determine compositional task difficulty. In addition, the number of instruction tokens (R 0.579; < 0.01) and the max depth of HTML subtrees (R -0.333; < 0.01) show negative correlations, which suggests the high complexity of observation and long instructions make the compositional tasks hard to resolve. In contrast, synthesized success rate from human performance, the number of HTML tokens, and elements in HTML just show relatively weaker correlations.
G Failure Examples with Advanced Language Models
We provide several failure episodes with advanced language models such as gpt-4 and text -oxt-davinci-003 in Table 12. The left columns have correct plans, and the right columns have failure plans. Qualitatively, LMAs based on advanced models manage reduce the ratio of failure modes in gpt-3.5-turbo (Table 3), such skipping necessary intermediate steps and hallucination of unnccossary actions. However, they tend to suffer from tiny errors more such as capitalization (RCI, AdaPlanner) or attributes in HTML (Synapse).
25

---

Published in Transactions on Machine Learning Research (12/2024)
X 1. click //"aadd 0QmOEUeT
2. click // //"ded ueername"
4. click (1"ddd== peaeword"
click /!|[odd= "abitto"
Select dQuiEE and then cater the wsemenu "cristin and the toxt poess. login
type cristin
5. type MS
Synapee (Zhven al. 20223)
click (feee "00m0EDE /mmput
2. click ff (podd "ueeraamee ]
ciick (/(dd "paasword""
click //(eddd "sabbrn")
Table 12: Failure examples in CompWoB with advanced models (gpt-4, text text-daviie davinciiini---ii 0033..
type MS
type
click Hieann teet() "angge" click !/" Bdatto tydto--yppe typer button")
Click on the link Augue and then click on a "battoo widget
AdaPlanner (Sun a), 20125)
click /eppnnteett O) "Angue" click ("(ddats-typee button]]
eboc Heganttortt toxt "adipiecingg" click //button[tedd (bex][]== 'Submit]
Click on the Btokk "uclppiocting" and then click: on the "culoot butfon
X
RCI (Kim on al, 20023)
dbek /asa teet "adipiocingg
cbick (text "uboait
26

---

Published in Transactions on Machine Learning Research (12/2024)
H
Per- Task Performance on MiniWoB and CompWoB
Task
HTML-T5 (Gur et al,, 2023)
HTML- (Ours)
book-flight
0.99
0.99
thoose-date
0.16
1.00
thoose-date date--da---
1.00
1.00
choose-date-medum
0.56
1.00
choose-list
dlck-bbttoon
1.00
1.00
clcckbbutonn-eeqeecee
1.00
1.00
clickccheccbooes
1.00
1.00
cicckcchellboore large
0.90
0.97
cick-cleckboxee-sfftt
1.00
blick-checdboxxeerrannnn
cllck-cllaeeiile
1.00
1.00
click-collapsible22
0.93
0.96
click-color
1.00
1.00
click-dialog
1.00
1.00
clickdlaaog-2
0.74
1.00
click-liuk
1.00
click-menu
0.37
0.96
click-option
1.00
1.00
click-pie
0.96
0.94
click-scroll-list
0.99
1.00
click-shades
1.00
click-shape
0.95
rlick-tab
1.00
1.00
click-tab-2
0.94
0.98
click-tab-2-hard
0.88
0.96
click-test
1.00
1.00
click-test-2
1.00
elick-widget
1.00
1.00
ronntshape
0.67
0.92
email-inbso
1.00
0.98
esnail inbox forward-in nl
1.00
1.00
email-inhox-tee www-i-nnoooaarteeee wow.woooww--tttteee nl--turk
1.00
1.00
emaillinbox--ll-tun
1.00
ester-date
enter-pasee enter-password
1.00
1.00
enter-text
1.00
1.00
euter-texd--yyaande
1.00
1.00
enter-time
1.00
1.00
focus-text
focus-Lexi-2
grid coordinate
1.00
1.00
muess-mmbbr
0.13
1.00
sdeutil--shape
1.00
1.00
login user
1.00
1.00
login- pppppppp-p
multt-layouus
1.00
Guult--uderiiin
1.00
1.00
nav havigate-tres
0.99
1.00
seacch-nngiue
0.93
0.96
social mediaa
0.99
1.00
medlal-edda--al
0.31
beial-media-sooe.
tios tac-toe
0.57
0.55
use-sutocomplete
0.97
0.99
use-spianer
0.07
0.06
Average
0.856
0.952
Table 13: Per-task average rate 56 tasks from MiniWoB++. We refer to Gur al. (2023) for the baseline performance.
27

---

Published in Transactions on Machine Learning Research (12/2024)
134
Abraua AaPlie
640
1800
6.00
6.00
6.00
1.0
6.0
1.0
13M
Syloe 1g2-4
170
1100
6100
0.66
6000
6000
110
6410
6.20
010
6100
5110
190
0901
0800
9800
1880
035
0.50
1900
1000
0100
190
1109
6.38
606
170
6008
6100
6758
6008
1000
6100
11:00
580
6211
500
Sriape
371
110
110
100
6100
6100
4118
4008
6108
610
6218
6110
676
6100
0182
Souges
Table Parkfask snocess rate on Comp WoBB
090
100
6155
67%
636
630
675
610
610
110
0.30
6.99
610
800
8440
0.305
10:4
14
830
234
23
880
330
6.54
6.0
5.00
180
840
110
6200
1100
5000
6880
530
620
500
600
240
8400
610
618
3-3
17
610
110
5100
608
622
0.25
143
114
108
110
113
100
1110
120
615
104
148
146
18
180
103
14.80
672
17%
168
1624
1350
162
1488
1088
24
15
170
5.99
476
170
1100
611
604
620
41%
1.24
28

---

Published in Transactions on Machine Learning Research (12/2024)
01.00
0100
61700
1150
01.00
1100
(100)
0100
0100
0100
0.300
11000
11,00
(1,00
10000
0.000
0.000
0.00
01.00
0.000
(1000
1..00
6.16
0.10
0.000
00000
0.000
Sycapen (oee)
0.000
0.500
6AT1
0.55
0160
0.00
0100
0.00
0.30
0100
0100
0.382
0200
0100
0200
0300
0180
0/00
0.000
0.000
0100
0100
01600
0100
07800
0300
01000
0.000
0:500
0,00
0,00
0200
0200
0,000
0200
01000
8:536
01318
6198
1-576
01988
6198
01900
0191
022
0.30
6191
0.341
R4900
11010
0100
0120
0193
01090
11:00
01031
0.00
6120
0.23
Syrsqee (1ee)
0.70
0:000
0.30
0100
0.08
0100
0100
0100
01000
0.200
0100
01000
0.00
0.00
0.06
0.000
6,00
0.00
01000
0.000
0,090
0100
01000
01000
0,000
0.00
0.00
0/00
0.000
0100
0.008
NC1 (21-1)
0.30
0.0
0.500
6550
0100
0.78
0.00
100
040
0.00
000
0:98
0
0100
0.00
0.000
0108
0140
0.00
0.100
0:000
0180
0100
01000
0:000
0.003
0.68
0.70
0.000
0108
0158
0.00
01-00
Table 15: Per-task success rate on CompWoB with reverse order instructions.
NC1 (ran))
0.5g
6030
10000
0000
1000
0:00
000
0100
0.30
6750
1080
(080)
0000
0000
0000
0060
0000
6150
0350
0080
10000
0100
6.00
0.00
0200
0000
0U0
0080
00754
0.722
RXT cwnrd
6:21
659
632
R-mu
0.60
680
620
699
609
0.310
R40
RGM
6GM
6100
0330
660
63:00
015M
0014
0:30
8.0
699
6131
00200
0.503
NeT (te))
5.50
0:36
0.00
0.50
0300
710
1:00
0106
0100
0100
0.299
0.06
0110
0.00
0.06
0:00
0:20
0100
0.10
6:20
0135
0100
0180
0100
6.000
0:00
676
5:00
016
0100
01000 0.770
NCT Det0
6155
(60))
06.00
0000
000
0.60
0101
0000
000
0.252
16:03
1090
0000
0000
0122
0000
01.00
(1:33
(115)
10000
1030
0.00
012
0/0
000
030
0090
0000
01408
0:53
0:3
0:33
6118
0.28
0:38
0.50
035
055
0.75
0.9
0164
0.361
01220
0.988
010
0.99
0:340
1.350
0:200
1391
01:25
0000
6338
0.50
033
0.11
0.30
0000
0-203
0.243
0.00
4110
0160
122
0100
0.70
6.20
1100
110
0:00
0.31
0/86
0.270
0.000
0.02
0.00
0.mm)
01000
0.000
0,044
0:300
1:00
0,08
0:27
1:300
0.30
0.200
0,307
0.307
WAIRUM
6330
0.500
4000
1090
0.80
01
0338
0990
0336
039
0.260
6009
10000
1,000
0098
00380
0390
0000
03300
00500
11.20
1:000
0.350
0555
013
(139)
0030
01990
402
d8 femee sopeoopee iit-iii viten 194 ferm Mio deddoooe A tytii 100 dhaiig legiin ven
(5h bettor wopeonoooon...... (04 allent de Ihs. d8 betten des deabboo 1004 spfor 100 Eliie
di0- Si-iigt aideiti 2300 bottoo 18 daccacc 2ib- cooppto.. dis diabe
Sodeettieen www.uliio shioestrt, 28 dab-dtiooooo parpartdd Sa Sto.
tanoonn gont mtablto Kowambel aade innaaii
be mare mnnoperee dibcliabs
da lasoros
Avergg (1moeett (egsterdim))
1300 wappoeeeppaaoo OB bopppi
be weelleeeed nelireus
nigg pooved
Beleeie des ual Sand
355 qiie srrgatii ine
1800 dee
wwe.eutteee 155-tnn the
355 230 riggt
330-ppi ligit oos
kgo oe sargan toe
Amirag (narwa)
da thib de amtee
Aeongge (trestio))
aito-aat, Vighnee
(bmi))
loge cag-aangoo
Thal
29

---

Published in Transactions on Machine Learning Research (12/2024)
1:3
100
1.0
0.5
Seacu Aaftaas Atllgan
140
241
190
246
1124
0.34
135
196
196
1226
130
170
5.11
610
1.4
130
103
103
1000
1.30
Table 16: Optimal success rate on CompWoB (synthetically calculated a product of buse task success rate Mini MiiiiWBB..
0.56
Staggs
410
410
6.30
600
670
679
0.8
610
4400
4100
4mm
6400
6000
6100
11.11
600
120
450
100
110
6.00
0.59
114
116
18
16
1.10
470
1600
6.00
303 HumE 121
10.
476
100
0.0
110
130
4.00
6.0
424
664
610
680
487
210
1600
6.54
301 101 303
160
264
230
180
200
260
1600
4.55
X00 Lumal
532
0.30
0.59
14
303 Jahee Spncum Andiaaan 3013
688
180
3.00
0.50
0.40
160
180
136
1399
144
146
360
1550
210
240
146
176
14 ann aaa
44 Ma
30

---

Published in Transactions on Machine Learning Research (12/2024)
Prompted Language Model Agents with Oracle Exemplars
In this paper, we assume, as realistic and important constraint, that: (1) prompted or finetuned LMAs are deployed as a service in the real world; (2) users are only allowed to interact with the agents vía providing task instruction, and (3) not allowed to intervene the backend system or prompts. Providing exemplars or preparing finetuning demonstrations for every compositional problem is infeasible given the huge space of web automation problems. Moreover, significantly hinders the user experience that the agents would be good at some specific format of instructions and that their violation of those affects the performance even if semantically the same.
As sanity check of the environments and baseline agents, we here demonstrate that prompted LMAs can change their behaviors adaptively by modifying their prompts and demonstrations. We evaluate the performance of RCT with gpt-4 and oracle exemplars on 20 two-way tasks CompWoB. We provide two demonstrations per task the prompls. Figure shows that RCI with gpt-4 and oracle exemplars achieves 82.9% success rate, which is the best among the baselines, such HTML-T5++ (73.9%), RCI with 10t-3.5-ture and oracle exemplars (56.8%), RCI with combination exemplars (gpt-3. gpt-4: 71.5%). See Table 14 for other baselines. This ensures that the tasks are feasible, and that if a prompt includes how to perform compositional lasks, the performance gets better. In contrast, while oracle demonstrations help improve performance, they could not fully resolve the issues from reverse-order instructions.
46.9%,
Through the experiments in Section 6, we have intended shed light on their zero-shot generalization capabilities of dealing with unknown sequential-task compositions. For reliable and robust web automation agents, might need to leverage both prompting and finetuning approachos at the different development stages. For instance, prompted LMAs might work well adaptive automated data collectors to the novel domains and finetuned LMAs as deployed agents for service.
CompWoB
Reverse-Order
Avg. Success Rate (%)
Avg. Success Rate (%)
82.9
80
80
73.9
71.5
71.8
60
56.8
60
57.3
49.4
46.9
440
440
36.4
33.6
HTML-T5++
RCI (comb)
RCI (oracle)
HTML-T5++
RCI (comb)
RCI (oracle)
gpt-3.5-turbo
gpt-4
w/ oracle exemplars
Figure 7: Average success rate of LMAs in 20 two-way tasks from CompWoB. RCI with gpt-4 achieves the best performance when the oracle exemplars provided (82.9%) in the prompt. While oracle demonstrations help improve performance, they could not fully resolve the issues from reverse-order instructions (for instance, 82.9% - 71.8% in RCI with gpt-4).
31

---

Published in Transactions on Machine Learning Research (12/2024)
J Task Complexity Analysis with Language Model Agents
Following the recent work in deep reinforcement learning literature (Furuta et al., 2021), we measure average performance as a proxy of oracle task solvability. If many kinds of agents perform poorly, such tasks are regarded as challenging. This can shed light on"task" or "environment" themselves, rather than "language model agents" or "prompting methods", while such an analysis has been overlooked so far. Additionally, while the trend with average might be the same, distributional characteristics for cach language model agent could be different. We extend our analysis by reporting the individual performances of each agent in Figure 8. Our results still indicate that while all the language model agents (HTMI-T5++, RCI, AdaPlanner, Synapse) often show negative correlations wetween the success rate and instruction tokens or max subtree depth with statistical significance, the trends for other statistics may differ among the methods.
0.49,
-0.427.p=000
-0 p.0.002
R= 00205.p-0000
3.0
0,386, 0.00
R=
0.332.p 0.05
0.045, p 0.755
R 0.079.p 0.587
0,398, 0.0.01
R= 0..333 p 0.0.1
0.236 P 0.100
R- 0,292 0.00
60
1000
2000
p.0..0
-0236, p 0.100
P 0.177
-0.183, p 0.207
Success
Instruction Tokenss 93 330
4.
HMML Subtree
20
10 00
Elements HTML
60
523
1100
HITML Talees
2010
30
AdsPanner
Synause
Figure 8: 2D-scatter plots between the success rate for each LMA (y-axis) and each statistic of compositional task (x-axis), such as the number of instruction tokens, max depth of HTML subtrees, the number of elements HTML, and the number of HTML tokens. The results imply that while all the language model agents (HTML-T5++, RCL, AdaPlanner, Synapse) show negative correlations between the success rate and instruction tokens with statistical significance, the trends for other statistics may differ among the methods.
32

---

Published in Transactions on Machine Learning Research (12/2024)
K Finetuning May Generally Outperform Prompting in Compositional Tasks
In this scction, we examine the capability of finetuned LLMs and prompted LLMs for compositional generalization in natural language reasoning tasks, rather than agentic tasks. We employ Last Letter Concatenation (Wei et al., 2022; Zhou et al., 2023a) as a benchmark, where if LLMs take "hans, structure" as a question, LLMs are asked to answer it as The last letter of "hans" " is "s" The last letter of "structure" is "e". Considering "s", "e" leads "se". So, "hans, structure" outputs "se". This reasoning problem exhibits a question. For finetuned L.MMs, we
compositional nature we increase the number of words provided adopt 1 11B-parameter Flan-T5-XXI, (Chung et al., 2022; Raffel et al., 2020) base model, and prepare 2000 examples per n-letter for training dataset. For prompted LLMs, we employ Flan-PaLMMSSB, 62B, and 540B (Chung et al., 2022; Chowdhery et al., 2022); and prepare randomized 4-shot CoT prompts from the same training dataset. We randomly use one of them during inference. Furthermore, we use up to 8 letters as a training dataset and few-shot exemplars and also use 9-12 letters as a hold-out test dataset to investigate the compositional generalization in natural language reasoning lasks.
Table 17 shows that finetuned LLM (Flan-T5-XXL.) almost always outperforms prompted LLM (Flan-U- PaLM-540B) on average (76% v.s. 44% in an exact match) and also in both in-distribution (2-8 letters) and out-of-distribution (9-12 letters) settings. This implies that finetuned LLMs may generally outperform prompted LLMs in compositional tasks, which can also lead to the better performance of finetuned LMAs than prompted LMAs in sequential web automation tasks, as shown in the main text.
# of Letters
Models
2
4
5
6
7
10
11
12
Ave.
Flan-PaLM-8B (Prompted)
0.10
0.01
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.01
Flan-PaLM-628 (Prompted)
0.48
0.16
0.05
0.03
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.07
Flan-U-PaLM-540 (Prompted)
1.00
1.00
0.75
0.76
0.57
0.41
0.16
0.08
0.06
0.00
0.01
0.44
Flax-T5-XXL (1IB; Finetuned)
1.00
0.98
0.99
0.98
0.98
0.96
0.96
0.83
0.47
0.20
0.05
0.76
Table 17: The performance of finetuning (Flan-115XXL (Chung et al., 2022; Raffel et al., 2020), 11B parameters) and (Chung et al., 2022; Chowdhery et al., 2022)) in Last Letter Concatenation (Wei
prompting (Flan-PaaM--81) PPLLM88B662B//8588 et al., 2022; Zhou et al., 2023a). We measure the performance with an exact match. We use up to 8 letters training dataset and few-shot exemplars and also use 9-12 letters as a hold-out test dataset to investigate the compositional generalization natural language reasoning tasks. The results show that finetuned LLM (Flan-T5-XXI) almost always outperforms prompted LLM Mlaa-U-PPLLM-50 in both in-distribution (2-8 letters) and oit-of-distiibution (9-12 letters) settings.
33


Reviews:
[{"review_id": "d1J1CdB6ZU", "reviewer": "Reviewer_QndN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n1. The paper makes it clear that current model fails in the compositional web automation tasks, and builds benchmark based on previous dataset to evaluate this point. \n2. The proposed data rebalancing method for finetuning HTML-T5++ achieve improvement on zero-shot performance of CompWob. \n\nWeakness: \n1. While CompWoB is a valuable addition to the field, it has quite a narrow scope and the data is from the exisiting dataset MiniWoB. I beileve more diverse data and tasks can be added to the benchmark. \n\n2. I'm having trouble understanding why prompt-based methods can fail on the compositional tasks. For example, in Synapse, the instruction is revised by LLM to more detailed step. And in Fig 1, it seems the difference between base task and compositional one is only adding more task instructions sequentially. In this way, the prompt-based method should still understanding each of them and revise into a longer list of instructions. However, the performance drops significantly, so maybe authors can provide more explanation and insights on that. \n\n3. For the data-rebalance method, it makes sense that the proposed method improve the sucess rate on MiniWoB, since more complex tasks are chosen and used for fine-tuning. But I'm wondering why this can help with the compositional case. From my perspetive, the compositional benchmark is to evaluate the multi-task capability of the model, and it can't be enhanced by training with more difficult single-task examples. Or does the training also include compositional samples in HTML-T5++?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. This paper shows that language model agents (LMAs) have a drastic decrease in success rates on compositional web automation tasks. \n\n2. The authors introduce a new benchmark, CompWoB, to assess the effectiveness of language model agents (LMAs) in handling complex, compositional web automation tasks. \n\n3. HTML-T5++—a model trained with a balanced dataset—outperforms other models by achieving human-level proficiency on MiniWoB and the highest zero-shot success rate on CompWoB.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the \"Weakness\" part above. The author may provide more explanation or results on the points: \n1. Why the previous method can solve single-task very well while fail on the compositional tasks? It seems the multiple tasks are independent from each other, so what is the reason that makes the previous model can't solve it step by step? \n\n2. The connection between compositional task and data-rebalance method is unclear to me. The authors can talk more about why this can help.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eS7EzhEOEr", "reviewer": "Reviewer_L9c7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths inlcude:\n- this benchmark task is seems like a useful ramp-up in difficulty from the single tasks in MiniWob, representing more realistic real-world chains of tasks.\n- categorization of the compositional task helps determine where agents are strong or weak specifically.\n- the authors test various planners with strong llm backbones, using multiple prompting strategies per LMA.\n- authors designed the task compositions to be realistic, e.g. pop-up windows following a login interaction.\n- the work sheds some light on the effect of prompt ordering and the effect of task complexity for recent LMAs and advanced proprietary models, highlighting challenges for even the most powerful LLMs.\n\nWeaknesses include\n- This work tests transfer of a model finetuned on base tasks (HTML-T5++), to compositional tasks from CompWoB. It is unclear to me whether this improvement comes from simply finetuning on the base tasks that are now part of the test set, or whether the LMA has become better at compositional tasks. Does the \"transferred\" dataset exclude the MiniWoB tasks that were used to create CompWoB? If not, the LMA may be trained on the test set. \n- Similarly, it isn't immediately clear whether the LMA fails because it fails at one of the base tasks in the compositional tasks, or if does not produce meaningful actions altogether for conjunctions. From the user point of view there may be no difference, but if a LMA always fails at a particular base task, then it may automatically fail at the composed tasks as well. Is the performance on just the subset of miniwob tasks used in CompWoB so high that this is not an issue?\n- Section 5.2 mentions manual curation of tasks, but section 5.1 mentions a selection process based on brute force metrics. Which is it? \n\nLastly, as with most LMA research, the base model is for most experiments a powerful but proprietary LLM. This makes it hard if not impossible to reproduce this work, and we cannot exclude that these models have been trained on MiniWoB (which is open source). I do not believe this makes the investigation invalid, the results are still insightful given that multiple planners share the same base model, but it does mean current benchmark scores cannot easily be used as ground truth baseline in future work. As evaluations cost around 3.6k USD, I hope this is not a limiting factor.\n\nI believe this investigation is a potentially useful benchmark task expanding MiniWoB. The innovation may be limited, but the evaluation seems rigorous. Assuming the point above on transferred models is resolved I believe this benchmark is relevant for a subset of the TMLR audience.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a new benchmark for language model agents (LMAs), and evaluates various LMAs with a gpt-3.5-turbo , gpt4 or html-t5++ base model.\n\nThe benchmark includes combinations of tasks from the previously established MiniWoB++ benchmark, by creating conjunctions of individual tasks after filtering. Authors also experiment with reverse-order composed instructions (do A, after doing B). Identifying which tasks can be chained is not straightforward, as some tasks may only make sense in a specific order, or be too challenging in a conjunction. Composing of base tasks therefore required some curation, and the authors describe their selection procedure. This means it may be possible to extend the compwob benchmark, as new tasks may be selected to follow a similar distribution. \n\nThe authors test both zero-shot performance of off-the-shelf LMAs and performance of agents that can learn from interaction. Primary findings are that LMAs that train on multiple examples of (individual) benchmark tasks in the prompt are better at handling composition, and that the instruction order has a big effect on performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- please provide more information about the exact setup for the transferred model. Does it ever see base tasks that are part of the composition test set? If so, is it still a meaningful comparison?\n- although compositions may not make sense for some environments, what would be the downside of chaining arbitrary combinations of tasks, and training on the resulting data? Would this allow transferred models to learn to handle both regular compositions and reverse-order instructions?\n- in section 5.2, the number 50 is mentioned a few times, but the phrasing can be improved. I initially understood that you added 50 train and 50 hold-out test tasks due to the sentence \"our addition of a hold-out test set\", but this is still talking about the same 50 tasks. \n- what is the intuition for larger degradation for reverse-order instructions for some columns in Figure 3? Section 6.2 concludes that all LMAs are susceptible, but there seems to be a big difference.", "broader_impact_concerns": "I think automating web navigation has many potential useful applications, but there is a big risk that a LMA makes critical mistakes, e.g. when filling in wrong user information, submitting incomplete forms. The only potential concern of any benchmark IMO is then that it gives a false sense of security; in this case, this work demonstrates that LMAs are far from secure, so I don't think that's an issue currently.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KGbkgnsmbT", "reviewer": "Reviewer_U8ks", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The design principles of the study have been fully described, including how the authors create the benchmark and its dataset. This is very helpful to understand this paper.\n- It is reasonable to use the multi-step decision making environment as a benchmark instead of real-world environments. Such an intermediate-level benchmark is necessary to gain finer understandings of the capabilities of existing LMAs.\n\n# Weaknesses\n- As far as I remember, the authors have not discussed why the performance of the finetuned LMAs were more robust than that of the prompted LMAs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors are interested in the capability of language model agents (LMA) in solving multi-step decision making tasks. To this end, the authors propose a new benchmark suite called CompWoB-50, which consists of multi-step web automation tasks, which are synthesized by composing base tasks from MiniWoB. By evaluating existing LMAs designed for MiniWoB in a zero-shot manner, the authors evaluate their generalization capability.\n\nThe experimental results suggest that\n1. while prompted LMAs perform better in a single-step decision making than finetuned LMAs, prompted ones are more robust to the multi-step setting as well as reversed-order instructions,\n1. the performance is related to the estimated success rate of the composed tasks, the number of instruction tokens, and the depth of HTML subtrees.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to request the authors to discuss why the performance of the finetuned LMAs were more robust than that of the prompted LMAs (if it is discussed, please let me know where it is, and please forget this request). In my opinion, such a discussion with preferably some empirical evidences is one last missing piece of this study, and is mandatory for acceptance.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Before providing additional comments, I'll describe the required revisions for the paper's final acceptance:

- Clarify the hyperparameter optimization process for rebalancing.
- Discuss whether the hold-out test set was used to determine the best rebalance. Avoid conducting new experiments, as the current ones have already been reviewed. Instead, qualify the observations based on the existing results.
- Mention hyperparameter optimization and data generation in the bullet points list preceding Section 2 (Related Works). This clarification will help readers better understand the contribution. For example, the change could clarify that the reported success rate for HTML-T5++ represents an upper bound achievable with data rebalancing.
- Provide estimates of the computational cost associated with generating the rebalancing data and finding the optimal balance.

Now, I'll proceed to add some additional comments.

While it could be argued that generalization should ideally be tested on a wide variety of realistic examples, the reviewers and I agree that the composition in this paper effectively allows the observation of deterioration phenomena in a controlled setting. This setup facilitates a more insightful analysis of the impact of composition.

One reviewer pointed out that each additional task corresponds to another instruction, which might make the task easier for fine-tuned models. I agree that this is an interesting question. The paper argues that selecting the relevant information is related to the so-called reasoning capabilities of LLMs. The higher generalization of fine-tuned models is supported by experiments on a task called "last letter concatenation." Another related observation from a different reviewer is that inverse instructions might feel somewhat artificial. I find the conclusions for "last letter concatenation" and inverse instructions to be slightly circumstancial but not to the detriment of the paper’s value.

As a final comment, while achieving human-level performance on a specific benchmark is typically seen as a positive development, I find it concerning in this context. Unlike domains such as Go, chess, or protein folding, solving web-based tasks remains fundamentally a task where humans perform better. If a model achieves human-level performance on these tasks, it could signal that the benchmark is insufficiently challenging or that the model is exploiting artifacts. However, I am not requesting that this issue be addressed in the final version.

### Additional Issues

- Table 11 is not referenced in the text.
- The red numbers in Figure 3 are not explained in the caption.
Decision: Accept (poster)