SteP: Stacked LLM Policies for Web Actions
Authors: Paloma Sodhi
S.R.K Branavan
Yoav Artzi
Ryan McDonald
Venue: COLM (2024)
Keywords: LLMs, web actions, policy composition
TL;DR: Dynamically compose LLM policies using a control stack to solve a diverse set of web tasks

Abstract: Performing tasks on the web presents fundamental challenges to large language models (LLMs), including combinatorially large open-world tasks and variations across web interfaces. Simply specifying a large prompt to handle all possible behaviors and states is extremely complex, and results in behavior leaks between unrelated behaviors. Decomposition to distinct policies can address this challenge but requires carefully handing off control between policies. We propose Stacked LLM Policies for Web Actions (SteP), an approach to dynamically compose policies to solve a diverse set of web tasks. SteP defines a Markov Decision Process where the state is a stack of policies representing the control state, i.e., the chain of policy calls. Unlike traditional methods that are restricted to static hierarchies, SteP enables dynamic control that adapts to the complexity of the task. We evaluate SteP against multiple baselines and web environments including WebArena, MiniWoB++, and a CRM. On WebArena, SteP improves (14.9\% to 33.5\%) over SOTA that use GPT-4 policies, while on MiniWob++, SteP is competitive with prior works while using significantly less data. Our code and data are available at https://asappresearch.github.io/webagents-step.

Published as a conference paper at COLM 2024
SteP: Stacked LLM Policies for Web Actions
Paloma Sodhi" S.R.K. Branavan', Yoav Artzi! Ryan McDonald
ASAPP Research, NY, USA
"Corelll University, NY, USA
{psodhi, branavan, rmcdonald)@asspp..com com, yoav@cs caav@ss.cornell... wwwnelc.ceeeeooooooooo..........
Abstract
Performing tasks on the web presents fundamental challenges to large lan- guage models (LLMss, including combinatorially large open-world tasks and variations across web interfaces. Simply specifying a large prompt to handle all possible behaviors and states extremely complex, and results in behavior leaks between unrelated behaviors. Decomposition to distinct policies can address this challenge but requires carefully handing off control between policies. We propose Stacked LLM Policies for Web Actions (SteP), an approach to dynamically compose policies to solve a diverse set of web tasks. SteP defines Markov Decision Process where the state is slack of policies representing the control state, i.e., the chain of policy calls. Unlike traditional methods that are restricted to static hierarchies, SteP enables dymamic control that adapts to the complexity of the task. We evaluate SteP against multiple baselines and web environments including WebArena, and CRM. On WebArena, SteP improves (14.9% to 33.5%) over SOTA that uses GPT-4 policies, while on MiniWob+++ SteP is competi- tive with prior works while using significantly less data. Our code and data are available at https: /awpp//paaappppaaa///aaaaaaeaeaa weetaarrchhhhhhthh 10 wwwwebagents---ee step.
1 Introduction
While large language model (LLM) agents have shown impressive decision-making capabil- ities (Yao et al., 2022b; Huang et al., 2022b), the web remains a challenging domain achieving much lower success rates compared to other benchmarks (Akter al., 2023; Zhou et al., 2023). The web contains a combinatorially large open-world space of tasks such as booking flights, purchasing items or making appointments. Web interfaces also differ substantially from one website to another, for instance, the task of purchasing item on Amazon looks different from purchasing it on eBay.
There fundamental challenges to designing singular LLM policy to solve all possible web tasks. First, the policy requires instructions and examples to cover all variations in tasks and websites. Second, solving longer horizon tasks requires keeping around long history of previous actions and observations in context. Longer contexts make harder to pay attention to salient information leading to more errors and costs (Liu et al., 2023).
Instead, a natural solution to decompose the problem into distinct policies (Khot et al., 2023). Each policy provides dedicated instructions and examples for particular subprob- lem, such as searching a list or finding a page. However, this typically requires manually specifying decomposition hierarchy that hands off control between policies (Prasad al., 2023; Zhou et al., 2021; Song et al., 2023). This restricts control to static hierarchy that fails to adapt to varying task complexity.
Our key insight is to enable dynamic control, where any policy can choose to invoke any other policy. Such expressiveness is crucial for solving web tasks that require policies operate at multiple levels of abstraction. Consider Fig. 1 where the agent must find all commits
*Corresponding author: Paloma Sodhi <psodhigasapp.com comP

---

Published as a conference paper at COLM 2024
SteP control flow
Objective: "Find all commits made by <user> across all repositories"
==
r=3
search Jist:
find_page ('commit
search, listt cuser=" "commit lirt))
commits"
"repostiory
Policy Stack
find pagel..)
search, list...)
Policy Library
search. lisTt.)
find
fill form
pagel..)
lisst..)
listt..)
Figure 1: SteP composes policies to solve complex task, where policies can invoke each other. SteP uses a policy stack to keep track of the dynamic control state. Given an objective "Find all commits made by <user> across all repositories" SteP intializes with search_ list() to search over all repositories, which in turn invokes another search_ list() to search overall commits in repository.
made by user across all repositories. A search_liste policy must first iterate over all repositories. For every repository, it must recursively call another march_listt () that iterates over all commits in that repository. This can only be solved by an architecture where policies can call each other, including themselves.
We propose Stacked LLM Policies for Actions on the Web (SteP), method to perform diverse set of web tasks by dynamically composing policies. SteP defines a Markov Decision Process (MDP) where the state is stack of policies. The stack stores the dynamic control state capturing the chain of policy calls that evolves over time. At every time step, the policy on the top of the stack either acts directly that gets pushed onto the stack, or terminates and pops out of the stack. For instance, in Fig. task, the stack initializes with search_. list () policy, which can both act the web page or instantiate another search_ (ist() policy with different set of arguments.
the web page, invokes new policy
Our key findings that dynamically composing policies (SteP) significantly outperforms both prior works (0.15 + 0.33) and single policy baselines (0.23 -> 0.33). SteP achieves this while using 2.3x lesser tokens per trajectory, resulting in lower overall costs. We also show several ablations on the effect of varying contexts, in-context examples, and CoT reasoning.
Our key contributions are:
1. A novel framework SteP that defines an MDP over stack of policies enabling dynamic composition of policies to solve complex web tasks.
2. Experimental validation on range of web benchmarks: WebArena, MiniWoB++, and an airline CRM simulator. On WebArena, SteP improves (0.15 0.33) over prior works that use few-shot LLM (GPT-4) policies, while on MiniWob++, SteP is competitive with prior works while using significantly less data.
3. Implementation of SteP as meta-policy that wraps around any existing policy class.
2 Related Work
Language models for web tasks. Early work mapping natural language instructions into actions (Branavan et al., 2009; Artzi & Zettlemoyer, 2013) has rapidly evolved into the field of LLM agents (Wang al., 2023b). Broadly, methods include training RL agents to navigate web interfaces (Humphreys et al., 2022; Liu et al., 2018; Shi et al., 2017), in-context learning with large language models (Zhou et al., 2023; Zheng et al., 2024; Kim al., 2023), or finetuning language models on web tasks (Deng et al., 2023; Furuta et al., 2024; Yao et al., 2022a). With in-context learning, using a single LLM policy that contains all instructions and examples results in long contexts that can be error-prone. Recent approaches (Zheng et al., 2024; Kagaya et al., 2024) counter this by retrieving trajectories from a database. However, covering the combinatorial space of tasks requires a large dataset from many tasks. Instead, our work leverages a library policies, each with dedicated instructions and examples, and composes policies cover such tasks.
2

---

Published as a conference paper at COLM 2024
Language models for decision making. Instruction following LLMs have shown impressive decision making capabilities (Huang et al., 2022a; Brown et al., 2020) and tool use (Schick et al., 2023; Yang et al., 2024) by chaining together reason and actions (Yao et al., 2022b). However, for long-horizon tasks, single policy with a long chain of reason and action can be error-prone. Broadly works deal with such issues by hierarchical planning (Prasad et al., 2023; Khot et al., 2023; Zhou et al., 2021), defining state machines (Ma et al., 2023), generating code (Wang et al., 2023a; Liang et al., 2022), or by self-correction (Shinn et al., 2023). Hier- archical decision-making has rich history in AI (Sutton, 1998), where high-level policy chooses either from library of skills (Tessler et al., 2017), or predicts subgoals (Nachum et al., 2018) or predicts rewards (Vezhnevets et al., 2017) for a low-level policy. While these methods focus on efficiently learning policies, they restrict the decomposition to a prede- fined hierarchy of 2 or 3 levels. Instead, our framework allows any policy to dynamically call any other policy in the library, with control state being tracked using policy stack. Com- pared to decomposition approaches like DecomP (Khot et al., 2023) which generates static program with subroutines that cannot invoke each other, SteP decomposes dynamically based on observations and allows policies to invoke each other. Compared to hierarchical planning approaches like ADalT (Prasad et al., 2023), which predicts a plan, SteP instead predicts policies that can react to failures more quickly.
Automata and Transition-based Parsing. The fact that the main control structure of our formalism stack relates our algorithm to Pushdown Automata (Hopcroft & Ullman, 1969). However, in Pushdown Automata, the input tape is static and processed sequentially. In contrast, for web actions the input tape is dynamic due to new observations that arise. These observations dynamically alter the context (or input string) that the policies work with. While different in scope, our work does draw inspiration from transition -basd dependency parsing systems, specifically stack-based algorithms (Nivre, 2008). In particular, our algorithm consists of states whose main data structure is a stack and states transition to new states via finite set of well defined actions (see Sec. 4.1).
3 Problem Formulation
Given a natural language instruction, such as "Book me flight from NYC to BOS", our objective is to learn policy π that execute this task on web environment. This can be formulated as a Partially Observable Markov Decision Process (POMDP), denoted as (S, A, A, r)) At each time step t, the policy (or agent) performs action a given partial observation 0,, resulting in a new state 5 and observation 0++1, where:
State, s S is the current state of the web environment, including the current webpage contents and results from previous interactions, e.g., a new repository created in a GitLab environment.
Action, a E A(s) denotes the possible actions that can be performed in the current state, such as clicking, scrolling, typing on specific web elements. These are represented as click [id], type [id] [value], where id refers to a specific element on the webpage. The action space is typically large due to the many elements on a web page. Observation, o O is the current observable aspect of the state, i.e., the current webpage Document Object Model (DOM) serialized as text.
Transition function, (('' is deterministic function modeling the change in the webpage resulting from an action determined by the underlying website. Reward, r(s, a) is awarded for reaching set of subgoals, e.g. canceling a flight has subgoals like finding the booking and then canceling it.
As the state is partially observable, the policy maps history of observations and actions to the current action a,, i.e. π h4 + a;. As discussed before, learning single LLM web policy is challenging. We next look at composing multiple policies.
h; =
Approach
We present a framework, Stacked LLM Policies for Web Actions (SteP), that performs a range of web tasks by dynamically composing policies. As previously discussed, designing a single policy that solves all tasks is challenging. Instead, we utilize library of policies II which we compose to solve task. Fig. 2 shows an illustration of SteP solving a web task
3

---

Published as a conference paper at COLM 2024
Browser Page
Aine
Syyert Greal
Coo
Policy Stack
find reviewi..)
find review...)
find reviews..)
search customent
search, customeri.
search. customeri...
cms. agent(..)
crms agentti..)
cmss agentE.))
cms. agentr...
ciss, agent(.)
cms. agentC..)
cms agenti.))
cms. agentt.))
agentt.))
web. agent(..)
web. agentC..)
web. agentt.)
web. agentl..)
web. agent(.)
web. agentt..)
web. agenri.)
web. agent...)
agentr...)
Actions
spology to custamer unhaopy with
Fiview"FFnd review of custumer unhapny fleece")
typ [L1]
really disappointed with
('Hannah Lim")
dick type (45)
stop
Found]]
type [1A] lWe are snrry hear you Cliee
stop ["Done']
[Cire Flance)
the Circe "1
[hannah Lim)
mar("Hannah
ems_agemtt...
Lini))
Figure 2: Example of SteP solving web task on a Customer Management System (CMS) website. SteP dynamically composes policies from library, using policy track to keep track of active policies. At every timestep, SteP either acts on the webpage, modifies the stack to add/ remove policies.
by dynamically stacking policies from the library. Sec. 4.1, 4.2 discusses the stacked policy model and SteP algorithm respectively.
4.1 Stacked Policy Model
At any given time, represent the control state stack Σ which denotes the chain of policy calls Σ = | We extend the MDP in Sec. to include a stack of policies:
State. The MDP state is augmented with stack Σ (Tg n |r|l... ... (T, of invoked policies. The top of the stack गT; is the current active policy. Each policy in the stack maintains its own history of observation, reason, action. The stack is initialized with base policy Σ [mo)..
Action. We augment the original MDP actions with two new actions invoke new policy E II or terminate the current policy π; with a return value Vj.
Transition. Suppose at timestep f, the current stack is Σ, = (7T ... [k)) The policy on top of the stack, 71, can take one of three actions leading to different state transitions:
1. Issue an action: It can issue an action a, along with reason This is sent to the environ- ment, which updates its state S141 and responds with an observation 0++1. The action, reason, and observation is appended to the history maintained by π,, The set of policies in the stack remains unchanged, only the history for the current policy updates.
(Tol |1....
[h)) + (To |.
[Iih + hU (a, (++1)))
(1)
2. Invoke another policy: can choose invoke a new policy π4+1- The new policy is initialized with an empty history and is pushed onto the stack.
(nol ... (m4) - (Tol 7z, / | (++1)
(2)
No action is sent to the environment.
3. Terminate and hand back control: can choose to terminate with a return value v; which in case is optional response returned by the policy once it finishes executing. The policy π. is popped off the stack, and the response is added to the history of π;-1-
(Tg) π -4, [4]] (nol m4 ... 14 h hub;))
(3)
No action is sent to the environment.
Reward. The reward functions the same the original MDP.
While this is a novel decision making model, it has close ties to transition-based dependency parsing systems in NLP and hierarchical decision making in RL. See Sec. 2.
4

---

Published as a conference paper at COLM 2024
Algorithm 1 SteP: Dynamically compose policies to solve web task
class SteP(Policy):
def predict, (sliionsee observation):
if self wwt.sstac..o. em_t ():
root_policy = cellfinnit_ppoiiie
self self.stack.. welf..stack.push/eeee (psh(rooo_p)licy)
while not self wwff.stack.... wtt.st.i..ectrrr.......
policy = self stt.k.t.cckoooo..
action, reason policy seliiyypredict_attionn predict_action (observvtii)
# Issue an environment action
if self salf.is_enviree wess_envvroometttce (atpemettt_ttiooaaaiioooo
return action, reason
# Invoke new policy
if self sss.policy_actionaaattttt
new_policy self (olicc
self.stack. pasacccppshheee_ooliigg
continue
Termsnate and hand back control
if self is_polscyy__ccc sorccy_dooeeee (dnde(actiion)
self stack stackkpop()
policy self stack top()
policy.appent repponddrreppossacccctt if policy else None
continue
return action, reason #Temminttion action root policy
def main()
policy SteP()
observation, done env resetts, False
while not done:
action, reason policy.predicts aetpficttactiooobbertettee observation,
env. step(action)
4.2 SteP Algorithm
Algorithm 1 presents the pseudo-code for SteP. SteP is a meta-policy that wraps around typical Policy class. SteP maintains self stack variable store the set of active policies. We focus on the predict_actionn () that takes as input the observation and returns an action and a reason. begins by initializing the stack with a root policy. It then takes the policy at the top of the stack and invokes its predict action( function.
The policy can perform one of three actions. First, if the action is an environment action, e.g. click [id], it returns that directly. Second, if the action is a call to another policy, e.g. search.custtoer
search_. customer(, (. .),,, initializes the corresponding policy, pushes it the top the stack, and assigns it control. Third, if the action indicates that the current policy is done acting, e.g. stop [response]] ], it pops the current policy out the stack, sends the response to the next policy the stack, and assigns it control.
Key features. Several characteristics emerge from such an approach, notably:
1. Dynamic Composition. Policies are composed dynamically at test time based on observa- tions from the environment. The space of possible control states are defined by each policy's s action space, i.e. other policies they can transition to. The stack can having varying depth that adapts to task difficulty. Compared to prior works (Akter et al., 2023; Zhou et al., 2023) that use single policy, composition allows for more adaptability. Scalability. Adding a new policy to SteP is easy. The user constructs the prompt for the policy and adds it to the library with description. This policy becomes available as part of the action space for any other policy, without requiring any change to the code. 3. Modularity. Each policy tracks only the local context of the specific subproblem it is solving. Once terminates, it hands back control to the previous policy in the stack without reasoning about the global context. This allows reusing policies in different context, e.g. the same fill_form form( !) can be used by book flights or make appointments. Compared to prior work (Zheng al., 2024) that require demonstration for entire tasks, modularity requires demonstrations for subtasks thus being more sample efficient.
5

---

Published as a conference paper at COLM 2024
5 Experiments
5.1 Experimental Setup
Environments. We evaluate across multiple distinct web environments listed below.
WebArena (Zhou et al., 2023). A recent benchmark with complex web tasks across mul- tiple domains like shopping, software development, content management. WebArena websites are highly realistic with tasks mirroring those that humans routinely perform on the internet. We evaluate across all 804 tasks in the benchmark.
MiniWoB++ (Liu et al., 2018; Shi et al., 2017). Compared to WebArena, this is a simplified web environment covering interactions like form filling, search, choosing dates. We evaluate across all 45 tasks that don't rely on vision and average over 50 seeds per task. AiMM We develop new CRM simulator (Appendix C) modeled after customer service workflows on popular airline websites. Compared to MiniWoB++, this contains longer-horizon tasks. We evaluate across 5 tasks averaged over 20 scenarios per task. Finally we test on live website environments and show results in Appendix D.
Policies. We use a library of 14 policies for WebArena, each covering multiple intents. Con- structing policy is straightforward: we use templated prompt with general instructions, action space definition, and place holders for policy specific instructions and examples. To design policies, we cluster intents that are functionally equivalent, e.g., searching over orders or listing products. See Appendix A.2.
Baselines. We compare against various baselines including prior state-of-the-art on We- bArena (Zhou et al., 2023; Akter et al., 2023) which design single web agent policy following ReAct (Yao et al., 2022b) style chain-of-thour of-thoughtt (CoT) prompting. On MiniWob++, compare against recent fine-tuning (Furuta et al., 2024; Gur et al., 2022a; Humphreys et al., 2022) and in-context learning (Zheng et al., 2024; Sun et al., 2024; Kim et al., 2023) works.
Additionally, we create baselines to study the following effects: (i) Single us Decomposed prompt (Flat vs SteP). Flat is single policy that concatenates instructions and examples from all the policies in the library into a single prompt. (ii) Varying context lengths of prompts (Flat-4k vs Flat 8k). Since the typical policy prompt is less than 4000 tokens, we create two baselines F that caps the prompt size to 4000 tokens and Flat-8K that caps it to 8000. (ii) Effect of in-context examples (Zero- shot vs Few- sewt). We study effect of adding observation action examples that help associate language instructions to webpage elements.
We study (i) on all datasets, (ii) on WebArena, where context lengths are longer due to more complex webpages, and (iii) on MiniWob++, AirlineCRM where the stripped down webpages have ambiguous elements (e.g. missing aria-labels) and benefit from examples.
Models. For models, on WebArena we evaluate with 1ot-4-turbi (OpenAT, 2023) since the tasks are complex, while for MiniWob++ and AirlineCRM we evaluate with either the instruction fine-tuned
or 3 turruboo (Ouyang et al,, 2022).
Metrics, We define metrics: Success Rate (suct), Task Progress (progt), and Number Actions (#act). suct is either 0 or 1 depending on the task being completed successfully. #act is the number of actions taken. On airline CRM, we also compute progt a number between 0 and indicating progress towards completing the task.
5.1.1 Overall Results
On WebArena, SteP outperforms prior works (0.15 -> 0.33) with improvements on every
environment: Shopping (0.2 + 0.37), CMS (0.10 -> 0.24), Reddit (0.11 - 0.59), Gitlab (0.14 - 0.32, Maps (0.15 - 0.30). See Sec. 5.1.2.
SteP achieves greater accuracy over Flat-8t 8k (0.23 + 0.33) while using 2.3x smaller
context lengths per episode. See Sec. 5.1.3, 5.1.4.
On MiniWob++, SteP Few- shot is competitive to prior works while using significantly less data. See Sec. 5.1.2.

---

Published as a conference paper at COLM 2024
0.25
Flat Bk
(2023)
SteP (Ours)
Acter (2023)
0.20
Hat-4
Rat
Prgbmbbilit
SheP (Daes)
Success
0.110
0.05
0.0
Shopping
CMS
Reddff
Gitabb
Maps
0.00
20k
40k Input takens (per trajectory)
63k
80k
100k
(a)
(b)
Figure 3: (a) Success rates of SteP against all baselines 5 different WebArena websites. (b) Distribution of input tokens per trajectory of SteP Flat 8k WebArena. SteP achieves higher success rates while needing less input tokens resulting in lower costs per trajectory.
Sampled Task Intents
Zhou et al. (2023)
Akter et al. (2023)
Flat-4k
Flat-8k
SteP
(samples website)
suct
suct
suct
#act
suct
#act
suct
#act
customers complaint about (items]
0.00
0.00
0.00
4.00
0.00
4.00
1.00
15.0
Shopping
Config of the (product] bought (time)
0.20
0.00
0.20
11.0
0.00
4.20
0.60
11.6
Show most recent (status] order
0.00
0.40
4.40
0.40
4.00
0.20
2.00
Summarize main criticisms product
0.00
0.20
0.60
2.00
0.40
2.20
0.00
10.4
Show (product) listings by order)
0,20
0.20
0.20
2.00
0.20
2.17
0.20
12.2
Mean all 48 Shopping intents
0.14
0.20
0.28
6.66
0.30
4.83
0.37
8.99
Update order #(order) with tracking
0.00
0.00
0.00
1.20
0.00
4.40
0.20
15.0
CMS
Tell reasons customee sustomess like (product)
0.00
0.00
0.00
1.20
0.20
4.80
0.00
8.80
Notify (namel: (message)
0.00
0.00
0.40
1.60
0.40
3.80
0.80
Find customer by Num)
0.20
0.00
0.60
3.00
0.40
5.00
0.40
4.60
How many reviews received on (time]
0.60
3.20
0.60
2,80
0.20
6.80
Mean all 41 CMS intents
0.11
0.10
0.17
1.75
0.18
5.42
0.24
11.6
Post | notice in (subreddit)
0.00
0.00
0.00
13.2
0.00
20.0
0.60
7,80
Reddit
Like submissions (user] in (subreddit)
0.17
0.50
4.67
0.33
9.83
0.50
14.5
Create new [forum] with [description]
0.00
0.20
0.60
7,00
0.40
0.76
0.80
9.00
Post review [book) with (comment).
0.20
0.20
0.20
10.8
0.80
17.60
0.90
8.00
Reply (post) with (content)
0.20
0.60
0.60
9.00
0.60
13.6
0.00
3.00
Mean all 21 Reddit intents
0.06
0.11
0.17
9.86
0.30
12.09
0.59
9.26
Create new lgroupl with (members)
0.00
0.00
0.00
9.80
0.00
10.0
0.80
14.2
Gitlab
Commits (user] made (repo)?
0.20
0.20
0.40
3.00
0.40
3.00
0.80
3.80
Check latest issue (keyword) if closed
0.00
0.00
0.00
2,80
0.00
3.00
0.20
6.80
Check out the most recent open issues
0.00
0.00
5.00
0.50
2.50
0.50
2.00
Open an issue to lissue] in (repo)
0.17
0.33
0.33
4.40
0.33
7,00
0.80
10.4
Mean all 41 Gitlab intents
0.15
0.14
0.20
5.48
0.23
6.65
0.32
9.34
Nearest [location] from [location2]
0.00
18.0
0.00
20.0
0.00
6.00
Maps
Closest (ss to (place2)
0.00
0.00
0.00
17.0
0.20
16.8
0.60
10.8
Driving time [cityl] to [city21
0.00
0.00
0.75
4.75
0.75
4.50
0.75
5.00
From [hotel], time to reach (placel
0.20
0.40
0.40
5.00
0.40
5.00
0.40
8.20
Find the (space) around [location]
0.40
0.20
0.40
10.6
0.40
10.0
0.40
6.20
Mean Maps intents
0.30
Mean all WebArena intents
0.12
0.15
0.20
6.44
0.23
7.25
0.33
10.0
Table 1: WebArena Success Rates over 804 tasks categorized by task intents across different websites. Values shown for 5 intents sampled from 5 quantiles, average values shown for all intents for each website. Each intent consists of 3-5 tasks. SteP caps each policy prompt to 4k tokens.
In-context examples help in addition to instructions by associating language instructions with corresponding web elements. Moreover, SteP Few-shot uses these examples more effectively by having them in dedicated policies. See Section 5.1.5.
We provide ablations on CoT reasoning and model scales in Appendix B.
5.1.2 Comparison prior works
On WebArena, Fig. 3(a), Table 1 show that SteP outperforms prior works (Zhou et al., 2023; Akter et al., 2023) that use a single GPT-4 policy on all environ- ments. By having library of only 14 policies (see Appendix A.2), SteP cov-
7

---

Published as a conference paper at COLM 2024
at least 50 intents out of total of 170 intents. Most significant gains come
from Shopping, Reddit where the policies cover a larger percentage of intents. On MiniWob+t, Table 2 shows that SteP out- performs all baselines that fine-tune models. uses only 10 demon- stration trajectories (24 observation-action exam- ples) compared to the most recent baseline (Furuta et al., 2024) that trains on 347K trajectories. It is also competitive to recent in-context learning works while using fewer trajectories. For instance, Synapse (Zheng et al., 2024) uses ~ 100 trajectories with 2-3 exemplars sampled from each of 48 tasks. In comparison, SteP uses 10 trajectories from only 6 tasks. SteP generalizes to remaining tasks through composition, i.e., by breaking them down into subtasks soluble by existing library policies. Thus, unlike prior work, SteP does not need to see exemplars from every task is required to solve.
Method
Models
Training trajectories
Success Rate
WGE (Liu al., 2018)
12K+
0.76
CC-Net (SL) (Humphoeys 2022)
ResNet
24M
0.36
CC-Net
(Humphheys al, 2022)
Res
0.96
WebN-T3 (Gur al., 2022b)
T5-XL
12K
0.56
WcbGUM (HTML.) (Porata et 2024)
Flan-T5-XI.
RCI (Kim et al,, 2023)
gpt-4
21
0.94
AdaPlanner (Sun et 2024)
text-davinci-008
0.93
Synapse (Zheng al., 2024)
gpt-3.5-turbo
100
0.98
SteP (Ours)
text-davinci-008
10
0.96
Table 2: Comparison prior works with success rates averaged across MiniWoB++ tasks. SteP achieves competitive success rates while using significantly less data than prior works.
5.1.3 Why does library of policies help over single policy?
We observed that in prior works (Akter et al., 2023; Zhou et al., 2023), common failure mode was an inability to navigate the website correctly to solve a complex, multi-step task. As natural first solution, how to solve such tasks. We created two baselines, Flat-4k and Flat-8k, containing such instructions and examples up context limit of 4000 and 8000 respectively.
add instructions and examples to the prompt to teach it
In Table. 1, we see that Flat-4k improves upon prior work (0.15 0.20). However, when we go from Flat-4k to Flat-8k, the performance gains in- crease only marginally (0.20 + 0.23) even though we double context lengths. On some intents, suc- cess rates even regress, e.g. on Shopping
Task
Zera-shar shat #act
Faw-shat
Zera-shat
Fow shot
(0.2 + 0, 0.6 + 0.4),
suct
suct
#act
suct
#act
suct
#act
on Reddit (0.5 + 0.33,
click-option
0.76
3,68
1.00
2.62
0.80
2:94
1.00
1.94
0.6 + 0.4). This is because additional instructions create larger prompts, making difficult for the model to pay attention.
simple
click-dialog
1.00
enter-date
1.00
3.00
1.00
2.00
0.00
4.08
1.00
2,00
login-user
0.96
3.42
1,00
3.00
1,00
3.06
1.00
3.00
grid-coordinnae
copy-paste-2
0.54
7.66
0.56
4.00
0.48
3.84
0.96
2.04
find-word
0.22
2.62
0.26
5.18
0.12
2.92
0.98
2.00
choose mediumm
Blikkcheckhoves-larg
0.00
8.40
0.20
8.40
0.00
7.00
1,00
comple
click checkboxes transfer
0.40
4,50
0.40
3.90
0.54
3.20
0.94
2.84
email-inbox
7.00
4.50
0.00
SteP introduces
simple-algebra
0.14
8.80
0.30
6.78
0.04
4.38
0.74
2.00
library of 14 policies, each with small set of dedicated instruc- tions and examples with prompt lengths Table 3: Task-wise performance breakup MiniWoB++ subset of 15 tasks. See Appendix A.4 for full breakup over 45 tasks.
search engine book flight
login user popup
0.38
0.46
3.64
6.28
0.38
0.46
3.16
3.52
0.46
0.26
5.82
4.46
1.00
1.00
4.88
4.30
0.10
11.10
0.00
Mean (all 45 tasks)
0.60
3.70
0.72
3.38
0.65
3.43
0.96
2.89
under 4000 (see Ap- pendix A.2). This
results in smaller prompts that make fewer errors. Table 1 shows that SteP outperforms both Flat 4k (0.20 + 0.33) and Fl (0.23 0.33). A key feature that helps SteP scale that a single policy can cover multiple intents, e.g. search order covers intents consisting of 30 tasks.
8

---

Published as a conference paper at COLM 2024
Comparable performanca simple tasks
Fat
ShoP Few-SShot
SteP doas better on complex tasks
1.0
Quccess Raie
0.8
0.6
0.4
0.2
of latu sotn pant--glla
dick-lnt clcck option Soute bed bade di lick the bark--t
0.0
diokdattio Autto
2
Pet Retuu
aauuiiii eedus
soari C popettii le dole ail
chocti dill dhg por
mim di
book
Figure 4: Success rate comparisons between Flat few-shot and SteP shot broken down over 45 MiniWob++ tasks (averaged over 50 seeds per task).
Task
Metric
Flat
Zero-shot:
Feu-shot
Zezo-shot
Pey-shoo:
Fuer: OV0N/003
ouct
1,00
0.20
100wa00
Figu
progt
0.15
1.00
0.80
1.00
Select your daparting flight
fct
11.2
6.00
11.3
6.00
suct
0.00
0,90
1,00
1.00
progt
0.00
0.90
1,00
1.00
fac
11.0
4.10
3.00
3.00
suct
0.00
0.00
0.00
1.00
progt
0.50
0.60
0.60
1.00
Paseenger
tac
11.0
11.0
11.0
5.00
PANENSOA
DATATA
suct
0,00
0,50
0,30
0.65
5.bla
progt
0.00
0,90
0.60
0.90
fact
16.0
11.1
14.3
11.8
BOOk
auct
0.00
0.00
progt
fact
0.55
26.0
0.40
26.0
25.7
0,40
0.82
22.2
126m
fight
MLAN
suct
0.02
0.48
0.30
0,86
Rakuring Ri
progt
0.24
0.76
0.68
0.94
fact
11.6
11.1
Figure 5: (a) Evaluation on 5 airline CRM tasks averaged over 20 randomized scenarios per task. (b) Simulator visualization of a book-flight task consisting of >20 steps. More details in Appendix C.
Mistakintly clicks on the 7th link in the current webpaae
SteP Zoro-shot:
click voult.
sreae "Seerch"
'Wayy"
Like
"Riley cicc saari metul.
SteP Few-shot: Navigates webpages and counts until it reaches 7th link
Search
Riley
Search
LLM Peason:
(iue the todlects "Raayy petarm dun diak uuirch
LLM Reason
Martho end preve
78 naarth veutt
and
den a ose vouk
Der kodb 'Wiley
Liee
siok 7th search reack
'Rlly"
inatuction
cosing the benoner ronteet
snerch with the inder 7. Since the set and
and
Rilay
Bearch
Dalows Chheme 7he mores the indor the linss as Dolores a 1- Cheree
Riley
Bearch
Riley
Search
Riley
Search
aeri kartis
provided of the tne with the of 17.
Joak
in
page To go the nuxt page. have click on the commsponding rign, commsoonds
Zo Crave
the next
Volkipe:: pelersacaa
(4g Mtare dorieels
123
122
1123 3
112
CLICK Joyn
CLICK 25
CLOCK
CLECK EiLlay
Figure 6: SteP Few-shot vs SteP Zero-shot on a search-engine task. The instruction asks to find the 7th link, however, it is ambiguous what 7 refers to. SteP Few- with an in-context example is able to ground the task in the UI and reason that the 7th link lies in the 2nd webpage.
On MiniWob++, in Table 3, we see similar trend where SteP improves over Flat Few-shot (0.72 - 0.96). Fig. 4 shows comparisons across individual tasks. While per- formance is comparable on simpler tasks, as the task complexity increases SteP outperforms by greater margins. SteP breaks down complex tasks into smaller sub-tasks covered by poli- cies in the library, e.g. book_ flight () is broken down into fill_ texttttxe( (), choose ().

---

Published as a conference paper at COLM 2024
5.1.4 How context efficient is SteP?
trade-off to SteP is that by introducing a library of policies, there is an overhead in passing control back and forth amongst policies. This leads to more number of calls to the model. However, since the prompt for each policy is significantly smaller, the total tokens ends up being smaller. For instance, when SteP solves a task it never has to see instructions and examples for policies not required in that task. Fig. 3(b) shows histogram of context lengths for SteP and Flat-8k 8k across all successful WebArena trajectories. SteP is distributed around smaller number of tokens, averaging 22.7K compared to 52K. Hence total cost and inference time for SteP is lower than Flat-8k.
5.1.5 What is the effect of in-context examples?
We observe that while instructions are often sufficient to solve many web tasks, exam- ples can provide a significant performance boost, particularly when these elements are stripped down lacking meaningful aria labels. On MiniWob++ Table 3, we see that few-shot examples provide performance gains for both Flat (0.60 > 0.72) and SteP (0.65 + 0.96).
We see similar trend in AirlineCRM in Fig. for both Flat (0.24 + 0.76) and SteP (0.68 - 0.94). Examples help associate language instructions with webpage elements, particularly for simplified pages when these are ambiguous. Fig. 6 shows MiniWob++ search-engine task, where it is un- clear what the 7th link implies. Fig. 7 shows SteP vs Flat with varying number of in-context exam- ples on MiniWob++. We provide a maximum of 21 examples. We observe two main sources of improvement: (1) For the same number of exam- ples (< 7), improvements come from decompos- ing task instructions into granular policy instruc- tions (2) Each policy prompt contains dedicated in-context examples, allowing for more in-context examples (> 7) in each prompt.
Flat Few-shot
0.8
Few-shat
0.6
Success
0.2
6
15.16
Number examples
Figure 7: SteP vs Flat with varying in- context examples subset of MiniWob++ tasks. Yellow region shows extra examples SteP packs in policies.
Limitations
We present SteP that dynamically composes policies to perform a diverse set of web tasks. Our results show that SteP outperforms both prior works and single policy baselines while being more context-efficient. However, SteP has several important limitations:
(1) Manually defined policies. For any new domain, the user needs to identify commonly occurring subtasks that are shared across multiple tasks. For each subtask, they then need to write a policy prompt that contains instructions on how solve the subtask. Both identifying subtasks and writing policy prompt pts require domain knowledge. Once the policy prompts are written, however, SteP can automatically decide when to compose the appropriate policies from the library. The benefit of composition is that small set of policies can cover a combinatorially large number tasks. Automatically discovering useful policies from experience or demonstration data is an important direction of future work and would enable scaling to a large number of domains and websites automatically. (2) Communicattin overhead. By decomposing tasks into smaller policies, we also incur communication overhead between policies that adds to inference times. One interesting solution would be to use smaller models for simpler policies and only escalate to larger models as needed. (3) Incomplete information. Finally, there are situations where a policy is unable to solve a subtask due to inadequate information being passed it. Moreover, fails to communicate what is missing to the parent policy, which can create an endless loop. An interesting future direction would be to explore how policies can share and update common belief state to prevent such errors.
10

---

Published as a conference paper at COLM 2024
Acknowledgements
We thank Kilian Weinberger, Michael Griffiths, Ramya Ramakrishnan, Adrian Botta, and the rest of ASAPP research and UI automation teams for their insightful feedback and suggestions. We also thank the anonymous reviewers for their constructive feedback, which greatly enhanced the quality of this paper.
Ethics Statement
Equipping LLMs with the ability to carry out web tasks opens up a variety of possibilities for societal benefits and change. These range from reducing cognitive burden on humans of doing repetitive tasks to enabling greater accessibility for elderly and individuals with disabilities. However, we acknowledge the ethical implications that come with the use of such technologies, and list some of them below:
1. Safety and Reliability. LLMs automating web tasks raises concerns regarding misuse, including malicious automation aimed at spamming, phishing, or manipulating online systems. To mitigate these risks, it is crucial to implement stringent safeguards. These may include developing sophisticated detection algorithms to identify and block au- tomated actions that exhibit patterns of misuse, ensuring that LLMs operate within ethical boundaries. Moreover, rigorous testing and validation protocols would ensure the technology's reliability and safety in open-world web environments.
2. Privacy and Data Security. LLMs interacting with web interfaces introduces potential risks such as unauthorized data access and privacy breaches. To mitigate these risks, is crucial to place safeguards such encrypting sensitive data, enforcing strict access controls, and adhering to strong privacy practices. Such transparent data handling policies are essential to maintain trust in these systems with end users.
3. Employment Impact. While automating web tasks with LLMs can enhance efficiency and accessibility, it is vital consider the impact on employment. Care must be taken that the deployment of such technologies is to augment human capabilities rather than replace them, helping free them up for more creative and nuanced interactions. Addressing this requires holistic approaches, including policies to support workforce transition through re-skilling and up-skilling, and encouraging the development of new roles that leverage the unique strengths of human creativity.
Reproducibility Statement
To promote reproducibility and transparency, have taken several steps to ensure that SteP and our findings can be replicated and validated by the broader research community:
1. Open source code. We have linked the SteP implementation, including necessary code and documentation. We also include the prompts, the raw model predictions, and relevant noteboooks for greater reproducibility.
2. Experimental Details. Our paper includes detailed descriptions of the experiments along with additional details on hyperparamaters, promptss, environments included in the Appendix.
3. Benchmarks and Models. We clearly specify the web benchmarks and models used in our evaluations, including appropriate references and links to those. Since OpenAT models continue to evolve, we have included the raw model predictions so that the results are reproducible.
4. Results and Ablations. We present detailed results, including performance metrics, ablation studies, and comparisons with state-of-the-art methods. Our aim is to provide clear and honest assessment of SteP's capabilities and limitations.
5. Limitations. Acknowledging the importance of transparency in scientific communi- cation, we discuss the limitations of our approach and discuss directions for future research.
11

---

Published as a conference paper at COLM 2024
References
Syeda Nahida Akter, Zichun Yu, Aashiq Muhamed, Tianyue Ou, Alex Bäuerle, Ángel Alexan- der Cabrera, Krish Dholakia, Chenyan Xiong, and Graham Neubig. An in-depth look at gemini's language abilities. Xiv preprint ar arXio:2312.11444 2023.
Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics (TACL), 2013.
Satchuthananthavale RK Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay. Reinforcement learning for mapping instructions to actions. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. Association for Computational Linguistics, 2009.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web. arXiv preprint arXiv:2306.0607, 2023.
Hiroki Furuta, Ofir Nachum, Kuang-Huei Lee, Yutaka Matsuo, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with custruction-fineturee foundation models. International Conference Learning Representations (ICLR), 2024.
Izzeddin Gur, Natasha Jaques, Yang Miao, Jiayuan Choi, Manik Tiwari, Honglak Lee, and
Alex Faust. Environment generation for zero-shot compositional reinforcement learning. Advances in Neural Information Processing Systems, 34, 2022a.
Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. Understanding HTML with large language models. arXiv preprint arX:::2210003945, 2022b.
John E Hopcroft and Jeffrey D Ullman. Formal languages and their relation to automata. Addison-Wesley Longman Publishing Co., Inc., 1969.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning (ICML), 2022a.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint ar Xfc::2207.00666 2022b.
Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillicrap. A data-driven approach for learning to control computers. In International Conference on Machine Learning (ICML), 2022.
Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou, Jayashree Karlekar, Sugiri Pranata, Akira Kinose, Koki Oguri, Felix Wick, and Yang You. Rap: aatrivval-augmenter planning with contextual memory for multimodal 11m agents. arx preprint ar 2xX::22402.03610, 2024.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. International Conference on Learning Representations (ICLR), 2023.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. Advances in Neural Information Processing Systems (NeurIPS), 36, 2023.
12

---

Published as a conference paper at COLM 2024
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arv preprint ar Xio:2209.07753. 2022.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforce- ment learning on web interfaces using workflow-guided exploration. In International Conference on Learning Represenlations (ICLR), 2018.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arxiv 2023. Transactions of the Association for Computational Linguistics (TACL), 2023.
Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman I'a,, and Dong Yu. Laser: Llm agent with state-space exploration for web navigation. ar preprint arXiv:230.008172 2023.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hier- archical reinforcement learning. Advances in neural information processing systems, 31, 2018.
Joakim Nivre. Algorithms for deterministic incremental dependency parsing. Computational Linguistics, 34(4):513-553, 2008
OpenAl. GPT-4 technical report, 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. arXiv preprint ar xxxv:2203.00155, 2022.
Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. Adapt: As-needed decomposition and planning with language models. arXiv preprint ar xrXiv:2311.05772, 2023.
Timo Schick, Jane Dwivedi-Yos Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.0476) 2023.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference Machine Learning (ICML), 2017.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. (2023). Xiv preprint cs.A1/2303.11366 2023.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wii-Lun Chao, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. In Intl. Conf. on Computer Vision (ICCV), 2023.
Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang Adaplanner: Adaptive planning from feedback with language models. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
Richard S Sutton. Between mdps and semi-mdps: Learning, planning, and representing knowledge at multiple temporal scales. 1998.
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor. A deep hierarchical approach to lifelong learning in minecraft. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International conference on machine learning, pp. 3540-3549 PMLRR 2017.
13

---

Published as a conference paper at COLM 2024
Huaxiaoyue Wang, Gonzalo Conzalez-Pumariieg Yash Sharma, and Sanjiban Choudhury. Demo2code: From summarizing demonstrations to synthesizing code via extended chain- of-thought. In Advances in Neural Information Processing Systems (NeurlPS), 2023a.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. arXiv preprint Xri::2230 114322 2023b.
Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction. Advances in Neural Information Processing Systems (NeurIPS), 2024.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems (NeurIPS), 2022a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022b.
Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Thajectory-as-exemplaa prompting with memory for computer control. International Conference on Learning Representations (ICLR), 2024.
Shuyan Zhou, Pengcheng Yin, and Graham Neubig. Hierarchical control of situated agents through natural language. Xrv preprint ar arXi:2200900211 2021.
Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. WebArena: A realistic web environment for building autonomous agents. ar Xrv preprint ar arXi::2307.11333 13854, 2023.
14

---

Published as a conference paper at COLM 2024
Appendix
Table of Contents
A Experiment Details
16
A.1 Hyper-parameters
16
A.2 WebArena Policies and Prompts
16
A.3 MiniWob++ Policies
................................
24
A.4 Task-wise Performance on MiniWoB++
24
B Ablations
25
B.1 Effect of Chain-of-Thought Reasoning
................................
25
B.2 Model Scaling
27
B.3 Evaluation with Llama model
28
C Airline CRM Simulator
30
C.1 Generating scenarios
30
Helper APIs
30
D Live Websites Evaluation
32
D.1 Collecting Human Task Demos
32
D.2 Parsing Browser Content
32
D.3 Live Website Results
33
15

---

Published as a conference paper at COLM 2024
A Experiment Details
A.1 Hyper-parameters
We use OpenAI API for model calls, gpt ---urbo previewwwwwwwwwweeoeeeeee as the default model for We- bArena and text certtdavvnci---00 for other environments. We use temperature of 0.3 and set the number of calls to 3. Below are the exact API calls,
1f (model_ type == "gpt-4- hopt-4-turbol turbo-ppreiiee
response openai. Opemai.ChatComplette Chattompletion.co ( model=model_type
messages=[ role"" "user", "content": prompt)),
5
temperature=0.33 top-p=1, n=3,
toxensssns-.x tokens max
10
response esspons..cooices
elif (model_typp type 2ott3.5--urbbo
response openai.ChatComee CwwtCompletion create( (
14
model=model_tyye
messages=[(" role"" "user", "content": prompt})], temperature=0 top-p=1, n=3, tokensmkess tokens
19
response cusponse.choices//e
cesssage.ontentt.strii..
23
elif (model. type == "tettdaavinci-093))
response Compaai.oompettinn create(
model=model_ type,
prompt=prompt, temperature-0.3,
28
best_of=3,, n=3,
max_ max_ttokes==mx.tee tokens
32
response response. eepoossecchoicsslll.tttttttttttt
Listing Hyper-parameters for different models
A.2 WebArena Policies and Prompts
We provide below the prompt template for WebArena that contains the list of policies, the base actions, examples for how to use the policies, and the base instruction template.
policies **.
2 Subroutine Actions:
3 commits [query]': Given you are in a project page, this Gitlab subroutine searches for commits made to the project and ret the query.
information about commit. This function returns the
7 "search issues [query]": Given you are in my issue page, this Gitlab subroutine searches issues that matches the query. Any objective that says "open my latest issue" "open issue with <keyword> in the title" must passed through this subroutine.
"crette project [queryl": Given are in the create new project page, this Gitlab subroutine completes the act of creating a project, adding members etc.
16

---

Published as a conference paper at COLM 2024
create_groui [queryl': Given you are in the create new group page, 1s this Gitlab subroutine completes the act of creating a group, adding members etc
[query]': This Reddit subroutine finds subreddit corresponding to the query. query can either the name the subreddit or vague 20 description of what the subreddit may contain. The subroutine hands back control once it navigates to the subreddit.
"find_use [user_ = This Reddit subroutine navigates the page of with user_name. The page contains
the posts made by the user.
wowtdlussttmereeeee review [queryl': This CMS subroutine finds customer reviews for particular product using the query to specify the kind of review.
29 find_order [queryl": This CMS subroutine finds an order corresponding to particular customer or order number.
"search_customer customer [query]": This CMS subroutine finds customer given details about them such as their phone number.
34
search [question]": This Shopping subroutine searches orders to answer question about my orders
[list_producee [query]': This Shopping subroutine find products that match query
revrchs [queryl": This Shopping subroutine searches reviews to answer question about reviews
42
find_ find__ireede dicetionns [queryl": This Maps subroutine finds directions between two Locations to answer the query
search nearest place [query]*: This Maps subroutine find places near given location
50 example_actions
actions = ***
click [7]
type [15] [Carnegie Mellon University] [1]
54
55 stop [Closed]
hover [15]
scroll [down]
note [Spent $10 on 4/1/2824]
find_coomitt [How many commits did user make to di diffussioProject on 03/23/2023?]
64
search_ issues [Open my latest updated issue that has keyword "better" in its title to check if it is closed]
68 create create_project [Create a new public project "awesome- llms" and add primer, convexegg, abishek as members]
create group [Create group "coding. friends" with members qhduan, Agnes-UJ
73 find_ [books]
find user [AdamCannon]
17

---

Published as a conference paper at COLM 2024
find_ customer_revieer [Show me customer reviews for Zoe products]
78
find order [Most recent pending order by Sarah Miller]
sustocr [Search customer with phone number 8015551212]
83
search [How much I spend on 4/19/2023 on shopping at One Stop Market?]
List products [List products from PS4 accessories category by ascending price]
87 search_ reviews [List out reviewers, if exist, who mention about ear cups being small]
find directions [Check if the social security administration in Pittsburgh can be reached one hour by car from Carnegie Mellon University]
[Tell me the closest cafe(s) to CMU Hunt library]
96
Page Operation Actions:
"click [id] This action clicks element with specific
id on the webpage.
100
"type [id] [content] [press_ entessseeterreee | 1] Use this to type the content into the field with id. By default, the "Enter" key is pressed
101
102
after typing unless press- afee is set to 0.
hhover [id]": Hover
an element with id. press [key_comb]': Simulates the pressing
104
key combination
105
on the keyboard (e.g., Ctrl+v).
106 sccoll [direction=down|up)) up]" Scroll the page or down.
note [content]": Use this to make personal note of some content you would like remember. This shows up your history previous actions so you can refer to it.
10s
109
110 go_ back" Navigate to the previously viewed page.
111
113 are an AI assistant performing tasks on a browser.
114
To solve these tasks, you will issue specific actions.
115
116
The actions you perform fall into several categories:
117 )ction)
118
119 (policies)
120
(example.action)
122
123
You will be provided with the fol following,
124
OBJECTIVE:
The goal you need to achieve.
126
OBSERVATION:
127
simplified text description of the current browser content, without formatting elements. URL:
128
129
The current webpage URL
131
PREVIOUS ACTIONS:
132
A list of your past actions with an optional response,
133
e.g. 1 find commits [query]
You need to generate response in the following format. Please issue only a single action at a time.
136
137
REASON:
138
Your reason for selecting the action below
ACTION:
18

---

Published as a conference paper at COLM 2024
Your action
141
142
153
144
Tab Management Actions:
146
"new_ tab": Open a new, empty browser tab.
147
"tab_ focus [tab_index]: Switch the browser's focus to a specific
168 tab using its index.
"cooe tab": Close the currently active tab.
151
Navigation Actions:
152
goto [url]": Navigate to specific URL.
153 go_ bockback Navigate to the previously viewed page.
go_ forward": Navigate to the next page (if previous back'
155
action was performed).
156
157
Completion Action:
stop [answer]": Issue this action when you believe the task is complete. If the objective is to find text-based answer, provide the answer the bracket.
160
161
We provide the prompts for all 14 commonly used policies below and for the reset, refer the reader to the code base.
find commits
"instruction": **"
3
Please follow these general instructions:
To find list of all commits, you must navigate the commits section of the repository
7 Look at the first and last date in your observation to know if the desired date is in the range
If it's in the range but not visible, that means commits were made that date
the date outside of the range, you need to scroll up/down to get to desired date range. Scrolling down takes you to date earlier in time (e.g Feb 2023 is earlier in time than Mar 2023)
To count commits from specific author, count the number of times their avatar (e.g. img "<author> avatar") appears in the observation.
11
"examples"; [...]
}
16 search search_issues =
"instruction": <eeneral_iistructiion..nn..
20 Please follow these general instructions:
First navigate the Issues page
Once you are in the Issues page, you MUST first navigate to all issues so that you see both open and closed issues for solving the objective 23 You may not see all issues listed at once, use the search bar to search for appropriate keywords and filter down to relevant set of issues * If the objective says to "Open open the specific issue page by clicking it. Do not stop [] until you have navigated to the specific issue page.
issue, check if it is X", you must first
25
Once you are on the issue page, return the appropriate status In your status, if the objective is to check if an issue is open or clossed, respond as though you are answering question, e.g. "No, it is open", "Yes it is closed
19

---

Published as a conference paper at COLM 2024
29 "examples": [...]
}
preate_project ={
"instruction": "*.
36 [generalLinntrrutiioon.nemm.
Please follow these general instructions:
To add members, you have created the project, click Project
Information in sidebar be guided to link with memmbers. When adding members, first type their name, then click on their name from the down down. Consult PREVIOUS ACTIONS to see if you have typed and selected the names.
.""
40
"examples": [...]
}
create group {
45
"instruction": "*"
geeneraliinstructiion..emmm
Please follow these general instructions:
To add new members, click on the Members tab the side pane. you don't
see it, click on Group Information in the sidebar to be guided to link with th memmbers.
When adding members, first type their name, then click their from the down down. Consult PREVIOUS ACTIONS see you have typed selected the names.
"examples"; [...]
55
find subreddit =
" "instruction": **n
seeneratllisstuucciion.tellttt))
59
Please follow these instructions to solve the subtask: The objective subreddit [query] asks you to navigate to the subreddit that best matches the query. The query can specific or vague.
62 The first step is to navigate to Forums to see the list of subreddits. However if you have done this already (indicated as non empty PREVIOUS ACTIONS), do not repeat this step.
Under forums, you will of subreddits, you need navigate the Alphabetical option.
only subset of subreddits. To get the full list
61 To know you can see the full list of subreddits, you will see 'All Forums' in the observation
Often you will not find focused subreddit that exactly matches your query. In that case, go ahead with the closest relevant subreddit.
66 To know that you have reached a subreddit successfully, you will see '/f/ subreddit_ name in the observation.
* Once you have navigated to any specific subreddit, return stop [N/A]. Even if the subreddit is generally related and not specific to your quwey, stop here and do not try search again for another subreddit.
68
"examples": [...]
}
73 find_used = {
"instruction": template)
20

---

Published as a conference paper at COLM 2024
Please follow these instructions solve the subtask: The objective find_user [aser_name] asks you to navigate the page of a user with user-name
78
To do so, look at the current base URL (e.g.
awsdev. wwsdev.asapp.ccm com) and add suffix /user/user_ name, i.e. goto Inttps://webarena-ent rwwdet. asapp. aswpp.com/use com/user/user_nmee Once you have navigated to the user page (as seen in your past actions), return stop [N/A]
81
84 "examples: [...]
}
find review
"instruction": "*"
89
(generallinsttuctiionteenp
Please follow these instructions to solve the subtask: The objective find_ revve [query] asks you to navigate to the product
page containing customer reviews.
To navigate to a review, first click on REPORTS in the side panel
Once you have clicked on REPORTS, and you see the Reports panel with Marketing
Sales, Reviews, Customers etc, click on By Products under Customers. Once you are in the Product Reviews Report, you need to locate the product by searching for it. Use the gridcell below Product to search for a product.
Do not use other search boxes. Look at the example below where I show you how to search for Zoe in the correct gridcell.
When searching for product, search the first word only Like Zoe, Antonia or Chloe.
97 Once the product shows up, click on 'Show Reviews'.
Once all the reviews show up, return stop [N/A] to hand back control to the agent that queried you.
...
100
"examples": [...]
101
}
102
find_order
104
"instonction" .*.
105
Eieneraalinsstuuttion..mmmm.p__.
106
107
Please follow these instruction to solve the subtask: The objective find order [query] asks you navigate the order page corresponding to the query To navigate to orders, first go to SALES in the side panel
108
109
113 Once you have clicked on SALES, go to Orders
111
Once you in the orders page, you have to the 'Filter' button to filter down desired criteria
112 Desired criterias include filtering down to a specific order ID field or Name field. ONLY use fields that are in the objective
113 You MUST use Filter to find orders instead of using the search bar to clear them before entering your
114 If there are any active filters,
filter criteria
115 In your filtered list of orders, if you don't find the desired order, make sure to scroll down till you find the order or reach end of page (typically indicated by 'Copyright ... in the observation) 116 Once you have found the order, go to View open the order
117 Once you are in the desired order page (as noted by "Order Account
Information") you MUST return stop [N/A] to hand back control to the agent that queried you. Do not go back to another page.
119
"examples": [...]
120 }
121
search. customer
21

---

Published as a conference paper at COLM 2024
"instruction":
124
renneral_instructinn templatte
125
126
Please follow these instructions to solve the subtask:
The objective search_ customer [query] asks you to search for customer details
corresponding to the query
128
To navigate customers, first click on CUSTOMERS the side panel
129
Once you have clicked on CUSTOMERS, click on ALL Customers.
130
* Once you are in the customers page, you have to use the 'Search by keyword' text box to search for your customer. Always be sure to search first. For example, for find_ order [Search customer with phone number 8015551212], search 8015551212.
131
* If the page shows a number has already been searched, click on Clear All first Then proceed with the search.
Once you done with the search, and the customer with matching query shows up, you MUST return stop [N/A] hand back control the agent that queried you. Do not go back to another page.
133
131
"examples"; [...]
136
137
search.order =
138
"instruction":
(generalinnstuctiion.teemtm
160
141 Please follow these GENERAL INSTRUCTIONS:
142
* Navigate to My Account, then My Orders to access all orders.
The orders are sorted by descending date. Click on Page Next to go to earlier date. Click 144 you don see an order for a date, and the first order on the page after the date, the last order on the page is before the date, then it means there is no order for the date.
Page Previous to go to earlier date.
155 In your REASON, state what is the current range, what range you are looking for, and whether you should search for an earlier or later date. * If you have find the total amount you spent on orders that span multiple pages, use note (Spent $10 on 4/1/2024] to make personal note before moving on to the next page. When you are done, you can look at PREVIOUS ACTIONS to find all notes.
146
147 When you are adding numbers, work out each addition step by step REASON.
148 * Use bo_back to go back to a previous page from an order. But before you do, use note [] to make note that you checked the page, e.g. note [Checked order 11/29/2023, no picture frame.]
159 If you are in an order page and need to go back, issue go_back, Don't click on My Orders else you have to start from all over again.
150 * Do not keep visiting the same order page over and over again!
151 To prevent this, whenever you visit page, always make note. For example note [Nothing relevant purchased September 29, 2022]
152
note [] see what dates you visit, be sure not visit that page again.
153 * Once you are done visiting all the pages, return stop [answer] with the answer to the query.
If the question is how much did I spend date, and I didn't spend anything return stop [$0]
155 * If the status of an order shows cancelled, that means did not spend that money
If you asked to change the delivery address order, you can't. Reply stop [N/A]
157
158
159
"examples": [...]
}
161
162 list-products =
163 "instruction":
22

---

Published as a conference paper at COLM 2024
165
166
Please follow these instructions solve the subtask:
167
To find a product category, you MUST use hover [id] to expand the popup Menu till you find the leaf element that has no popup. Then click [id] on the leaf element.
For exmaple, to find PS accessories you must hover hover over Playstation 4, then click on Accessories. Use note [] eveytime you hover on an item, and don't find the category. This is to ensure you don't keep trying the same category repeatedly.
Video Games, then
109
170 To sort current list of products
price and in ascending order, you MUST use the goto [url] action by appending ?product_ list litpict__ssttrrdrerpppppe ordersprice to the current URL. For example:
171 * To sort in descending order, you MUST use the goto [url] action by appending product_ wiaicctttstttordrrrreee order==riccepproducct Dostuctlistt.o. dir=desr,,, e.g.
To list all items less than particular price, e.g. $25, you MUST the goto [url] action by appending ?price=0-25
173 * Once you are done in stop [N/A]
174 * If the OBJECTIVE asks you to show the most expensive product, you must click
on the product.
.."
176
177 "examples": [...]
178 }
180 reviews =
181 "instruction": **"
182
(gsneral template)
Please follow these instructions to solve the subtask:
185
If you are not the product page, you can search the product using the search bar.
* To find the list of reviews, search for link with Reviewers. If you can't find it, scroll down to search for it. 187 Iterate over all reviews. For every relevant review, a note [ reviewer_ name: review info]. Record the relevant reviewer name name review VERBATIM. Once you are done with all the reviews in a page, scroll down to access more reviews.
186
Refer PREVIOUS ACTIONS know which reviews you have noted already. If you have noted a review already, look for the next review your current
OBSERVATION or scroll down.
189 * Do NOT repeat the note [] action for the same review.
Not all reviews will be visible on the reviews page. You MUST scroll down till you reach the end of the page. You will know that you have reached the end of the page if you see Contact Us in the OBSERVATION.
191
Once you have scrolled through all reviews, combine all your noted reviews that you can find under PREVIOUS ACTIONS. To combine, create a list of dicts where every dict has name and review key. Be to capture ALL the reviews in your note. Return that stop [[name: reviewer_name_11 review: review.1l, {name: reviewer_name.2, 2, review: review_2), ..]
192
193
"examples"; [...]
195
196
197
find directions = "instruction": (general_instruccioo..ee. template)
200
201
Please follow these instructions to solve the subtask:
202
First click on "Find directions between two points", then enter From and To Fields, and click search.
203
If you have find directions social security administration Pittsburgh search for it in a structured format like Social Security Administration, Pittsburgh.
23

---

Published as a conference paper at COLM 2024
205
206
"examples": [...]
207
}
208
209
search_nearees nearest-placee
210
"instruction":
211
seenerallinstructiion..etptppp
212
213
Please follow these instructions to solve the subtask:
211
For searches that refer to CMU, e.g. "find cafes near CMU Hunt Library" You have first center your map around location. If you have to find cafes near CMU Hunt Library, the first step is to make sure the map centered around Carnegie Mellon University. To do that, first search for Carnegie Mellon University and then click [] on list of location that appears. You MUST click the Carnegie Mellon University location to center the map. Else the map will not centered. E.g click [646]
215
216 Now that your map is centered around Carnegie Mellon University, directly
search for "cafes near Hunt Library". Do not include the word CMU in the search itee.
The word CMU cannot be parsed by maps and will result in invalid search. c. When your search returns list elements, return them structured format like stop [A, B, C]
218
219 * For searches that don't refer to CMU
220
No need to center the map. Directly search what is specified in OBJECTIVE, g. "bars near Carnegie Music Hall"
221
when your search returns a list elements, return them structured format like stop [A, B, C]
222 Be sure to double check whether the OBJECTIVE has CMU not and then choose
between instruction and 2.
223
Remember that the word parsed by maps.
cannot be typed in search bar as it cannot
224 * Remember that if you want to center your map around Carnegie Mellon University you have to click it after you search for it. Check your PREVIOUS ACTIONS confirm you have so, e.g. click [646] should be the previous actions.
225
226
eexmmples: [...]
A.3 MiniWob++ Policies
On MiniWob++, we identified set of policies that solve commonly recurring subtasks, e-g. filling different text boxes, choosing dates from date picker, processing emails, etc. For each of these policies, we collect few in-context examples to cover the subtask. At test time, SteP composes these policies to solve a wide number of tasks. Table 4 below shows the various policies and tasks they cover.
A.4 Task-wise Performance on MiniWoB++
Table shows task-wise performance breakup on the MiniWoB++ benchmark for various models. We choose a set of 45 tasks that don't require visual reasoning. We run 50 seeds per task and report the average success rates and number of actions. We divide tasks into simple and complex, where complex tasks might require multiple policies to solve the task. While on simple tasks almost all methods are equivalent, on complex tasks SteP Few-shot outperforms Flat Fewtshot every task.
Fig. shows an example of SteP for book-flight task. Book-flight a relatively complex task, requiring filling in drop down text boxes and filling dates. SteP outperforms Flat (0.10 + 0.90), where both methods have access to a single demonstration of book flight. SteP composes policies FILL (EXT() to fill in the Flight-from and Flight-to box, and then
24

---

Published as a conference paper at COLM 2024
Method
Examples
Tasks covered by examples
Flat
7
choose-date, book-flight
SteP
21
AGENT
3
FILL_
5
choose-date, book-flight search-engine, click-tab-2 click-checkbox, email- inbox
CHOOSE DATE
4
SEARCH. LINK
3
SEARCH. TAB
1
CLICK_ CHECKBOX
2
PROCESS. EMAIL
3
Table 4: On MiniWob++ library of policies used by SteP, number of in-context examples per policy and the tasks covered. Each example is state-action pair particular timestep. Each policy in SteP requires fewer in-context examples compared to Flat, but combined together they cover many more tasks.
CHOOSE CHOOSE_DATE() policies to pick a date from the datepicker. Each of these policies are more robust than Flat at solving the subproblems, resulting in higher overall success rate.
Browser UI
Starlin
Di0- High1
Lnvrogton MY (LE4
Lediggion KY..E.
01-
Kiseroee LASADO
Book
GROGI LIMTEE
catepiiinss
CLLER cliges)
Actons
CLOCK
790 mone 192011
het com Calhrreen
2iin de comppardigle S010000
Figure& 8: Outputs from SteP Few- shot on book-flight task showing hierarchical task planner actions, levelevel web policy actions, and LLM reasoning.
B
Ablations
B.1 Effect of Chain-of-Thought Reasoning
While we initially did not have chain-of-thought reasoning 2, we found that adding it in helped SteP shot rationalize a particular action well and boost performance. We wanted to understand which tasks in particular were helped by chain-of-thought reasoning, and what the trade-offs were.
We wanted to test the following hypothesis for SteP Few-shot with and without chain-of- thought:
1. Hypothesis 1: Chain-of-thought reasoning helps across all tasks. Even though chain-of-thought reasoning makes prompts slightly longer, the added step of ratio- nalizing actions should always boost performance.
2. Hypothesis 2: Chain-of-thought reasoning particularly helps in multi-step tasks, Multi-step tasks often require breaking down a problem into of steps and executing each step. While demonstrations certainly show how to break down task, adding cfain-of-thought better rationalizes this breakdown and helps generalize to new tasks not covered in the demonstrations.
"Chain-of-Thought Prompting elshttps:/ hwwp::///////vv///g/./ ab20///..//////////////
Elicits
Reasoning
in Large
Language
Mod-
25

---

Published as a conference paper at COLM 2024
Flat pero-shot
Flat Few-shot
SteP Bero-shot #act.
SteP Few-shot
Task
&suct
#act..
Asuct
#act.
&suct
ussct
#act.
click-link
0.94
1.00
1.00
1.00
1.00
1.00
1.00
1.00
click-option
3.68
2.62
2.94
focus-text
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
click-button
0.98
1.02
1.00
1.00
1.00
1.00
1.00
1.00
click-bution-sequenc
2.00
2.00
2.00
1.00
2.00
click-dialog
1.00
1.06
1.00
1.20
1.00
1.28
1.00
1.02
click-dialog-2
0.98
1.00
1.00
1.00
0.98
1.00
1.00
1.02
click-tab
1.00
simple
click-test
1.00
1.00
1.00
1.00
1,00
1.00
1.00
1.00
click beck-test-2
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
enter-text
2.50
2.00
2.10
1.00
2.00
cocus-text-2
1.00
1.00
1,00
1.00
1,00
1.00
1.00
1.00
enter-lexf-dynamic
0.98
2.44
1.00
2.00
0.98
2.06
1.00
2.00
enter-password
3.08
3.00
3.20
1.00
4.52
login-user
0.96
3.42
1,00
3.00
1,00
3.06
1.00
3,00
click-pie
1.00
3.00
1,00
3.00
1.00
3.00
1.00
3.00
enter-date
1,00
3.00
1,00
2.00
0.00
4.08
1.00
2.00
grid-coordinate
1,00
1.00
1.00
1.00
1,00
1.00
1.00
1.00
click-widget
0.94
1.00
0.94
1.00
0.94
1.00
1,00
1.00
email-inbox
7.00
0.70
4.50
0.00
3.00
0,90
5.20
email-inbox-nl-tun
0.40
7.20
0.50
6.00
0.00
2,90
1.00
4.58
email-nbox-forward-ree
0.30
5.08
0.40
4.80
0.00
3.50
0.90
4.30
multi-orderings
3.60
3.98
0.76
4.28
1,00
4.00
choose-date
0.20
3.00
0.94
3.60
0.20
5.80
1.00
5.40
clckk-oollapsible22
0.60
3.64
0.74
4.14
0.34
3.34
0.80
4.50
simple-arihmmttc
2.12
2.12
2.66
click-tab-2
0.76
5.58
0.88
4.62
0.88
2.84
1,00
2.24
click-tab-2-hard
0.68
3.36
0.88
3.84
0.76
3.06
1.00
2.42
multi-layouts
4.46
0.72
3.94
4.82
0,94
4.00
copy copy-paste
0.14
2.14
0.70
3.48
0.98
2.72
1.00
2.00
complex
click-coll collapsible
0.54
1.76
0.68
1.88
0.86
1.88
1.00
2.00
choose-date-easy
copy-paste-2
0.54
7.66
0.56
4.00
0.48
3.84
0.96
2.04
simple-algebra
0.14
8.80
0.30
6.78
0.04
4.38
0.74
2.00
click-cheekboxes
4,90
wall--chhebbxxxstrssssseee transfer
0.40
4,80
0.40
3.90
0.54
3.20
0.94
2.84
login-user-popup
0.46
6.28
0.46
3.52
0.46
5.82
1.00
4.88
click-checkooxes-soft
7,00
6.94
0.54
5.64
enter inter-text-2
0.00
2.60
0.40
5.20
0.40
2,00
1.00
2.00
mmail-inhox-forwanddn
0.10
5.04
0.10
4.58
0.00
3.24
0.74
4.74
search-engine
3.64
3.16
0.26
4.46
1.00
find-word
01.22
2.62
0.26
5.18
0.12
2.92
0.988
2.00
dhoose-aate-medium
2.90
0.20
2.76
0.20
9.26
1.00
3.86
click-checkboxes-layg
8,40
0.20
8.40
7.00
1.00
6.20
book-flight
0.00
16.00
0.10
11.10
0.00
13.52
0.90
9.14
Mean
0.60
3.70
0.72
3.38
0.65
3.43
0.96
2.89
Table 5: Task-wise performance breakup on MiniWoB++ benchmark on set of 45 tasks.
We compared SteP Few-shot with two versions having chain of thought, and not having. Fig. 9 shows a plot of success rate for each of the 3 clusters of tasks single, composite, multi.
Hypothesis Chain-of-thoughh reasoning helps across all tasks.. We find this be since for all tasks, chain-of-thon thought performs either equally or better. This confirms that the extra tokens consumed by the reasoning does not hurt performance and in fact helps significantly in some cases.
Hypothesis 2: Chain-of-thought reasoning particularly helps in multi-step tasks. We find this to be true well. Looking at the multi-step tasks, chain-of thhouugg has the largest performance gains compared to any other cluster. The performance gains are the largest in book-flight and search-engine where the horizon length is the largest. In comparison, for single and composite the performance gains vary, being higher for certain tasks like choose-date and find-word and zero for others like click tasks.
26

---

Published as a conference paper at COLM 2024
B.2 Model Scaling
In MiniWob++, while we developed the prompts with rext-davinci-003, we wanted to compare how performance varies with newer models gpt-3. and gpt-4. 3pt-3.5- turbo is also an InstructGPT model optimized for chat and trained at 1/10th the price of text- text-davinci-003 gpt 4 is a significantly larger model, and capable of solving more complex tasks.
We wanted test the following hypothesis:
1. Hypothesis 1: GPT-4 improves performance across all methods, but both decom- position and few-shot examples help. GPT-4 is powerful model and with an exhaustive set of instructions in the prompt, should be able to achieve perfect performance. However, designing such exhaustive prompts for all web tasks is challenging. We hypothesize that decomposition helps break down and scope instructions leading to better performance for GPT-4. Similarly, few-shot examples helps GPT-4 ground instructions the webpage.
2. Hypothesis 2: gpt-3.5-turbe slightly worse than text-davinci-003 given few-shot examples. Practioners have noted that while 3 t-rbbo has better shot performance, text-davinci-O5 is trained on a more diverse set of tasks and performs better with k-shot learning https://scale, com/ b
vs davinni##nnrroduction Since shot performance for web-
Success Rate
0.0
62
Single
dick didk-eidoe
opiion
bow-lod
didk www.dcccoooiicco targe
Chain-af- Frouggt
1.0
Success Rate
0.0
No Chun-----huggh
6.0
ospg soacing?
chorendabl
mipgle sipets
fndwodd
wris ledt dyname
mstan enta-onwwad
masii lpdodd
mullicoreenng
Eggin wiper
Composite
Success Rate
0.8
0.4
Hoce Sight
Nearch orgina
byti (0000))
dist
Figure 9: Success rate of SteP Few-shot with both thain-of-thought and without.
27

---

Published as a conference paper at COLM 2024
book- -lighhhtt Success Rate
02
pin, permator
dovetor
pAN Jonstie
Search- search-engine eh-igiiine Success Rate
6.
daiine 000
pin, pevoaaa
fst. Jownsst
Plor, Sovvtict
Simple- Hample-algenn algebra Success rate
aot po..-oo-ioo
66
Demabut
fer Sterhho
Stoeete
Figure 10: Success rate all methods with different models.
tasks is challenging without exhaustive instructions, we hypothesize that text-davinci--0: will perform better.
We compare three language models text-davinci- 003, gpt-3. 5-turbo and gpt-4 for all baselines on 3 tasks from MiniWoB++ -toooB+-bboooflight search-engine, simple-algebra. Fig. 10 shows a plot for each of these tasks.
Hypothesis 1: GPT-4 improves performance across all methods, but both decomposition and few-shot examples help. We find this to be true. GPT-4 is unambiguously better than any other model and improves all methods. However, as shown in book-flights and search-engine, both decomposition (SteP) and few-shot examples boost performance. In simpler reasoning tasks like simple algebra, it gets it correct right away.
Hypothesis 2: gpt-3.5 5- t-rubb slightly worse than text-davinci-003 given few-shot examples. We also find evidence for this. contt-dvviiii-io0 003 with few-shot examples always outperforms gpt 3.5 5 turbo. Although, in simple-algebra, zero shot performance of g 3. have observed in other tasks.
turbo is better than text text-davinci-on matching what other practitioners
B.3 Evaluation with Llama model
We also compare Flat and SteP with open-source models like LLaMA-17B, 13B). The models are all pre-trained instruction following models without any additional fine-tuning.
28

---

Published as a conference paper at COLM 2024
The prompts used are the same Flat Zero-sho shot and SteP shot. Overall, we find the performance to be lower than 8pt-(3,3.5,4). The drop in performance could be due to number of reasons such as the model size or the training data which the models are trained. However, we find that SteP still outperforms Flat for many tasks, which we discuss below.
StsP
1.0
Success Rate
0.5
0.0
dick-test
click-test-2
dlck-dialog
click button
collapbbee
dlck-dialog-2
clickrabb
enter-fext text
enter-text-2
ctcck Merl-buttonseee sequence
Figure 11: Evaluation with LlaMA-13B on subset of MiniWob++ tasks. Success rate breakdown of 10 tasks averaged 10 seeds over task.
Figure. 11 shows a comparison of Flat and SteP on subset of MiniWob++ tasks. We see that Flat matches SteP for simple CLICK tasks, it fails for slightly more complex tasks that require multiple actions, e.g. multiple CLICK or TYPE.
We cluster common failure modes for both Flat, SteP below:
1. Repeating the same action in multi-step tasks. For tasks that require sequence of actions, Flat tends to get stuck in the first step repeating the same action over and over. It fails to interpret the previous actions or changes to the browser content to update the predicted actions.
For example, in the click-buttee -ew--uuutto--seuenee sequence task, for the following input
CONTEXT:
Click button ONE, then click button TWO.
BROWSER CONTENT:
<body id=1 val= />
<div id=2 val=wrap />
<div id=3 val=area />
<button id=4 val=ONE />
<button id=5 val=TwO />
PREVIOUS ACTIONS:
CLICK
the model produces the following output
REASON:
To achieve the goal of clicking button ONE and then button TWO, we need to first click button ONE to activate it, and then click button TWO to activate it.
ACTION:
CLICK
when the correct output would have been CLICK 5, given that the previous actions show that CLICK 4 has already been executed.
2. Skipping over intermediate actions. faced with multistep tasks. For example, on a book flight task with browser content
We see that LLaMA-13B skips over actions when
<h2 id=5 val=Book Your One-Way Flight />
<input text id=7 val=flight--fre from />
29

---

Published as a conference paper at COLM 2024
the model generates an action like CLICK 5 directly before filling in the empty flight-from input box.
Not following the desired action formats. We observe that LLaMA-13B often times fails to follow the specified action formats for CLICK and TYPE. This problem occurs for both Flat and SteP.
For example, given the following browser content
<div id=2 val=wrap />Anslink id=4 val=justo. />...
the models predict CLICK #justo instead of CLICK 4.
Similarly, the model predicts TYPE "Ignacio" instead of TYPE 5 "Ignacio".
C Airline CRM Simulator
We constructed mock Airline CRM environment based on typical airline call-center work- flows, and modelled on public airline websites (e.g,, jetblue.com, aa.com). The website is built as a standard single-page web application, using light-weight web development stack. Figs 12, 13 shows sample screenshots from the CRM.
C.1 Generating scenarios
This CRM allows us to test scenarios that are more complex than those in MiniWoB++, and scenarios that cannot practically be run on public websites (e.g,, cancelling flight). The scenarios currently supported are described below.
1. Find one-way or return flight. This is a single step task, that requires the source destination airports and flight dates to be entered in the UI, and the search button to be clicked.
2. Book flight This is four step task:
(a) Find flight (scenario 1),
(b) Select desired outward and return flights click Confirm,
(c) Enter passenger details (Title, first name, last name, gender, date-of-birth) & click Save,
(d) Enter payment card details (card number, expiry, CVC/CVV) click Book flight.
3. Find a booking This is a single step task enter booking reference click Search.
4. Cancel booking This is three step task:
(a) Find booking (scenario 3),
(b) Click Cancel,
(c) Confirm cancellation by re-entering booking reference & click Cancel.
5. Modify passenger details on an existing booking This is a three step task:
(a) Find booking (scenario 3),
(b) Click Modify,
(c) Change any required passenger details & click Save.
6. Modify flights on an existing booking This is
(a) Find booking (scenario 3),
(b) Click Modify,
(c) Find flight (scenario 1),
(d) Select desired outward and return flights & click Save,
C.2 Helper APIs
In addition to supporting the above scenarios, the CRM also exposes a few helper APIs that make running and evaluating experiments easier. Two of these are of interest here:
30

---

Published as a conference paper at COLM 2024
booking
Help
Find flights
From
Depar
Setuin
Search
BOS
JFK
05/21/2023
03/26/2023
Select your departing flight
From
Highh mumber
Price
5:51am
7:00am
017
$100
sue
7:30am
8:40em
617
$162.00
bos
0:25am
1:31
717
598 00
$152.0
$173.06
6:00pm
7:20pm
255mm
9:48pm
8:25pm
9:48pm
Select your return flight
From
Fiihht number
Price
6:45am
7:50am
901
aFc
806
0:17am
1:24am
613
$144.00
Pic
2:42pm
1:40pm
1318
901.02
anC
1:54pmm
3:10pm
116
507 83
606
$117.75
$175.71
1:5ppmm
Cheskout
Figure 12: search flight screen of the mock airline CRM.
candom-scenarin This API returns scenario randomly selected from those listed above, along with all of the data required for completing that scenario the UI. Shown below is example of scenario returned by this API. In addition to the task specific data, the scenario includes unique id, and unique URL on which the task can be executed.
"scenario": FAADK
"id": "ylnjdGiuuqddcddrssss","
"url":
"details":
"flight": { "from": "JFK" "to": "FLL", "departure": "2023-07-07",
"return": "2023-09-13",
dentwardddeearture--ttee *7:01pm",
12
Noutward-arrivalltiim" "0:13pm".
13
"return-departure--tmm"" "6:00am"
14
"return-arrival--iine "8:43am"
31

---

Published as a conference paper at COLM 2024
Confirmation code
BT.COL
Passenger details
Glenna
Scone
glanna boonellshenw.rec
Ferulo
00/11/2003
Departing flight
255pm
Returning flight
MSJ 2003-05-25 34tem
2003-66-25 b0tperr
400
$71.05
Total
$122.83
Find altemate flights
Search
flights
Figure 13: find and modify booking screen the mock airline CRM.
This API provides a means of automatically evaluating the success rate and fask progress metrics for scenarios generated by the API above. Specifically, if the UI actions are performed on the unique URL returned by the generaae-randdon--ccmmn afterwards with the scenario id will return the metrics. These metrics are calculated with reference the gold standard actions required for the given scenario.
API, calling the evaluate API
D Live Websites Evaluation
D.1 Collecting Human Task Demos
We collected a dataset of human agents searching for flights across three websites: https:/ www. www.jetblus com/, https://ww/ aa. com/, https: / //ww:./wwwwww united com/en/us For each of the websites, a human agent was given a set of 10 task specifications as short conversations, e.g. "Book flight departing from <>, arriving at <>, leaving on and returning on <>". The actions of the human agent, e.g. the click and types, were recorded for each episode. Along with the actions, the raw DOM of the website was also recorded.
D.2 Parsing Browser Content
Given data point of raw website DOM (Doc- ument Object Model) and action, we make use of playwright pttps://ptl /pwpwwigghtttttt dev parse the DOM and extract a list of web elements. This process involves traversing the DOM tree and ex- tracting salient nodes, i.e. nodes with actionable elements like <input>, <button>> <link>, We also propagate up text attributes from child nodes to the salient parent nodes since the text label for element may occur inside the subtree. Every
12000
not compressed compressed
10000
Takens
6000
4000
2000
Miniwob
CRM
Livelveb
Jasks
Figure 14: Token counts for browser con- tent before and after compression on differ- ent environments.
32

---

Published as a conference paper at COLM 2024
web element in the iist represented by an id and optionally a value. The list of all such elements is concatenated to represent the browser content in natural text form. This is input as the browser observation in the LLM context. Natural language descriptions attached to different web elements helps it generalize across different websites since LLMs have been pre-trained on natural language data. This text form is included under Browser Content in the LLM context. We also convert the demonstrated actions to CLICK <id> or TYPE <id> "TEXT".
D.3 Live Website Results
Fig. 15 shows evaluation of SteP Few-shot and Flat Few-shot across 10 runs each on different live websites with task specification coming from short simulated conversations. What makes this task challenging is that the browser content from these websites have a lot of extraneous information that make challenging to parse the correct fields. Fig. 14 shows the extent of compression we perform to fit the browser content into the LLM's context space. For each run, we evaluate by comparing model performance against a reference human demonstration. In Fig. 15, SteP Few-sho shot is able to generalize to multiple websites even though it has demonstration from only one (i.e. jetblue.com) In contrast, Flat Few-shot fails to generalize from it's demonstration. Again SteP Few-shot, by decomposing the problem into policies able to solve the task.
Metric
SteP Few-shot
Few-shot
Rondly
Ssscf
0.60= oa5
1.00s
Train
sprost
4act.
8.00 0.00
7.00 0.00
Nsuct
0.20= 0.44
0.50s
Marage
sprost
0.47=001
fact.
8.00= 0.00
6.40 0.27
8sucf
0.00= 0.00
0.20s
sprost
0.04
#act.
6.00 0.00
6.00 0.00
Figure 15: (Left) Evaluation 3 live airline websites averaged over 10 runs per website. (Right) Difference train (jetblue) v/s test (united, aa) website UIs.
33


Reviews:
[{"review_id": "5CkU4aznXa", "reviewer": "Reviewer_piMY", "summary": "This paper proposes SteP, which dynamically composes policies from a policy library into a policy stack, to solve web tasks. The proposed model shows strong performance on WebArena, MiniWoB, and a CRM task.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback!\n\n**Task-specific, manual policy library**\n\nWhile we agree that adding a new policy requires manually constructing a prompt, it is no more additional work than prompt design common in current LLM agents. E.g., total human-written instruction characters in SteP policy library is similar to existing Flat/ReAct web agents, ***Flat-4k (7.5k chars, 0.20 success) -> Flat-8k (17.3k chars, 0.23 success) -> SteP (20.2k chars, 0.36 success)***\n\nWe don't write policy prompts to solve individual tasks as that requires a combinatorially large number of policies. Rather, each policy solves common subtasks shared across multiple tasks. E.g., ***SteP uses only 14 policies on 804 WebArena tasks***\n\nSteP's key contribution is its dynamic composition of policies, allowing policies to decide which other policies to call. This expands task coverage beyond the instructions in policy prompt and enables the LLM to contextually load instructions for specific subtasks.\n\nOur work is developed within the context of existing works designing prompts for LLM agents. Discovering useful policies or skills autonomously (a long standing challenge in hierarchical RL [3]) is an important but orthogonal problem to the general development of the field, outside our current scope.\n\n**Comparison to [1,2]**\n\nWe cover [1,2] in related work. Both don't address web automation, other important differences are:\n\nDecomP [1] vs SteP:\n* DecomP subroutines do not call each other (except for recursions), restricting it to a static hierarchy. SteP allows any policy to call any other, creating a dynamic hierarchy. \n* DecomP generates a static program upfront, whereas SteP does this dynamically, changing the policy call sequences based on observations. \n\nADaPT [2] vs SteP:\n* ADaPT predicts a plan and greedily decomposes it until one of the subtasks can be solved, without backtracking.\n* SteP uses policies instead of plans – allowing it to react to failures quicker by changing the decomposition.\n\n**Prompts to manage policy stack?**\n\nThe stack is managed outside of the LLM. We simply provide the LLM a list of available policies to choose from.\n\n**How was policy library chosen?** \n\nEach policy is designed by identifying commonly occurring subtasks from related task clusters, e.g. search_order() policy is shared by 69 tasks (20 clusters) like “Change delivery address..”, “Draft refund message..”\n\n***\n[3] Diversity is All You Need: Learning Skills without a Reward Function", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The stack formulation for composing policies is intuitive and effective.\n- The performance is strong on multiple tasks.\n- The paper is clear and well written.", "reasons_to_reject": "My main concerns are as follows:\n- The task-specific nature of the proposed method, as it relies on a manually crafted choice of policy library, as well as demonstrations for the policies. \n- A lack of novelty with respect to existing decompositional methods such as [1,2], as both methods explored the utilization of dynamic decomposition using a predefined skill library.\n\n[1] Decomposed Prompting: A Modular Approach for Solving Complex Tasks. Khot et al., ICLR 2023\n[2] ADaPT: As-Needed Decomposition and Planning with Language Models. Prasad et al., NAACL 2024", "questions_to_authors": "- What kind of prompts does the meta-policy need for each task, in order to manage the policy stack?\n- Can the authors share more details about how the policy library was chosen on each task?", "extra_scores": {}}, {"review_id": "IzTJPeweE8", "reviewer": "Reviewer_TTFb", "summary": "This paper proposed a method called Step, which aims to solve the complex web navigation tasks. Instead of relying on one global policy/prompt to control how agent interact with the environment, Step contains a set of policies that are tailored to solve different sub-tasks, and the Step agent dynamically composes/invokes different policies to solve the task at hand. In particular, each policy can choose to execute an action in the environment, call other policies or call itself, which offers great flexibility in agent’s exploration. The Step method is evaluated on popular web agent benchmarks including WebArena, MiniWob++ and CRM and a few real-world websites. The results show that it outperforms previous methods and baselines with a flat policy.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback!\n\n**Manually crafted policies**\n\nWhile we agree that adding a new policy requires manually constructing a prompt, it is no more additional work than prompt design common in current LLM agents. E.g., total human-written instruction characters in SteP policy library is similar to existing Flat/ReAct web agents, Flat-4k (7.5k chars, 0.20 success) -> Flat-8k (17.3k chars, 0.23 success) -> SteP (20.2k chars, 0.36 success). We do not write policy prompts to solve individual tasks as that requires a combinatorially large number of policies. Rather, each policy solves a commonly occurring subtask shared across multiple tasks.\n\nWhile the current instructions may appear specific to websites in WebArena, it is easy to extend them to cover more websites. E.g. search_order() policy instructions (shared by 69 tasks) can be thought of as hints (not hard rules) to the LLM agent on how to locate the page containing order, e.g. My Account -> My orders, or click profile picture -> My Orders. Ultimately, the LLM agent chooses appropriate actions given webpage observation. We already do this for our LiveWebsite evaluation (Appx E) where the same choose_date() policy generalizes across different websites like JetBlue, American, United that have different UIs. \n\nWe do observe that the ***majority of the gains (8.1% -> 47.9%) come on tasks when policies are invoked*** vs (19.2% -> 25.8%) when policies are not. Note that a library of only 14 policies covers 47.5% of the total 804 WebArena tasks, indicating our policies are not task-specific.\n\nWe also note that our work is developed within the specific context of existing works designing prompts for LLM agents. Automatically discovering useful policies is an important, orthogonal problem not only to our work but to the general development in the field, outside our current scope.\n\n**Comparison to related work**\n\nThank you for the reference, we will be sure to include it.\n* LASER define a finite state machine (FSM) where every state corresponds to an LLM agent performing a subtask. However, they do not hierarchically decompose a task into a set of subtasks. This would require tracking which agent called which other agent, not expressed in the FSM (but expressible in SteP as the policy stack)\n* SteP is evaluated on WebArena (5 domains) while LASER on WebShop (1 domain) making direct comparison difficult. We compare against prior works on WebArena, and ***achieve the top result on WebArena.***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method is very intuitive that the complex tasks can be decomposed into simpler ones, and each simple task can be handled by dedicated policies. The results on web agent benchmarks are also pretty promising. \nOverall the Step agent uses less tokens in its prompt to complete the task compare to previous baselines, hence that it is more cost effective.", "reasons_to_reject": "My biggest concern is about the construction of policies. As the authors admitted in the limitations, the Step method requires user to manually write detailed prompts for every policy. This implies that the method cannot be easily scaled to more websites/tasks. Moreover, from the examples shown in Appendix B.2 we can see that these policies/prompts are very tailored towards specific websites. For example, search order policy informs the agent to first go to My Account and then go to My orders. I’m not sure if the prompt can transfer well out of the box to different websites. For example, what if a website do not have the My Account button but the user actually needs to click their profile picture to get to the account details? \nIn section 5.1.2., the authors mention that the manully written policies coveres  at least 50 intent templates from WebArena. Have you checked if the gains on WebArena are mainly from these covered intents? \n\nMissing references:\nThe idea of constructing specific policies/prompts at different states to control the web agent has been explored in Ma et al. 23. It would be good to compare your method with their work as well. \n\nMa, Kaixin, Hongming Zhang, Hongwei Wang, Xiaoman Pan and Dong Yu. “LASER: LLM Agent with State-Space Exploration for Web Navigation.” ArXiv abs/2309.08172 (2023): n. pag.", "questions_to_authors": "Please see the limitation section", "extra_scores": {}}, {"review_id": "NheYx4ABew", "reviewer": "Reviewer_BzFx", "summary": "They advance the research on the line of performing web tasks with LLMs by decomposing them into different policies (instead of using a single prompt). In particular, they develop an approach based on a Markov Decision Process where the state is compounded by a stack of policies, which represent the sequence of policy calls. Evaluation of their methodology across key benchmarks such as WebArena, MiniWoB++, and a CRM simulator reveals promising results, demonstrating efficiency in data consumption.", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback!\n\nWe will be sure to include a conclusion section containing the following key takeaways:\n\n* LLMs face significant challenges solving web tasks, including combinatorially large open-world tasks and variations across web interfaces. This often results in performance substantially lower than human performance.\n* Simply specifying a large prompt to handle all possible tasks and behaviors is complex, and results in behavior leaks between unrelated behaviors.\n* SteP successfully performs a diverse set of web tasks by decomposing them into distinct policies, each solving a specific subtask. At test time, it dynamically composes these policies to solve a new task.\n* Our key findings include,\n  * On WebArena, SteP outperforms (0.15 -> 0.36) prior works that use few-shot LLM (GPT-4) policies, while on MiniWob++, SteP is competitive with prior works while using significantly less data.\n  * SteP achieves this while using 2.3x lesser tokens per trajectory, resulting in lower overall costs.\n* SteP opens up possibilities for interesting future research, such as the potential to orchestrate models of varying complexity for different policies or exploring automatic policy discovery.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors introduce an innovative method for addressing the challenge of executing web tasks with LLMs. \n- They conduct a comprehensive comparison with other baseline methods across major benchmarks.\n- Their approach demonstrates superior performance over the state-of-the-art in certain benchmarks, all while necessitating less data. \n- The paper is well structured and presents information in a clear and accessible manner. \n- Following acceptance, the code will be made publicly accessible.\n- I believe it encompasses a satisfactory amount of work and quality to merit acceptance at COLM.", "reasons_to_reject": "- The absence of a Conclusion section, which consolidates the conclusions drawn from preceding sections, is noted. However, aside from this observation, I do not identify any other issues with this paper.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents SteP, a novel approach for web task automation that dynamically composes policies from a policy library into a policy stack. The reviewers agree that the proposed method demonstrates significant improvements over existing methods on benchmarks such as WebArena, MiniWoB++, and a CRM task. The intuitive and effective stack formulation, combined with strong performance results, highlights the method's potential. However, concerns were raised about the task-specific nature of the policy library and the manual effort required to construct it. The authors' rebuttal clarified the scalability and flexibility of the policy design and distinguished their approach from similar works. Overall, the paper is well-written and contributes valuable insights to the field, meriting acceptance.
Decision: Accept