SteP: Stacked LLM Policies for Web Actions

Authors: paloma  sodhi
s r k branavan
yoav  artzi
ryan  mcdonald
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 14:28:28

Abstract: Performing tasks on the web presents fundamental challenges to large language models (LLMs), including combinatorially large open-world tasks and variations across web interfaces. Simply specifying a large prompt to handle all possible behaviors and states is extremely complex, and results in behavior leaks between unrelated behaviors. Decomposition to distinct policies can address this challenge but requires carefully handing off control between policies. We propose Stacked LLM Policies for Web Actions (SteP), an approach to dynamically compose policies to solve a diverse set of web tasks. SteP defines a Markov Decision Process where the state is a stack of policies representing the control state, i.e., the chain of policy calls. Unlike traditional methods that are restricted to static hierarchies, SteP enables dynamic control that adapts to the complexity of the task. We evaluate SteP against multiple baselines and web environments including WebArena, MiniWoB++, and a CRM. On WebArena, SteP improves (14.9\% to 33.5\%) over SOTA that use GPT-4 policies, while on MiniWob++, SteP is competitive with prior works while using significantly less data. Our code and data are available at https://asappresearch.github.io/webagents-step.

Main paper: 
 introduction: 
 While large language model (LLM) agents have shown impressive decision-making capabilities (Yao et al., 2022b; Huang et al., 2022b) , the web remains a challenging domain achieving much lower success rates compared to other benchmarks (Akter et al., 2023; Zhou et al., 2023) . The web contains a combinatorially large open-world space of tasks such as booking flights, purchasing items or making appointments. Web interfaces also differ substantially from one website to another, for instance, the task of purchasing an item on Amazon looks different from purchasing it on eBay.There are fundamental challenges to designing a singular LLM policy to solve all possible web tasks. First, the policy requires instructions and examples to cover all variations in tasks and websites. Second, solving longer horizon tasks requires keeping around a long history of previous actions and observations in context. Longer contexts make it harder to pay attention to salient information leading to more errors and costs (Liu et al., 2023) .Instead, a natural solution is to decompose the problem into distinct policies (Khot et al., 2023) . Each policy provides dedicated instructions and examples for a particular subproblem, such as searching a list or finding a page. However, this typically requires manually specifying a decomposition hierarchy that hands off control between policies (Prasad et al., 2023; Zhou et al., 2021; Song et al., 2023) . This restricts control to a static hierarchy that fails to adapt to varying task complexity.Our key insight is to enable dynamic control, where any policy can choose to invoke any other policy. Such expressiveness is crucial for solving web tasks that require policies to operate at multiple levels of abstraction. Consider Fig. 1 where the agent must find all commits 1 Published as a conference paper at COLM 2024 Objective: "Find all commits made by <user> across all repositories"Policy StackFigure 1 : SteP composes policies to solve complex task, where policies can invoke each other. SteP uses a policy stack to keep track of the dynamic control state. Given an objective "Find all commits made by a <user> across all repositories", SteP intializes with a search _ list() to search over all repositories, which in turn invokes another search _ list() to search overall commits in a repository.made by a user across all repositories. A search _ list() policy must first iterate over all repositories. For every repository, it must recursively call another search _ list() that iterates over all commits in that repository. This can only be solved by an architecture where policies can call each other, including themselves.We propose Stacked LLM Policies for Actions on the Web (SteP), a method to perform a diverse set of web tasks by dynamically composing policies. SteP defines a Markov Decision Process (MDP) where the state is a stack of policies. The stack stores the dynamic control state capturing the chain of policy calls that evolves over time. At every time step, the policy on the top of the stack either acts directly on the web page, invokes a new policy that gets pushed onto the stack, or terminates and pops out of the stack. For instance, in Fig. 1 task, the stack initializes with a search _ list() policy, which can both act on the web page or instantiate another search _ list() policy with a different set of arguments.Our key findings are that dynamically composing policies (SteP) significantly outperforms both prior works (0.15 → 0.33) and single policy baselines (0.23 → 0.33). SteP achieves this while using 2.3x lesser tokens per trajectory, resulting in lower overall costs. We also show several ablations on the effect of varying contexts, in-context examples, and CoT reasoning.Our key contributions are:1. A novel framework SteP that defines an MDP over a stack of policies enabling dynamic composition of policies to solve complex web tasks. 2. Experimental validation on a range of web benchmarks: WebArena, MiniWoB++, and an airline CRM simulator. On WebArena, SteP improves (0.15 → 0.33) over prior works that use few-shot LLM (GPT-4) policies, while on MiniWob++, SteP is competitive with prior works while using significantly less data. 3. Implementation of SteP as a meta-policy that wraps around any existing policy class. 
related work: 
 Language models for web tasks. Early work mapping natural language instructions into actions (Branavan et al., 2009; Artzi & Zettlemoyer, 2013) has rapidly evolved into the field of LLM agents (Wang et al., 2023b) . Broadly, methods include training RL agents to navigate web interfaces (Humphreys et al., 2022; Liu et al., 2018; Shi et al., 2017) , in-context learning with large language models (Zhou et al., 2023; Zheng et al., 2024; Kim et al., 2023) , or finetuning language models on web tasks (Deng et al., 2023; Furuta et al., 2024; Yao et al., 2022a) . With in-context learning, using a single LLM policy that contains all instructions and examples results in long contexts that can be error-prone. Recent approaches (Zheng et al., 2024; Kagaya et al., 2024) counter this by retrieving trajectories from a database. However, covering the combinatorial space of tasks requires a large dataset from many tasks. Instead, our work leverages a library of policies, each with dedicated instructions and examples, and composes policies to cover such tasks.Language models for decision making. Instruction following LLMs have shown impressive decision making capabilities (Huang et al., 2022a; Brown et al., 2020) and tool use (Schick et al., 2023; Yang et al., 2024) by chaining together reason and actions (Yao et al., 2022b) . However, for long-horizon tasks, a single policy with a long chain of reason and action can be error-prone. Broadly works deal with such issues by hierarchical planning (Prasad et al., 2023; Khot et al., 2023; Zhou et al., 2021) , defining state machines (Ma et al., 2023) , generating code (Wang et al., 2023a; Liang et al., 2022) , or by self-correction (Shinn et al., 2023) . Hierarchical decision-making has a rich history in AI (Sutton, 1998) , where a high-level policy chooses either from a library of skills (Tessler et al., 2017) , or predicts subgoals (Nachum et al., 2018) or predicts rewards (Vezhnevets et al., 2017) for a low-level policy. While these methods focus on efficiently learning policies, they restrict the decomposition to a predefined hierarchy of 2 or 3 levels. Instead, our framework allows any policy to dynamically call any other policy in the library, with control state being tracked using a policy stack. Compared to decomposition approaches like DecomP (Khot et al., 2023) which generates a static program with subroutines that cannot invoke each other, SteP decomposes dynamically based on observations and allows policies to invoke each other. Compared to hierarchical planning approaches like ADaPT (Prasad et al., 2023) , which predicts a plan, SteP instead predicts policies that can react to failures more quickly.Parsing. The fact that the main control structure of our formalism is a stack relates our algorithm to Pushdown Automata (Hopcroft & Ullman, 1969) . However, in Pushdown Automata, the input tape is static and processed sequentially. In contrast, for web actions the input tape is dynamic due to new observations that arise. These observations dynamically alter the context (or input string) that the policies work with. While different in scope, our work does draw inspiration from transition-based dependency parsing systems, specifically stack-based algorithms (Nivre, 2008) . In particular, our algorithm consists of states whose main data structure is a stack and states transition to new states via a finite set of well defined actions (see Sec. 4.1).We see that LLaMA-13B skips over actions when faced with multistep tasks. For example, on a book flight task with browser content ... <h2 id=5 val=Book Your One-Way Flight /> <input _ text id=7 val=flight-from /> ... the model generates an action like CLICK 5 directly before filling in the empty flight-from input box.3. Not following the desired action formats. We observe that LLaMA-13B often times fails to follow the specified action formats for CLICK and TYPE. This problem occurs for both Flat and SteP.For example, given the following browser content ... <div id=2 val=wrap />\n<link id=4 val=justo. />... the models predict CLICK #justo instead of CLICK 4.Similarly, the model predicts TYPE "Ignacio" instead of TYPE 5 "Ignacio".We constructed a mock Airline CRM environment based on typical airline call-center workflows, and modelled on public airline websites (e.g., jetblue.com, aa.com). The website is built as a standard single-page web application, using a light-weight web development stack. Figs 12, 13 shows sample screenshots from the CRM.This CRM allows us to test scenarios that are more complex than those in MiniWoB++, and scenarios that cannot practically be run on public websites (e.g., cancelling a flight). The scenarios currently supported are described below. In addition to supporting the above scenarios, the CRM also exposes a few helper APIs that make running and evaluating experiments easier. Two of these are of interest here:Invoke another policy: It can choose to invoke a new policy π i+1 . The new policy is initialized with an empty history and is pushed onto the stack.https://platform.openai.com/docs/modelssubroutine searches issues that matches the query. Any objectivethat says "open my latest issue" or "open issue with <keyword> in the title" must be passed through this subroutine.`create _ project [query]`: Given you are in the create new project page, this Gitlab subroutine completes the act of creating a project, adding members etc. 
problem formulation: 
 Given a natural language instruction, such as "Book me a flight from NYC to BOS", our objective is to learn a policy π that execute this task on a web environment. This can be formulated as a Partially Observable Markov Decision Process (POMDP), denoted as ⟨S, A, O, T , r⟩. At each time step t, the policy (or agent) performs an action a t given a partial observation o t , resulting in a new state s t+1 and observation o t+1 , where:• State, s ∈ S is the current state of the web environment, including the current webpage contents and results from previous interactions, e.g., a new repository created in a GitLab environment. • Action, a ∈ A(s) denotes the possible actions that can be performed in the current state, such as clicking, scrolling, typing on specific web elements. These are represented as click [id], type [id] [value] , where id refers to a specific element on the webpage. The action space is typically large due to the many elements on a web page. • Observation, o ∈ O is the current observable aspect of the state, i.e., the current webpage Document Object Model (DOM) serialized as text. • Transition function, T (s ′ |s, a) is a deterministic function modeling the change in the webpage resulting from an action determined by the underlying website. • Reward, r(s, a) is awarded for reaching a set of subgoals, e.g. canceling a flight has subgoals like finding the booking and then canceling it.As the state is partially observable, the policy maps a history of observations and actions h t = {o t , a t-1 , o t-1 , ..} to the current action a t , i.e. π : h t → a t . As discussed before, learning a single LLM web policy π is challenging. We next look at composing multiple policies. 
approach: 
 We present a framework, Stacked LLM Policies for Web Actions (SteP), that performs a range of web tasks by dynamically composing policies. As previously discussed, designing a single policy that solves all tasks is challenging. Instead, we utilize a library of policies Π which we compose to solve a task. Fig. 2 shows an illustration of SteP solving a web task SteP dynamically composes policies from a library, using a policy track to keep track of active policies. At every timestep, SteP either acts on the webpage, or modifies the stack to add/remove policies.by dynamically stacking policies from the library. Sec. 4.1, 4.2 discusses the stacked policy model and SteP algorithm respectively.At any given time, we represent the control state as a stack Σ which denotes the chain of policy calls Σ = ⟨π 0 |π 1 | . . . |π i ⟩. We extend the MDP in Sec. 3 to include a stack of policies:State. The MDP state is augmented with a stack Σ = ⟨π 0 |π 1 | . . . |π i ⟩ of invoked policies. The top of the stack π i is the current active policy. Each policy in the stack maintains its own history of observation, reason, action. The stack is initialized with a base policy Σ = [π 0 ].Action. We augment the original MDP actions with two new actions -invoke a new policy π ∈ Π or terminate the current policy π i with a return value v i .Transition. Suppose at timestep t, the current stack is Σ t = ⟨π 0 |π 1 | . . . |π i [h]⟩. The policy on top of the stack, π i , can take one of three actions leading to different state transitions:1. Issue an action: It can issue an action a t along with reason r t . This is sent to the environment, which updates its state s t+1 and responds with an observation o t+1 . The action, reason, and observation is appended to the history maintained by π i . The set of policies in the stack remains unchanged, only the history for the current policy updates. Zhou et al., 2023) that use a single policy, composition allows for more adaptability. 2. Scalability. Adding a new policy to SteP is easy. The user constructs the prompt for the policy and adds it to the library with a description. This policy becomes available as part of the action space for any other policy, without requiring any change to the code. 3. Modularity. Each policy tracks only the local context of the specific subproblem it is solving. Once it terminates, it hands back control to the previous policy in the stack without reasoning about the global context. This allows reusing policies in different context, e.g. the same fill_form() can be used by book flights or make appointments. Compared to prior work (Zheng et al., 2024) that require demonstration for entire tasks, modularity requires demonstrations for subtasks thus being more sample efficient.EQUATIONAlgorithm 1 presents the pseudo-code for SteP. SteP is a meta-policy that wraps around a typical Policy class. SteP maintains a self.stack variable to store the set of active policies. We focus on the predict _ action() that takes as input the observation and returns an action and a reason. It begins by initializing the stack with a root policy. It then takes the policy at the top of the stack and invokes its predict _ action() function.The policy can perform one of three actions. First, if the action is an environment action, e.g. Zhou et al., 2023) that use a single policy, composition allows for more adaptability. 2. Scalability. Adding a new policy to SteP is easy. The user constructs the prompt for the policy and adds it to the library with a description. This policy becomes available as part of the action space for any other policy, without requiring any change to the code. 3. Modularity. Each policy tracks only the local context of the specific subproblem it is solving. Once it terminates, it hands back control to the previous policy in the stack without reasoning about the global context. This allows reusing policies in different context, e.g. the same fill _ form() can be used by book flights or make appointments. Compared to prior work (Zheng et al., 2024) that require demonstration for entire tasks, modularity requires demonstrations for subtasks thus being more sample efficient. 
experimental setup: 
 Environments. We evaluate across multiple distinct web environments listed below.• WebArena (Zhou et al., 2023) . A recent benchmark with complex web tasks across multiple domains like shopping, software development, content management. WebArena 5Environments. We evaluate across multiple distinct web environments listed below.• WebArena (Zhou et al., 2023) . A recent benchmark with complex web tasks across multiple domains like shopping, software development, content management. WebArena websites are highly realistic with tasks mirroring those that humans routinely perform on the internet. We evaluate across all 804 tasks in the benchmark. • MiniWoB++ (Liu et al., 2018; Shi et al., 2017) . Compared to WebArena, this is a simplified web environment covering interactions like form filling, search, choosing dates. We evaluate across all 45 tasks that don't rely on vision and average over 50 seeds per task. • AirlineCRM. We develop a new CRM simulator (Appendix C) modeled after customer service workflows on popular airline websites. Compared to MiniWoB++, this contains longer-horizon tasks. We evaluate across 5 tasks averaged over 20 scenarios per task. • Finally we test on live website environments and show results in Appendix D.Policies. We use a library of 14 policies for WebArena, each covering multiple intents. Constructing a policy is straightforward: we use a templated prompt with general instructions, action space definition, and place holders for policy specific instructions and examples. To design policies, we cluster intents that are functionally equivalent, e.g., searching over orders or listing products. See Appendix A.2.Baselines. We compare against various baselines including prior state-of-the-art on We-bArena (Zhou et al., 2023; Akter et al., 2023) which design a single web agent policy following a ReAct (Yao et al., 2022b) style chain-of-thought (CoT) prompting. On MiniWob++, we compare against recent fine-tuning (Furuta et al., 2024; Gur et al., 2022a; Humphreys et al., 2022) and in-context learning (Zheng et al., 2024; Sun et al., 2024; Kim et al., 2023) works.Additionally, we create baselines to study the following effects: (i) Single vs Decomposed prompt (Flat vs SteP). Flat is a single policy that concatenates instructions and examples from all the policies in the library into a single prompt. (ii) Varying context lengths of prompts (Flat-4k vs Flat-8k). Since the typical policy prompt is less than 4000 tokens, we create two baselines Flat-4K that caps the prompt size to 4000 tokens and Flat-8K that caps it to 8000. (iii) Effect of in-context examples (Zero-shot vs Few-shot). We study effect of adding observation action examples that help associate language instructions to webpage elements.We study (i) on all datasets, (ii) on WebArena, where context lengths are longer due to more complex webpages, and (iii) on MiniWob++, AirlineCRM where the stripped down webpages have ambiguous elements (e.g. missing aria-labels) and benefit from examples.For models, on WebArena we evaluate with gpt-4-turbo1 (OpenAI, 2023) since the tasks are complex, while for MiniWob++ and AirlineCRM we evaluate with either the instruction fine-tuned text-davinci-003 1 or gpt-3.5-turbo 1 (Ouyang et al., 2022) .Metrics. We define 3 metrics: Success Rate (suc↑), Task Progress (prog↑), and Number Actions (#act). suc↑ is either 0 or 1 depending on the task being completed successfully. #act is the number of actions taken. On airline CRM, we also compute prog↑ a number between 0 and 1 indicating progress towards completing the task.• On WebArena, SteP outperforms prior works (0.15 → 0.33) with improvements on every environment: Shopping (0.2 → 0.37), CMS (0.10 → 0.24), Reddit (0.11 → 0.59), Gitlab (0.14 → 0.32, Maps (0.15 → 0.30). See Sec. 5.1.2. • SteP achieves greater accuracy over Flat-8k (0.23 → 0.33) while using 2.3x smaller context lengths per episode. See Sec. 5.1.3, 5.1.4. • On MiniWob++, SteP Few-shot is competitive to prior works while using significantly less data. See Sec. 5.1.2. Values shown for 5 intents sampled from 5 quantiles, average values shown for all intents for each website. Each intent consists of 3-5 tasks. SteP caps each policy prompt to 4k tokens.• In-context examples help in addition to instructions by associating language instructions with corresponding web elements. Moreover, SteP Few-shot uses these examples more effectively by having them in dedicated policies. See Section 5.1.5. • We provide ablations on CoT reasoning and model scales in Appendix B.On WebArena, Fig. 3(a) , Table 1 show that SteP outperforms prior works (Zhou et al., 2023; Akter et al., 2023) that use a single GPT-4 policy on all environments. By having a library of only 14 policies (see Appendix A.2), SteP cov-ers at least 50 intents out of a total of 170 intents. Most significant gains come from Shopping, Reddit where the policies cover a larger percentage of intents. (Gur et al., 2022b) T5-XL 12K 0.56 WebGUM (HTML) (Furuta et al., 2024) Flan-T5-XL 347K 0.90 RCI (Kim et al., 2023) gpt-4 21 0.94 AdaPlanner (Sun et al., 2024) text-davinci-003 65 0.93 Synapse (Zheng et al., 2024) gpt-3.5-turbo 100 0.98 SteP (Ours) text-davinci-003 10 0.96We observed that in prior works (Akter et al., 2023; Zhou et al., 2023) , a common failure mode was an inability to navigate the website correctly to solve a complex, multi-step task.As a natural first solution, we add instructions and examples to the prompt to teach it how to solve such tasks. We created two baselines, Flat-4k and Flat-8k, containing such instructions and examples up to a context limit of 4000 and 8000 respectively.In On some intents, success rates even regress, e.g. on Shopping (0.2 → 0, 0.6 → 0.4), on Reddit (0.5 → 0.33, 0.6 → 0.4). This is because additional instructions create larger prompts, making it difficult for the model to pay attention.introduces a library of 14 policies, each with a small set of dedicated instructions and examples with prompt lengths under 4000 (see Appendix A.2). This results in smaller prompts that make fewer errors. Table 1 shows that SteP outperforms both Flat-4k (0.20 → 0.33) and Flat-8k (0.23 → 0.33). A key feature that helps SteP scale is that a single policy can cover multiple intents, e.g. search _ order() covers 6 intents consisting of 30 tasks. On MiniWob++, in Table 3 , we see a similar trend where SteP improves over Flat Few-shot (0.72 → 0.96). Fig. 4 shows comparisons across individual tasks. While performance is comparable on simpler tasks, as the task complexity increases SteP outperforms by greater margins. SteP breaks down complex tasks into smaller sub-tasks covered by policies in the library, e.g. book _ flight() is broken down into fill _ text(), choose _ date().A trade-off to SteP is that by introducing a library of policies, there is an overhead in passing control back and forth amongst policies. This leads to more number of calls to the model. However, since the prompt for each policy is significantly smaller, the total tokens ends up being smaller. For instance, when SteP solves a task it never has to see instructions and examples for policies not required in that task. Fig. 3 (b) shows a histogram of context lengths for SteP and Flat-8k across all successful WebArena trajectories. SteP is distributed around a smaller number of tokens, averaging 22.7K compared to 52K. Hence total cost and inference time for SteP is lower than Flat-8k.We observe that while instructions are often sufficient to solve many web tasks, examples can provide a significant performance boost, particularly when these elements are stripped down lacking meaningful aria labels. On MiniWob++ Table 3 , we see that few-shot examples provide performance gains for both Flat (0.60 → 0.72) and SteP (0.65 → 0.96). We see a similar trend in AirlineCRM in Fig. 5 for both Flat (0.24 → 0.76) and SteP (0.68 → 0.94).Examples help associate language instructions with webpage elements, particularly for simplified pages when these are ambiguous.  
limitations: 
 We present SteP that dynamically composes policies to perform a diverse set of web tasks.Our results show that SteP outperforms both prior works and single policy baselines while being more context-efficient. However, SteP has several important limitations:(1) Manually defined policies. For any new domain, the user needs to identify commonly occurring subtasks that are shared across multiple tasks. For each subtask, they then need to write a policy prompt that contains instructions on how to solve the subtask. Both identifying subtasks and writing policy prompts require domain knowledge. Once the policy prompts are written, however, SteP can automatically decide when to compose the appropriate policies from the library. The benefit of composition is that a small set of policies can cover a combinatorially large number of tasks. Automatically discovering useful policies from experience or demonstration data is an important direction of future work and would enable scaling to a large number of domains and websites automatically.(2) Communication overhead. By decomposing tasks into smaller policies, we also incur a communication overhead between policies that adds to inference times. One interesting solution would be to use smaller models for simpler policies and only escalate to larger models as needed.(3) Incomplete information. Finally, there are situations where a policy is unable to solve a subtask due to inadequate information being passed to it. Moreover, it fails to communicate what is missing to the parent policy, which can create an endless loop.An interesting future direction would be to explore how policies can share and update a common belief state to prevent such errors.Equipping LLMs with the ability to carry out web tasks opens up a variety of possibilities for societal benefits and change. These range from reducing cognitive burden on humans of doing repetitive tasks to enabling greater accessibility for elderly and individuals with disabilities. However, we acknowledge the ethical implications that come with the use of such technologies, and list some of them below:1. Safety and Reliability. LLMs automating web tasks raises concerns regarding misuse, including malicious automation aimed at spamming, phishing, or manipulating online systems. To mitigate these risks, it is crucial to implement stringent safeguards. These may include developing sophisticated detection algorithms to identify and block automated actions that exhibit patterns of misuse, ensuring that LLMs operate within ethical boundaries. Moreover, rigorous testing and validation protocols would ensure the technology's reliability and safety in open-world web environments. 2. Privacy and Data Security. LLMs interacting with web interfaces introduces potential risks such as unauthorized data access and privacy breaches. To mitigate these risks, it is crucial to place safeguards such as encrypting sensitive data, enforcing strict access controls, and adhering to strong privacy practices. Such transparent data handling policies are essential to maintain trust in these systems with end users. 3. Employment Impact. While automating web tasks with LLMs can enhance efficiency and accessibility, it is vital to consider the impact on employment. Care must be taken that the deployment of such technologies is to augment human capabilities rather than replace them, helping free them up for more creative and nuanced interactions. Addressing this requires holistic approaches, including policies to support workforce transition through re-skilling and up-skilling, and encouraging the development of new roles that leverage the unique strengths of human creativity.To promote reproducibility and transparency, we have taken several steps to ensure that SteP and our findings can be replicated and validated by the broader research community: 1. Open source code. We have linked the SteP implementation, including necessary code and documentation. We also include the prompts, the raw model predictions, and relevant noteboooks for greater reproducibility. 2. Experimental Details. Our paper includes detailed descriptions of the experiments along with additional details on hyperparamaters, prompts, environments included in the Appendix. 3. Benchmarks and Models. We clearly specify the web benchmarks and models used in our evaluations, including appropriate references and links to those. Since OpenAI models continue to evolve, we have included the raw model predictions so that the results are reproducible. 4. Results and Ablations. We present detailed results, including performance metrics, ablation studies, and comparisons with state-of-the-art methods. Our aim is to provide a clear and honest assessment of SteP's capabilities and limitations. 5. Limitations. Acknowledging the importance of transparency in scientific communication, we discuss the limitations of our approach and discuss directions for future research.We use OpenAI API for model calls, gpt-4-turbo-preview as the default model for We-bArena and text-davinci-003 for other environments. We use a a temperature of 0.3 and set the number of calls to 3. Below are the exact API calls, 1 if (model _ type == "gpt-4-turbo-preview"): Listing 1: Hyper-parameters for different modelsWe provide below the prompt template for WebArena that contains the list of policies, the base actions, examples for how to use the policies, and the base instruction template.1 policies = """ 2 Subroutine Actions:3 `find _ commits [query]`: Given you are in a project page, this Gitlab 4 subroutine searches for commits made to the project and retrieves 5 information about a commit. This function returns the answer to the query. `find _ customer _ review [query]`: This CMS subroutine finds customer reviews for a particular product using the query to specify the kind of review.`find _ order [query]`: This CMS subroutine finds an order corresponding to a particular customer or order number.`search _ customer [query]`: This CMS subroutine finds a customer given some details about them such as their phone number.` `click [id]`: This action clicks on an element with a specific id on the webpage.`type [id] [content] [press _ enter _ after=0|1]`: Use this to type the content into the field with id. By default, the "Enter" key is pressed after typing unless press _ enter _ after is set to 0.`hover [id]`: Hover over an element with id.`press [key _ comb]`: Simulates the pressing of a key combination on the keyboard (e.g., Ctrl+v).`scroll [direction=down|up]`: Scroll the page up or down.`note [content]`: Use this to make a personal note of some content you would like to remember. This shows up in your history of previous actions so you can refer to it.`go _ back`: Navigate to the previously viewed page.general _ instruction _ template = """ You are an AI assistant performing tasks on a web browser.To solve these tasks, you will issue specific actions.The actions you can perform fall into several categories: {base _ actions} {policies} {example _ actions} You will be provided with the following, OBJECTIVE:The goal you need to achieve.A simplified text description of the current browser content, without formatting elements.The current webpage URL PREVIOUS ACTIONS:A list of your past actions with an optional response, e.g. 1 = find _ commits [query] You need to generate a response in the following format.Please issue only a single action at a time.Your reason for selecting the action below ACTION:Your action """ Tab Management Actions:`new _ tab`: Open a new, empty browser tab. `tab _ focus [tab _ index]`: Switch the browser's focus to a specific tab using its index.`close _ tab`: Close the currently active tab.`goto [url]`: Navigate to a specific URL. `go _ back`: Navigate to the previously viewed page. `go _ forward`: Navigate to the next page (if a previous 'go _ back' action was performed).`stop [answer]`: Issue this action when you believe the task is complete. If the objective is to find a text-based answer, provide the answer in the bracket. """We provide the prompts for all 14 commonly used policies below and for the reset, we refer the reader to the code base.1 find _ commits = { 2 "instruction": """ 3 {general _ instruction _ template} 4 5 Please follow these general instructions: 6 * To find a list of all commits, you must navigate to the commits section of the repository 7 * Look at the first and last date in your observation to know if the desired date is in the range 8 * If it's in the range but not visible, that means no commits were made on that date 9 * If the date is outside of the range, you need to scroll up/down to get to the desired date range. Scrolling down takes you to a date earlier in time (e.g . Feb 2023 is earlier in time than Mar 2023) 10 * To count commits from a specific author, count the number of times their avatar (e.g. img "<author> avatar") appears in the observation. 11 """, On MiniWob++, we identified a set of 7 policies that solve commonly recurring subtasks, e.g. filling different text boxes, choosing dates from a date picker, processing emails, etc. For each of these policies, we collect a few in-context examples to cover the subtask. At test time, SteP composes these policies to solve a wide number of tasks. Table 4 below shows the various policies and tasks they cover.Table 5 shows a task-wise performance breakup on the MiniWoB++ benchmark for various models. We choose a set of 45 tasks that don't require visual reasoning. We run 50 seeds per task and report the average success rates and number of actions. We divide tasks into simple and complex, where complex tasks might require multiple policies to solve the task. While on simple tasks almost all methods are equivalent, on complex tasks SteP Few-shot outperforms Flat Few-shot on every task. CHOOSE _ DATE() policies to pick a date from the datepicker. Each of these policies are more robust than Flat at solving the subproblems, resulting in a higher overall success rate. While we initially did not have chain-of-thought reasoning 2 , we found that adding it in helped SteP Few-shot rationalize a particular action well and boost performance. We wanted to understand which tasks in particular were helped by chain-of-thought reasoning, and what the trade-offs were.We wanted to test the following hypothesis for SteP Few-shot with and without chain-ofthought:1. Hypothesis 1: Chain-of-thought reasoning helps across all tasks. Even though chain-of-thought reasoning makes prompts slightly longer, the added step of rationalizing actions should always boost performance.2. Hypothesis 2: Chain-of-thought reasoning particularly helps in multi-step tasks. Multi-step tasks often require breaking down a problem into a set of steps and executing each step. While demonstrations certainly show how to break down task, adding chain-of-thought better rationalizes this breakdown and helps generalize to new tasks not covered in the demonstrations. Table 5 : Task-wise performance breakup on MiniWoB++ benchmark on a set of 45 tasks.We compared SteP Few-shot with two versions -having chain of thought, and not having. Fig. 9 shows a plot of success rate for each of the 3 clusters of tasks -single, composite, multi.Hypothesis 1: Chain-of-thought reasoning helps across all tasks.. We find this to be true since for all tasks, chain-of-thought performs either equally or better. This confirms that the extra tokens consumed by the reasoning does not hurt performance and in fact helps significantly in some cases.Hypothesis 2: Chain-of-thought reasoning particularly helps in multi-step tasks. We find this to be true as well. Looking at the multi-step tasks, chain-of-thought has the largest performance gains compared to any other cluster. The performance gains are the largest in book-flight and search-engine where the horizon length is the largest. In comparison, for single and composite the performance gains vary, being higher for certain tasks like choose-date and find-word and zero for others like click tasks.The prompts used are the same as Flat Zero-shot and SteP Zero-shot. Overall, we find the performance to be lower than gpt-{3,3.5,4}. The drop in performance could be due to a number of reasons such as the model size or the training data on which the models are trained. However, we find that SteP still outperforms Flat for many tasks, which we discuss below. Figure . 11 shows a comparison of Flat and SteP on a subset of MiniWob++ tasks. We see that Flat matches SteP for simple CLICK tasks, it fails for slightly more complex tasks that require multiple actions, e.g. multiple CLICK or TYPE.We cluster common failure modes for both Flat, SteP below:1. Repeating the same action in multi-step tasks. For tasks that require a sequence of actions, Flat tends to get stuck in the first step repeating the same action over and over. It fails to interpret the previous actions or changes to the browser content to update the predicted actions.For example, in the click-button-sequence task, for the following input CONTEXT: Click button ONE, then click button TWO. BROWSER CONTENT: <body id=1 val= /> <div id=2 val=wrap /> <div id=3 val=area /> <button id=4 val=ONE /> <button id=5 val=TWO /> PREVIOUS ACTIONS: CLICK 4 the model produces the following output REASON: To achieve the goal of clicking button ONE and then button TWO, we need to first click button ONE to activate it, and then click button TWO to activate it. ACTION: CLICK 4 when the correct output would have been CLICK 5, given that the previous actions show that CLICK 4 has already been executed. 
 
 Appendix:  


Decision: The paper presents SteP, a novel approach for web task automation that dynamically composes policies from a policy library into a policy stack. The reviewers agree that the proposed method demonstrates significant improvements over existing methods on benchmarks such as WebArena, MiniWoB++, and a CRM task. The intuitive and effective stack formulation, combined with strong performance results, highlights the method's potential. However, concerns were raised about the task-specific nature of the policy library and the manual effort required to construct it. The authors' rebuttal clarified the scalability and flexibility of the policy design and distinguished their approach from similar works. Overall, the paper is well-written and contributes valuable insights to the field, meriting acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes SteP, which dynamically composes policies from a policy library into a policy stack, to solve web tasks. The proposed model shows strong performance on WebArena, MiniWoB, and a CRM task.
strength_weakness: strength_weakness: My main concerns are as follows:
- The task-specific nature of the proposed method, as it relies on a manually crafted choice of policy library, as well as demonstrations for the policies. 
- A lack of novelty with respect to existing decompositional methods such as [1,2], as both methods explored the utilization of dynamic decomposition using a predefined skill library.

[1] Decomposed Prompting: A Modular Approach for Solving Complex Tasks. Khot et al., ICLR 2023
[2] ADaPT: As-Needed Decomposition and Planning with Language Models. Prasad et al., NAACL 2024
questions: questions: - What kind of prompts does the meta-policy need for each task, in order to manage the policy stack?
- Can the authors share more details about how the policy library was chosen on each task?
paper_summary: paper_summary: This paper proposed a method called Step, which aims to solve the complex web navigation tasks. Instead of relying on one global policy/prompt to control how agent interact with the environment, Step contains a set of policies that are tailored to solve different sub-tasks, and the Step agent dynamically composes/invokes different policies to solve the task at hand. In particular, each policy can choose to execute an action in the environment, call other policies or call itself, which offers great flexibility in agent’s exploration. The Step method is evaluated on popular web agent benchmarks including WebArena, MiniWob++ and CRM and a few real-world websites. The results show that it outperforms previous methods and baselines with a flat policy.
strength_weakness: strength_weakness: My biggest concern is about the construction of policies. As the authors admitted in the limitations, the Step method requires user to manually write detailed prompts for every policy. This implies that the method cannot be easily scaled to more websites/tasks. Moreover, from the examples shown in Appendix B.2 we can see that these policies/prompts are very tailored towards specific websites. For example, search order policy informs the agent to first go to My Account and then go to My orders. I’m not sure if the prompt can transfer well out of the box to different websites. For example, what if a website do not have the My Account button but the user actually needs to click their profile picture to get to the account details? 
In section 5.1.2., the authors mention that the manully written policies coveres  at least 50 intent templates from WebArena. Have you checked if the gains on WebArena are mainly from these covered intents? 

Missing references:
The idea of constructing specific policies/prompts at different states to control the web agent has been explored in Ma et al. 23. It would be good to compare your method with their work as well. 

Ma, Kaixin, Hongming Zhang, Hongwei Wang, Xiaoman Pan and Dong Yu. “LASER: LLM Agent with State-Space Exploration for Web Navigation.” ArXiv abs/2309.08172 (2023): n. pag.
questions: questions: Please see the limitation section
paper_summary: paper_summary: They advance the research on the line of performing web tasks with LLMs by decomposing them into different policies (instead of using a single prompt). In particular, they develop an approach based on a Markov Decision Process where the state is compounded by a stack of policies, which represent the sequence of policy calls. Evaluation of their methodology across key benchmarks such as WebArena, MiniWoB++, and a CRM simulator reveals promising results, demonstrating efficiency in data consumption.
strength_weakness: strength_weakness: - The absence of a Conclusion section, which consolidates the conclusions drawn from preceding sections, is noted. However, aside from this observation, I do not identify any other issues with this paper.

Comments: title: Response to authors
comment: Thanks so much for your feedback. We'll be sure to extend the 5 policy prompts in Appendix B to include the remaining prompts from the full policy library.
Thanks for your response. I'm sure that incorporating a conclusion section will enhance the final version of the paper.
I appreciate the authors' response. I understand the distinction from DecomP and ADaPT better now. As I understand it, it offers a much more flexible control of policy composition than those methods.

That said, in order to be complete, I think paper needs to include the full policy library for all tasks, since these are all handcrafted for each task.

Overall, I think the method is interesting and I have increased my score accordingly.
Thanks for the reply.