AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents
Authors: Yao Fu
Dong-Ki Kim
Jaekyeom Kim
Sungryull Sohn
Lajanugen Logeswaran
Kyunghoon Bae
Honglak Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: large language model agents
sequential decision-making
Primary area: natural_language_processing
TL;DR: AutoGuide extracts a comprehensive set of context-aware guidelines to improve the sequential decision-making ability of large language model agents.

Abstract: Recent advances in large language models (LLMs) have empowered AI agents capable of performing various sequential decision-making tasks. However, effectively guiding LLMs to perform well in unfamiliar domains like web navigation, where they lack sufficient knowledge, has proven to be difficult with the demonstration-based in-context learning paradigm. In this paper, we introduce a novel framework, called AutoGuide, which addresses this limitation by automatically generating context-aware guidelines from offline experiences. Importantly, each context-aware guideline is expressed in concise natural language and follows a conditional structure, clearly describing the context where it is applicable. As a result, our guidelines facilitate the provision of relevant knowledge for the agent's current decision-making process, overcoming the limitations of the conventional demonstration-based learning paradigm. Our evaluation demonstrates that AutoGuide significantly outperforms competitive baselines in complex benchmark domains, including real-world web navigation.

AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents
Yao Fu 1+
Dong-Ki Kim 2+
Jaekyeom Kim
Sungryull Sohn 2 Honglak Lee 1,2
Lajanugen Logeswaran"
Kyunghoon Bae 2 LG AI Research
'University of Michigan
Abstract
Recent advances in large language models (LLMs) have empowered AI agents capable of performing various sequential decision-making tasks. However, ef- fectively guiding LLMs to perform well in unfamiliar domains like web naviga- tion, where they lack sufficient knowledge, has proven to be difficult with the demonstration-basee in-context learning paradigm. In this paper. we introduce a novel framework, called AUTOGUIDE, which addresses this limitation by automat- ically generating context-aware guidelines from offline experiences. Importantly, each context-aware guideline is expressed in concise natural language and follows a conditional structure, clearly describing the context where it is applicable. As a result, our guidelines facilitate the provision of relevant knowledge for the agent's current mecision-making process, overcoming the limitations of the conventional demonstration bbaed learning paradigm. Our evaluation demonstrates that AU- TOGUIDE significantly outperforms competitive baselines in complex benchmark domains, including real-world web navigation.
Introduction
Recent advances in large language models (LLMs) have empowered AI agents to address various sequential decision- decision-making making tasks and applications [1, 2]. The foundation these successes involves the planning and reasoning capabilities of pre-traing trainadddd LLMs, enabling agents to execute effective policies [3, 4]. The predominant approach leveraging these (typically closed source) models for sequential decision making tasks is to provide demonstrations in the form of in-context examples. However, direct application of this learning paradigm can be limited, especially in target domains where the LLM has insufficient prior knowledge such as in web navigation, where LLM agents generally achieve low success rates due to diverse and dynamic contents [5-8]. Providing all available experiences demonstrations prompt sensitivity, and difficulty with complex reasoning [9-12].
agent can further be unsuccessful due to context length limitations,
On the other hand, LLMs excel in interpreting concise instructions provided as natural language, an ability that also reinforced in the instruction-tuning phase of LLMs. Inpired by this, we explore data-driven strategies that leverage offline experiences to extract actionable knowledge to help guide LLM agents. offline experiences implicitly convey valuable knowledge about desirable and undesirable policies in domains, they promise to serve useful resource for improving an LIM agent's decision-makinn in situations where the pre-trained L.MM lacks understanding. Despite this potential benefit, critical challenge lies in effectively extracting the implicit information embedded in offline data.
Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

ReAct
AutoGuide With Context- Guidelines
Task:
Task:
discussion post "effectiveness
Create discussion about "effectiveness
online learning" in a relevant subreddit.
online learning" 'in relevant subreddit. In: Context Example!
In: Context Example!
Below are example trajectories.... ...
example trajectories...
X Action:
Current Trajectory:
Current Trajectory:
Observation 0:
[32) Maader/Atnntaaddaark
RootWeilrre 'Postmill' focused:
Click (66)
Identification
Context
[3] RoctWebbrrs 'Postmill' focused: True
Observation 0:
[32] HexderAsNentannMMakk
Action:
[33] link 'Homee'
likk 'Wkl'
[33] link 'Homee
Click [65] link
link 'Forums'
link 'Forums'
[56] link "Wik"
[86) link "Wilk"
'Forums'
Offline Experiences
Retrieval of Relevant Guidelines
Context:
On the Reddit main page.
Context-Alwaee Guidelines
When on the Reddit main page, you want to novigalte to specific forum, you should click on "link 'Forums" located above "link 'Wik" "
Contert--Aware Guidelines
Contaxti
On the Redelit main page.
Context-Aware Guidelline)
When on the Reddit main page, you want to navigate to specific forum, you should click on "link 'Farums" located right ahove "link "Wili".
Knowledge Extraction
Figure 1: AUTOGUIDE aims to extract the implicit knowledge embedded in offline experiences and help the secision-making process of an LLM agent. Specifically, our method generates a comprchensive set of context-aware guidelines from offline data and explicitly identifies when each guideline is applicable by generating its corresponding context. Our context-awar guidelines enable providing pertinent guidelines test time by identifying the context of the current trajectory, leading to correct precision-maaing compared to baselines without context-aware guidelines.
To address the challenge of extracting knowledge from offline data, we propose novel framework, called AUTOGUIDE. Specifically, AUTOGUIDE automatically derives a comprehensive set of context- aware guidelines from offline experiences. Our method applies these context-conditional guidelines to enhance the performance of an LLM agent by retrieving guidelines relevant to the agent's current state and incorporating them into the prompt during testing (see Figure 1). Notably, generate context-aw -warr guidelines in concise natural language statements, effectively compressing knowledge in offline data. Moreover, context-aware guidelines clearly describe the contexts where they are applicable, so AUTOGUIDE enables an LLM agent to select pertinent guidelines for its current Decision-making process. As result, AUTOGUIDE achieves the highest success rates compared to competitive baselines in complex sequential decision-making benchmark environments.
Our contribution. In summary, we present the following main contributions in this paper:
Principled method based context-aware guidelines (Section 3): We develop two modules to automatically generate context-aware guidelines from offline experiences: the context identification module for identifying the context of a given trajectory, and the guideline extraction module for ex- tracting desired guideline corresponding to that context. The outcome is a set of domain knowledge in concise natural language that enhances decision-making by providing pertinent information.
Comprehensive evaluation of AUTOGUIDE (Section 4.2): We show AUTOGUIDE's capability in extracting helpful context-aware guidelines in various interactive benchmark domains, includ- ing navigating real-world web domains (c.g., GitHub). Our results highlight the effectiveness of AUTOGUIDE, which significantly outperforms baselines without context-aware guidelines.
Analyses with important perspectives (Section 4.3): We study various aspects of AUTOGUIDE, such as the significance of determining the applicability of each guideline based on generated contexts. We also investigate the generalization ability of context-aware guidelines and demonstrate that our tasks.
guidelines enhance the performance across
2 Related Work
I.I.-based agents. Language models have recently been shown to possess strong priors for sequential decision- decision-making tasks, which has given rise to LLM-powered agents [1, 2, 13, 14]. Agents

---

Pair of Contrasting Trajectories
Task: Create discussion post about "effectiveness of online learning" in a relevant subreddit.
Context Identification Module
Context-Aware Guideline
Observation 0:
v Desired Action O:
When Reddit page. you to navigate to specific forum, you should click on "link "Forums" located right above "link "Wiki"!
[1] RootWebArea 'Postmill"
Click [65] link Forums
[32] HeaderAaNonlanddMark
[33] link 'Home'
[65] link 'Forums'
Undesired Action Click [66] link "Wiki'
Guideline Extraction
[66] link
Module
Figure 2: Context-aware guideline generation process based on a pair of contrastive trajectories T and rΣ. In this example, the two trajectories start deviating from each other at t= 0. The context identification module generates a description of the context at t = 0 given T.o. and the guideline extraction module generates the corresponding guideline for that context.
need to possess various skills to be effective in practice including planning [3, 15, 16], reasoning [4, 17], tool manipulation [18-21], code generation [22, 16], among others. In this work, we focus on building effective agents for web [23, 8] and embodied [24] environments.
Self-reflecion from past experiences. An important capability for agents to succeed is the ability to learn from past experiences and update their behavior based on feedback. Self-feedback [25-27] has emerged as effective technique where model inspects its own incorrect predictions, reflects on to identify what went wrong and attempts to improve its prediction. While self-feedback provides intra-task (i.e., per-episode) knowledge within task based on immediate feedback, approach offers an orthogonal and complementary aspect of inter-task knowledge (over multiple tasks) by considering multiple train tasks in offline data. AutoGuide enhances learning efficiency and credit assignment by utilizing detailed feedback from multiple tasks. However, these self- self-feedback approaches complementary AUTOGUIDE and can be used in conjunction with our approach, as shown in our experiments (see Section 4.2).
Leveraging natural language guidance. Natural Language can be a rich source of information for agents to learn to act efficiently. Prior work has explored the notion of learning from human- written text manuals, which describe details about the environment [28-30]. Recent work has explored automatically generating such guidance in the form of chain-of-thought reasoning [4, 23). which emulates thought process rationale for agent's 's predictions. In contrast approaches which generate such guidance dynamically on the fly by imitating example guidance demonstrations provided by human, our approach carefully compares trajectories in offline data to generate appropriate guidance and uses these guidelines for predicting better actions. ExpeL. [31] proposed related approach derive guidelines. In contrast to ExpeL, where all guidelines an agent as a prompt, our guideline selection process is contextual, where guidelines relevant to the agent's ss current state are retrieved and used for prediction. We show that this substantially improves over ExpeL's non-contextual guideline-bast approach.
provided to
3 AUTOGUIDE: Principled Method Based on Context-Aware Guidelines
Our work is motivated by the increasing availability of offline experiences that agents or humans naturally accumulate through their interactions with the environment. AUTOGUIDE aims to leverage this offline data to improve the decision- making an LLM agent by generating helpful context-aware guidelines. This section details how AUTOGUIDE automatically constructs these guidelines and applies them to guide action generation at test time.
3.1 Problem Statement
Formally, AUTOGUIDE is given offline data DD Durain (..,,,, that consist of N trajectories from training tasks. Each trajectory (zo, ag, ro...., FT) is a sequence of observations, actions, and rewards following the partially observable Markov decision process [32]. The return of a trajectory is defined as the sum of rewards obtained throughout the trajectory: R(T)= - Tt. The objective of AUTOGUIDE to distill knowledge from offline experiences into a useful natural language format, such that the extracted information helps to maximize the expected return E, [R(T)] during test time.
3

---

Algorithm Extracting context-aware guide- lines from offline data
Algorithm 2 Applying context-aware guidelines at test time
Input: Offline data Drrain context identification module Meontext, guideline extraction module
Context-aware guideline dictionary G, context identification module Moontext: guideline selection module Manlect, LLM agent policy
Input:
Mggideline
Initialize context-aware guideline dictionary 9 for Each pair T - Pirain do
Initialize test trajectory T = {2a}
for Each timestep do
# Identify the context from trajectory
Identify the current context from a trajectory CONTEXT + Mcontex:(()
Find the deviating timestep f from 74 and r
CONTEXT MMonntex (7:1
# If the current context matches any existing ones, perform top-k guideline selection if CONTEXT 9 then
# Check the current context matches any existing contexts
if CONTEXT & 9 then [[CONTEXT] ()
GUIDELINES f- Manlect (CONTEXT, G.
else
end
GUIDELINES + Ø
# Generate the context-aware guideline
end if
GUIDELINE<- Magidelinn (T (T_,7 T .ONTEXT)
Action selection based on guidelines
G[CONTEXT]s (CCONTExTs-C(CCONTEx]] U (GUIDELINE) end for
Gr ~ ग ((, CONTEXT, GUIDELINES)
Execute action a and observe d++1 Update trajectory T < U (CONTEXT, as, Fe+1] end for
Return Context-awas guideline dictionary
3.2 Extraction of Context-Aware Guidelines
AUTOGUIDE generates a set of context-aware guidelines by utilizing pairs of contrastive trajectories from offline data. Each context-aware guideline is expressed in concise natural language and follows a conditional structure, clearly describing the context in which the guideline is applicable. Intuitively, contrasting a pair of trajectories with different returns provides important information about when and which actions are effective or ineffective in maximizing expected returns. Building on this insight, we develop two modules for automatically extracting context-aware guidelines (see Figure 2):
Context identification module. This module is responsible for abstracting the given partial trajectory into its context, concise natural language description of the agent's state. More specifically, for a timestep / and the corresponding trajectory T := (Zo, Gos Co,..., Tz), we prompt LLMs to clearly describe the agent's status:
CONTEXT +- MMontext (T
(I)
Our prompt templates for the context identification module are shown in Appendix C.I.
Guideline extraction module. This module aims to generate a desired guideline corresponding to specific context. Let T and Tí represent contrasting pair of trajectories for the same task in offline data Dirain, where R(T1) > R(r²). ). We want to contrast the pair of trajectories to find desired behaviors at an important timestep. To do this, we compare these two trajectories to find the deviation timestep t at which they begin diverge due to different actions. Then we apply the context identification module to summarize the context for the shared part of the trajectory T. Eventually, extract useful natural language guideline by examining the paired contrastive trajectories -. and T with respect to the context:
GUIDELINE CUIDELINE-Mqquielltt (Tİ, -_, CONTEXT).
MMguideliee
where we refer to Appendix C.2 for our prompt template. As an example, the paired trajectories in Figure deviate from timestep = 0, for which the context is summarized as On the Reddit main page. This module then generates the following context-aware guideline: When the Reddit main page, you want to navigate to a specific forum, you should click on "link 'Forums" located right above "link "Wiki""
Construction of context-aware guidelines. We collect context-awan guidelines G by iterating through available pairs in the paired offline data and organize the guidelines in a dictionary format, using the context as the key and the corresponding guidelines as the value (see Algorithm 1). In particular, we observe that the context identification module occasionally produces contexts that describe the same situation but are expressed slightly differently. To minimize redundancy, we employ LLM to determine if the current context corresponds to any previously identified context.

---

ALFWorld:
Embodied Household Tasks
WebShop: Web Shopping Tasks
Task: a pan on diningtable.
Task: am looking for white curtains living room, price lower than dollars.
Observation:
Observation: [80735RVMRY]
The cabinet is closed.
Action: Open cabinet
MIUCO White Sheer Curtains
Observation:
Action: Clioon/ClickBBoo B0735R9RY
open
Observation:
The cabinet is open.
sice [37x63 inch]]37x8d inch)
nothing.
color [belge]lburgundyy. -..
WebArena:
Realistic Web Navigation Tasks
Real-World Websites: Multi-Modal Web Tasks
Task; Post review of my recent reading "Gone with the wind" in r/books with comment "it's a book with history".
Task: Find me flights from Rio de Janeiro to Dubai on Aug 29, 2024, show me buying options for the earllest arrival.
Observation: [4364] 'baltinore'
Multi-modal observation:
[12] [OUTTON] [1]
[4365] link 'books"
Fights ts
[13] [DV] [Econamy]
Action: (lick(4365)
[INPUT] [Where from?]
Action: Type [14] |Nio de Janeiro]
Observation:
[4862] heading /1/books'
Multiimodda observation:
[16] [INPUT] [Where to?]
[5342] link "ubmissions'
[17] [NPUT] [Departune) ...
Figure 3: Sequential decision decision-making benchmark domains considered in work: ALFWorld [24],
WebShop [33], WebArena [8], and multi-modal real-world websites. Graphic credit: [34, 33, 8, 35].
If a match is found, we reuse the existing context; otherwise, we introduce a new context into our dictionary G. The specific prompt template for this context- maathhtn procedure in Appendix C.3.
3.3 Applying Context-Aware Guidelines at Test Time
After extracting set of context-aware guidelines from offline experiences. our method employs these guidelines to enhance the decision- -aaiing of an LLM agent during testing. At each timestep, AUTOGUIDE identifies the CONTEXT of the current test trajectory up to timestep / (which represents the agent's interactions up to the current time-step) using our context identification module Mcontext- Our guideline selection module Muclect then selects relevant guidelines for CONTEXT from G. More specifically, the module applies CONTEXT as the key to fetch set of possible guidelines g G(CoNTEXT). If there are more than guidelines in G[CONTEXT]. Maelect prompts an LLM to choose top-k guidelines for the specific
RELEVANT GUIDELINES <- Mcslect(COOTEEE T: A)).
where Appendix C.4 details the prompt template for this selection procedure. Subsequently, AU- TOGUIDE incorporates both the context and relevant guidelines into the agent's s action generation prompt. Therefore, the agent selects an action by considering the provided context and guidelines (see Figure for an example). This process iterates until the end of the test trajectory (see Algorithm 2).
Key benefits of AUTOGUIDE. First, the extraction of context- aware guidelines in AUTOGUIDE offers the inherent benefit of providing relevant guidelines for the context of interest. This capability important since neglecting the specific context in which guideline applies can confuse the agent's 's decision-making process. The second key benefit is the generation of concise natural language guidelines, which can be seamlessly incorporated into any prompt-based L.MM agent. Lastly, AUTOGUIDE, generates guidelines at the individual context level rather than at the trajectory level. Given that single incorrect action can lead complete failure, is essential to provide detailed assistance in each action selection process. With these advantages, we demonstrate in the next section that our approach significantly enhances the performance of LLM agents.
4 Evaluation
This section demonstrates the efficacy of AUTOGUIDE by conducting experiments on diverse suite of sequential decision-making benehmark domains. We also perform important analyses about AUTOGUIDE, such the ablation study of different AUTOGUIDE components, comparison to
5

---

in-context learning, and generalization to out-of-domain tasks. We refer to Appendix B for additional experimental details.
4.1 Evaluation Setup
4.1.1
Sequential Decision-Making Benchmark Domains
We consider the following interactive sequential decision-making benchmarks to study various aspects of AUTOGUIDE (see Figure 3):
ALFWorld [24]: In this embodied benchmark, an LLM agent interacts with an environment to carry out household tasks, such placing pan the dining table. Observations and actions are expressed natural language statements, and the agent must navigate through the space and manipulate objects to successfully complete the tasks.
WebShop [33]: This interactive web environment simulates the task of online shopping an e-commerce website. The agent's s goal is to understand a text instruction and buy a product that meets specified criteria. This involves querying the website's 's search engine, understanding the descriptions and details each item, and selecting necessary options.
WebArena [8]: This web-based benchmark introduces realistic environments by replicating the functionality and data found in popular web domains (e.g.. Gitlab, Reddit, Wikipedia). Compared to WebShop, WebArena presents more challenges and difficulties for an LLM agent due to large observation and action space, along with tasks that involve longer planning horizons. We focus oe the Reddit domain for the main WebArena experiments.
Finally, we consider evaluating AUTOGUIDE on a variety of real-world website tasks. These span from a collaborative software development platform (e.g.. GitHub) to a flight search engine (e.g., Google Flights) and an online education platform (e-g.. Coursera). Please refer to Appendix B.4 for example tasks. In particular, in comparison to WebShop and WebArena, we design our tasks to be multi-modal such that the agent must consider both visual (e.g., images) and textual information (e.g., HTML) to complete these tasks.
Real-world multi-modal websites:
4.1.2 Baselines
We compare AUTOGUIDE against the following baseline approaches to study the effect of context- aware guidelines (refer to Appendix for more details):
ReAct [23]: This LLM-based planning method integrates reasoning and acting to address sequential decision-making tasks. However, does not leverage offline experiences and thus suffers from the limited understanding of pre-trained LLMs in downstream domains.
ExpeL [31]: This method also extracts natural language knowledge from offline data. However, it fails to consider the applicability of guidelines and does not generate context-aware guidelines. Instead, it provides all guidelines to an LLM agent without filtering out irrelevant ones based on the current context. ExpeL has two contributions, the guideline generation and in in-context example selection module. Because the latter is orthogonal to our analysis and can be seamlessly combined with our method, we consider Expel. with guidelines in our experiments.
Reflexion [27]: This approach converts environmental feedback into text statements assist an LLM agent (e.g., ReAct) in the next trial. The baseline generates valuable feedback about solving specific test task. We demonstrate how context- guidelines derived by AUTOGUIdE can be combined with the feedback.
4.1.3 Implementation
We collect offline experiences either by running ReAct and Reflexion, or incorporating human demonstrations. We use ReAct with GPT-3.5-turbo as our base LLM agent for WebShop and ALFWorld and GPT-4-turbo for WebArena. For each benchmark, we apply the same GPT model for action generation, context identification, and guideline selection. We extract context-aware guidelines from offline data with GPT-4-turbo and evaluate their effectiveness by applying them to the test set with non-overlapping tasks. We refer to Appendix B for more details.

---

ReAct Trajectory
ExpeL Tralectory
AutoGuide Trajectory
Observation: You the cabinet 4. The
Observation: The cabinet is open. you see candie 1, and tolletpaper Fixed guidelines!
Observation: On the garbagecan 1, you see nothing.
cabinet open, you see a seapbottie 1,
and tolletpaper 2.
Context: Looking an object, but it's not here.
Action: Think:
scapbar (1). Next,
item fails or does not progress the lask, reassess the situation consider alternative actions or
atterppt interact
Context aware Guidelines
need to take it.
When looking an object and you found the desired object in the expected locations, you should expand your search less abvicus places that have not been checked yet. example, you found a soaphar in the sinkhasin, bathtubbasin, garbagecan, shelves, proceed to check the tollet.
Observation:
Action: Take soapbar I from cabinet 4)
locations before repeating the same action. to specity the ite''s number and location when interacting with
When item found,
Observation: Nothing happens.
Action: Take soapbotile 1 from cabinet 4.
Observation: You pick up the scapbottle 1 from the cabinet.
Action: Take soapbar 1 from cabinet S)
Observation: Nothing happers.
Action: So to toilet
Fised guidelines:
Observation:
toilet 1, you see soapbar and a soapbettlc 2.
Reward: Falled
X
When an item is found, ensure to specify
an attempt interact with Item
Contexti Looking an object and found
desired object here.
Action: Think: tried to take the soapbar (1)
Context-awarc Guidelines
but failed, I specify the wrong item number,
When looking for an object and found the desired objett hern, you should only take the abject it visible and specify the correct location and number when taking For example, if you see a credit card on the dining table, should "take creditcard from diningtable 1'.
should redo by take soapbottle from cabinet 4.
Task: soapbar and put countertop.
Observation: OK.
Fixed guidellnas:
If attempt to interact with item- item is found, ensure to specify...
Action: Think: Now! find soapbar (2). This what looking for. Basad on the guideline, can take it.
Action:
Take spapbottle from cabinet 4)
Observation: Nothing happers.
Observation: DK.
X
Reward: Success
Figure 4: Trajectories of ReAct, ExpeL, and AUTOGUIDE from the same test task. ReAct (Left) chose the wrong item, consequently failing the task in the end. ExpeL (middle) was confused by guidelines that were irrelevant current context, leading to incorrect reasoning and actions. AUTOGUIDE (right) selects relevant guidelines to the agent's context, enabling the agent to accomplish the task.
Offline data?
Context aware?
ALFWorld [24]
WebShop [33]
WebArena [8]
Algorithm
Success Rale (SR)1
Rewardf
SR
SRT
ReAct [23]]
x
X
54.5%
66.4
30%
8.0%
Expel. [31] AUTOGUIDE
/
59.0%
60.9
35%
21.8%
/
79.1%
73.4
46 %
47.1 %
ReAct [23]
Reflexion [27]
x
67.2%
77.1
51%
N/A
Expel. [31] + Reflexion [27]
/
71.6%
71.7
42%
N/A
AUTOGUIDE + Reflexion [27]
/
88.1% %
81.4
57 %
N/A
Table 1:
Test reward and success rate on ALFWorld, WebShop, and WebArena. The base agent model for ALFWorld and WebShop is GPT-3.5-turbe and for WebArena is GPT-4-turbo. Reflexion is done by GPT-4-turbo for at most trials. In our experiments, due to token limit of GPT, we did not experiment with Reflexion on WebArena tasks.
4.2 Main Results
Q1. How effective is AUTOGUIDE compared baselines without context-aware guidelines?
To answer this question, we compare methods on ALFWorld, WebShop, and WebArena benchmarks. The performance on the test datasets is presented in Table 1. There are three notable observations:
1. Effectiveness of context-aware guidelines. Our approach surpasses baseline performance in both ALFWorld and WebShop, achieving the highest test rewards and success rates in Table 1. These results highlight the effectiveness of employing context-awar guidelines in language- based decision- making domains. To further examine the action selection differences among ReAct, ExpeL, and our method, we present their trajectories in Figure 4. We observed that ReAct makes common mistakes such as trying to take soapbar that is not visible, taking a soapbottle instead of soapbar due to their similar names. Both Expel. and AUTOGUIDE improve on this by extracting guidelines from similar mistakes in the offline experience. However, ExpeL often erroncously applies incorrect guidelines
7

---

GitHub Flights Coursera
WebShop Reward SR
WebShop
Algorithm
Algorithm
Top-k
SR
SR1
SR1
SR1
SoM [36]
2/30
5/20
1/20
ReAct (1-shot)
66.4
30%
h: == 0
30%
AUTOGUIDE
19/30
9/20
14/20
ReAct (2-shot)
66.0
35%
k
42%
ReAct (4-shot)
70.2
37%
k 46%
Table 2: Test results of AUTOGUIDE on real-world web domains within multi-modal settings. The base agent model runs with GPT-4V and applying context-aware guidelines significantly improves the performance.
ReAct (6-shot)
71.0
38%
k 47%
AUTOGUIDE 73.4
46%
A: 43%
Table 3: Analysis of AUTOGU- IDE against ReAct with varying numbers of in-context examples.
Table 4: Ablation study of AUTOGUIDE using various top-k values.
due to the availability of all guidelines at each timestep. In Figure 4, ExpeL mistakenly attends the second guideline "ensure to specify the item's 's number and location ...", leading to wrong reasoning and action. AUTOGUIDE presents relevant guidelines necessary moments, enabling accurate task completion by avoiding the mistakes seen in Expel. and ReAct.
Importance of providing pertinent knowledge. Expel. approach helps ReAct by extracting knowledge from offiin experiences, but its impact is not as significant as AUTOGUIDE, Recall that for ExpeL, the guidelines are neither generated for specific contexts at training time nor selected to only provide context-aware guidelines at test time. As a result, irrelevant guidelines can be introduced to an agent, potentially causing confusion for the agent. Consequently, the result highlights the significance of providing relevant guidelines conditioned on contexts for LLM agents.
3. Scalability to complex environments. We conduct experiments on WebArema-Redee which features diverse tasks on realistic and complex websites requiring longer action sequences. This domain has a larger observation space and a more complex action space (e.g.. scrolling). Table 1 presents the results, where AUTOGUIDE achieves the highest success rate with significant margin when compared to ReAct and Expel. We observe that ReAct scores low task success rate (8.0% in WebArena due the complex observation and action spaces and longer task horizon. In Expel., the issue of presenting all guidelines to an agent is exacerbated in the WebArena compared to simpler environments like ALFWorld and WebShop. WebArena's 's wide variety of tasks across different domains requires a larger number of guidelines to cover the knowledge needed for all tasks and domains. This results in either an overload of irrelevant guidelines that could mislead the agent or lack of crucial information when the number of guidelines is limited, suggested in Expel. [31]. In contrast, AUTOGUIDE achieves a more significant performance enhancement (47.1 %) compared to ExpeL (2188% by efficiently providing pertinent guidelines and minimizing the burden on context capacity. We refer to Figure 14 for list of example contexts and guidelines.
Q2. How does AUTOGUIDE perform when combined with test-time self-feedback approaches?
Our context-aware guidelines effectively provide inter-task knowledge by considering multiple tasks in offline data. Mcanwhile, self-feedback methods (c.g., Reflexion) offer intra-task knowledge based environmental feedback during test time. In this question, we explore the effectiveness of integrating both inter-task and intra-task information. The results presented in Table demonstrate that the combination of AUTOGUIDE with Reflexion achieves the highest performance in the WebShop and ALFWorld benchmarks. Hence, we find that our context-aware guidelines positively complement the intra-task knowledge of Reflexion. Another observation from Table 1 is that, while ExpeL + Reflexion outperforms Expel. alone, this combination is not as effective as other approaches. This limitation may stem from Expel. introducing irrelevant knowledge, potentially leading to conflicts with Reflexion's 's feedback and having an adverse impact on the decision-making process.
Q3. Can AUTOGUIDE generate context-aware guidelines for multi-modal inputs?
Going beyond text-only inputs is an essential step toward building capable agents for solving real- world environments and tasks. We test AUTOGUIDE in complex multi mudti- setting, where each observation includes image and text information. Specifically, we introduce a set of real-world website navigation tasks in 3 domains: GitHub, Google Flights, and Coursera. For these multi-modal tasks, we employ the Set-of-Marks (SoM) agent [36, 5] as our base method. The SoM prompting improves the visual grounding capabilities of large multi-modal models such as GPT-4V by adding visually distinguishable marks to image inputs [36]. We apply AUTOGUIDE with GPT-4V to generate natural language context-awand guidelines from collected trajectories with both image and text observations.
8

---

Algorithm
CI
GES
WehShop SR
Algorithm
WcbArena-Shopping
ReAct
X
X
30%
SR1
ReAct + CI
/
X
36%
ReAct 10.2%
ReAct + GES
X
37%
AUTOGUIDE 20.4%
AUTOGUIDE
/
46%
Table 5: Out-of-distribution generalization of context-aware guidelines from WebShop the 98 WebArena- -hopping tasks that have a product in the intent template.
Table 6: Ablation study of AUTOGUIDEE, analyzing each module's contribution in WebShop. CI denotes our context identification module, and GES denotes the guideline extraction and selection modules.
Table 2 shows the effectiveness of AUTOGUIDE, demonstrating its generalization ability to complex real-world multi-modal settings. We refer to Figure 15 for example context-aware guidelines.
4.3 Analyses of AUTOGUIDE
Q4. How does AUTOGUIDE compare to ReAct with varying numbers of in-context examples?
Table 3 shows that, while increasing the number of in-context examples for ReAct gradually improves performance, there is plateau a certain number of shots. Additionally, ReAct with more than shots often exceeds the token limit of GPT-3.5-turbo. These results indicate that directly inputting raw trajectories into ReAct for in-context learning is not an effective way to fully leverage offline data. In contrast, AUTOGUIDE extracts knowledge from entire training trajectories by summarizing them into concise cotextt-wware guidelines, making them easy to integrate with prompt- agents.
Q5. How does altering the number of top-k guidelines impact the performance of AUTOGUIDE?
We conducted ablation study WebShop using various values of l in Table 4. We find that employing context-aware guidelines consistently outperforms the no-guideline baseline (k- = 0: ReAct). The k 3 yields the best performance. The largest & value of 5 can lead an LLM agent to overthink, potentially resulting in a slight decrease in performance. Conversely, smaller k, like k= 1, may cause LLM overlook additional helpful guidelines, leading to slightly worse performance.
Q6. How do AUTOGUIDE 's context-aware guidelines generalize to out-of-domain environments?
We conduct an experiment to further demonstrate AUTOGUIDE's off-oof--mminn capability across different domains but relevant tasks. We extract context-aware guidelines from WebShop and apply them to WehArena-Shopee Shooping, which is a distinct domain with variations in observation/action spaces, task intentions, and episodic horizons. For this domain adaptation case, we additionally incorporate a grounding module to align the context-aware guidelines from WehShop to WebArena's observations based on GPT--4 T-rrbo. As shown in Table 5, the transferred guidelines bring a notable improvement in success rates compared to the ReAct baseline in WebArena Shopping.
Q7. How does each component of AUTOGUIDE contribute to the final results?
We evaluate the impact of different components within AUTOGUIdE on its performance in WebShop, as detailed in Table 6. We examine two variants: ReAct+CI and ReAct+GES. The ReAct+CI, which incorporates contexts into observations without guidelines, shows improvement over ReAct. This suggests that contexts enhance decision-making by verifying the current state before action selection. ReAct+GES, which generates guidelines from trajectories without contexts and employs GPT-3.5-turbo for guideline selection, also enhances performance but is less effective than the full AUTOGUIDE. This indicates that choosing relevant guidelines based on the trajectory alone is more challenging than using contexts. Therefore, integrating both context summaries and guidelines is crucial for maximizing the benefits of AUTOGUIDE.
5 Conclusion
We present AUTOGUIDE, an effective framework for exploiting important domain knowledge from offline experiences for improving decision decision-making with pre-trained LLMs. We proposed to generate context-award guidelines that can incorporated into prompts for LLM agents. As AUTOGUIDE extracts the guidelines by contrasting trajectories in offline data, the resulting context-aware guidelines carry critical information for preventing failures the domains. For inference, it provides the guidelines pertinent to each of the different context that LLM agents encounter, which can make

---

pre-trained LLMs strong decision- makinng agents in the downstream domains. Empirically, we showed that AUTOGUIDE outperforms strong baselines by large margin and achieves outstanding performance in decision-making benchmarks.
6 Acknowledgements
This work was supported in part by LG Research.
References
[1] Lei Wang, Chen Ma. Xueyang Feng, Zeyu Zhang, Hao Yang. Jingsen Zhang, Zhiyuan Chen, Jiakai Tang. Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents, arXiv preprint arXiv:2308. 11432, 2023.
[2] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864 2023.
[3] Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as say: Grounding language in robotic affordances. In Conference on Robot Learning, pages 287-318. PMLR, 2023.
[4] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thougle prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 24837, 2022.
[5] Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents realistic visual web tasks. arXiv preprint arXiv:2401. 13649, 2024.
[6] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web. arXiv preprint arXiv:2306.06077 06000, 2023.
[7] Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowd- hery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. Understanding HTMIL with large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Asso- ciation for Computational Linguistics: EMNLP 2023, pages 2803-2821, Singapore, December 2023. Association for Computational Linguistics.
[8] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307. 13854, 2023.
[9] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACI., 2022.
[10] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint arXiv:2301.0m 000334, 2022.
[11] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP, 2022.
[12] Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models, 2023.
[13] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) generalist web agent, if grounded. arXiv preprint arXiv:2401.01614, 2024.
10

---

[14] Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong. and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. arXiv preprint 2Xiv:2310.12823 2023.
[15] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118-9147. PMLR, 2022.
[16] Lajanugen Logeswaran, Yao Fu, Moontae Lee, and Honglak Lee. Few-shot subgoal planning with language models. In NAACL: HLT, 2022.
[17] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference Machine Learning, pages 10764-10799. PMLR, 2023.
[18] Yujia Qin, Shihao Liang, Yining Ye. Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16788 2023.
[19] Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.1533 2023.
[20] Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arXiv preprint 220. 2022.
[21] Timo Schick, Jane Dwivedi-Yo Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.044::
2023.
[22] Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models. arXiv preprint arXiv:2305. 16653, 2023.
[23] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
[24] Mohit Shridhar, Xingdi Yuan, Mlec-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In ICLR, 2021.
[25] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303. 17651, 2023.
[26] Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. arXiv preprint 2rXiv:2303.17490 2023.
[27] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In shirty- Conference on Neural Information Processing Systems, 2023.
[28] SRK Branavan, David Silver, and Regina Barzilay. Learning to win by reading manuals in a monte-carlo framework. Journal of Artificial Intelligence Research, 43:661-704, 2012.
[29] Austin W Hanjic, Victor Zhong, and Karthik Narasimhan. Grounding language to entities and dynamics for generalization in reinforcement learning. In International Conference on Machine Learning, pages 4051-4062. PMLR, 2021.
[30] Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. Rtfm: Generalising to new environ- ment dynamies via reading. In ICLR, pages 1-17. ICLR, 2020.
[31] Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Lim agents are experiential learners. arXiv preprint 2308.2200..0144 2023,
11

---

[32] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
[33] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-207525 2022.
[34] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
[35] Google Flights. https:/ wwwp:/oogee.coo//rree///e/eeeeae
Accessed: 2024-05-21.
[36] Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint 20.i:::3/1.11444: 2023.
12

---

A Limitation and Broader Impacts
Limitation. The performance of AUTOGUIDE depends on the diversity of offline experiences. As such, one important direction for improvement is to automatically collect diverse offline experiences through continual learning, where we iteratively generate guidelines and use them to gather more trajectories with high rewards. Another avenue is the need for quantifying the quality of generated contexts and guidelines. Currently, apart from applying context-aware guidelines to ReAct and measuring test time performance, there lacks standardized method for quantifying the quality of generated contexts and selected guidelines. Introducing a quantifiable metric to approximate the quality could pave the way for new optimization approaches such as reinforcement learning.
Broader impact. This paper introduces research aimed at enhancing the decision makinn capabilities of LLMs. In terms of societal impact, while we develop generic LLM-based autonomous agent, having biased offline datasets may lead to making decisions with suboptimal outcomes. Additionally, autonomous agents may be misused for malicious applications. To mitigate these risks, potential solutions would include diversifying datasets, implementing ethical oversight, ensuring transparency and accountability, engaging with stakeholders for broader perspective, and incorporating secu- rity measures to prevent misuse. We believe that the research community, including ourselves, should responsibly advance L.Md-based agent research, prioritizing societal well-being and ethical considerations.
B Evaluation Details
B.1 ALFWorid [24]
B.1.1 Environment details
Each task in ALFWorld starts with description of the specific environment and the goal to achieve. At each timestep, an agent can choose one of the following actions interact with the objects and receptacles in the environment:
go to [recep]
take [object] from [recep]
put [object] in/on [recep]
open/close/use [recep]
clean/heat/cool [object] with [recep]
Alternatively, the agent can generate think action for planning and reflection, which helps with decision-makir but does not change the environment itself. After one action performed. the environment returns an observation that describes view changes.
Following ReAct, concatenate list of (observation, action) pairs to show the entire trajectory up to the current timestep for LLM agents to generate the next action. We experiment on 134 un- seen test tasks with 6 categories of pick_and alace, pick_cleas wee.keeeenttheellleeeeeeeee theneplace, pick heatsheetthheee -hhentllaceeee.....eo pick_coo) cocc_coo__hheeee _the__the__gggg whtettllaceeelppppppppppppia place, look at .obj, and pick_two_okd For each task, the agent is allowed to take maximum of 50 actions.
B.1.2 Baseline and Models
For ALFWorld tasks, we follow the same setting as ReAct by providing 2 in-context examples for each of the 6 task categories. The original results of ReAct in their paper are produced based on Dext-Davinci-002 However, this GPT version is no longer available, so we apply gpt-3.5-turbo instruct instead to generate actions. For Expel. we directly take the guidelines from their appendix and append them the ReAct agent at time.
B.1.3 Implementation details ofAUTOGUIDE
We run the first 100 training tasks of ALFWorld to collect (T+, pairs with Refllxinn and extract context-dependant guidelines on the collected data. For context identification, provide
13

---

2-shot demonstrations for each of the 6 task categories. The corresponding prompt templates can be found in appendix C. All parameter details are shown in table 7.
Parameter name
Value
Allowed Episode Length
50
n-shots
Agent Model
301-3.5tturbo-inse
Context Identification Model
apt-3.5-turbo-ing
Guideline Selection Model
3pt-3.5-turbo-inste
Guideline Extraction Model
gpt-4- gpt-4-1106-previns 106-preview
Reflexion Model
gpt-4-1106: 106-preview
top-k guideline selection
Table Experiment hyperparameters on ALFWorld. The maximum allowed episode length and n-shots follow the same setup in ReAct.
B.2
WebShop [33]
B.2.1
Environment details
WebShop provides an e-commerce environment, where the objective is to find and buy the product that matches the task-specific Instruction. The agent can select one of the following actions to perform:
search[query]
click[button]
Following ReAct, the agent generate think actions do planning reflection. After buying product, the environment returns a reward showing how well the bought product matches the target one in type, price, buying options, and attributes. The reward is calculated by:
Wan O Yasl +|Upps n Yopel [Mprice S Upprioe]
Ttype"
|Vat Uppt
where y is the bought product and u is the target product. Same as ALFWorld, for WebShop, the agent takes (obs,, act,) pairs for every previous timestep f input generate the next action.
B.2.2 Baseline and Models
Following ReAct, experiments are done a one-shot setting. We apply spt-3.5-turbo-0613 to generate actions, but when the token number exceeds the token limit (for example, for the n-shot ReAct experiments in table 1), we use the 16k version of apt-3.5-turbo-061 instead. For ExpeL. we could not find how many training tasks the framework used for training. Therefore, directly apply the guidelines from the appendix of their paper at test time. We only consider Expel with guidelines, ExpeL with in-context example selection in our experiments for a fair comparison. The in-context example selection method is orthogonal to our work and can be easily combined with our method. For Reflexion, shown in their paper, their 2-shot Reflexion prompt does not work well WebShop. Therefore, we re-write the prompt and apply gpt-4- 106-preview to generate episode-level reflections for all Reflexion experiments. Following Reflexion and ExpeL., the evaluation is done on 100 test tasks. The maximum number of allowed actions for each task is 15. At the same time, each search action shows the top products for the search query. Please refer to Table 8 for more details about the experiments.
B.2.3 Implementation details of AUTOGUIDE
We randomly sample training tasks from the training set of WebShop, on which we run Re- Act+Reflexion to collect pairs and generate guidelines. The context identification prompt is one-shot, which is shown in Appendix C. At test time, we ask gpt-3.5-turbo--00 to select each state's most relevant top 2 guidelines.
14

---

Parameter name
Value
Allowed Episode Length of Search Results
15
n-shots
Agent Model
gpt-3.5-turb--613
Context Identification Model
gpt-3.5-turbo-0613
Guideline Selection Model
upt-3.5-turbo-011
Guideline Extraction Model
gpt-4-1 1014--110-prreview
Reflexion Model
gpt-4-1106-preveew
top-k guideline selection
Table 8: Experiment hyperparameters on WebShop. The maximum allowed episode length, the number of search results per page, and n-shots follow the same setup in ReAct.
B.3
WebArena [8]
B.3.1
Environment details
WebArena provides web-based benchmark environments that closely follow the data and functionality of real-world websites. Unlike other benchmarks like WebShop that provide clean text of website information as observation, WebArena's 's wehpage content is represented as an accessibility tree, which is subset of the DOM tree with useful elements of webpage. For our expeirments, we focus on WebArena Reddit, which simulates the real Reddit websites with users, forums, and posts with abundant text information.
For each task in WebArena, the agent is expected to achieve a task-specific intent. At each timestep, WebArena provides a list of opened tabs, the accessibility tree of the focused webpage, and the URL of the current page observation. For WebArena, each observation is long, Therefore, following the baseline in WebArena, at each timestep, we only provide the observation of the current timestep to the agent. We additionally provide up to 5 past actions for the agent to understand what it did in the past. The allowed actions WebArena include the following:
goto [url]
click [element_id id]
type [element id] [text] [I for enter or 0 for not enter]
press coembnnabittion
scroll [up down]
back
The maximum allowed number of actions for single task 20. Note that WebArena does not provide training tasks, but the work provides 19 demonstrations for Reddit, each of a different category. Therefore, we set these 19 tasks as the training tasks and then test on the rest 87 tasks.
B.3.2 Baseline and Models
We directly run the two-shot ReAct-style baseline in the official codebase of WehArena using gpt4- preview-1 106.. For Expel., the original paper does not include experiments WebArena, therefore we try our best to implement our own version and run on the same training tasks as our method,
B.3.3 Implementation details of AUTOGUIDE
We directly run ReAct on the tasks with T4 collect actions and generate guidelines correspond- ingly. We provide a 5-shot prompt for the context identification module, which is shown Figure At test time, the top guidelines at each timestep selected to guide action generation.
15

---

Parameter name
Value
Allowed Episode Length
20 2
n-shots
Agent Model
gpt-4-1 106-previeww
Context Identification Model
gpt-4- 1116-peeeiiewww
Guideline Selection Model
gpt-4- 104-1100-pmaa 111--peeiiewww
Guideline Extraction Model
1044--106-ppeeieee
top-k guideline selection
Table 9: Experiment hyperparameters on WebArena. The number shots follows the same setup in ReAct.
B.4
Real Websites
B.4.1
Environment details
We design a set of real-world website navigation tasks from domains: Software Development (GitHub), Travel (Google Flights), and Education (Coursera), which have 30, 20, and 20 test tasks accordingly. Here are some example tasks:
GitHub
Navigate to the repository for the Python library Seaborn with the most stars and show me all the open issues labeled with bug.
Go the GitHub org for Spotify and open the pinned project with the most stars for me.
Google Flights
Find me the one-way flight from Hong Kong to Paris departing on Oct 15th 2024 with the least emmisions.
Show me the booking options of the one-way flight departing from Auckland on September 11, 2024, and arriving in Rome with the earliest departure time on that day.
Coursera
Show me a Cybersecurity course that can finish within month and show me all the reviews for the selected course.
-Find me Coursera guided project that covers Unity and show its main page.
We follow the action space design of Visual WebArena [5], which has the following action types available:
goto [url]
click [element id]
hover [element_id]
type [element id] [text] [1 for enter for not enter]
press [key_on combination
scroll [up or down]
tab-focus focus [tab_index]
close_tab
go back
forfooddrd
As the real websites are constantly and dynamically changing, we evaluate the completed task with human experts.
B.4.2 Baseline and Models
We directly run the two-shot SoM algorithm the official codebase of Visual WebArena. The only modification we made from the original codebase is the bounding box detection algorithm, in
16

---

which we further filter invisible bounding boxes from the list and add the list elements as interactable elements to consider.
B.4.3 Implementation details of AUTOGUIDE
We provide a total of 6 training tasks (3 for GitHub, 2 for Google Travel, and 1 for Coursera), on which we collect human demonstration as T+ and run SoM to collect T-'s. From the pairs with multi-modal observations we generate text-based guidelines to guide action selection. Both context identification and guideline extraction are done by gpt-4-vision-rres prrvvie,, and we provide 3-shot prompt for context identification. All parameter details are shown in Table 10.
Parameter
Value
Allowed Episode Length n-shots
15 2
Agent Model
gpt-4-vision-previee
Context Identification Model
gpt-4-vision-prevveew
Guideline Selection Model
gpt-4-vision-preview
Guideline Extraction Model
gpt-4-vision-previes prssion-ppeviiwwww
top-k guideline selection
Table Experiment hyperparameters on multi-modal real-world website tasks.
C Prompt Templates
C.1 Context Identification
We present prompt templates for the context identification module Moontext (Equation (I)) for WebShop, ALFWorld, and WebArena in Figures to 7, respectively. In ALFWorld, there exist six categories of tasks, and context identification prompting with 2-shot examples for each task, following the practice by [23]. Figure shows one example for the pick and place tasks.
C.2 Guideline Extraction
Figures to 10 detail our prompt templates Mguideline for extracting guidelines (Equation (2)) in the WebShop, ALFWorld, and WebArena domains.
C.3 Context Matching
Figure 11 shows our prompt template for matching the generated context with one of the existing contexts if there any similar context, for the construction of the set context-aware guidelines (Section 3.2) and the retrieval of relevant guidelines during testing (Section 3.3) in all the three domains: WebShop, ALFWorld, and WebArena.
C.4 Guideline Selection
For selecting only k most relevant guidelines in case there are more corresponding context-aware guidelines during testing (Equation (3)), we use the prompt with Figure 12 for WebShop and ALFWorld and Figure 13 for WebArena.
D Example Context-Aware Guidelines
In Figures 14 and 15, we show a list possible contexts and context-aware guidelines on WebArena and real-world websites, respectively.
17

---

Context Identification Prompt for WebShop
Instruction
Now you'll snipoet trajectorv. job brief and general summarization of the current general, any information specific instructions, products, buying options. "nsttrcciion' with the same status, the summarization should be the same.
In-Context Example
Webshop
[Features]
Instruction:
[Reviews]
would like bottle of bright citrus deodorant under 50
[Astributes] Now]
[Seurch)
SUMMARZATION: The assistant on product's main
SUMMARIZATION: The assistant the search
[Search]
Action: dlik["bright citrus' "(pack of 1/) Observation: Invalid Action!
Actioni search/bright dendorant)
Observation:
SUMMARIZATION: The assistant falled click on button and "rwalldd Action!:
[Back to Search]
Page (Total 3)
[Next So5]
Action: dlick[bright citrus] Observation: You have dliced bright citrus.
[D678GWRCUJ]
Bright Citrus Deodorant
SUMMARZATION assistant has sucoessfully dlicked on button.
$10.99
[8078GTKVAY]
Ginger Fresh Deodorant
Action: click(Description
$10.99
Observation:
Search]
Barrel and Oak: Aluminum- Free Deodorant
[$c5 Prev]
$15.95
Bright Citrus Dendorant sensitive skin
SUMMARIZATION: assistant
result page
SUMMARZATION assistant product's subpagelone of [Description), [features], [Rediews], [Atributes]].
navigated
with with list of products.
Action: Glek[B078GWRCCJ]]
Observations
Action: dick[Buy New
Search]
Action!
[$o$ Prev]
SUMMARZATION The assistant foiled dlick on button and "nwvaldd Action!..
(assorted scents]lbright citruslipinger fresh)
size [trovel set [l-puck][[lpack 3)][ne (2 (2-pock)]
Bright Citrus Deadorant
Action: searchibright citrus deodorant under 50] Observations Invalid Action! SUMMARIZZTTON: The assistant falled search on the page and "rwalld Actionl:.
$10.99
Rating: N.A. [Description)
Input
your (Current trajectory)
Figure 5: Our prompt template for context identification (Equation (1)) in the WehShop domain.
18

---

Context Identification Prompt for ALFWorld
Instruction
'to''l get snippet of trajectory of textlbased ALFRED task. Your job is to generate brief and general summariztion of current status following SUMMARIZATION:
Keep broad and general, avaid any information about spezific objects and locations. Here are two examples:
In-Context Example
You are in the middle of room. Looking quickly around cabinet 1, a coffeemachine 1, a countertop 1, a diningtable 3, ciningtable diningtable drawer fridge 1, garbageran 1, microwave 1, sidetable 1, a sinkbasin 1, and touster 1.
diningtable
you, you
the diningtable 1, nothing.
SUMMARDATIIN Looking for an object, but it's not here.
the garbagecan 1, you see apple 3, egg 3.
SUMMARZATION: Looking for object and found the
Your tack find some apple and sidetable. think: To solve the task, need to find and take an apple, then put in sidetable.
desired object think: Now! find an apple (3), Next, need take take apple from garosgecan
up
garbagecan
think: First I need to find an apple. apple is more likely
SUMMMARGZZIIN Successfully took an object.
in fridges diningtables (1:3), sidetables countortops (1), sinkbasins (3), garbagecan can one by one, starting with fridge 1.
think: Now find spple Next, need In/on OK:
go sidetable
go to fridge
On the sidetable 1, you see nothing.
The fridge dlosed.
SUMMAROZTTIN: Looking for a location object
SUMMARIZATION: Looking
here,
dooeed.
put apple 3 in/on sidetable Nothing happers.
fridge
You open the fridge fridge open. you see
SUMMARZATION: Tried to put an object infon location but falled.
lettuce 2, 2, potato and tomato 1,
SUMMARIZATION; Looking object,
sidetable
> to table
You put the apple 3 in/on the sidetable 1. SUMMARIZTION: Successfully
Nothing happens.
object Inyon
SUMMARIZATION: go
location.
In-Context Example
Input
New it's your turn:
(Current trajectory)
Figure 6: Our prompt template for context identification (Equation (1)) in the ALFWorld domain.
19

---

Context Identification Prompt for WebArena
Instruction
You autonomous inteligent tasked with ravigating web browser.. You will be provided with the following information:
1. 1st of context summarizations you have seen the past.
snippet
page's accessibiiity simplified
webpage
the current web pages' URL.
Please
summarizziion the current obsenvation after SUMMARIZATION:
Here are some requirements:
Raquirement 1: Different types of webpages should have clearly different summaribations. For example, for GitHub there can repository,
the page of GitHub. etc, should clearly categorize those and make sure not to mix them up. information, instead, websites that page of every reditt forum should be categoried as the same context summarization: On the main page of Reddit forum. You spectfic Requirement 3: The URLs will be very useful for you determine the summarization. Requirement the
page
page
page
The summarization should be same categories should have same summarization, example, main
user/abject/task spacific
summarization.
the
matching by word.
In-Context Example
Observation:
The current web access 0 (current]:
[1] RoosWebArea "Pssmill' focused: True
Header&
[32] link Home'
[SS] link 'Formms
[56] link Wik!
[64] searchbox 'Search query'
(0)'
[66] link 'Summit"
MarvekSrantMann3k focused: hasPopup: True
[243] link Profile'
[239) link accoune'
[245] "Uer settings'
[255] link Block list'
orientation: horizontal
[270] button Dark made'
URL: http::/reediit.com
SUMMARIZATION:
main page.
In-Context Example
Input
Here the
(Seen list] (Current trajectory
Figure 7: Our prompt template for context identification (Equation (1)) in the WebArena domain.
Guideline Extraction Prompt for WebShop
Instruction
[Task description). You will provided with desired and undesired trajectory of the same task, What the first action that trajectories?
trajectory failed answes, generate an action guideline make future task avoid the same mistake. The guideline should specifl what to do what situation the format of "When what option buttons buying options such as sizes, colors, scents, and flavors', and clearly say
you
successful?
should (or should notl.... On product's
with product
information, strictly refer buying options are not subpages like [Description] and [Attributes] when you mention buying options, Your guideline must be ganeral enough requierments ments in Instruction. Strictly follow what the destred trajectory does and never suggest actions that the desired trajectory didn't do. When refenning to actions, use the allowed action format, You should make your answer concise, limit your answer
tack, therafore
include task specific information, Instead, refer
the
answer within tokens, Guideling: ...!.
RRasoning:...
Input
Trajectory:
(Desired trajectory Undesired Trajectory: (Undesired trajectory)
Figure 8: Our prompt template for guideline extraction (Equation (2)) in the WebShop domain.
20

---

Guideline Extraction Prompt for ALFWorld
Instruction
[Task description] You will be provided with desired and undesired trajectory of the same task, What the first action that ciffers besween the two trajectories? Why do you think it makes one trajectory failed and the other successful? Based on your same
answes, generate action guideline what situstion in the format of "When what status, (optional: you want to ...I you should (or should noc)... (optional: a short example for demonstration." For the 'When in what status' part, directly use the words SUMMARIZZATON. Here two examples:
guideline should specify
do
esample When looking for an object, if you want to find a kitchen-related object like a spatula, you should start from the most possible locations.
Example When looking an object found the desired object the location, You should only take the essect object that you want.
Stricity follow to actions, use the allowed action formet. You should make your answer concise, limit your answer within 128 tokens, and put your answer this format: 'Reasoning: Guideliner....
desired trajectory and suggest
desired trajectory didn't do. referring
Input
Desired Trajectory:
(Desired trajectory)
Undesired Trajectory: (Undesired trajectory)
Figure 9: Our prompt template for guideline extraction (Equation (2)) in the ALFWorld domain.
Guideline Extraction Prompt for WebArena
Instruction
[Task description) 'ou just finished task but falled. For this failed task, we you. Please compare the demonstration with your generated action at each step, resson about the intention of the correct action, and then geneate action guideline future same mistake and make the future tasks successful.
"link "Forums" or "button 'Creste submission", you should spectfically include them in the exect text your guideline. When referring detailed as possible only replace the task specific information palceholder, example. replace forum Iphone sforum nomes specify the in full, starts with http:///
provide human demonstration
your guideline, specify it
to avoid
The guidelina should be less than tolons.
Here's the information you'll hive:
Please refer to "the previous actions" and "Demonstration actions in later stops" to generate more accurate descr descriptions of your purpose achieve the purpese. make sure to emphasice the order the actions, do not .., for example singl action, and put them you typed
1. The current obsorvation:
The objective: task 'e trying complete. The current web page's accersibility tree: simplified
sequence
representation the webpoge, providing key information. current web page's URL: the link the page you're carrently on.
text, you should do these sequentially: 1. ... 2 You must strictly follow the order"
The tabs: the tabs provious actions: sequence of past actions that you performed.
opened.
When you mention muttiple steps actions, mention the guideline that you should refer to the PREVIDUS which
you and what you should do next. Specify that you should not repeatedly issue the same action, but should move action instead.
ACTIONS
2. The action you generated in the failed run.
the next
3. correct action that you should take.
Only speicily what to do or what not to do, don't explain
4. Demonstration actions steps
pago.
is important to clearly specify when to issue stop action when the stop action is either the correct action or in the "Demonstration actions later steps on same page not spectfy the 'answe' in 'stop lanswer]" because answer is different for different tasks, and about stop action is netther in correct action you should take." nor "Demonstration actions loter steps page.".
Based on the information, please generate a short and correct action.
concise guideline that quide you Important: The guideline should be general enough to generalize to similar tasks, not this task, Therefore do not include any task-spectfic information your guideline, for example user name, a specific forum, the specific
mention anything
in front of each element, for example (123), the numbers are randomly generated therefore never include them your guideline, However,
enter,
number
Please strictly adhere the 'correct action that you should take', do not propose other actions.
non-specific elements
Input
Hare are the information you need: (Observation)
action you generated failed
(Predicted action)
The correct action that should take:
(Demo action)
Demonstration actions in later steps on the same page:
(Later action)
Please put your anower this format: Reasoning ... Guideline: ...
Figure 10: Our prompt template for guideline extraction (Equation (2)) in the WebArena domain.
21

---

Context Matching Prompt
Instruction
(Task description)
task trajectory can be long. Therafore the assistant summarizes the status of each step.
different task with the status, summarizztion should be the therefore instructions products.
information about
You will be provided with the following:
list summariaations assistant
2. nowly generated summarization.
Please determine summarication from the list matches the exact same status the newly generated one.
corresponding summarisation, example "Answer: 2"; otherwise,
Input
Seen Summarizations:
(List of contexts)
Figure 11: Our prompt template for context matching (Sections 3.2 and 3.3) in all the WebShop, ALFWorld, and WebArena domains.
Guideline Selection Prompt for WebShop and ALFWorld
Instruction
[Task description) You equipped
following resources:
1. A list of action gurdelines with valuable guidelines.
Trajectory history, includes
observations actions,
Not guidelines are useful to generate the next action. Please select the guidelines that are useful and relevant to the next action given the to directly you selected guidalines in a python list. for example you select guideline 1, 5, answer: [1, 5). none of them are useful far next action, answer the emoty [).
and recent observations. To generate next action, which guidelines from the provided guidelines for
action? can
guidelines,
the
Input
(List of guidelines) (Current trajectory)
Figure 12: Our prompt template for selecting the most relevant context-aware guidelines during the test time (Equation (3) from Section 3.3) in the WebShop and ALFWorld domains.
Guideline Selection Prompt for WebArena
Instruction
(Task description) each time step, you need generate one action given the current olservation..
equisged
following resources:
1. A action gurdelines with valuable guidelines.
2. The intent of the task, which objective/gval that should achieve.
Trajectory history,
includes current observation sequence past actions.
Not all puidelines are useful generate the next action, Please select the guidelines that are useful and relevant to the next observation
the To generate the next action, which guidelines from the provided guidelines are most useful to directly tell you what to for the next action? You can select guidelines (or less there are less than guidelines), and put the number indices of the guideline answer 5).
guidelines python relovant, answer ().
you
are
Input
(List of guidelines) (Current trajectory)
Figure 13: Our prompt template for selecting the most relevant context-aware guidelines during the test time (Equation (3)) in the WebArena domain.
22

---

Context: the Reddit main page.
Context-Awware Guidelines
When on the Reddit
page, you want to change your bio, you should click dropdown, right ahove the "link "My account". The correct action format to do this is "'click [proflle_iind, _i_nk__id""""._
'Profile",, which located in user menu
When on the Reddit main page, If you want to navigate to a specific forum, you should click on the "link "Forums", which is located at the early part of the observation, right above the link 'Wiki. The correct action format to this ""clik [linkid]"". Id]".
When on the Reddit main page, if you want to create new forum, you should dlikk on the "link "Forums", which located near the top of the observation, right above the "link 'Wiki". The correct action format to do this is ""lick [link_jd When on the Reddit main page, you want to like all submissions created by a specific user in specific subreddit, you can directly
navigate to the user's page with the action format goota [url]"", replacing [url] with the user's page URI, formatted as http http:///saatttoosoe/ corommddomdmmooooo domains/user//eee /uuss///ee/ssmmme....
Context: On the overview page of Reddit user.
Aonte Guidelines
When on the overview page of Reddit user, if you want to interact with submissions from specific subreeddit you should first navigate to the 'Submissions' tab to filter the content the user's submissions, The correct action format do this is ""cick [link, id]"", where
[link id] the ID of the 'Submissions' link in the main content area of the page.
Contexti On the biography edit page of Reddit user.
Context-Aware Guidelines
When on the biography edit page of Reddit user, If you want to change the blography text, you should do these sequentially: 1. Click on the textbox 'Blography' focus it, located in main section of page, with action ""llck [textbow id]"". 2. Select all text inside the textbox using the action "press [Meta+a]"". 3. Clear the selected text with the action ""press [Backspacel" 4. Type the new biography content into the textbox 'Biography' with action ""type [textbox_ id] [new] [oo__ottttt]]] [1]"", S. Click on the button 'Save', located below the textbox BBoggrahhy,, to submit the changes with action "dlcc [button_id] "idtoon_dd""""" After these steps, issue stop action when the task is complete.
Context:
On the page listing all forums on a heddit-like platform.
Context-Aware Guideline:
When on the page listing all forums on a Reddit-like platform, if you want to navigate to a specific forum, you should do these sequentially: "link 'Niphabetical", which located the main area, forums alphabetically. The correct action format do this "click (link id]" 2. After the forums are sorted, click on the specific "link 'cforum, names" that you wish to navigate to. The correct "click [linkjid]
1. click
action format
Contexti the
page create submission.
Context: Aware Guideline:)
When on the Reddit page create submission, If you have filled in the 'Title' and 'Body' textboxes but do not see the button 'Create
submission', you should scroll down reveal more the The correct action format do this is ""scroll [down]"" After scrolling, you want to submit the post, you should click on the button 'Create submission', which is located after the 'Body' textbox, The correct ""lick [button_ id]"
action format submit the
Contexti main page of
forum.
Context-Aware Guidelinesl
When on the main page a Reddit forum, you want posts related to a spocific topic among top posts, you should first
posts by their popularity ensure you are viewing the most relevant content. To do this, you can: dlick on the "button 'oort by: Hot",,
which is located in the main section of the page, below the forum heading and above the first article. The correct action format to do this is option under the sorting
""click [id]"
After the sorting options have expanded, click on the "link Top"", which will appear
button. This action should be repeated once. The correct action format to do this is "'click [id]"".
When on the main page a Reddit forum, if carly part of the observation, right below the "link NNotificatonn 10]". The correct action format to da this is ""click id]""
create new post, should click on the "link 'Submit", which located the
Contexti On the submissions page a Reddit user.
-ontext-Aware Guidelinei
When on the submissions page of Reddit user, if you want to perform an action on specific subreddit submissions but do not see them, you should down reveal more submissions. The correct action
do is "'scroll [down]"" After scrolling, you find submission from the desired subreddit, such as 'UplifingNews, and need to downvote it, dlick on the button 'Downnvote' located at the end the submission's details. The correct action format downvote ""clic teauwwote_button "uuttonidd"". Once all required actions are performed the submissions, issue stop action with the format ''soop D""
Contexti On the submissions page of a Reddit forum sorted by top.
Context- Guideline:)
When on the submissions page of Reddit forum sorted by top, if you want to review posts but see "There's nothing here...', you shauld expand the time range to view more posts. Do this by diicking on button 'From: 24 hours", which is located in main section of the page, below the heading /ttbbook'' and above the StaticTest 'There's nothing here..... The correct action format to do this is "'click [button id] After expanding the time range, if you find post that meets the task criteria, issue a stop action.
Contexti On the submissions page of Reddit forum sorted by Context Aware Guidellnee)
When the submissions page Reddit forum sorted new, you upvote
post and you have already clicked the upvote button for the first article entry, you should issue the stop action. The correct action format to do this is ""stop".
Contexti On the page of a Redoit post.
Context Awaare Guidelines
When on the page a Reddit past, If you have already navigated to the 'Submit' link, filled in the image URL and title, and clicked the 'Create submission' button, you should consider the task complete. The correct action format do this ""stop""
Figure 14: Example contexts and corresponding guidelines for WebArena.
23

---

Context: On the main page of GitHub.
Context-Aware Guidelines
When an the main page of GitHub, if you want to search for a repasitory, you should type the search query into the search bar at the top of the page, which visually identillable and typically labeled with text like 'Search jump to.... Do not type into any other input fields.
Perform the action as follows: "type [search bar id) [search query] [1]
Context: On the
page of GitHub repository.
Context-Aware Guidelines)
the issues page of GitHub regository, you want to issues specific label, you can tyge filter the search input field, which usually at the top of the current webgage and shown as the first appeared "[INPUT] [I' in observation. The correct action the number closed issues, which is labeled as "[A] [num Clased]" in observation, located near the search input field. The correct action format "click [link id]
format do is ""type [id] [label:" caapllsssppi_aaaabeeeaaaa (1) After applying filter: Click link
Context: On the search result page of GitHub.
Context-Aware Guidelines)
the search result page GitHub, you to filter search results find specific organization, after you have typed
organization's 's name in the search bar, you can do these sequentially: 1. click on the "[UJ [Users]" in the left sidebar, which is visually text "Users" and first "[Li] [Users]" in observation, by action ""click [i [I]id]"" filter applied, click on the organization's name, which is visually represented by the blue text "Meta' under the "Users" section, by action "'click [linkid]]
represented by the
user organization
Contexti On search result page Coursera.
Context Aware Guidelines)
search bar, you can these sequentially: 1. click on the filter for courses, which represented by "[INPUT] [I" and visually lorated in the filter section on the left side of the webpage, by action ""elick linput id] 2. click on the course link, which represented by "[A] [course, name]" and visually located in the main content area of the webpage, by action "cclick [link_id]" This action should be repeated twice. switch to the new tab that contains the course details by ""page, focus [1]"".
search result page Coursera, if you want select course, you have typed the course
Contexti On the main search page of Google Flights.
Context-Aware Guideline:
On the main search page of Google Flights, if you want to search for one way flight with specific departure and destination airports and date, you can these sequentially: type departure airport code "[INPUT] [Where from?]", which the input box at the top of the search area, by action ""type [element_jd) [d] [sirport_ (1] 2. the destination airport code "[INPUT] [Where to?]", which is next to '[INPUT] [Where from?]". by action "type [element id] Idestination airport code] [1] click on "[DIV] [Round trip]" to change the trip type, located the of the search area, by action ""dick [clement__ie [d" click on "[u] [One way]" select the way trip option, which appears after clicking "[DIV] [Round tripl", by action "click Jelement idl" 5. type the departure date "[INPUT] [Departure]", which the date, which appears after entering the departure date, by action ""llick (element [d] 7 7. dlick on "[BUTTON] [Search]" to perform the flight search, which below search fields, by action ""dlik (element_id]"
to "[INPU] [Where to?)", action "type (element [date] [3] click "[BUTTON] [Done]" to confirm
Contexti the search result page Google Flights with list Best departing flights and Other deporting flights. Context-Aware Guidelines)
search result page Google Flights with list of Best departing flights and Other departing flights, if want select flight, can click on the first flight option under the "Best flights" section. The correct action format to do this ""click Briiht_pttion____ _id]" " where [flight_optio__]] id] the the [L] element corresponding to the flight listed. This element visually located the top of the list of flights and contains the departure and arrival times, airline name, flight duration, and other details.
departing flights and Other degarting flights, if you want with the least emissions, you can do these sequentially: 1. Click on the sort options button, which is visually located at the top of the flight shown as "[BUTTON] [Sort by]" observation, by action "'lick (sort. button id] 2. Then, on the emissions sort option, which visually located in the sort options dropdown and shown as "[u] [Emissions]" in observation, by action ""dick [emissions_one _otttooot_o____ "d] Finally, click on the first flight listed under the "Best flights section, which visually located at the top of the list and shown as "[u] [flight, details"" in observation. action "dlcc [firtt, best, flight, "i]"" Repeat this action twice.
search result page Google Flights
flight
Figure 15: Example contexts and corresponding guidelines for real-world websites.
24

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification: the abstract and introduction, we clearly outline our main contribution: extracting context-aware guidelines from offline data to enhance the decision- maisiinnmaking of LLM-based agents. We support the effectiveness of our method through empirical evalua- tions in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We outline potential limitations of AUTOGUIDE in Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Liiiitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer: [NA]
Justification:
This paper does not include theoretical contributions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 4.1 I and Appendix B provide detailed information (e.g.. model ver- sions, hyperparameters, number training data) for reproducing our experimental results. Additionally, Appendix C contains our prompt templates.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, other means that appropriate the research performed.
also be provided via detailed instructions
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either be a way to access this model for reproducing the results or way reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path reproducing verifying the results.
5. Open access to data and code
26

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We intend to release the source code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co/ guides/ Laaleeee/oodSSuumssiioooooiiooi
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https: tuipscce/ubblig /ilie//u htapll//ggiiee//leeuubbiiieieee CooleSubmissionPolleyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We detail our experimental settings in Section 4.1 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We provide the average rewards and success rates calculated across test tasks in our results, but we do not include the variance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, least for the experiments that support the main
claims of the paper.
The factors of variability that the emor bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
27

---

The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g,, Normally distributed errors). of errors is not verified.
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1- I-sigma error bars, but one should state it. The authors should preferably report 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer:
[Yes]
Justification: Our experiments are based on GPT, and we detail the versions of GPT in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https:/ /twurpusuucccee wwwlic//btttupeee wwwlic//thicsCuuicceee /EthicsCuideelnees
Answer: [Yes]
Justification: Our work conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethies.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide the broader impacts of our work in Appendix A.
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairnsss considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications. let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper makes the algorithmic contribution and does not release new data models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We reference the original papers and sources throughout the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, saperswithcode.com wwwcchod....o.aaa.eeeeeeeeeeeeee /datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
29

---

For existing datasets that are packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/mooel as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "x1nwBrJcKg", "reviewer": "Reviewer_zgYy", "summary": "This paper introduces AUTOGUIDE, a framework for enhancing large language model agents' performance in sequential decision-making tasks by automatically generating context-aware guidelines from offline experiences. The method consists of two main components: a context identification module and a guideline extraction module. These modules work together to create concise, natural language guidelines that are relevant to specific contexts. At test time, AUTOGUIDE identifies the current context and selects pertinent guidelines to assist the LLM agent in decision-making. The authors evaluate AUTOGUIDE on various benchmark domains, including ALFWorld, WebShop, WebArena, and real-world multi-modal websites.", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Context-awareness is noteworthy because it addresses a fundamental challenge in leveraging offline experiences for LLM agents. By generating guidelines that are explicitly tied to specific contexts, AUTOGUIDE can provide more targeted and relevant assistance during decision-making. This is evident in the example shown in Figure 4, where AUTOGUIDE's context-aware guideline helps the agent locate a soapbar in a less obvious place (the toilet).\n- Ablation study is pretty strong. The authors test their method across a diverse range of benchmark domains, including both text-based and multi-modal environments, demonstrating AUTOGUIDE's versatility and effectiveness.", "weaknesses": "- The method can be summarized as ReAct with context identification and guideline retrieval. And from the ablation study, it seems like the GES component is really important. For each different task, the guidelines have to be constructed, and that can introduce noise. This may hinder the effectiveness of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your positive evaluations of our paper and insightful feedback. Below, we respond to your valuable comment. We will also carefully incorporate your feedback into an updated version of our main paper and appendix.\n\n> “For each different task, the guidelines have to be constructed, and that can introduce noise. This may hinder the effectiveness of the method.”\n\nFirst, we hope to clarify how AutoGuide generates context-aware guidelines across different domains and tasks:\n- For each domain (e.g., ALFWorld, WebArena, WebShop), a new set of contexts and guidelines is extracted.\n- Within a domain, contexts and guidelines are extracted from training tasks and then used for test tasks.\n- For each state of a test task, the context is identified and the most helpful guidelines are selected.\n\nAutoGuide can experience noise from three sources: 1) noise in the context identification module, 2) noise in guideline selection module, and 3) noise in guideline extraction module. In this rebuttal, we conduct an additional analysis to demonstrate that the noise in each module is minimal.\n\nFor the noise sources of 1) and 2), we provide statistics on the rate at which the context identification module incorrectly identifies contexts and the guideline selection module selects incorrect guidelines on ALFWorld through manual evaluation. This analysis demonstrates that both the context identification and guideline selection modules produce results with low noise:\n\n| Module | Error Rate (%) |\n| :---------------- | :----: |\n| Context Identification | 3.2% |\n| Guideline Selection | 1.1% |\n\nFor the noise source of 3), it is challenging to determine the correctness of the guideline extraction module compared to the first two modules (i.e., context identification and guideline selection). Therefore, we qualitatively examine each extracted guideline in Figures 14-15. Our observations indicate that these guidelines contain helpful and effective knowledge, as demonstrated in our empirical experiments. Having a standardized metric for quantifying the quality of generated guidelines would be beneficial, and this is part of our planned future work, as noted in Appendix A (Limitation and Broader Impacts).\n\nLastly, AutoGuide employs principled methods to systematically address potential noise. Specifically, AutoGuide includes a context-matching procedure (Lines 128-131) to determine if the currently generated context matches any previously identified contexts. Additionally, AutoGuide uses the guideline selection module (Section 3.3) to choose up to $k$ relevant guidelines based on the current context. If no guidelines are deemed relevant, this guideline selection module can opt not to select any guideline.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RD4iteyHAO", "reviewer": "Reviewer_rEoJ", "summary": "The paper introduces AUTOGUIDE, a novel framework designed to enhance LLM agents' performance in unfamiliar domains like web navigation by automatically generating context-aware guidelines from offline experiences. These guidelines are expressed in concise natural language and follow a conditional structure, clearly describing the applicable context. The framework includes two key modules: the context identification module, which abstracts the agent's state into a concise description, and the guideline extraction module, which generates the desired guideline for a specific context. The evaluation demonstrates that AUTOGUIDE significantly outperforms competitive baselines in complex benchmark domains, including real-world web navigation.", "questions": "1. How is the number of training tasks determined for different domains?", "limitations": "The authors have discussed the limitations well.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method proposed is reasonable. It acts like a kind of \"library learning\" that summarizes past experiences into reusable modules and improves the agents' ability in unfamiliar domains.\n2. The experimental results are promising, showing the advantage of the proposed method.\n3. The writing is clear.", "weaknesses": "1. The author claims that their method and the in-context example selection module in ExpeL are orthogonal, so they only compared ExpeL with guidelines. This point requires experimental verification.\n2. The author should provide the number of tokens used or the number of times the LLM is called (in training and inference), to more comprehensively compare the various methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review and helpful questions. We have addressed each comment individually and conducted additional experiments based on your insightful feedback. We will also carefully incorporate your feedback into an updated version of our main paper and appendix.\n\n> “The author claims that their method and the in-context example selection module in ExpeL are orthogonal, so they only compared ExpeL with guidelines. This point requires experimental verification.”\n\nIn response to your insightful feedback, we conduct two additional experiments on ALFWorld: AutoGuide with the In-Context Example Selection Module and ExpeL with the In-Context Example Selection Module. The results are summarized in the table below, leading to the following conclusions:\n- AutoGuide can be readily expanded with other techniques, such as in-context example selection.\n- AutoGuide consistently outperforms ExpeL, both with and without the in-context example selection module.\n\n| Algorithm | Success Rate (%) |\n| :---------------- | :----: |\n| ExpeL | 59.0% |\n| ExpeL + In-Context Example Selection | 61.9% |\n| AutoGuide | 79.1% |\n| AutoGuide + In-Context Example Selection | 85.8% |\n\n> “The author should provide the number of tokens used or the number of times the LLM is called (in training and inference), to more comprehensively compare the various methods.”\n\nPlease refer to the tables below for the number of tokens used during training and inference. We compute these statistics using a subset of training and testing tasks on WebArena. For training, we observe that AutoGuide requires more tokens than ExpeL due to the additional context generation. However, please note that this context generation enables the filtering of irrelevant guidelines during inference. As a result, AutoGuide uses a comparable number of tokens to ExpeL (which provides all guidelines at inference to an LLM agent) and achieves higher test performance than ExpeL.\n\n| Algorithm | Average Token per Training Data |\n| :---------------- | :----: |\n| ReAct | N/A |\n| ExpeL | 3888.2 | \n| AutoGuide | 4873.4 |\n\n| Algorithm | Average Token per Inference Step |\n| :---------------- | :----: |\n| ReAct | 1734.6 |\n| ExpeL | 5125.5 |\n| AutoGuide | 4809.1 |\n\n> “How is the number of training tasks determined for different domains?”\n\nGenerally, the more training tasks available, the better, as a larger offline dataset can enhance AutoGuide's performance. In practice, we determine the number of training tasks for each domain based on several factors: the number of available offline experiences/tasks and budget considerations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nOOk23BsNS", "reviewer": "Reviewer_wFet", "summary": "This paper proposes a decision making framework called AudoGuide. It leverages domain knowledge from offline experiences to generate context-aware guidelines. This framework improves LLM agents in downstream decision-making tasks.", "questions": "See above.\n\nA typo: line 110 Appendix C.1: -> C.1.", "limitations": "See above.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.Design of context aware guidelines is ingenious.\n2. The proposed framework outperforms competitive baselines in complex benchmarks.", "weaknesses": "1. The two advantages when comparing with related works seems contradictory, e,g., inter-task knowledge compared with Reflexion (line 69), context-aware guideline compared with ExpeL (line 83). As the context-aware guideline and inter-task knowledge both contribute to the performance, does it imply that there're sharing contexts across different tasks? Then what's the percentage of inter-task sharing CONTEXT, and inter-task guideline retrieved by guideline selection?\n\n2. Analysis in section 4.2.Q1 (line 222-227) is inconsistent with Table 1. It seems to analyze \"ReAct+AutoGuide\" over \"ReAct+ExpeL\" by claming \"ExpeL approach helps ReAct by ... but is not as significant as AUTOGUIDE\". But I can't find \"ReAct+ExpeL\" setting in Table 1. It's confusing since following discussion also centers around this setting.\n\n3. Lack of explanation about the setting \"AUTOGUIDE + Reflexion\" in section 4. It's simply mentioned in section 4.1.2 (line 197). How is it implemented?\n\n4. \"Context Identification\" module necessitates handcrafted context definitions per task. Since guideline selection relies on context matching, will the number of demonstrations affect task performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the positive evaluation of our paper and providing constructive comments. We have addressed individual comments below and conducted additional evaluations. We will also carefully incorporate your feedback into an updated version of our paper.\n\n> “... As the context-aware guideline and inter-task knowledge both contribute to the performance, does it imply that there're sharing contexts across different tasks? Then what's the percentage of inter-task sharing CONTEXT, and inter-task guideline retrieved by guideline selection?”\n\nWe hope to clarify that both contexts and guidelines are shared across different tasks. Below, for each context, we count the number of tasks where the same context is shared and then calculate the average percentage in ALFWorld. We apply the same calculation for guidelines:\n\n| | Average Shared Percentage (%) |\n| :---------------- | :----: |\n| Context | 48.82% |\n| Guideline | 43.64% |\n\n> “... But I can't find \"ReAct+ExpeL\" setting in Table 1 …”\n\nThank you for your feedback. The \"ExpeL\" baseline in Table 1 corresponds to the \"ReAct + ExpeL\" baseline. We will correct this confusion by changing the \"ExpeL\" baseline in Table 1 to \"ReAct + ExpeL\" in the paper. \n\n> “Lack of explanation about the setting \"AUTOGUIDE + Reflexion\" in section 4”\n\nWe appreciate your feedback. We will provide more detailed information about our implementation of Reflexion, including the maximum number of trials (3), the GPT version (GPT-4-Turbo), the temperature setting (0.0), and the number of shots (2 for ALFWorld and 0 for WebShop), in Section 4.\n\n> \"Context Identification\" module necessitates handcrafted context definitions per task. Since guideline selection relies on context matching, will the number of demonstrations affect task performance?\n\nIn response to your insightful comment, we introduce a new experiment that eliminates the use of handcrafted context definitions in our context identification module. Specifically, we devise a general prompt (please refer to below “Context Proposal Prompt for WebArena”), where GPT-4-Turbo proposes a possible set of context summaries. Then, we use GPT to replace the human-designed list of context summaries (e.g., “On the main page of reddit” in Figure 7) with the ones proposed by GPT (e.g., “On a forum main page with navigation links and user menu”; please refer to below “GPT Proposed Contexts for WebArena”). We show that employing AutoGuide with GPT-proposed contexts in the few-shot examples (rather than human-designed contexts as in Figure 7) achieves comparable results to the original AutoGuide on WebArena:\n| Algorithm | Success Rate (%) |\n| :---------------- | :----: |\n| AutoGuide – Human-Designed Contexts | 47.1% |\n| AutoGuide – GPT-Proposed Contexts | 46.0% |\n\nWe additionally conduct a new experiment on WebArena, varying the number of demonstrations in the human-designed context identification prompt. The result shows that our context identification module is robust to the number of demonstrations:\n| Number of Demonstrations ($n$) | Success Rate (%) |\n| :---------------- | :----: |\n| $n=1$ | 44.9% |\n| $n=3$ | 47.1% |\n| $n=5$ (Original Hyperparameter) | 47.1% |\n\n> ‘A typo: line 110 Appendix C.1: -> C.1.”\n\nThank you for noting this. We will fix this typo in the paper. \n\n---\n### Context Proposal Prompt for WebArena\n```\n(Instruction) You are an autonomous intelligent agent for general-purpose text-based interactive tasks. You will be provided with a few trajectories of tasks from an interactive environment, each trajectory includes a goal, and a sequence of observations and actions. For each observation, we want you to abstract the underlying context at that step into a short, concise, and general context summarization, which describes the context given the goal and target.\nFor example: \n1. On the main page of Google Scholar with a [search] box. \n2. Attempted to chop an object but failed, therefore getting an error message as observation. \nNotice: Your summarization must be general enough, so that similar situations, especially similar ones in consecutive steps, get summarized into the same context identification. Therefore, please do not include any task-specific information, for example, do not mention a specific user name, specific product category, or specific object type, instead conclude them to a broad and general category: a user, a product or an object. Please refer to attempted actions in the past tense and use the same words for observation and actions in the trajectories. Please list all the unique context summarizations in a python list: Answer: ['context_summarization1', 'context_summarization2', 'context_summarization3', ...]. Please don't put a context_summarization that doesn't appear in the provided trajectories. \n\n(Trajectory Format Specification) The goal is after \"Intent:\", observations are after \"Observation: \", including an accessibility tree of a focused webpage and its corresponding URL, and actions are after \"Action: \".\n\n(Input) Here is the input: {Trajectories}\n```\n\n### GPT Proposed Contexts for WebArena\n```\nOn a forum main page with navigation links and user menu\nOn a forum list page with navigation links and sorting options\nOn a forum category page with navigation links and submission options\nOn a submission preview page with options to edit or submit\nOn an alphabetical list of forums with navigation options\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a method (autoguide) to improve LLM agent performance by generating context-aware guidelines. 

Strengths: 
S1. All reviewers like the idea of context-aware guideline. 
S2. Strong experimental results on multiple benchmark tasks. 

Weakness: 
W1. Some connections and comparisons with ReAct, Reflexion, ExpeL need further evidence. 

Overall, it is a clearly-written paper with an interesting idea (context-aware guideline).
Decision: Accept (poster)