AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents

Authors: yao  fu
dong-ki  kim
jaekyeom  kim
sungryull  sohn
lajanugen  logeswaran
kyunghoon  bae
honglak  lee
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-02 03:25:59

Abstract: Recent advances in large language models (LLMs) have empowered AI agents capable of performing various sequential decision-making tasks. However, effectively guiding LLMs to perform well in unfamiliar domains like web navigation, where they lack sufficient knowledge, has proven to be difficult with the demonstration-based in-context learning paradigm. In this paper, we introduce a novel framework, called AutoGuide, which addresses this limitation by automatically generating context-aware guidelines from offline experiences. Importantly, each context-aware guideline is expressed in concise natural language and follows a conditional structure, clearly describing the context where it is applicable. As a result, our guidelines facilitate the provision of relevant knowledge for the agent's current decision-making process, overcoming the limitations of the conventional demonstration-based learning paradigm. Our evaluation demonstrates that AutoGuide significantly outperforms competitive baselines in complex benchmark domains, including real-world web navigation.

Main paper: 
 introduction: 
 Recent advances in large language models (LLMs) have empowered AI agents to address various sequential decision-making tasks and applications [1, 2] . The foundation of these successes involves the planning and reasoning capabilities of pre-trained LLMs, enabling agents to execute effective policies [3, 4] . The predominant approach to leveraging these (typically closed source) models for sequential decision making tasks is to provide demonstrations in the form of in-context examples. However, direct application of this learning paradigm can be limited, especially in target domains where the LLM has insufficient prior knowledge such as in web navigation, where LLM agents generally achieve low success rates due to diverse and dynamic contents [5] [6] [7] [8] . Providing all available experiences as demonstrations to an agent can further be unsuccessful due to context length limitations, prompt sensitivity, and difficulty with complex reasoning [9] [10] [11] [12] .On the other hand, LLMs excel in interpreting concise instructions provided as natural language, an ability that is also reinforced in the instruction-tuning phase of LLMs. Inpired by this, we explore data-driven strategies that leverage offline experiences to extract actionable knowledge to help guide LLM agents. As offline experiences implicitly convey valuable knowledge about desirable and undesirable policies in domains, they promise to serve as a useful resource for improving an LLM agent's decision-making in situations where the pre-trained LLM lacks understanding. Despite this potential benefit, a critical challenge lies in effectively extracting the implicit information embedded in offline data.Figure 1 : AUTOGUIDE aims to extract the implicit knowledge embedded in offline experiences and help the decision-making process of an LLM agent. Specifically, our method generates a comprehensive set of context-aware guidelines from offline data and explicitly identifies when each guideline is applicable by generating its corresponding context. Our context-aware guidelines enable providing pertinent guidelines at test time by identifying the context of the current trajectory, leading to correct decision-making compared to baselines without context-aware guidelines.To address the challenge of extracting knowledge from offline data, we propose a novel framework, called AUTOGUIDE. Specifically, AUTOGUIDE automatically derives a comprehensive set of contextaware guidelines from offline experiences. Our method applies these context-conditional guidelines to enhance the performance of an LLM agent by retrieving guidelines relevant to the agent's current state and incorporating them into the prompt during testing (see Figure 1 ). Notably, we generate context-aware guidelines in concise natural language statements, effectively compressing knowledge in offline data. Moreover, context-aware guidelines clearly describe the contexts where they are applicable, so AUTOGUIDE enables an LLM agent to select pertinent guidelines for its current decision-making process. As a result, AUTOGUIDE achieves the highest success rates compared to competitive baselines in complex sequential decision-making benchmark environments.Our contribution. In summary, we present the following main contributions in this paper:• Principled method based on context-aware guidelines (Section 3): We develop two modules to automatically generate context-aware guidelines from offline experiences: the context identification module for identifying the context of a given trajectory, and the guideline extraction module for extracting a desired guideline corresponding to that context. The outcome is a set of domain knowledge in concise natural language that enhances decision-making by providing pertinent information.• Comprehensive evaluation of AUTOGUIDE (Section 4.2): We show AUTOGUIDE's capability in extracting helpful context-aware guidelines in various interactive benchmark domains, including navigating real-world web domains (e.g., GitHub). Our results highlight the effectiveness of AUTOGUIDE, which significantly outperforms baselines without context-aware guidelines.• Analyses with important perspectives (Section 4.3): We study various aspects of AUTOGUIDE, such as the significance of determining the applicability of each guideline based on generated contexts. We also investigate the generalization ability of context-aware guidelines and demonstrate that our guidelines enhance the performance across out-of-domain tasks. 
related work: 
 LLM-based agents. Language models have recently been shown to possess strong priors for sequential decision-making tasks, which has given rise to LLM-powered agents [1, 2, 13, 14] . Agents In this example, the two trajectories start deviating from each other at t = 0. The context identification module generates a description of the context at t = 0 given τ i :0 , and the guideline extraction module generates the corresponding guideline for that context. need to possess various skills to be effective in practice including planning [3, 15, 16] , reasoning [4, 17] , tool manipulation [18] [19] [20] [21] , code generation [22, 16] , among others. In this work, we focus on building effective agents for web [23, 8] and embodied [24] environments.Self-reflection from past experiences. An important capability for agents to succeed is the ability to learn from past experiences and update their behavior based on feedback. Self-feedback [25] [26] [27] has emerged as an effective technique where a model inspects its own incorrect predictions, reflects on it to identify what went wrong and attempts to improve its prediction. While self-feedback provides intra-task (i.e., per-episode) knowledge within a task based on immediate feedback, our approach offers an orthogonal and complementary aspect of inter-task knowledge (over multiple tasks) by considering multiple train tasks in offline data. AutoGuide enhances learning efficiency and credit assignment by utilizing detailed feedback from multiple tasks. However, these self-feedback approaches are complementary to AUTOGUIDE and can be used in conjunction with our approach, as shown in our experiments (see Section 4.2).Leveraging natural language guidance. Natural Language can be a rich source of information for agents to learn to act efficiently. Prior work has explored the notion of learning from humanwritten text manuals, which describe details about the environment [28] [29] [30] . Recent work has explored automatically generating such guidance in the form of chain-of-thought reasoning [4, 23] , which emulates a thought process or rationale for agent's predictions. In contrast to approaches which generate such guidance dynamically on the fly by imitating example guidance demonstrations provided by a human, our approach carefully compares trajectories in offline data to generate appropriate guidance and uses these guidelines for predicting better actions. ExpeL [31] proposed a related approach to derive guidelines. In contrast to ExpeL, where all guidelines are provided to an agent as a prompt, our guideline selection process is contextual, where guidelines relevant to the agent's current state are retrieved and used for prediction. We show that this substantially improves over ExpeL's non-contextual guideline-based approach.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We outline potential limitations of AUTOGUIDE in Appendix A. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
autoguide: principled method based on context-aware guidelines: 
 Our work is motivated by the increasing availability of offline experiences that agents or humans naturally accumulate through their interactions with the environment. AUTOGUIDE aims to leverage this offline data to improve the decision-making of an LLM agent by generating helpful context-aware guidelines. This section details how AUTOGUIDE automatically constructs these guidelines and applies them to guide action generation at test time.Formally, AUTOGUIDE is given offline data D train = (τ 1 , ..., τ N ) that consist of N trajectories from training tasks. Each trajectory τ = (x 0 , a 0 , r 0 , ..., r T ) is a sequence of observations, actions, and rewards following the partially observable Markov decision process [32] . The return of a trajectory is defined as the sum of rewards obtained throughout the trajectory: R(τ ) = T t=0 r t . The objective of AUTOGUIDE is to distill knowledge from offline experiences into a useful natural language format, such that the extracted information helps to maximize the expected return E τ [R(τ )] during test time. AUTOGUIDE generates a set of context-aware guidelines by utilizing pairs of contrastive trajectories from offline data. Each context-aware guideline is expressed in concise natural language and follows a conditional structure, clearly describing the context in which the guideline is applicable. Intuitively, contrasting a pair of trajectories with different returns provides important information about when and which actions are effective or ineffective in maximizing expected returns. Building on this insight, we develop two modules for automatically extracting context-aware guidelines (see Figure 2 ): Context identification module. This module is responsible for abstracting the given partial trajectory into its context, a concise natural language description of the agent's state. More specifically, for a timestep t and the corresponding trajectory τ i :t := (x 0 , a 0 , ..., x t ), we prompt LLMs to clearly describe the agent's status:EQUATIONOur prompt templates for the context identification module are shown in Appendix C.1.Guideline extraction module. This module aims to generate a desired guideline corresponding to a specific context. Let τ i + and τ i -represent a contrasting pair of trajectories for the same task i in offline data D train , where R(τ i + ) > R(τ i -). We want to contrast the pair of trajectories to find desired behaviors at an important timestep. To do this, we compare these two trajectories to find the deviation timestep t at which they begin to diverge due to different actions. Then we apply the context identification module to summarize the context for the shared part of the trajectory τ i :t . Eventually, we extract a useful natural language guideline by examining the paired contrastive trajectories τ i + and τ i -with respect to the context:EQUATIONwhere we refer to Appendix C.2 for our prompt template. As an example, the paired trajectories in Figure 2 deviate from timestep t = 0, for which the context is summarized as On the Reddit main page. This module then generates the following context-aware guideline: When on the Reddit main page, if you want to navigate to a specific forum, you should click on "link 'Forums'" located right above "link 'Wiki'".Construction of context-aware guidelines. We collect context-aware guidelines G by iterating through available pairs in the paired offline data and organize the guidelines in a dictionary format, using the context as the key and the corresponding guidelines as the value (see Algorithm 1). In particular, we observe that the context identification module occasionally produces contexts that describe the same situation but are expressed slightly differently. To minimize redundancy, we employ an LLM to determine if the current context corresponds to any previously identified context.Task: Put a pan on the diningtable.Task: Post a review of my recent reading "Gone with the wind" in r/books with comment "It's a book with history". WebShop [33] , WebArena [8] , and multi-modal real-world websites. Graphic credit: [34, 33, 8, 35] .If a match is found, we reuse the existing context; otherwise, we introduce a new context into our dictionary G. The specific prompt template for this context-matching procedure is in Appendix C.3.After extracting a set of context-aware guidelines G from offline experiences, our method employs these guidelines to enhance the decision-making of an LLM agent during testing. At each timestep, AUTOGUIDE identifies the CONTEXT of the current test trajectory τ up to timestep t (which represents the agent's interactions up to the current time-step) using our context identification module M context . Our guideline selection module M select then selects relevant guidelines for CONTEXT from G. More specifically, the module applies CONTEXT as the key to fetch a set of possible guidelines G[CONTEXT]. If there are more than k guidelines in G[CONTEXT], M select prompts an LLM to choose top-k guidelines for the specific τ :EQUATIONwhere Appendix C.4 details the prompt template for this selection procedure. Subsequently, AU-TOGUIDE incorporates both the context and relevant guidelines into the agent's action generation prompt. Therefore, the agent selects an action by considering the provided context and guidelines (see Figure 1 for an example). This process iterates until the end of the test trajectory (see Algorithm 2) .Key benefits of AUTOGUIDE. First, the extraction of context-aware guidelines in AUTOGUIDE offers the inherent benefit of providing relevant guidelines for the context of interest. This capability is important since neglecting the specific context in which a guideline applies can confuse the agent's decision-making process. The second key benefit is the generation of concise natural language guidelines, which can be seamlessly incorporated into any prompt-based LLM agent. Lastly, AUTOGUIDE generates guidelines at the individual context level rather than at the trajectory level.Given that a single incorrect action can lead to a complete failure, it is essential to provide detailed assistance in each action selection process. With these advantages, we demonstrate in the next section that our approach significantly enhances the performance of LLM agents.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: This paper does not include theoretical contributions. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
evaluation: 
 This section demonstrates the efficacy of AUTOGUIDE by conducting experiments on a diverse suite of sequential decision-making benchmark domains. We also perform important analyses about AUTOGUIDE, such as the ablation study of different AUTOGUIDE components, comparison to in-context learning, and generalization to out-of-domain tasks. We refer to Appendix B for additional experimental details.We consider the following interactive sequential decision-making benchmarks to study various aspects of AUTOGUIDE (see Figure 3 ):• ALFWorld [24] : In this embodied benchmark, an LLM agent interacts with an environment to carry out household tasks, such as placing a pan on the dining table. Observations and actions are expressed in natural language statements, and the agent must navigate through the space and manipulate objects to successfully complete the tasks.• WebShop [33] : This interactive web environment simulates the task of online shopping on an e-commerce website. The agent's goal is to understand a text instruction and buy a product that meets specified criteria. This involves querying the website's search engine, understanding the descriptions and details of each item, and selecting necessary options.• WebArena [8] : This web-based benchmark introduces realistic environments by replicating the functionality and data found in popular web domains (e.g., Gitlab, Reddit, Wikipedia). Compared to WebShop, WebArena presents more challenges and difficulties for an LLM agent due to its large observation and action space, along with tasks that involve longer planning horizons. We focus oe the Reddit domain for the main WebArena experiments.• Real-world multi-modal websites: Finally, we consider evaluating AUTOGUIDE on a variety of real-world website tasks. These span from a collaborative software development platform (e.g., GitHub) to a flight search engine (e.g., Google Flights) and an online education platform (e.g., Coursera). Please refer to Appendix B.4 for example tasks. In particular, in comparison to WebShop and WebArena, we design our tasks to be multi-modal such that the agent must consider both visual (e.g., images) and textual information (e.g., HTML) to complete these tasks.We compare AUTOGUIDE against the following baseline approaches to study the effect of contextaware guidelines (refer to Appendix B for more details):• ReAct [23] : This LLM-based planning method integrates reasoning and acting to address sequential decision-making tasks. However, it does not leverage offline experiences and thus suffers from the limited understanding of pre-trained LLMs in downstream domains.• ExpeL [31] : This method also extracts natural language knowledge from offline data. However, it fails to consider the applicability of guidelines and does not generate context-aware guidelines.Instead, it provides all guidelines to an LLM agent without filtering out irrelevant ones based on the current context. ExpeL has two contributions, the guideline generation and in-context example selection module. Because the latter is orthogonal to our analysis and can be seamlessly combined with our method, we consider ExpeL with guidelines in our experiments.• Reflexion [27] : This approach converts environmental feedback into text statements to assist an LLM agent (e.g., ReAct) in the next trial. The baseline generates valuable feedback about solving a specific test task. We demonstrate how context-aware guidelines derived by AUTOGUIDE can be combined with the feedback.We collect offline experiences either by running ReAct and Reflexion, or incorporating human demonstrations. We use ReAct with GPT-3.5-turbo as our base LLM agent for WebShop and ALFWorld and GPT-4-turbo for WebArena. For each benchmark, we apply the same GPT model for action generation, context identification, and guideline selection. We extract context-aware guidelines from offline data with GPT-4-turbo and evaluate their effectiveness by applying them to the test set with non-overlapping tasks. We refer to Appendix B for more details. ALFWorld [24] WebShop [33] WebArena [8] Success Reflexion is done by GPT-4-turbo for at most 3 trials. In our experiments, due to token limit of GPT, we did not experiment with Reflexion on WebArena tasks.To answer this question, we compare methods on ALFWorld, WebShop, and WebArena benchmarks. The performance on the test datasets is presented in Table 1 . There are three notable observations:1. Effectiveness of context-aware guidelines. Our approach surpasses baseline performance in both ALFWorld and WebShop, achieving the highest test rewards and success rates in Table 1 . These results highlight the effectiveness of employing context-aware guidelines in language-based decisionmaking domains. To further examine the action selection differences among ReAct, ExpeL, and our method, we present their trajectories in Figure 4 . We observed that ReAct makes common mistakes such as trying to take soapbar that is not visible, or taking a soapbottle instead of soapbar due to their similar names. Both ExpeL and AUTOGUIDE improve on this by extracting guidelines from similar mistakes in the offline experience. However, ExpeL often erroneously applies incorrect guidelines SR↑ k = 0 30% k = 1 42% k = 2 46% k = 3 47% k = 5 43%Table 4 : Ablation study of AUTOGUIDE using various top-k values.due to the availability of all guidelines at each timestep. In Figure 4 , ExpeL mistakenly attends to the second guideline "ensure to specify the item's number and location ...", leading to wrong reasoning and action. AUTOGUIDE presents relevant guidelines at necessary moments, enabling accurate task completion by avoiding the mistakes seen in ExpeL and ReAct.2. Importance of providing pertinent knowledge. ExpeL approach helps ReAct by extracting knowledge from offline experiences, but its impact is not as significant as AUTOGUIDE. Recall that for ExpeL, the guidelines are neither generated for specific contexts at training time nor selected to only provide context-aware guidelines at test time. As a result, irrelevant guidelines can be introduced to an agent, potentially causing confusion for the agent. Consequently, the result highlights the significance of providing relevant guidelines conditioned on contexts for LLM agents.3. Scalability to complex environments. We conduct experiments on WebArena-Reddit, which features more diverse tasks on realistic and complex websites requiring longer action sequences. This domain has a larger observation space and a more complex action space (e.g., scrolling). Table 1 presents the results, where AUTOGUIDE achieves the highest success rate with a significant margin when compared to ReAct and ExpeL. We observe that ReAct scores low task success rate (8.0%) in WebArena due to the complex observation and action spaces and longer task horizon. In ExpeL, the issue of presenting all guidelines to an agent is exacerbated in the WebArena compared to simpler environments like ALFWorld and WebShop. WebArena's wide variety of tasks across different domains requires a larger number of guidelines to cover the knowledge needed for all tasks and domains. This results in either an overload of irrelevant guidelines that could mislead the agent or a lack of crucial information when the number of guidelines is limited, as suggested in ExpeL [31] . In contrast, AUTOGUIDE achieves a more significant performance enhancement (47.1%) compared to ExpeL (21.8%) by efficiently providing pertinent guidelines and minimizing the burden on context capacity. We refer to Figure 14 for a list of example contexts and guidelines.Q2. How does AUTOGUIDE perform when combined with test-time self-feedback approaches?Our context-aware guidelines effectively provide inter-task knowledge by considering multiple tasks in offline data. Meanwhile, self-feedback methods (e.g., Reflexion) offer intra-task knowledge based on environmental feedback during test time. In this question, we explore the effectiveness of integrating both inter-task and intra-task information. The results presented in Table 1 demonstrate that the combination of AUTOGUIDE with Reflexion achieves the highest performance in the WebShop and ALFWorld benchmarks. Hence, we find that our context-aware guidelines positively complement the intra-task knowledge of Reflexion. Another observation from Table 1 is that, while ExpeL + Reflexion outperforms ExpeL alone, this combination is not as effective as other approaches. This limitation may stem from ExpeL introducing irrelevant knowledge, potentially leading to conflicts with Reflexion's feedback and having an adverse impact on the decision-making process.Q3. Can AUTOGUIDE generate context-aware guidelines for multi-modal inputs?Going beyond text-only inputs is an essential step toward building capable agents for solving realworld environments and tasks. We test AUTOGUIDE in a complex multi-modal setting, where each observation includes image and text information. Specifically, we introduce a set of real-world website navigation tasks in 3 domains: GitHub, Google Flights, and Coursera. For these multi-modal tasks, we employ the Set-of-Marks (SoM) agent [36, 5] as our base method. The SoM prompting improves the visual grounding capabilities of large multi-modal models such as GPT-4V by adding visually distinguishable marks to image inputs [36] . We apply AUTOGUIDE with GPT-4V to generate natural language context-aware guidelines from collected trajectories with both image and text observations. 2 shows the effectiveness of AUTOGUIDE, demonstrating its generalization ability to complex real-world multi-modal settings. We refer to Figure 15 for example context-aware guidelines. Table 3 shows that, while increasing the number of in-context examples for ReAct gradually improves performance, there is a plateau at a certain number of shots. Additionally, ReAct with more than 6 shots often exceeds the token limit of GPT-3.5-turbo. These results indicate that directly inputting raw trajectories into ReAct for in-context learning is not an effective way to fully leverage offline data. In contrast, AUTOGUIDE extracts knowledge from entire training trajectories by summarizing them into concise context-aware guidelines, making them easy to integrate with prompt-based agents.We conducted an ablation study on WebShop using various values of k in Table 4 . We find that employing context-aware guidelines consistently outperforms the no-guideline baseline (k = 0; ReAct). The k = 3 yields the best performance. The largest k value of 5 can lead an LLM agent to overthink, potentially resulting in a slight decrease in performance. Conversely, a smaller k, like k = 1, may cause LLM to overlook additional helpful guidelines, leading to slightly worse performance.We conduct an experiment to further demonstrate AUTOGUIDE's out-of-domain capability across different domains but relevant tasks. We extract context-aware guidelines from WebShop and apply them to WebArena-Shopping, which is a distinct domain with variations in observation/action spaces, task intentions, and episodic horizons. For this domain adaptation case, we additionally incorporate a grounding module to align the context-aware guidelines from WebShop to WebArena's observations based on GPT-4-Turbo. As shown in Table 5 , the transferred guidelines bring a notable improvement in success rates compared to the ReAct baseline in WebArena Shopping.We evaluate the impact of different components within AUTOGUIDE on its performance in WebShop, as detailed in Table 6 . We examine two variants: ReAct+CI and ReAct+GES. The ReAct+CI, which incorporates contexts into observations without guidelines, shows improvement over ReAct. This suggests that contexts enhance decision-making by verifying the current state before action selection. ReAct+GES, which generates guidelines from trajectories without contexts and employs GPT-3.5-turbo for guideline selection, also enhances performance but is less effective than the full AUTOGUIDE. This indicates that choosing relevant guidelines based on the trajectory alone is more challenging than using contexts. Therefore, integrating both context summaries and guidelines is crucial for maximizing the benefits of AUTOGUIDE.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Section 4.1 and Appendix B provide detailed information (e.g., model versions, hyperparameters, number of training data) for reproducing our experimental results. Additionally, Appendix C contains our prompt templates. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 We present AUTOGUIDE, an effective framework for exploiting important domain knowledge from offline experiences for improving decision-making with pre-trained LLMs. We proposed to generate context-aware guidelines that can be incorporated into prompts for LLM agents. As AUTOGUIDE extracts the guidelines by contrasting trajectories in offline data, the resulting context-aware guidelines carry critical information for preventing failures in the domains. For inference, it provides the guidelines pertinent to each of the different context that LLM agents encounter, which can make pre-trained LLMs strong decision-making agents in the downstream domains. Empirically, we showed that AUTOGUIDE outperforms strong baselines by a large margin and achieves outstanding performance in decision-making benchmarks.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We intend to release the source code upon acceptance. 
acknowledgements: 
 This work was supported in part by LG AI Research.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We detail our experimental settings in Section 4.1 and Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We provide the average rewards and success rates calculated across test tasks in our results, but we do not include the variance.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Our experiments are based on GPT, and we detail the versions of GPT in Appendix B.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper makes the algorithmic contribution and does not release new data or models. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We reference the original papers and sources throughout the paper. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Limitation. The performance of AUTOGUIDE depends on the diversity of offline experiences. As such, one important direction for improvement is to automatically collect diverse offline experiences through continual learning, where we iteratively generate guidelines and use them to gather more trajectories with high rewards. Another avenue is the need for quantifying the quality of generated contexts and guidelines. Currently, apart from applying context-aware guidelines to ReAct and measuring test time performance, there lacks a standardized method for quantifying the quality of generated contexts and selected guidelines. Introducing a quantifiable metric to approximate the quality could pave the way for new optimization approaches such as reinforcement learning.Broader impact. This paper introduces research aimed at enhancing the decision-making capabilities of LLMs. In terms of societal impact, while we develop a generic LLM-based autonomous agent, having biased offline datasets may lead to making decisions with suboptimal outcomes. Additionally, autonomous agents may be misused for malicious applications. To mitigate these risks, potential solutions would include diversifying datasets, implementing ethical oversight, ensuring transparency and accountability, engaging with stakeholders for a broader perspective, and incorporating security measures to prevent misuse. We believe that the research community, including ourselves, should responsibly advance LLM-based agent research, prioritizing societal well-being and ethical considerations. Each task in ALFWorld starts with a description of the specific environment and the goal to achieve. At each timestep, an agent can choose one of the following actions to interact with the objects and receptacles in the environment:• go to [recep] • take [object] from [recep] • put [object] in/on [recep] • open/close/use [recep] • clean/heat/cool [object] with [recep]Alternatively, the agent can generate a think action for planning and reflection, which helps with decision-making but does not change the environment itself. After one action is performed, the environment returns an observation that describes view changes.Following ReAct, we concatenate a list of (observation, action) pairs to show the entire trajectory up to the current timestep for LLM agents to generate the next action. We experiment on 134 unseen test tasks with 6 categories of pick and place, pick clean then place, pick heat then place, pick cool then place, look at obj, and pick two obj. For each task, the agent is allowed to take a maximum of 50 actions.For ALFWorld tasks, we follow the same setting as ReAct by providing 2 in-context examples for each of the 6 task categories. The original results of ReAct in their paper are produced based on Text-Davinci-002. However, this GPT version is no longer available, so we apply gpt-3.5-turboinstruct instead to generate actions. For ExpeL, we directly take the guidelines from their appendix and append them to the ReAct agent at test time.We run the first 100 training tasks of ALFWorld to collect (τ + , τ -) pairs with ReAct+Reflexion and extract context-dependant guidelines on the collected data. For context identification, we provide WebShop provides an e-commerce environment, where the objective is to find and buy the product that matches the task-specific Instruction. The agent can select one of the following actions to perform:• search[query] • click[button]Following ReAct, the agent can generate think actions to do planning or reflection. After buying a product, the environment returns a reward showing how well the bought product matches the target one in type, price, buying options, and attributes. The reward is calculated by:r = r type • |U att ∩ Y att |+|U opt ∩ Y opt |+1[y price ≤ u price ] |U att |+|U opt |+1where y is the bought product and u is the target product. Same as ALFWorld, for WebShop, the agent takes (obs t , act t ) pairs for every previous timestep t as input to generate the next action.Following ReAct, experiments are done in a one-shot setting. We apply gpt-3.5-turbo-0613 to generate actions, but when the token number exceeds the token limit (for example, for the n-shot ReAct experiments in table 1), we use the 16k version of gpt-3.5-turbo-0613 instead. For ExpeL, we could not find how many training tasks the framework used for training. Therefore, we directly apply the guidelines from the appendix of their paper at test time. We only consider ExpeL with guidelines, not ExpeL with in-context example selection in our experiments for a fair comparison. The in-context example selection method is orthogonal to our work and can be easily combined with our method. For Reflexion, as shown in their paper, their 2-shot Reflexion prompt does not work well on WebShop. Therefore, we re-write the prompt and apply gpt- WebArena provides web-based benchmark environments that closely follow the data and functionality of real-world websites. Unlike other benchmarks like WebShop that provide clean text of website information as observation, WebArena's webpage content is represented as an accessibility tree, which is a subset of the DOM tree with useful elements of a webpage. For our expeirments, we focus on WebArena Reddit, which simulates the real Reddit websites with users, forums, and posts with abundant text information.For each task in WebArena, the agent is expected to achieve a task-specific intent. At each timestep, WebArena provides a list of opened tabs, the accessibility tree of the focused webpage, and the URL of the current page as observation. For WebArena, each observation is long. Therefore, following the baseline in WebArena, at each timestep, we only provide the observation of the current timestep to the agent. We additionally provide up to 5 past actions for the agent to understand what it did in the past. The allowed actions in WebArena include the following:• goto [url]• click [element id]• type [element id] [text] [1 for enter or 0 for not enter]• press [key combination]• scroll [up or down]• go backThe maximum allowed number of actions for a single task is 20. Note that WebArena does not provide training tasks, but the work provides 19 demonstrations for Reddit, each of a different category. Therefore, we set these 19 tasks as the training tasks and then test on the rest 87 tasks.We directly run the two-shot ReAct-style baseline in the official codebase of WebArena using gpt4preview-1106. For ExpeL, the original paper does not include experiments on WebArena, therefore we try our best to implement our own version and run on the same training tasks as our method.We directly run ReAct on the tasks with τ + to collect τ -actions and generate guidelines correspondingly. We provide a 5-shot prompt for the context identification module, which is shown in Figure 7 . We follow the action space design of Visual WebArena [5] , which has the following action types available:As the real websites are constantly and dynamically changing, we evaluate the completed task with human experts.We directly run the two-shot SoM algorithm in the official codebase of Visual WebArena. The only modification we made from the original codebase is the bounding box detection algorithm, in which we further filter invisible bounding boxes from the list and add the list elements as interactable elements to consider.We provide a total of 6 training tasks (3 for GitHub, 2 for Google Travel, and 1 for Coursera), on which we collect human demonstration as τ + 's and run SoM to collect τ -'s. From the pairs with multi-modal observations we generate text-based guidelines to guide action selection. Both context identification and guideline extraction are done by gpt-4-vision-preview, and we provide a 3-shot prompt for context identification. All parameter details are shown in Table 10 .Allowed We present our prompt templates for the context identification module M context (Equation ( 1)) for WebShop, ALFWorld, and WebArena in Figures 5 to 7 , respectively. In ALFWorld, there exist six categories of tasks, and we use context identification prompting with 2-shot examples for each task, following the practice by [23] . Figure 6 shows one example for the pick and place tasks.Figures 8 to 10 detail our prompt templates M guideline for extracting guidelines (Equation ( 2)) in the WebShop, ALFWorld, and WebArena domains.Figure 11 shows our prompt template for matching the generated context with one of the existing contexts if there is any similar context, for the construction of the set of context-aware guidelines (Section 3.2) and the retrieval of relevant guidelines during testing (Section 3.3) in all the three domains: WebShop, ALFWorld, and WebArena.For selecting only k most relevant guidelines in case there are more corresponding context-aware guidelines during testing (Equation ( 3)), we use the prompt with Figure 12 for WebShop and ALFWorld and Figure 13 for WebArena.In Figures 14 and 15 , we show a list of possible contexts and context-aware guidelines on WebArena and real-world websites, respectively. Your task is to: find some apple and put it in sidetable. > think: To solve the task, I need to find and take an apple, then put it in sidetable. OK. > think: First I need to find an apple. An apple is more likely to appear in fridges (1), diningtables (1-3), sidetables (1), countertops (1), sinkbasins (1), garbagecan (1) . I can check one by one, starting with fridge 1.The fridge Please put your answer in this format: Reasoning: ... Guideline: ...You just finished a task but failed. For this failed task, we provide a human demonstration for you. Please compare the demonstration with your generated action at each step, reason about the intention of the correct action, and then geneate an action guideline for future tasks to avoid the same mistake and make the future tasks successful.Here's the information you'll have: 2. The action you generated in the failed run.3. The correct action that you should take.4. Demonstration actions in later steps on the same page.Based on the information, please generate a short and concise guideline that guide you to issue the correct action. Important: The guideline should be general enough to generalize to all similar tasks, not only this task. Therefore do not include any task-specific information in your guideline, for example a user name, a specific forum, the specific text you want to enter, or any number ID in [] in front of each element, for example [123], the numbers are randomly generated therefore never include them in your guideline. However, for other non-specific elements like "link 'Forums'" or "button 'Create submission'", you should specifically include them in the exact text in your guideline.When referring to a url in your guideline, specify it as detailed as possible, only replace the task specific information as a palceholder, for example, replace a forum iphone with <forum_name> and specify the url in full, starts with http://. The guideline should be less than 128 tokens. Please refer to "the previous actions" and "Demonstration actions in later steps" to generate more accurate descriptions of your purpose and the sequence of actions to achieve the purpose. make sure to emphasize the order of the actions, do not miss any single action, and put them in 1. 2. {Task description} A task trajectory can be long. Therefore the assistant summarizes the status of each step.For different task with the same status, the summarization should be the same, therefore please ignore any information about instructions or products.You will be provided with the following: 1. A list of summarizations the assistant saw in the past. Justification: In the abstract and introduction, we clearly outline our main contribution: extracting context-aware guidelines from offline data to enhance the decision-making of LLM-based agents. We support the effectiveness of our method through empirical evaluations in Section 4.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: This paper proposes a method (autoguide) to improve LLM agent performance by generating context-aware guidelines. 

Strengths: 
S1. All reviewers like the idea of context-aware guideline. 
S2. Strong experimental results on multiple benchmark tasks. 

Weakness: 
W1. Some connections and comparisons with ReAct, Reflexion, ExpeL need further evidence. 

Overall, it is a clearly-written paper with an interesting idea (context-aware guideline).

Reviews: review: paper_summary: paper_summary: This paper introduces AUTOGUIDE, a framework for enhancing large language model agents' performance in sequential decision-making tasks by automatically generating context-aware guidelines from offline experiences. The method consists of two main components: a context identification module and a guideline extraction module. These modules work together to create concise, natural language guidelines that are relevant to specific contexts. At test time, AUTOGUIDE identifies the current context and selects pertinent guidelines to assist the LLM agent in decision-making. The authors evaluate AUTOGUIDE on various benchmark domains, including ALFWorld, WebShop, WebArena, and real-world multi-modal websites.
strength_weakness: strength_weakness: - The method can be summarized as ReAct with context identification and guideline retrieval. And from the ablation study, it seems like the GES component is really important. For each different task, the guidelines have to be constructed, and that can introduce noise. This may hinder the effectiveness of the method.
questions: questions: N/A
limitations: limitations: N/A
paper_summary: paper_summary: The paper introduces AUTOGUIDE, a novel framework designed to enhance LLM agents' performance in unfamiliar domains like web navigation by automatically generating context-aware guidelines from offline experiences. These guidelines are expressed in concise natural language and follow a conditional structure, clearly describing the applicable context. The framework includes two key modules: the context identification module, which abstracts the agent's state into a concise description, and the guideline extraction module, which generates the desired guideline for a specific context. The evaluation demonstrates that AUTOGUIDE significantly outperforms competitive baselines in complex benchmark domains, including real-world web navigation.
strength_weakness: strength_weakness: 1. The author claims that their method and the in-context example selection module in ExpeL are orthogonal, so they only compared ExpeL with guidelines. This point requires experimental verification.
2. The author should provide the number of tokens used or the number of times the LLM is called (in training and inference), to more comprehensively compare the various methods.
questions: questions: 1. How is the number of training tasks determined for different domains?
limitations: limitations: The authors have discussed the limitations well.
paper_summary: paper_summary: This paper proposes a decision making framework called AudoGuide. It leverages domain knowledge from offline experiences to generate context-aware guidelines. This framework improves LLM agents in downstream decision-making tasks.
strength_weakness: strength_weakness: 1. The two advantages when comparing with related works seems contradictory, e,g., inter-task knowledge compared with Reflexion (line 69), context-aware guideline compared with ExpeL (line 83). As the context-aware guideline and inter-task knowledge both contribute to the performance, does it imply that there're sharing contexts across different tasks? Then what's the percentage of inter-task sharing CONTEXT, and inter-task guideline retrieved by guideline selection?

2. Analysis in section 4.2.Q1 (line 222-227) is inconsistent with Table 1. It seems to analyze "ReAct+AutoGuide" over "ReAct+ExpeL" by claming "ExpeL approach helps ReAct by ... but is not as significant as AUTOGUIDE". But I can't find "ReAct+ExpeL" setting in Table 1. It's confusing since following discussion also centers around this setting.

3. Lack of explanation about the setting "AUTOGUIDE + Reflexion" in section 4. It's simply mentioned in section 4.1.2 (line 197). How is it implemented?

4. "Context Identification" module necessitates handcrafted context definitions per task. Since guideline selection relies on context matching, will the number of demonstrations affect task performance?
questions: questions: See above.

A typo: line 110 Appendix C.1: -> C.1.
limitations: limitations: See above.

Comments: title: Response to Reviewer rEoJ
Response to Reviewer wFet
Response to Reviewer wFet
comment: Thank you for reading our rebuttal. We also appreciate your positive review and kind support of our paper.
We're glad our responses addressed your concerns. We also appreciate your positive evaluation of our paper. Yes, based on your insightful and helpful feedback, we will include the implementation details in the revised version.
Thanks for the response! I've read it and will keep the score for acceptance.
Thank you for the explanation! 

My concerns are addressed.  Please also provide the implementation details in the revised version.
We are pleased that our rebuttal addressed concerns #1, #2, and #4. We also appreciate the reviewer's clarification of concern #3. Please find further details on the AutoGuide + Reflexion approach below.

In the initial episode of a given test task, we begin by using AutoGuide to solve the task. If the initial episode fails, we then use Reflexion to generate reflective feedback based on the test reward. This feedback suggests the plan the agent should attempt in the next episode (or trial). In our AutoGuide + Reflexion setting, **this reflective feedback is appended to the action generation prompt, immediately following the context-aware guidelines**. This approach enables the agent to consider both the context-aware guidelines and the reflective feedback when solving the test task in the next episode. As in Reflexion, when there are multiple past episodes, we concatenate the reflection feedback from each episode. This process continues until the maximum number of episodes (or trials) is reached.
Thank you for your thorough explanation and experiments! My concerns towards weakness 1/2/4 are all addressed.

Regarding the setting "AUTOGUIDE + Reflexion", my question is that how to combine/integrate them together, since they're different frameworks with multiple individual components.