AutoManual: Generating Instruction Manuals by LLM Agents via Interactive Environmental Learning

Authors: minghao  chen
yihang  li
yanting  yang
shiyu  yu
binbin  lin
xiaofei  he
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 12:52:17

Abstract: Large Language Models (LLM) based agents have shown promise in autonomously completing tasks across various domains, e.g., robotics, games, and web navigation. However, these agents typically require elaborate design and expert prompts to solve tasks in specific domains, which limits their adaptability. We introduce AutoManual, a framework enabling LLM agents to autonomously build their understanding through interaction and adapt to new environments. AutoManual categorizes environmental knowledge into diverse rules and optimizes them in an online fashion by two agents: 1) The Planner codes actionable plans based on current rules for interacting with the environment. 2) The Builder updates the rules through a well-structured rule system that facilitates online rule management and essential detail retention. To mitigate hallucinations in managing rules, we introduce a *case-conditioned prompting* strategy for the Builder. Finally, the Formulator agent compiles these rules into a comprehensive manual. The self-generated manual can not only improve the adaptability but also guide the planning of smaller LLMs while being human-readable. Given only one simple demonstration, AutoManual significantly improves task success rates, achieving 97.4\% with GPT-4-turbo and 86.2\% with GPT-3.5-turbo on ALFWorld benchmark tasks. The code is available at https://github.com/minghchen/automanual.

Main paper: 
 introduction: 
 Recently, autonomous agents based on Large Language Models (LLM), e.g., ReAct [33] , Reflexion [16] , SayCan [1] , WebGPT [10] , and Voyager [24] , have demonstrated their potential to complete long-horizon tasks in grounded environments. These LLM agents operate by generating thoughts and actions that are executable in the environment. For customized environments, such as robotics [1, 7, 18, 23] and games [14, 24, 39] , prior methods provide detailed instructions and in-context examples to familiarize LLM with action functions (API) and the target environment. However, unlike these agents, humans can autonomously build and update their understanding of an unfamiliar environment through dynamic interaction.Several existing methods enable LLM agents to reflect on feedback [16, 21] or save successful experiences as skills [21, 24, 35] to enhance the performance and reduce the reliance on humanprovided examples. However, these reflections and skills have not been well exploited to foster a deeper understanding of the environment. As a result, directly using saved skills as in-context examples can lead to the Path Dependence problem, i.e., the agent blindly replicates the paths of previous successes, failing to appropriately adapt to new scenarios. Such problems are more severe in real-world situations characterized by high variability.A previous work, ExpeL [35] , gathers the trajectories of LLM agents and extracts cross-task rules from them. However, these rules are extracted offline, making ExpeL suffer from the same distributional shift problem as Offline RL [6] . Meanwhile, due to the simplicity of rule management, its rules are always armchair general and unhelpful for the Path Dependency problem. In this paper, we propose a novel framework called AutoManual to build a well-organized understanding of the environment that can guide multi-task planning effectively. AutoManual leverages a dynamic rule system that not only extracts valuable experience, including skills and reflections, into different types of rules but also allows for continuously updating these rules in response to new situations. Additionally, error-prone details are explicitly described in the rules to improve the robustness of planning.AutoManual follows two alternating iterative processes to optimize the rules. First, given the observation and task of an episode, the Planner agent utilizes currently discovered rules to write free-form code as an actionable plan. The interaction between the environment and the Planner will loop until the episode ends. Second, based on this trajectory, the Builder agent will update relevant rules through the rule system. This online updating mechanism can timely verify whether the rules have deviations and are applicable to the Planner. After rules optimization, the Formulator agent categorizes these rules according to their application scenarios and compiles a comprehensive manual in Markdown format.The challenge lies in enabling the Builder to accurately extract applicable rules from a long trajectory, as LLM are prone to generating hallucinations. To address this, we employ a case-conditioned prompting strategy, which directs the Builder to focus on specific rules according to the case of the trajectory. For example, if errors occurred in a trajectory, the Builder is first asked to determine which caused the error: an unrecorded situation occurred, or the Planner failed to follow existing rules.Based on this answer, the Builder will be given corresponding prompts to update relevant rules.To summarize, our contributions are the following:• We adopt actionable code as the way for the Planner agent to interact with the environment.We introduce a structured rule system that allows the Builder agent to manage multiple types of knowledge from these code-based interactions.• We propose an alternating process between the Planner and Builder agents to optimize rules in an online manner and resolve the Path Dependency problem. To improve readability, the Formulator agent is introduced to reorganize and formalize the rules into a Markdown manual.• To facilitate rule management, we employ a case-conditioned prompting strategy, which guides the Builder to manage specific types of rules for different trajectory cases.• Starting from a single demonstration, AutoManual can generate detailed instruction manuals for complex environments like ALFWorld and MiniWoB++. These manuals allow LLM agents to achieve remarkable success rates of 97.4% with GPT-4-turbo and 86.2% with GPT-3.5-turbo on ALFWorld, 98.3% with GPT-4-turbo and 92.7% with GPT-3.5-turbo on MiniWoB++.2 Related Works 
llm for agents planning: 
 Large Language Models (LLM) exhibit powerful reasoning and planning capabilities [11, 12, 28, 33, 39] while requiring much fewer demonstrations than traditional learning methods. With this planning capability as the core, LLM agents are being developed for use in robotics [1, 7, 18, 20, 23] , game-playing [14, 24, 27, 39] , software development [3, 15] , and other fields [30] . Prior studies [16, 21, 33] allow agents to adjust actions or plans based on environmental feedback to improve planning performance. Given the powerful programming capability of LLM, several works, e.g., CodeAsPolicy [7] , ProgPrompt [18] and AdaPlanner [21] , propose to use Python code as the plan of LLM agents. This form of output can automatically respond to in-plan feedback and achieve better performance than the action and JSON format [21, 25] .Embodied agent research has long sought to enable agents to self-improve through interactive experiences. Unlike traditional learning-based agents that require extensive iterations for optimization, Reflexion [16] allows LLM agents to reflect on previous failures and quickly improve their plans. Some works [32, 34, 36] combine tree search with reflection to deliberately seek a better solution.Apart from failure experiences, prior studies [21, 24, 39] utilize successful experiences as skills to assist future planning. Voyager [24] stores generated and verified programs into the skill library as a new skill for more complex tasks. AdaPlanner [21] also discovers and archives successful programs into skill memory for future similar tasks. However, these methods stop updating skills after storing them, which inevitably leads to the Path Dependency problem.Another series of works [26, 31, 38] employs LLM as a prompt optimizer to enhance its own performance. In contrast to our approach, which addresses challenges in unfamiliar environments, these studies focus on enhancing LLM reasoning performance. As a result, their optimized prompts are typically brief and lack environmental knowledge.For LLM agents, learning from past experiences can also be viewed as managing the episodic memory [16] . CLIN [9] proposes to keep updating a memory centered on causal abstractions for new trials. Retrieval-Augmented Planning (RAP) [4] retrieves past experiences corresponding to the current situation. MemGPT [13] allows LLM to select content to retain in working memory and to search for information in long-term memory. Generative Agents [14] retrieve memories based on recency, importance, and relevance to the current situation. Generative Agents also generate tree-structured reflections, but they focus on a continuous scenario rather than task-oriented rules.Several recent works also investigate the rule discovery capabilities of LLM. Zhu et al. [40] propose Hypotheses-to-Theories (HtT), enabling LLM to induce and deduce rules for basic reasoning tasks.For LLM agents, ExpeL [35] gathers the trajectories of Reflexion agents and extracts cross-task rules from them. Furthermore, AutoGuide [2] generates state-aware rules and retrieves rules relevant to the test-time state. Unlike ExpeL and AutoGuide, which extract rules from offline experiences, we update rules in an online manner, verifying their reliability and applicability. For more discussion of differences, refer to Appendix C.-**rule_1**: This rule provides a 'find_object' method to search for objects, which the agent used correctly to find the mug. -**rule_2**: This 'Success Process' rule outlines the steps to put an object in a receptacle after finding and manipulating it. However, it does not explicitly state that the agent must be at the location of the appliance to use it, which is relevant for the heating step. -**rule_3**: This rule reminds the agent to open closed receptacles to search for objects, which the agent did correctly.3. There is a 'Success Process' rule applicable to this type of task (rule_2), but it does not explicitly cover the precondition for heating an object, which is to be at the same location as the appliance.4. The fault in this scenario can be attributed to:*Imperfect Rules*: The existing 'Success Process' rule (rule_2) does not fully document the precondition for using appliances, such as the microwave. It does not state that the agent must be at the location of the appliance to use it for heating an object.Based on the analysis, the conclusion is that the fault stems from *Imperfect Rules*. The rules need to be updated to include the precondition for using appliances, which is a critical step in the 'Success Process' for tasks involving heating objects. Builder: Potential Rules: 1. *Corrected Error*: When attempting to use an appliance (e.g., microwave), the agent must be at the same location as the appliance. This rule is related to the existing 'Success Process' rule (rule_2), but it adds a specific precondition for using appliances that was not previously documented.2. *Success Process*: The existing 'Success Process' rule (rule_2) should be updated to include the precondition of being at the location of the appliance before using it. This is a critical step in tasks involving heating objects.Check Difference:-The potential 'Corrected Error' rule targets the specific precondition for using appliances, which is a new addition and not covered by existing rules. -The update to the 'Success Process' rule expands on the existing steps to include the precondition for using appliances, which is a refinement of the existing rule.Check Existing Rules: -Rule_2 needs to be updated to include the precondition for using appliances, as this was the key oversight that led to the agent's initial error.Codes: '''python # Update the existing 'Success Process' rule to include the precondition for using appliances rule_system.update_rule( rule_id='rule_2', rule="If the task is to heat an object and put it in a receptacle, the agent must first find the object, ensure they are at the location of the heating appliance before using it, and then place the object in the target receptacle.", example="""# For example, to heat a mug and put it in a cabinet: # [Step 1] Use 'find_object' method to search all receptacles # [Step 2] Take the mug # [Step 3] Go to the heating appliance (e.g., microwave) and ensure you are at its location # [Step 4] Heat the mug with the appliance # [Step 5] Find an empty cabinet or any cabinet to place the heated mug in # [Step 6] Place the mug in the cabinet""", validation_record="Updated to include the precondition for using appliances based on the error and subsequent success in epoch_1." ) # Write a new 'Corrected Error' rule for the precondition of using appliances rule_system.write_rule( rule="Before using an appliance (e.g., microwave), the agent must go to the location of the appliance .", type="Corrected Error", example="""# Corrective code: # Go to the microwave before attempting to heat the mug observation = agent.go_to('microwave_1') # Heat the mug with the microwave observation = agent.heat_with('mug_1', 'microwave_1')""", validation_record="Derived from the error and correction in epoch_1 where the agent initially failed to heat the mug due to not being at the microwave's location." ) rule_system.stop_generating() '''Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See Limitation Section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
automanual overview: 
 Our AutoManual framework, shown in Fig 1 , consists of three main stages. Building stage: The Planner agent and Builder agent collaborate to build rules from the interactive environment. The Consolidator agent merges or deletes redundant rules when the rules exceed the maximum rule number. Formulating stage: The Formulator agent categorizes the rules, summarizes the key points, and formulates them into a manual in Markdown form. Testing stage: Based on the generated manual, a test-time Planner agent will be evaluated through test tasks and scenarios.Formally, an Interactive Environment can be modeled as a Partially Observable Markov Decision Process (POMDP): (S, A, T , G, O). At the start of each episode, a scenario s 0 2 S will be initialized, a text-grounded task g 2 G and the initial observation o 0 2 O (processed into textual form) will be given. The environment can be interacted with through permissible actions (API) set A. After executing an action a 2 A, the environment will return the result of the action and the new observation o 0 based on the dynamics T (s 0 |s, a) 2 T and O(o 0 |s 0 ). Finally, when the episode is done, a binary reward r 2 { 1, 1} indicating the failure or success of the task will be returned.We approach the learning of environmental rules as an optimization problem:EQUATIONwhere ⇥ denotes all rules in our rule system, ⇢(•|⇥) denotes the policy of the Planner given the current rules ⇥ and ⌧ ⇢ denotes a trajectory of ⇢(•|⇥) starting from [o 0 , g]. Classic policy gradient methods [29] solve such problems through stochastic gradient ascent, i.e., executing the current policy to obtain the episodic reward and back-propagating gradients to update the parameters. The Planner agent interacts with the environment by coding actionable plans. After receiving the current trajectory of the Planner, the Builder agent manages rules through the online rule system. ( 2) Formulating Stage: The Formulator agent formulates the resulting rules into a Markdown manual.Introduction ----, ------- Included Rules rule_1: ----, ------- rule_2: ----, -------(3) Testing Stage: A test-time Planner agent utilizes the manual to complete testing tasks.Inspired by this online reinforcement learning paradigm, we follow two alternative processes to optimize the rules ⇥: 1. The Planner practices the current rules through interaction within an episode.2. The Builder updates the rules ⇥ based on this trajectory. Compared to traditional parameter optimization, sample-inefficient gradient ascent is replaced by text-based rule management. We design a well-structured rule system described in Section 3.3 to ensure the rule updating contributes to rewards. Additionally, to limit the role of human expertise, we only provide a simple example demonstrating the output format to agents. Then, manually derive several initial rules from this example as the starting point of the optimization.As demonstrated by the success of Voyager [24] and AdaPlanner [21] , code-based planning can leverage the powerful programming capability of LLM and automatically react to in-plan feedback.Voyager and AdaPlanner output and refine a complete solution function for the task, which is potentially reusable. However, this function-form output is difficult to adjust in response to environmental feedback, as it requires maintaining the integrity of the plan throughout.Our Planner Agent outputs free-form code as its plan, which aligns more with the natural programming capabilities of LLM [7, 22] . This form simplifies planning by only generating code necessary for the current environmental situation and feedback without the overhead of integrating previously executed code. As shown in Fig 2 , at the start of a new episode, the Planner receives system prompts, current rules ⇥, relevant samples from the skill and reflection libraries, the target task g, and initial observation o 0 . System prompts contain the role, permissible actions A, response guidelines, and a simple example (detailed in Appendix H). The output of the Planner is structured into four segments during each cycle:1. Analysis: The understanding of the current situation and reflection on previous errors if exist.2. Related Rules: Rules (along with their IDs) that need to be considered in this situation.The general plan to complete the task.Code: A block of Python code divided into steps. The Planner is encouraged to define helpful functions in the code, which might be reusable in similar scenarios.We denote this response of the Planner as [thought t , code t ], where thought t denotes the first three segments. code t executed in the environment is followed by feedback c t , which informs the subsequent output cycle. This process iterates until the episode ends or a response limit is reached. The TrajectoryThe Planner Trajectory: Given the current task and rules, the Planner will interact with the environment through free-form code. Based on the trajectory result, the Planner will generate a corresponding conclusion, which will be saved in the skill or reflection library.As shown in Fig 2 , according to the episodic reward, we categorize the result into Direct Success, Indirect Success (errors occurred but were solved later), and Failure. In the case of Direct or Indirect Success, the Planner will be prompted to organize its previous code into a code block. For Indirect Success, it additionally summarizes the mistakes and misunderstandings that cause errors. For the Failure case, the Planner will be prompted to reflect on the reason for the failure carefully, suggest reasonable corrections, and specify the code segment that caused the error. We denote this response of the Planner as conclusion. Finally, we obtain a trajectory of the Planner:EQUATIONSkill Library and Reflection Library: Apart from rules, we also manage and transmit conclusions from previous episodes, which provide essential details for generating planning code. In the case of Direct or Indirect Success, we save the code block in conclusion as a skill for that task type1 into the skill library [21, 24] . In the Failure case, we save its conclusion as a reflection for that task type into the reflection library. When a new task comes, the code block of the most similar task is retrieved from the skill library. If there is no existing skill for the new task type, the reflection for that task type will be returned. As mentioned in the Introduction, compared with rules, these skills and reflections contain more programming details but are less generalizable to new scenarios, i.e., the Path Dependence problem.Cooperation between Agents: In our framework, rule management is not solely the responsibility of the Builder; the Planner also plays a critical role by explicitly identifying the rules it engages in its response. This cooperation is facilitated by including the Planner's thoughts within the trajectory ⌧ , which is provided to the Builder. This synergy enhances the identification and adjustment of problematic rules. In addition, conclusion from the Planner contains the detailed success process or reflections on errors, which further assist the Builder in managing corresponding types of rules.Upon receiving the trajectory ⌧ ⇢ , the Builder has to manage the rules through the rule system. Given the current trajectory, the Builder classifies the cause of the major error as "Imperfect Rules" or "Imperfect Agents". Then, the Builder will get the base prompt and corresponding prompt to guide its rule management.Rule System: We intuitively identify rules as the kinds of knowledge that help task completion, including the analyses of the observed phenomenon T (o 0 |o, a), the mechanism T (s 0 |s, a), and the correlation between the reward r and ⌧ ⇢ , i.e., the success process or the occurred error. Therefore, unlike ExpeL [35] and AutoGuide [2] , which derive general insight from the trajectory, our system categorizes six specific rule types to extract environmental knowledge that targets different aspects of the trajectory. Furthermore, each rule in our system is enhanced with Example attribute to illustrate its application and important details, making it grounded and well-understood. Specifically, each rule in the rule system has these four attributes:1. Rule Type: The type of the rule, chosen from ["Special Phenomenon", "Special Mechanism", "Useful Helper Method", "Success Process", "Corrected Error", "Unsolved Error"];2. Rule Content: A description of the rule, beginning with the scope of its applicable scenarios;3. Example: An example or code from the trajectory demonstrates this rule, where additional remarks, e.g. error-prone details, can also be added to it;4. Validation Logs: Logs that track the rule's application and updates, including episode and rule IDs that trace the rule's evolution, serving as a reference for the Builder and Consolidator.The Builder manages the rules through the following functions of the rule system:• write_rule(**rule_attributes): Write down a new rule with its four attributes.• update_rule(rule_id, **rule_attributes): Rewrite the attributes of a existing rule.• stop_generating(): When the trajectory is not needed or insufficient to derive any more new rules, the function should be called.Similar to hierarchical reflections in Generative Agents [14] , we allow the Builder to utilize existing rules to induce more general or deeper rules and record their dependence in Rule Content or Validation Logs, more discussed in Appendix D.To mitigate the risk of erroneous rule creation, such as deriving rules of success from a failed trajectory, we employ case-conditioned prompts. As illustrated in Fig 3 , the Builder first analyzes and determines if the major errors stem from "Imperfect Rules" or "Imperfect Agent". Based on this analysis and the trajectory results, targeted prompts guide the Builder in rule management 2 . For example, in a case of indirect success due to imperfect rules (Case 2), the prompts will guide the Builder to extract or update the success process, helper methods, and error reflections in corresponding rule types. Finally, the Builder responds with the potential rules detailing their relation with existing rules and uses the functions of the rule system to manage rules.Rule Consolidation: When the number of rules in the rule system exceeds N max , the Consolidator agent steps in to consolidate related rules and delete redundant rules. It uses three functions of the rule system: get_trajectory(episode_id), update_rule(rule_id, **rule_attributes) and delete_rule(rule_id).Given the current rules, the Consolidator identifies potentially relevant or overlapped rules, uses get_trajectory function to investigate the trajectories they depend on, and finally calls the remaining functions to manage the rules. During the management, the Consolidator ensures that consolidation retains details of rules and examples.Once the building stage is complete, we can obtain a set of rules targeted to different situations, whose applicability has been validated through online optimization. Our next goal is to enhance their readability and global understanding. To achieve this, we introduce the Formulator agent, designed to transform these rules into a user-friendly manual, analogous to a teacher imparting a wealth of knowledge through easily digestible lessons. As depicted in Fig 1 , the Formulator begins by categorizing all rules based on their target scenarios. This categorization aids in structuring the manual and ensures that related rules are discussed together, which enhances the logical flow and accessibility of the information. For each category, the Formulator drafts an introduction, summarizing the rules it contains and highlighting the key points and overall principles that govern the specific scenarios. Finally, the Formulator compiles the rules and their introductions into a comprehensive manual formatted in Markdown.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 In line with prior works [2, 21] , we conduct the experiments on three interactive environments: (1) ALFWorld [17] is a text-based virtual household environment containing six distinct task types.We run the building stage on 36 tasks (6 tasks for each task type) sampled from the training set of ALFWorld, and each task is run only once. Following previous works [16, 21, 33] , we run the testing stage on the validation unseen set containing 134 tasks across these six types. (2) MiniWoB++ [8] is a simulated web environment where agents complete diverse tasks on the Internet by performing keyboard and mouse actions. Prior works [5, 21] selects 9 task types with environmental feedback and 44 task types without feedback from MiniWoB++ tasks. We perform experiments on 9 task types with feedback or on all 53 task types. At each stage, we randomly sample 6 tasks for each task type. (3) WebArena [37] introduces realistic web environments by emulating the functionality and data of popular websites. This benchmark poses significant challenges for LLM agents due to its large observation and action space, along with tasks that require longer planning horizons. Following AutoGuide [2] , our experiments focus on the Reddit domain within WebArena.During the building and formulating stages, we use GPT-4-turbo (gpt-4-1106-preview) for all agents. At the testing stage, we equip the Planner agent with GPT-4-turbo or GPT-3.5-turbo (gpt-3.5-turbo-1106), to evaluate the effect of generated manuals on relatively smaller LLM.Compared Methods: In the experiments, we compare AutoManual with the following methods of LLM Agent: (1) ReAct [33] prompts LLM to generate the reasoning trace using CoT [28] agents are also provided with one human demonstration. To reduce randomness, we performed each experiment three times and reported the average. More details of the implementation and prompts for AutoManual can be found in the Appendix. Main Results on ALFWorld: As shown in Tab. 1, AutoManual significantly outperforms the existing methods, evidenced by overall success rates of 86.2% when using GPT-3.5-turbo for the testing stage and 97.4% when using GPT-4-turbo. Noticeably, AutoManual requires little expert prior knowledge about the environment and is only provided with one human example to achieve excellent results. In comparison, the rules induced by ExpeL hardly improve performance, as its offline trajectories are composed of individual actions rather than code. We find the performance of AdaPlanner is lower than reported. One reason is that AdaPlanner requires LLM to output specific formats to complete its function-form code, which is difficult for creative LLM, e.g., GPT-4-turbo. In addition, AdaPlanner and Planner+Lib. are inferior to AutoManual because they only store successful paths as skills and inevitably face the Path Dependence problem. Especially, tasks in Put Two have various scenarios, such as "two objects can occur at the same receptacle or different receptacles", that require different processes to solve (Appendix G shows an example). Furthermore, Planner+Lib. often does not mark error-prone points in its skills, such as "target objects may appear in unconventional locations". Main Results on MiniWoB++: As shown in Tab. 2, the performance of AutoManual exceeds the previous methods and Planner+Lib. by a large margin. Especially in 9 task types with feedback, these tasks have higher diversity and require LLM agents to cope with various situations. For example, the tasks in login-user-popup type will interrupt the agent's plan at any time, requiring the agent to cope with unexpected situations. Therefore, solely imitating previous successful experiences without extracting targeted rules will lead to task failure. Additionally, due to the flexibility of free-form codes, our method shows better adaptability while requiring fewer expert examples than prior methods.Learning Curves. We show the success rate curves (testing with GPT-4-turbo or GPT-3.5-turbo) when gradually increasing the tasks of the building stage in Fig 4 . In the left image, we share rules across all task types (Cross-task Type), as in AutoManual, or each task type builds a separate set of rules (Single-task Type) during the building stage. Planner+Lib. on 9 tasks with feedback in MiniWob++. We find that Planner+Lib. tends to get stuck with a lower success rate. In the face of highly diverse scenarios, Skill Library cannot express the rules behind the environment, thus falling into the Path Dependency problem.In this ablation study, we quantify the impact of each core component of the AutoManual framework on performance, specifically focusing on success rates and error reduction during task execution.Since we allowed the Planner to replan up to 3 times, each task could have up to 4 error steps.Online v.s. Offline Rule Management: We perform offline AutoManual by collecting all trajectories and then managing rules from them. As Tab 4 shows, without online rule management, the generated manual can only slightly improve planning (from 88.1% to 90.7%). This is because more mundane mistakes and fewer direct successes will occur in the trajectories (the distributional shift problem), and the rules cannot be verified by feedback from the environment.Skill&Reflection Libraries: Retrieving historical conclusions is essential for correct planning, as they record massive interacting details that can complement the rules. Without them, there will be more errors in the details, and the success rate drops from 97.4% to 89.5%. However, as discussed previously, using plain experiences without inducing rules will lead to Path Dependency.Case-Conditional Prompts: This strategy further improves the rule management process by reducing the hallucination, as evidenced by an increase in success rate from 93.8% to 97.4%. These prompts ensure that the Builder updates rules reasonably and grounded.Effect of Manual Formulation: The final formulation of rules into a comprehensive manual contributed to the success rate of 97.4% and decreased average error steps, demonstrating the effectiveness of presenting rule-based knowledge in an organized and accessible format. It not only aids the Planner in mastering multiple rules but is also friendly for human reading. 
conclusion: 
 In  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper does not provide pretrained language models, image generators, or scraped datasets.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have stated and cited the asset used in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Despite the significant contributions of the AutoManual framework, several limitations warrant further discussion. First, our method heavily relies on the capabilities of GPT-4-turbo to generate reliable rules, which may limit the framework's applicability with less advanced language models.Secondly, the current implementation places all rules directly within the context of LLM, which, while effective in smaller or moderately complex environments, may not scale well to larger, more dynamic settings. For such expansive environments, integrating the rule system with Retrieval-Augmented Generation (RAG) techniques, similar to the approach taken by AutoGuide [2] , could enhance performance by dynamically selecting relevant rules based on the context, thereby managing the cognitive load on the LLM more efficiently.Thirdly, for complex and challenging tasks, the agents of AutoManual are insufficient in exploring the environment, as they only attempt solutions based on current knowledge. To enhance agents' exploration of unfamiliar environments, it may be necessary to endow the agent with curiosity [24] or combine it with tree search algorithms [32] .Lastly, there remains a challenge in ensuring that the Planner agent consistently adheres to the rules outlined in the manual. In practice, the Planner may sometimes ignore these rules (or have hallucinations about observations) and cannot generate the ideal solution code that can be applied to different situations. This indicates a need for additional mechanisms to enforce or verify rule compliance during operations. This issue underscores the potential necessity for developing more sophisticated methods to ensure rule adherence or to introduce more robust validation steps within the planning process.The AutoManual framework, leveraging LLM agents, presents positive and negative impacts on safety. On the positive side, by autonomously generating reliable manuals, our method enhances the predictability and reliability of LLM behaviors in dynamic environments, potentially reducing errors and increasing operational safety. However, relying on LLMs also introduces risks of unpredictable behaviors when agents encounter unanticipated scenarios or when rule adherence is not fully ensured.Furthermore, the manuals generated by our method can be invaluable tools for human workers. They encapsulate a distilled form of interaction-based learning that can aid in training, provide decision support, and improve task efficiency in various domains. This can not only enhance productivity but also ensure that human workers are better informed and prepared to manage the complex systems with which they interact.Lastly, AutoManual's ability to generate structured, context-aware manuals from interactive experiences suggests a promising avenue for constructing comprehensive knowledge bases for AI. These manuals can contribute to a global knowledge base shared by LLMs of different sizes, promoting broader AI research and development. It offers a method to systematically organize and retrieve complex interaction data in a way that is accessible and useful for both machines and humans.We compare AutoManual with prior methods that extract rules from experiences, i.e., ExpeL [35] and AutoGuide [2] , and discuss all differences here:1) Interactive Form. ExpeL and AutoGuide, following Reflexion [16] , each time output the thought and next-step action to interact with the environment. Our Planner agent interacts using free-form code. As evidenced by previous works [7, 25] , using code as the plan is more efficient because the code will automatically perform the in-plan actions and require LLM responses far fewer times.Additionally, planning with code can enjoy the powerful programming capability of GPT. More importantly, compared with action sequences, code is easier to generalize to similar scenarios, facilitating the management.2) Online v.s. Offline. ExpeL and AutoGuide extract rules from offline experiences of Reflexion agent. We update rules online by alternating rule practice and rule management. This online rule management can verify the reliability and applicability of the rules in a timely manner, forbidding rules to be armchair general. Additionally, online-update rules can help planners continuously improve their trajectories, making it easier for higher-quality success processes to emerge.3) Collaboration Between Agents. Building rules online also enables collaboration between the Planner and Builder agents. In our AutoManual, the Planner is prompted to describe the rules considered and analyze special phenomena, which ease the management of the Builder. In contrast, the builder in ExpeL and AutoGuide can only receive action sequences from the Planner.4) Rule System. In ExpeL and AutoGuide, each rule has only two attributes: the content and score. In our rule system, each rule has four attributes: "Type", "Content", "Example", and "Logs". These attributes provide a comprehensive representation of rules and facilitate the management and usage of rules. Moreover, we allow rules to build on top of other rules.Handling excessive rules. ExpeL and AutoGuide utilize the rule scores to delete the low-score rules. However, we found that rule scores are unreliable in the experiments because the Builder tends to give overconfident scores to all rules. Instead, our AutoManual employs a Consolidator agent to merge and delete redundant rules.In the building stage of all experiments, the maximum number of rules was set to 12 to balance the context length and rule diversity. We use OpenAI's Assistant API for all agents to save their history and prevent duplicate inputs. We set all LLMs' temperatures to 0 and maximum context length to 16000. Reflexion agents are allowed to try at most 3 trials for each task. For AdaPlanner and AutoManual, we allow the Planner agent to replan at most 3 times on ALFWorld and 6 times on Table 5 : The GPT models used for each method in our implementation when using GPT-3.5-turbo or GPT-4-turbo as test-time LLM. Here GPT-3.5-turbo denotes gpt-3.5-turbo-1106 and GPT-4-turbo denotes gpt-4-1106-preview.Test-Time LLM GPT-3.5-turbo GPT-4-turbo ReAct [33] Actor: GPT-3.5-turbo Actor: GPT-4-turbo Reflexion [16] Actor: GPT-3.5-turbo Actor: GPT-4-turbo Self-reflection: GPT-3.5-turbo Self-reflection: GPT-4-turbo ExpeL [35] Offline Trajectory: Offline Trajectory: Actor: GPT-3.5-turbo Actor: GPT-4-turbo Self-reflection: GPT-3. 5 Testing Stage: Planner: GPT-3.5-turbo Planner: GPT-4-turbo MiniWob++ in response to the environmental feedback. In the Building stage on ALFWorld, we use 6 tasks for each task type, a total of 36 tasks, by default. We shuffle all tasks, and when tasks in a task type succeed three times in a row during building, we consider that this task type has been solved and will no longer run it. The API call cost for building and formulating stages is about $14 in total.We made a slight modification to the text format of ALFWorld to make it more suitable for code: for each object, "object id" was changed to "object_id", and all preceding articles were removed. The maximum action step for a task is set to 50.For the 9 tasks with feedback on Miniwob++, we find that "email-inbox-nl-turk" and "email-inboxforward-nl-turk" are repetitions of "email-inbox" and "email-inbox-forward-nl". Therefore, we only used 7 task types in the building stage, while in the testing stage, all 9 types were evaluated. In the building stage, we run a total of 42 tasks (6 tasks for each type).For all 53 tasks on Miniwob++, since running the building directly will lead to a large number of rules, and tasks without feedback have low variability, we adopt a two-step building strategy: we first run our building stage on 9 task types with feedback and then only update the skill and reflection libraries on 44 task types without feedback. The 4 examples for experiments on all task types are from the following task types: 'click-menu', 'enter-date', 'social-media-some'. We chose these tasks mainly because they are difficult for GPT-4-turbo to try out due to the lack of HTML feedback. For example, the HTML in 'enter-date' will not display the cursor but requires the cursor at the correct position to enter.Environmental feedback on executing code in MiniWob++ is as follows: Whenever one of the actions is executed, we will log whether the action was executed successfully or failed. Finally, the results of these actions are concatenated together, and used as feedback of the code along with the HTML text.For AdaPlanner [21] and RCI [5] , we use their official code on GitHub to implement their methods. We fix their bugs on matching text patterns but still find their performance is much lower than they reported. This may be because they have not released the code used in their papers, or their methods are greatly affected by the GPT version (newer versions of GPT will be more creative).We present the generated manual3 on ALFWorld by AutoManual in Fig 5 . As shown in the manual, the Formulator agent categorizes the rules into four categories: "Navigation and Search", "Object Interaction and Location Management", "Task-Specific Processes", and "Correctness and Validation" with clear introductions. We find our agents have found and recorded lots of important rules that directly impact task completion. For instance, "the agent should include all possible locations in its search" in rule_0, "The agent can only hold one object at a time" in rule_3, and "When multiple items of the same type are present at a location, the agent may have to choose one to interact with or examine." in rule_5. For tasks that can be solved by a fixed strategy, such as "Cool", "Heat", "Clean" and "Examine", AutoManual provides clear Success Process type rules. For the complex task, "Put Two", AutoManual conducts classified discussions in its Success Process rule.These demonstrate that AutoManual resolves the Path Dependency problem of skills by digging deeper into mechanisms, updating and incorporating success processes, and annotating important details.Moreover, we prompt the Builder to break down large phenomena into specific rules and derive deeper rules from them later. In the initial rules 9, we give a simple example of how to build rules upon rules. In the manual 5, we find that the Builder can correctly utilize rules recording basic mechanisms to build more complex rules. For example, rule_4 uses rule_3 to induce a solution: "If all objects are found at the same location, handle them sequentially according to rule_3." However, we find the generated manual is still not perfect. There are some unnecessary duplicates between rules, such as rule_7 and rule_8, which are both Success Process type rules for "Examine" tasks, but they are divided into two rules. This manual is intended to assist the housekeeper agent in the successful execution of tasks within a simulated environment. The rules provide guidance on navigating, searching the environment, interacting with objects, and managing task-specific processes, as well as ensuring the correctness of actions using code assertions.These rules provide guidance on how to search for objects, including the use of helper methods to streamline the process and ensure thoroughness.Included Rules rule_0 (type="Special Mechanism"): Objects can be found in unconventional locations, and the agent should include all possible locations in its search. For example, In epoch_9, the agent found a soapbar on the toilet, which is an unconventional location for storing such items.rule_1 (type="Useful Helper Method"): If there are multiple receptacles to be search, the agent can write and use 'find_object' method as shown in the example. For example,These rules inform the agent on how to interact with objects, from taking and placing items to handling multiple items of the same type.Proper location management is crucial for successful task execution.rule_2 (type="Special Phenomena"): When using a microwave, the agent can interact with it (e.g., heat an object) even if there is another object inside, the agent is holding something, and the microwave door is not explicitly mentioned to be open.For example, In epoch_1, the agent was able to heat the mug with the microwave even though there was an egg inside the microwave and the agent was holding the mug.The agent can only hold one object at a time and must put down any held object before taking another.For example, In epoch_2, the agent was holding statue_4 and attempted to take statue_3 without putting down statue_4 first, resulting in a 'Nothing happens' observation.rule_4 (type="Success Process"): When tasked with placing multiple objects in/on a receptacle, the agent can either collect all objects before attempting to place them or find and place them one by one, ensuring they revisit locations with multiple objects if necessary. If all objects are found at the same location, handle them sequentially according to rule_3.For example, In epoch_15, the agent should have revisited sidetable_1 to collect the second pencil before attempting to place it in coffeetable_1. In epoch_23, the agent failed to collect all required statues from coffeetable_1 because it did not revisit, is also addressed by this rule. #The agent must interact with a receptacle to observe its contents, which includes going to the receptacle and opening it if it is closed. Before performing a put or take action, the agent must ensure it is at the correct location. When multiple items of the same type are present at a location, the agent may have to choose one to interact with or examine.For example, In epoch_16, the agent had to open several closed cabinets (e.g., cabinet_1, cabinet_2) to find items such as the mug. In epoch_21, the agent observed multiple alarm clocks on desk_1 and selected one ('alarmclock_4') to interact with.This category outlines the steps required to complete specific tasks, such as heating, cooling, and examining objects with another object's assistance.rule_6 (type="Success Process"): If the task involves cooling or heating an object before placing it, the steps are: (1) search for the object using 'find_object' in rule_1, (2) take the object, (3) cool/heat it as required, (4) go to the target receptacle, and (5) put the object. Ensure the agent's location and the state of the environment are updated after each action. For example, rule_7 (type="Success Process"): When tasked with examining an object under a desklamp, the agent should first find the desklamp and the object, ensure the desklamp is on, take the object, and then use the desklamp to examine the object. For example, rule_8 (type="Success Process"): When tasked to look at an object under a desklamp, ensure the lamp is on before using it to examine the object. For example,Instructions on asserting code to confirm state changes and enhance the reliability of the agent's actions.rule_9 (type="Corrected Error"): Assertions in the agent's code should confirm state changes such as location or held objects, rather than rely on specific phrases in observations.For example, Instead of asserting 'You are at' in the observation, the agent should assert the location and held object state changes. Also, when handling multiple required objects at the same location, the agent should manage them sequentially without unnecessary variables.# For example, to cool a mug and put it in a coffeemachine: The well-structured rule system is essential for the rule management and usage. We perform the ablation study of our rule system in Tab 7.• AutoManual without "Type": we remove the "Type" attribute of each rule in the rule system and the instruction for the Builder to manage various types of rules. As shown in Tab 7, without the "Type" attribute, the performance drops significantly, from 86.2% to 74.6% and 97.4% to 91.5%, as the Builder cannot manage specific types of rules and loses specific instruction on each rule.• AutoManual without "Example": we remove the "Example" attribute of each rule in the rule system. The performance also drops by a large margin, as the Builder cannot specify necessary details in rules, and rules with lower understandability can sometimes be misleading to the Planner.• AutoManual without "Validation Logs" has little impact on performance, but "Validation Logs" of rules are very useful during debugging.We also conducted some detailed ablation studies of our AutoManual in Tab 7.• Helper methods defined by humans and the Planner can serve as the solution for a subgoal of a task that can be used in multiple tasks. AutoManual without "Useful Helper Method": we remove the "Useful Helper Method" in the "Type" attribute, the human example, and initial rules, and we no longer encourage the Planner to write helper functions when writing code. The performance drops from 86.2% to 79.6% and 97.4% to 94.3%, demonstrating that extracting reusable helper methods can facilitate the programming.• AutoManual without Cooperation between Agents: we remove the Planner's thoughts in the trajectory when providing it to the Builder and no longer require the Planner to output Relevant Rules. The performance decreases significantly, indicating the thoughts and conclusions of the Planner are helpful for the Builder to manage rules.• Case-Conditioned Prompts without Classification: we remove the requirement to classify the error reason to "Imperfect Rules" or "Imperfect Agents" and only use the prompts for "Imperfect Rules" during case-conditioned prompting. As demonstrated in Tab 7, the results are inferior to using all 5 cases because the analysis and classification of error reason can boost the Builder to manage rules accurately.We analyze the sensitivity of our rule optimization to the initial condition, i.e., the human example and initial rules. In the experiments on 9 task types with feedback of MiniWob++, we use an example of search-engine task and the corresponding initial rules, which are shown in Listing 8 and 9. In these initial rules, we extract the successful process of search-engine task and a helper method to turn to the next page. Here, we provide AutoManual with an example of enter-text task, the simplest task type in MiniWob++, as shown in Listing 10.The results in Tab 8 demonstrate that even if given less initial knowledge, AutoManual can still automatically learn the required knowledge through online optimization, which shows that AutoManual has good robustness to the initial rules. In addition, we found that the knowledge it learned about search-engine is highly similar to the standard knowledge provided by humans, such as the need to turn pages for the search index larger than 3.In the following example, the existing skill can correct solve the "Put two" task when the two objects occur at the same receptacles. However, in the new scenario, the Planner still replicates the strategy but fails to complete the task, as the target objects are at different receptacles.Here is the code for a relevant skill: You are in the middle of a room. Looking quickly around you, you see bed_1, desk_1, drawer_5, drawer_4, drawer_3, drawer_2, drawer_1, dresser_1, and garbagecan_1. The task is to: put two cellphone in bed. '''python # Helper method to manually parse the observation for cellphones def parse_for_cellphones(observation):# This is a simple parser that looks for the word 'cellphone' followed by an underscore and a number return # Take and place the cellphones in the bed. for cellphone in found_cellphones[:2]: # Take only two cellphones # Go to the receptacle where the cellphone was found. observation = agent.go_to(receptacle) # Take the cellphone. observation = agent.take_from(cellphone, receptacle) # Go to the bed and put the cellphone in it. observation = agent.go_to('bed_1') observation = agent.put_in_or_on(cellphone, 'bed_1') ''' Please pay close attention to the process and details of this successful code when writing code. Also, be aware of potential randomness (the current environment may differ from this one).You are in the middle of a room. Looking quickly around you, you see armchair_1, cabinet_4, cabinet_3, cabinet_2, cabinet_1, drawer_5, drawer_4, drawer_3, drawer_2, drawer_1, dresser_1, garbagecan_1, safe_1, shelf_12, shelf_11, shelf_10, shelf_9, shelf_8, shelf_7, shelf_6, shelf_5, shelf_4, shelf_3, shelf_2, shelf_1, sidetable_1, and sofa_1. Your task is to: put two pillow in sofa. Planner: ### Understanding of the task: The task is to find two pillows in the environment and place them in or on the sofa. The environment contains various receptacles and surfaces where the pillows might be located.### Plan for the task: 1. Define a list of all receptacles and surfaces to check for pillows. 2. Search each receptacle for pillows.3. Take two pillows once found. 4. Move to the sofa's location. 5. Place each pillow on the sofa.Here is the code for the task:'''python # Helper method to manually parse the observation for pillows def parse_for_pillows(observation): # This is a simple parser that looks for the word 'pillow' followed by an underscore and a number return [ Here, we present a trajectory of the Planner and the corresponding responses of the Builder for a task in ALFWorld. In this "Indirect Success" trajectory, the Planner made the mistake of heating the mug without having it next to the microwave. The Planner then corrected the mistake and provided organized code in its conclusion. After observing this trajectory, the Builder keenly found the cause of the error: the existing "Success Process" rule did not indicate that the agent must go to the object before interacting with the object. The Builder updated the "Success Process" rule and added a "Corrected Error" rule to clarify the mistake.Initial observation and the task: You are in the middle of a room. Looking quickly around you, you see cabinet_13, cabinet_12, cabinet_11, cabinet_10, cabinet_9, cabinet_8, cabinet_7, cabinet_6, cabinet_5, cabinet_4, cabinet_3, cabinet_2, cabinet_1, coffeemachine_1, countertop_1, diningtable_1, drawer_1, fridge_1, garbagecan_1, microwave_1, shelf_3, shelf_2, shelf_1, sinkbasin_1, stoveburner_4, stoveburner_3, stoveburner_2, stoveburner_1, and toaster_1. Your task is to: heat some mug and put it in cabinet. Planner: ### Understanding of the observation: The agent is in a room with numerous cabinets, appliances, and surfaces. The task is to heat a mug and then place it in a cabinet. This requires finding a mug, heating it, and then placing it in any cabinet.-**rule_0**: Ensure all receptacles and surfaces are considered when searching for the mug.-**rule_1**: Use the 'find_object' method to search for the mug.-**rule_2**: After finding the mug, heat it and then put it in a cabinet.-**rule_3**: Open closed receptacles to search for objects.### Plan for the task: 1. Use the 'find_object' method to search for a mug in all receptacles and surfaces. 2. Once the mug is found, take it.3. Heat the mug with an appropriate appliance (e.g., microwave). 4. Find an empty cabinet or any cabinet to place the heated mug in. 5. Place the mug in the cabinet.Here is the code for the task: '''python The agent successfully found a mug on shelf_1 and took it. However, when attempting to heat the mug with the microwave, the action failed with "Nothing happens." This indicates that the agent tried to use the microwave while not being at the microwave's location. The agent must go to the microwave before attempting to heat the mug.### Mistakes in the previous code and the reason for errors:The error occurred because the agent attempted to heat the mug without being at the microwave's location.The precondition for heating an object with a receptacle is that the agent must be at the same location as the receptacle.### Rules to consider: -**rule_0**: Ensure all receptacles and surfaces are considered when searching for the mug.-**rule_1**: Use the 'find_object' method to search for the mug.-**rule_2**: After finding the mug, heat it and then put it in a cabinet.-**rule_3**: Open closed receptacles to search for objects.### Plan for the task: 1. Since the mug has already been found and is being held by the agent, proceed to the microwave. Analyzing the scenario, we can identify the root cause of the observed mistakes and relate them to the existing rules:1. The agent initially attempted to heat the mug without being at the microwave's location, which resulted in the action "Nothing happens." This mistake was due to the agent not following the precondition for the 'heat_with' action, which requires the agent to be at the same location as the receptacle (in this case, the microwave).Listing 3: System Prompts for Planner Agent[Role] You are an excellent household agent in a simulated environment. Given a household environment and a task, you write Python code to accomplish the task.[Actions] For each task, an agent is created in an environment, and the initial observation and global memory are printed.# The agent is an instance of Agent class, which includes the state of the agent ( its location, what it's holding) and the actions it can take.class Agent: def __init__(self, env: InteractiveEnvEngine): self.location = env.agent_location self.holding = "nothing" ... # Here are the admissible actions the agent can take; all action functions return an observation string of the result of the action. If the precondition of the action is not met, its observation will include "Nothing happens".# Go to a receptacle and update the agent's location. # For example, 'On countertop_1, you see candle_1, cloth_2, and soapbar_1.' = go_to('countertop_1') # For example, 'On sidetable_2, you see nothing.' = go_to('sidetable_2') def go_to(self, receptacle):... # Take an object from a receptacle if the agent is not holding anything. # For example, 'You take soapbar_1 from towelholder_1.' = take_from('soapbar_1', 'towelholder_1') def take_from(self, object, receptacle):... # Put an object in or on a receptacle if the agent is holding it. # For example, 'You put soapbar_1 in/on cabinet_1.' = put_in_or_on('soapbar_1', 'cabinet_1') def put_in_or_on(self, object, receptacle):... # Use a lamp. # For example, 'You turn on desklamp_1.' = use('desklamp_1') def use(self, object): ... # Clean an object with a receptacle. # For example, 'You clean soapbar_1 using sinkbasin_1.' = clean_with('soapbar_1 ', 'sinkbasin_1') def clean_with(self, object, receptacle):... # Heat an object with a receptacle. # For example, 'You heat tomato_1 using microwave_1.' = heat_with('tomato_1', ' microwave_1') def heat_with(self, object, receptacle):... # Cool an object with a receptacle. # For example, 'You cool pan_2 using fridge_1.' = cool_with('pan_2', 'fridge_1') def cool_with(self, object, receptacle): ... # A useful function that you can use. # Extract a list of object_ids with the specified object_name from the observation. def get_object_with_id(observation, object_name): ...You will be given rules discovered by colleagues, summarizing game mechanisms and experiences of success and failure. Use these rules to guide your coding efforts.Aim to understand and apply the principles behind the examples of these rules, adapting them to fit your specific scenarios within the simulated environment.Output Format Instructions: 1. First, explain your understanding of the task and the current observations, then describe rules (with their IDs) that need to be considered and plan for the task. Then, write your code between "'''python" and "'''". No text should follow after the code block. 2. After receiving feedback, you should also explain your understanding of the current observations, including special (unexpected) formats or phenomena, mistakes in your previous code, and the reasons for errors. Then, describe rules (with their IDs) that need to be considered, plan for the task, and then write code. Follow these instructions: 1. DO NOT USE an undefined function or attribute of the agent. Your code must be directly executable in the given environment. I won't implement any placeholders in your code for you. 2. Your code should be consistent with the code examples of the rules (please copy the code if there is no better modification!), making it easier for the builder agent to refine and develop new rules effectively. 3. In your code, you are encouraged to define helpful functions, which should be general and reusable in different scenes. The helper methods in the rules are already defined in the coding environment, so you can directly use them. If you don 't need to modify them, don't redefine them. 4. After defining helper methods, your code should be divided into steps (marked with "[Step]" in comments) and write assertions at the end of each step to ensure that each step achieves its subgoal. Follow these instructions. Don't give up or change the task.Please summarize the misunderstandings and mistakes you made, and then organize your code into a code block. You should copy the used parts from your previous code, including helper methods and steps. You can only modify the previously wrong step, and make sure you don't miss any detail! So that I can use it with similar but not the same scenarios.You failed to complete the task. Now carefully review the trajectory of the executed actions and the corresponding observations, then identify the reasons for the failure. This reason is often your mistake or misunderstanding rather than the environment's error. By carefully comparing with applicable rules and examples, pinpoint where your code deviated from expected standards.If there exist coding errors, specify the segments of the problematic code and elucidate how they contributed to the errors. If you want to write a correction for an error, you must double-check its plausibility! Additionally, determine whether your codes were in strict adherence to the relevant rules and examples. This reflection and documentation will serve as a reminder for completing future tasks.Listing 6: The Put Example for ALFWorld You are in the middle of a room. Looking quickly around you, you see cabinet_4, cabinet_3, cabinet_2, cabinet_1, countertop_1, garbagecan_1, sinkbasin_2, sinkbasin_1, toilet_2, toilet_1. Your task is to: find some spraybottle. ### Understanding of the observation: ... ### Rules to consider: ... object_ids = get_object_with_id(observation, object_name) return object_ids, receptacle return None, None ''', "validation_record": "Essential helper method provided by User."}, "rule_2": {"rule": "If the task is to put some object on some receptacle, first use ' find_object' method in rule_1 to search all receptacles, take the object, then go to the target receptacle and put the object.", "type": "Success Process", "example": ''' # For example, to put some spraybottle on toilet: # For MineWob++, the Planner is allowed to complete the task in steps: initially output segments of planned steps and pause to observe their outcomes to inform subsequent steps. Here is a code of my plan: '''python You are observing a housekeeper agent as it codes and acts within a simulated environment (game). Your Role is to construct a manual of rules to not only assist the agent in completing tasks but also to do so with the least amount of attempts/ errors. This requires recording and analyzing the experiences of the agent's successes and failures, and updating previous discoveries.[Functions] You will be presented with the current epoch's trajectory. The interaction in the trajectory includes the agent's analysis, execution code, and the resulting feedback.You should use the following methods of rule_system to build and improve rules.rule_system.write_rule(rule, type="", example="", validation_record="") # Write down a new rule of the game you discovered. # Parameters: # rule: a rule of the game you discovered. # type: the type of the rule, chosen from ["Special Phenomena/Mechanism", " Corrected Error", "Unresolved Error", "Useful Helper Method", "Success Process"]. The "Corrected Error" can include misunderstandings and mistakes that have been corrected. # example: an example (or code) from the trajectory demonstrates this rule. You can add detailed information in the comment. # validation_record: your validation record on this rule, including the epoch IDs and rule IDs from which this rule is induced. rule_system.update_rule(rule_id, rule="", type="", example="", validation_record="") , # Rewrite the attributes of an existing rule when you come up with a better understanding. # Input only the attributes you want to rewrite. Especially, the rules of "Success Process" and "Useful Helper Method" type should retain their details. Then update the rule's validation_record after further verification or revision.Listing 14: Case 1 Prompts for Builder **Rules for Success** You should extract "Useful Helper Method" and "Success Process ": * For each useful helper function identified: If it is not already included in a rule, create a rule of type "Useful Helper Method" and record its code unchanged in the rule's example section. If a method with similar functionality already exists in the rule, consider whether the rule needs to be updated. * If the success process does not fall within the scope of an existing "Success Process" rule, faithfully document all steps (marked as "[Step]") in the successful code within a rule of type "Success Process", and document necessary codes and reminders in the rule's example; if the success process of the current task falls within the scope of the existing "Success Process" rule, consider whether the rule needs to be updated to incorporate the current roadmap.Listing 15: Case 2 Prompts for Builder **Rules for Success** You should extract "Useful Helper Method" and "Success Process ": * For each useful helper function identified: If it is not already included in a rule, create a rule of type "Useful Helper Method" and record its code unchanged in the rule's example section. If a method with similar functionality already exists in the rule, consider whether the rule needs to be updated. * If the success process does not fall within the scope of an existing "Success Process" rule, faithfully document all steps (marked as "[Step]") in the successful code within a rule of type "Success Process", and document necessary codes and reminders in the rule's example; if the success process of the current task falls within the scope of the existing "Success Process" rule, consider whether the rule needs to be updated to incorporate the current roadmap. **Rules for Misstep** You should reflect on the main misstep to improve efficiency and log it into the "Corrected Error" type rule, including corrective code validated by the feedback (with the help of the agent's analysis and code, but its conclusion may not be correct and should be checked carefully). Remember that the rules of "Success Process" and "Useful Helper Method" type should retain their details.Listing 19: System Prompts for Consolidator[Role] You are observing a housekeeper agent as it codes and acts within a simulated environment (game). Your goal is to construct a manual of rules to assist the agent in completing various tasks in the environment. Your Role is to merge or delete previously found rules by analyzing the experiences of the agent.[Functions] You will be presented with the current found rules. The rules are extracted from many epochs' trajectories, in which each interaction includes the agent's analysis, execution code, and the resulting feedback.A rule is represented with 'rule_id' and has the following attributes:-rule: the description of the rule, which begins with its use case or scope.-type: the type of the rule. -example: an example (or code) from the trajectory demonstrates this rule. You can add detailed information in the comment. -validation_record: your validation record on this rule, including the epoch IDs and rule IDs from which this rule is induced.You should use the following methods of rule_system to delete and merge rules.rule_system.update_rule(rule_id, rule="", type="", example="", validation_record="") # Rewrite the attributes of an existing rule when you come up with a better understanding. # Input only the attributes you want to rewrite. Listing 20: System Prompts for Formulator[Role] You are observing a housekeeper agent as it codes and acts within a simulated environment (game). Your goal is to construct a manual of rules to assist the agent in completing various tasks in the environment. Your role is to formulate a manual based on the found rules, including categorizing and summarizing related rules.[Functions] You will be presented with the current found rules. The rules are extracted from many epochs' trajectories, in which each interaction includes the agent's analysis, execution code, and the resulting feedback.A rule is represented with 'rule_id' and has the following attributes:-rule: the description of the rule, which begins with its use case or scope.-type: the type of the rule. -example: an example (or code) from the trajectory demonstrates this rule. You can add detailed information in the comment. -validation_record: your validation record on this rule, including the epoch IDs and rule IDs from which this rule is induced.[Actions] At each epoch, an agent is created in an environment. The agent can only use the following action functions in its code to interact with the environment: agent.go_to(receptacle) # Go to a receptacle and update the agent's location. agent.open(receptacle) # Open a receptacle and observe its contents. agent.close(receptacle) # Close a opened receptacle. agent.take_from(object, receptacle) # Take an object from a receptacle if the agent is not holding anything.agent.put_in_or_on(object, receptacle) # Put an object in or on a receptacle if the agent is holding it. agent.use(object) # Use a lamp. agent.clean_with(object, receptacle) # Clean an object with a receptacle. agent.heat_with(object, receptacle) # Heat an object with a receptacle. agent.cool_with(object, receptacle) # Cool an object with a receptacle. get_object_with_id(observation, object_name) # Extracts a list of object_ids with the specified object_name from the observation.[Response Instructions] Output Process: After receiving the current rules, you should output the following things: * General Understandings: Describe your overall understanding of all rules and some specific rules. * Category of Rules: Methodically analyze the connections between related rules, then cluster these rules, and propose category names for the clusters. Make sure each rule must belong to one and only one category! * The Manual: Finally, sequentially write a structured manual within ''''markdown' and '''''. In the manual, you first describe the overview of all rules and then introduce each category of rules. In each category, you should list the rules and write rule_id within ** and **.Detailed instructions:1. Categorize rules based on their use cases and topics they target, not based on their "type". 2. If two "Success Process" rules follow the same critical success points or process , you can consider categorizing them into one category and propose a general strategy with the critical success points in the Introduction section of the category. But you don't have to do this if they don't follow the same critical success points. 3. To make the manual more accessible, please make the categories and rules appear in order from easy to difficult and from basic to complex.NeurIPS Paper Checklist Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.All tasks in ALFWorld are divided into 6 task types, e.g., pick_heat_then_place, look_at_obj_in_light. For each task type, we store only one skill code.Notice: These prompts for the Builder are environment-independent and shared across all environments.The manual doesn't include Validation Logs of the rules, which are not visible for the Planner. 


Decision: This paper introduces AutoManual, which is a framework built upon LLMs to facilitate planning using three agents; Planner, Formulator, and Builder. This planning framework automatically improves with experience by codifying (Formulator) and improving/revising (Builder) a set of planning rules which can then be used by the Planner to accomplish a goal. AutoManual is validated on two benchmarks against various baselines.

The reviewers were mixed but not negative. Reviewers shared concerns that the > 50 tasks evaluated were 'homogeneous' in nature, which called into question how generalizable the approach was. There were also remaining questions about how much sensitivity there was to prompt engineering due to the temperature = 0.

The reviewers agreed that there was originality, positive results (including ablations), and sound results. While concerns remain about generalizability, I am inclined to view this paper on the positive side due to the careful way in which the approach was evaluated, the positive results, and the contribution that this paper makes towards open-ended learning with LLMs for planning, which is a difficult problem for LLMs.

Reviews: review: paper_summary: paper_summary: The authors propose AutoManual, a method for improving LLM agents by collecting and updating rules online via rule-assisted interactions in the environment, where code form planning is used for the interactions. In addition to the management of rules, AutoManual also maintains skill and reflection libraries per task type, where successful trajectories form the skill libraries and failed trajectories form the reflection libraries. The authors empirically show that AutoManual outperforms the compared baselines in ALFWorld and MiniWoB++ environments.

**Post-Rebuttal/Discussion**: While I appreciate the detailed author response, I don't think it fully addresses the major concerns I had. For example, by "maintaining the skill and reflection libraries per task type (L166)", I meant there exist explicitly and discretely isolated skill and reflection libraries, one for each task type. This design is made by the humans, not the LLMs. It may be doable for ALFWorld, which is comprised of clearly separable task types, but it would require more advancement for more complex environments. Therefore, I maintain my original score.
strength_weakness: strength_weakness: - While the presented empirical evaluations mainly compare the number of examples used as a measure of the amount of domain or human knowledge provided to the methods, such knowledge can be injected in different forms, e.g., input prompts or the exposure of LLMs such as GPT to the benchmarks and corresponding execution codes during their training. But another point I notice is that, this work injects such knowledge in a more explicit or discrete form by maintaining the skill and reflection libraries per task type (L166), which contributes most to the performance according to Table 3.
- Over-claiming statements need to be clarified or toned down. For instance, the Abstract claims that LLM agents "typically require elaborate design and expert prompts" in specific domains and suggests AutoManual as a solution. However, it appears that AutoManual leverages human-designed prompts for each domain as well as domain knowledge for design (as mentioned above).
- The experimental settings for the GPT-3.5-turbo results may not be entirely fair. According to Table 1 from Appendix D, the "GPT-3.5-turbo" runs with all the baselines are actually based on GPT-3.5-turbo except for the Builder component for ExpeL. It means that the Actor and Self-reflection for offline trajectories for ExpeL are also equipped with GPT-3.5-turbo, unlike Planner and Builder for the building stage for AutoManual.
- The manuscript can be improved in terms of presentation. especially the choice of technical vs non-technical terms. For instance, "free-form code" appears multiple times, but I couldn't find clarifications on what exactly "*free-form* code" is and whether (and how) it is different from just "code", which I assume is not. I have a similar concern about "natural programming capabilities" (L141).
questions: questions: * Are the "initial rules" made or selected by humans or fully automatically derived by Builder?
* Please take a look at the Weaknesses section above.
limitations: limitations: The authors discussed the fair limitations in the Limitations section.
paper_summary: paper_summary: The authors propose a new approach to interactive environment interactions by LLMs which they dub AutoManual. The approach takes an LLM and uses it in several different roles, differentiated by what context is included in their prompts, and some number of saved rules. The fundamental idea of the approach is to optimize over the saved rules in order to improve the performance. The authors demonstrate that AutoManual can perform well with a single human-example in both ALFWorld and MiniWob++ compared to baseline approaches and they demonstrate the importance of its various components through an ablation study.

Edit: I have updated my review based on the authors' response.
strength_weakness: strength_weakness: I have one concern with the paper and a few minor concerns.

My major concern has to do with the reported performance of AdaPlanner. The authors acknowledge that they could not replicate the performance but justified this that AdaPlanner's format requirements were too strict. The original authors, Sun et al. identified that the issue with 3.5-turbo (and one would assume later models) was their tendency to hallucinate. If AutoManual is robust to these hallucinations that would be excellent, but it would make sense to include an AdaPlanner implementation with gpt-3 given that is the stated preferred models by AdaPlanner's authors. Further, the results with gpt-3 appear to meet or exceed the performance of AutoManual. This is by far my largest concern with the current paper draft and I'd strongly encourage the authors to include an AdaPlanner gpt-3 baseline or explain their reasoning not to.

In terms of minor concerns, there are a number of claims that are somewhat overblown in the current paper text such as "to enhance their readability and global understanding" when no results demonstrate improved readability or global understanding. Similarly, the authors claim the approach is "user-friendly" and "like a teacher" in the same paragraph, given that the authors do not include any human subject study or other user tests these are unsupported in the current draft. "making it grounded and well-understood" would be yet another unsupported claim. Another minor concern is that some implementation details and associated design decisions are not given, such as not specifying the value of Nmax used for any of the experiments. Finally, the language is somewhat off in certain parts such as "the function should be called"->"this function should be called", "like a teacher conveys"->"like how a teacher conveys", and so on.
questions: questions: 1. Why didn't the authors include a gpt-3 baseline for AdaPlanner?
2. Do the authors feel AutoManual handles hallucinations better? If so, why?
3. Can the authors given the value of N_max and other hyperparameters missing from the paper?
limitations: limitations: Yes the authors have adequately addressed the limitations.
paper_summary: paper_summary: LLM agents show promise in many environments but require expert prompts in specific environments to perform well. This paper introduces the AutoManual framework which autonomously builds a 'manual' of rules through interactions to help adapt to new environments. The framework consists of 3 agents: the Planner which generates code to carry out a plan, the Builder which updates rules, and a Formulator which consolidates the rules into a manual. The Builder updates rules through a original case-conditioned prompting strategy to mitigate hallucinations. AutoManual is evaluated on 2 benchmarks: AlfWorld and MiniWoB++ and achieves high performance.
strength_weakness: strength_weakness: This paper is pretty strong already so there are only 2 things I wanted to point out that could make it stronger.

1. The ablations are very helpful in justifying design decisions but the table looks odd with seemingly random kinds of ablations. I understand if there is a concern with resource-constraints running all 16 combinations of ablations but why this set of 7 ablations?
2. Temperature 0 for OpenAI is typically a red flag to me that performance is brittle since there are no guarantees of determinism [1]. Despite experiments being run 3 times and averaged, I've experienced varying performance across different days. I'd be interested to see if a subset of the results remains relatively the same.

[1] https://community.openai.com/t/why-the-api-output-is-inconsistent-even-after-the-temperature-is-set-to-0/329541
questions: questions: Suggestions
- Provide an explanation/justification for the incomplete ablation table in the paper
- Provide a subset of results (e.g. a single column) to verify performance isn't brittle due to temperature 0

Questions
-  Manuals are constructed by sampling specific task types in both environments. I am curious whether you experimented with constructing manuals with limited task types and testing on unseen task types. In other words, can the manuals be used to perform other tasks (e.g. for AlfWorld only constructing a manual out of Put but testing on Put Two)
- Despite the case conditioned prompting to mitigate hallucinations for the Builder, the Planner can ignore these rules as noted in the limitations. Is this the most common failure case and, if so, what were the practical solutions that were taken to make the Planner pay attention to the Builder's rules?
- As a follow-up, AlfWorld and MiniWoB++ are environments where there is likely overlap in the LLM's training data and the benchmarks (home environment and web-based environments); however, how would AutoManual perform in environments that may not align with the LLMs priors (e.g, Atari games or Sokoban)
limitations: limitations: The limitations section is transparent and the broader impacts section addresses societal impact.
paper_summary: paper_summary: This paper introduces AutoManual, a framework enabling LLM agents to autonomously build understanding of new environments and generate instruction manuals through interactive learning. The system comprises three main components: a Planner agent generating code-based plans for environment interaction, a Builder agent updating a structured rule system based on observed trajectories, and a Formulator agent compiling rules into a comprehensive manual. To mitigate hallucinations in rule management, the authors implement a case-conditioned prompting strategy for the Builder. The framework is evaluated on the ALFWorld and MiniWoB++ benchmarks, demonstrating high success rates. AutoManual aims to address limitations of prior approaches such as path dependency and reliance on extensive human-provided instructions, potentially improving the adaptability and generalization of LLM agents in diverse environments.
strength_weakness: strength_weakness: 1. Heavy reliance on very capable LLMs like GPT-4, and dependency on oracle feedback during rule construction process limits broader applicability, do we have results by fine-tuning smaller models on the generated data?
2. Scalability to larger, more complex environments is unclear - the current approach puts all rules in LLM context. It’s not clear how easily the method can be generalized to other environments, for example, SWE-Bench.
3. Limited evaluation of the quality and usefulness of the generated manuals for humans. Lack of theoretical analysis on the rule learning process.
questions: questions: 1. How is the initial rules and examples selected and how sensitive is the performance to the quality of the initial rules/examples provided? How this initialization needs to be changed when we switch to different environments?
2. In Section 3.2 Skill and Reflection library construction, the most similar task is retrieved from the skill library when new task comes, how is the similarity between task been defined?
3. In 3.3 Rule types definition, how does the 6 rule types come up?  
4. Table 2 shows the success rate of LLM agents on 9 task types with feedback, what are those feedback, and is the feedback rely on an oracle who know the correct answer/solution for the given query?
limitations: limitations: The authors provide a reasonable discussion of limitations, including reliance on GPT-4 and potential scaling challenges. However, more concrete mitigation strategies are missing. They also touch on broader impacts, noting both positive potential (e.g., improved safety, worker training) and risks (e.g., unpredictable behaviors). 

Side note: should the limitation and broader impact discussions be included within the main 9-page scope rather than in the Appendix?

Comments: title: Reviewer Engagement
Response to Authors
Re: Rebuttal by Authors
Code for "AutoManual on WebArena (Reddit)"
Publicly available code of AdaPlanner; The issue with reproducing AdaPlanner; Code for "3- GPT-3 Tends to Copy Curated Examples"
comment: Dear Reviewer,

Please remember to engage with the authors.

Thanks!

AC
Thank you for the response. I appreciate running the temperature experiment - I am happy to see that the results are relatively robust and am excited to see this applied to other domains in the future! I have increased my score.
In order to reproduce Alfworld, the following packages are required to be adjusted:
```
pip install Werkzeug==2.2.2
pip install langchain-community langchain-core langchain_openai
```

I appreciate the authors' rebuttal, as they have addressed my questions about the framework and method. That being said, I would recommend including the above clarification in the final version. Meanwhile, I maintain a different perspective on the interaction complexity of tasks such as SWE-Bench and the coding tasks in AgentBench. In my view, these environments involve significant interaction, particularly in the form of responding to automatically generated feedback, such as stdout.

A critical characteristics of a LLM agent is the agent's ability to interpret and act upon system feedback in different modalities. For instance, in the coding environment, when an agent misuses a command, the terminal often suggests using `<command> --help` for more information. However, many off-the-shelf agents struggle to leverage these suggestions, instead opting to generate new responses without gathering the available additional information.

I believe the mechanism proposed in this paper has the potential to address these information retrieval behaviors under certain conditions. This is precisely why I strongly recommend expanding the evaluation to include environments with different modalities. This would demonstrate that the AutoManual's success extends beyond navigation-like tasks (such as household household navigation in ALFWorld or web navigation in MiniWoB++ and WebArena) to a broader range of real agent tasks.
Testing the proposed method in these varied environments would provide compelling evidence of its versatility and effectiveness across different types of agent interactions. 

Given these considerations, I maintain my current score for the paper.
We have updated its access permissions, and you can download traces at https://send.cm/d/1DdXT
Thanks to the authors' for their detailed response! My major concern around Adaplanner has been addressed. I would suggest including a truncated version of that in the next version of the paper. As such, I have increased my score. 

I disagree with the characterization that the claims around human judgement are intuitive. I also do not agree that reviewers' subjective opinions would be sufficient to support these claims from the provided examples. I believe that arguments around human response to any system or system output require a human subject study for support.
The traces of the test results requires email and access request.
We provide the code and trajectories at https://anonymous.4open.science/r/automanual-3860.
The publicly available code of AdaPlanner [https://github.com/haotiansun14/AdaPlanner]

The issue of reproducibility on AdaPlanner's GitHub [https://github.com/haotiansun14/AdaPlanner/issues/2]

The code and traces for the experiment in "3- **GPT-3 Tends to Copy Curated Examples**" are available at [https://anonymous.4open.science/r/adaplanner_alfworld-F4B7].