AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning
Authors: Minghao Chen
Yihang Li
Yanting Yang
Shiyu Yu
Binbin Lin
Xiaofei He
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
AI Agents
planning
decision making
programming
Primary area: other
TL;DR: The paper introduces AutoManual, a framework that enables LLM agents to autonomously adapt to new environments by interacting and generating comprehensive instruction manuals through online rule optimization.

Abstract: Large Language Models (LLM) based agents have shown promise in autonomously completing tasks across various domains, e.g., robotics, games, and web navigation. However, these agents typically require elaborate design and expert prompts to solve tasks in specific domains, which limits their adaptability. We introduce AutoManual, a framework enabling LLM agents to autonomously build their understanding through interaction and adapt to new environments. AutoManual categorizes environmental knowledge into diverse rules and optimizes them in an online fashion by two agents: 1) The Planner codes actionable plans based on current rules for interacting with the environment. 2) The Builder updates the rules through a well-structured rule system that facilitates online rule management and essential detail retention. To mitigate hallucinations in managing rules, we introduce a *case-conditioned prompting* strategy for the Builder. Finally, the Formulator agent compiles these rules into a comprehensive manual. The self-generated manual can not only improve the adaptability but also guide the planning of smaller LLMs while being human-readable. Given only one simple demonstration, AutoManual significantly improves task success rates, achieving 97.4\% with GPT-4-turbo and 86.2\% with GPT-3.5-turbo on ALFWorld benchmark tasks. The code is available at https://github.com/minghchen/automanual.

AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning
Minghao Chen',
Yihang Li?, Yanting Yang', Shiyu Yu², Binbin Lin Xiaofei He²
School of Computer Science, Hangzhou Dianzi University
State Key Lab of CAD&CG, Zhejiang University
Schooo of Software Technology. Zhejiang University Fullong Inc. 'NinnBo Port Group ninghaocheen0@@gaal1
Abstract
Large Language Models (LLM) based agents have shown promise in autonomously completing tasks across various domains, e.g., robotics, games, and web navigation. However, these agents typically require elaborate design and expert prompts to solve tasks in specific domains, which limits their adaptability. We introduce AutoManual, a framework enabling LLM agents to autonomously build their un- derstanding through interaction and adapt new environments. AutoManual categorizes environmental knowledge into diverse rules and optimizes them in an online fashion by two agents: 1) The Planner codes actionable plans based on current rules for interacting with the environment. 2) The Builder updates the rules through a well-structured rule system that facilitates online rule management and es- sential detail retention. To mitigate hallucinations in managing rules, we introduce a case-conditioned prompting strategy for the Builder, Finally, the Formulator agent compiles these rules into comprehensive manual. The self-generated manual can not only improve the adaptability but also guide the planning of smaller LLMs while being significantly improves task success rates, achieving 97.4% with GPT-4-turbo and 86. 2% with GPT-3.5-turbo on ALFWorld benchmark tasks. The code is available at btttps://gite tiub........ htt////nnnhhee/atttaaaeea
Given only one simple demonstration, AutoManual
Introduction
Recently, autonomous agents based on Large Language Models (LLM), e.g., ReAct [33], Re- flexion [16], SayCan [1], WebGPT [10], and Voyager [24], have demonstrated their potential to complete long-horizon tasks in grounded environments. These LLM agents operate by generating thoughts and actions that are executable in the environment. For customized environments, such as robotics [1, 7, 18, 23] and games [14, 24, 39], prior methods provide detailed instructions and in-context examples to familiarize L.LM with action functions (API) and the target environment. However, unlike these agents, humans can autonomously build and update their understanding of an unfamiliar environment through dynamic interaction.
Several existing methods enable LLM agents reflect on feedback [16, 21] save successful experiences as skills [21, 24, 35] to enhance the performance and reduce the reliance on human- provided examples. However, these reflections and skills have not been well exploited a deeper understanding of the environment. As a result, directly using saved skills as in-context examples can lead the Path Dependence problem, i.e., the agent blindly replicates the paths of previous successes, failing to adapt appropriately to new scenarios. Such problems are more severe in real-world situations characterized by high variability.
foster
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

A previous work, ExpeL [35], gathers the trajectories of LLM agents and extracts cross-tass task rules from them. However, these rules are extracted offline, making ExpeL suffer from the same distributional shift problem as Offline RI. [6]. Meanwhile, due to the simplicity of rule management, its rules are always armehair general and unhelpful for the Path Dependency problem. In this paper, we propose a novel framework called AutoManual to build well-organized understanding of the environment that can guide multi-task planning effectively. AutoManual leverages a dynamic rule system that not only extracts valuable experience, including skills and reflections, into different types of rules but also allows for continuously updating these rules in response to new situations. Additionally, error- pron-prone details are explicitly described in the rules to improve the robustness of planning.
AutoManual follows two alternating iterative processes to optimize the rules. First, given the observation and task of an episode, the Planner agent utilizes currently discovered rules to write free-form code as an actionable plan. The interaction between the environment and the Planner will loop until the episode ends. Second, based on this trajectory, the Builder agent will update relevant rules through the rule system. This online updating mechanism can timely verify whether the rules have deviations and are applicable to the Planner. After rules optimization, the Formulator agent categorizes these rules according to their application scenarios and compiles a comprehensive manual in Markdown format.
The challenge lies in enabling the Builder to accurately extract applicable rules from a long trajectory. as LLM are prone to generating hallucinations. To address this, we employ a case-conditioned prompting strategy, which directs the Builder to focus on specific rules according to the case of the trajectory. For example, if errors occurred in a trajectory, the Builder is first asked to determine which caused the error: an unrecorded situation occurred, or the Planner failed to follow existing rules. Based on this answer, the Builder will be given corresponding prompts to update relevant rules.
To summarize, our contributions are the following:
We adopt actionable code as the way for the Planner agent to interact with the environment. We introduce a structured rule system that allows the Builder agent to manage multiple types of knowledge from these code-based interactions.
We propose an alternating process between the Planner and Builder agents to optimize rules in an online manner and resolve the Path Dependency problem. To improve readability, the Formulator agent is introduced to reorganize and formalize the rules into a Markdown manual.
To facilitate rule management, we employ case-conditioned prompting strategy, which guides the Builder to manage specific types of rules for different trajectory cases.
Starting from a single demonstration, AutoManual can generate detailed instruction manuals for complex environments like ALFWorld and MiniWoB++. These manuals allow LLM agents to achieve remarkable success rates of 97.4% with GPT-4-turbo and 86.2% with GPT-3.5-turbo on ALFWorld, 98.3% with GPT-4-turbo and 92.7% with GPT-335-turke MiniWoB++.
2 Related Works
2.1 LLM for Agents Planning
Large Language Models (LLM) exhibit powerful reasoning and planning capabilities [11, 12, 28, 33, 39] while requiring much fewer demonstrations than traditional learning methods. With this planning capability as the core, LLM agents are being developed for use in robotics [1, 7, 18, 20, 23], game-playing [14, 24, 27, 39], software development [3, 15], and other fields [30]. Prior studies [16, 21, 33] allow agents to adjust actions or plans based on environmental feedback to improve planning performance. Given the powerful programming capability of LLM, several works, e.g.. CodeAsPolicy [7], ProgPrompt [18] and AdaPlanner [21]. propose to use Python code as the plan of LLM agents. This form of output can automatically respond to in-plan feedback and achieve better performance than the action and JSON format [21, 25].
2.2 Self-improvement of LLM Agents
Embodied agent research has long sought to enable agents to self-improve through interactive experiences. Unlike traditional barning-based agents that require extensive iterations for optimization,

---

Reflexion [16] allows LLM agents to reflect on previous failures and quickly improve their plans. Some works [32, 34, 36] combine tree search with reflection to deliberately seek better solution. Apart from failure experiences, prior studies [21, 24, 39] utilize successful experiences as skills to assist future planning. Voyager [24] stores generated and verified programs into the skill library as a new skill for more complex tasks. AdaPlanner [21] also discovers and archives successful programs into skill memory for future similar tasks. However, these methods stop updating skills after storing them, which inevitably leads to the Path Dependency problem.
Another series of works [26, 31, 38] employs LLM as prompt optimizer to enhance its own performance. In contrast to our approach, which addresses challenges in unfamiliar environments, these studies focus on enhancing LLM reasoning performance. As result, their optimized prompts are typically brief and lack environmental knowledge.
2.3 Memory Management of LLM Agents
For LLM agents, learning from past experiences can also be viewed as managing the episodic memory [16]. CLIN [9] proposes to keep updating memory centered causal abstractions for new trials. Retrieval-Augmented Planning (RAP) [4] retrieves past experiences corresponding to the current situation. MemGPT [13] allows L.M to select content retain in working memory and to search for information in long-term memory. Generative Agents [14] retrieve memories based on recency, importance, and relevance the current situation. Generative Agents also generate tree-structured reflections, but they focus on a continuous scenario rather than task-oriented rules.
2.4 LLM for Rule Discovery
Several recent works also investigate the rule discovery capabilities of LLM. Zhu et al. [40] propose typotthesesto--hheoriss (HtT), enabling LLM to induce and deduce rules for basic reasoning tasks. For LLM agents, ExpeL. [35] gathers the trajectories of Reflexion agents and extracts cross-tast rules from them. Furthermore, AutoGuide [2] generates state-aware rules and retrieves rules relevant to the test-time state. Unlike ExpeL. and AutoGuide, which extract rules from offline experiences, we update rules in online manner, verifying their reliability and applicability. For more discussion of differences, refer to Appendix C.
3 Methods
3.1 Auto.annua Overview
Our AutoManual framework, shown Fig 1, consists of three main stages. Building stage: The Planner agent and Builder agent collaborate to build rules from the interactive environment. The Consolidator agent merges or deletes redundant rules when the rules exceed the maximum rule number. Formulating stage: The Formulator agent categorizes the rules, summarizes the key points, and formulates them into a manual in Markdown form. Testing stage: Based on the generated manual, test-time Planner agent will be evaluated through test tasks and scenarios.
Formally, an Interactive Environment can be modeled as a Partially Observable Markov Decision Process (POMDP): (S, A, T, G, O). At the start of each episode, a scenario so E S will be initialized, text-grounded task g E and the initial observation Og E O (processed into textual form) will be given. The environment can be interacted with through permissible actions (API) set A. After executing an action a E AA the environment will return the result of the action and the new observation of based on the dynamics T(s'|
E T and O(d'|s') Finally, when the episode is done, binary reward r E {,1, }} indicating the failure or success of the task will be returned.
We approach the learning of environmental rules as an optimization problem:
max Et.. (p((o))(TT)
where denotes rules in our rule system, p(- denotes the policy of the Planner given the current rules θ and Tp denotes trajectory of p(-10) starting from (oo, g). Classic policy gradient methods [29] solve such problems through stochastic gradient ascent, i.e., executing the current policy obtain the episodic reward and back prok-propgggting gradients to update the parameters.
3

---

Interactive Environment
Anappa
Coea
Trajectory
Anayea
Sode
Trajectory
Planner
Pln Generated Mansal
Planner
Planner
Testing Stage
PUn
PUn
Consolidator
M
Reaorr wiiting
Marged
Builder
Update
Updatt
Categeriza
Online Rule System
Rule
Formulator
Building Stage
Formulating Stage
Figure 1: AutoManual Overview:
AutoManual operates in three stages: (1) Building Stage:
The Planner agent interacts with the environment by coding actionable plans. After receiving the current trajectory of the Planner, the Builder agent manages rules through the online rule system. (2) Formulating Stage: The Formulator agent formulates the resulting rules into a Markdown manual. (3) Testing Stage: A test-time Planner agent utilizes the manual to complete testing tasks.
Inspired by this online reinforcement learning paradigm, we follow two alternative processes to optimize the nules The Planner practices current rules through interaction within an episode. 2. The Builder updates the rules @ based on this trajectory. Compared to traditional parameter optimization, sample-inefficiee gradient ascent is replaced by text-based rule management. We design well-structured rule system described in Section 3.3 to ensure the rule updating contributes to rewards. Additionally, to limit the role of human expertise, we only provide a simple example demonstrating the output format to agents. Then, manually derive several initial rules from this example as the starting point of the optimization.
3.2 Planner Agent for Interactive Planning
As demonstrated by the success of Voyager [24] and AdaPlanner [21], code-based planning can leverage the powerful programming capability LLM and automatically react to in-plan feedback. Voyager and AdaPlanner output and refine complete solution function for the task, which is poten- tially reusable. However, this function-form output is difficult to adjust in response to environmental feedback, as it requires maintaining the integrity of the plan throughout.
Our Planner Agent outputs free-form code as plan, which aligns more with the natural programming capabilities of LLM [7, 22]. This form simplifies planning by only generating code necessary for the current environmental situation and feedback without the overhead of integrating previously executed code. As shown in Fig 2, at the start of current rules θ, relevant samples from the skill and reflection libraries, the target task g. and initial observation Op- System prompts contain the role, permissible actions A, response guidelines, and simple example (detailed Appendix H). The output of the Planner is structured into four segments during each cycle:
episode, the Planner receives system prompts,
Analysis: The understanding of the current situation and reflection on previous errors exist.
2. Related Rules: Rules (along with their IDs) that need to be considered in this situation.
3. Overall Plan: The general plan to complete the task.
4. Code: A block of Python code divided into steps. The Planner encouraged to define helpful
functions in the code, which might be reusable in similar scenarios.
We denote this response of the Planner as [thought, codes], where thought, denotes the first three seg- ments. code, executed in the environment is followed by feedback 6,, which informs the subsequent output cycle. This process iterates until the episode ends response limit is reached.

---

System Prompts
samples library
Current Rules
Initial obi
Task: heat some mag and put cabiret
Analysis:..
Skill Library
Reflection Library
mic J: Ensure all receptacles are seunched Overall Plan
der tire, axcectiopett, receg, ra caes, as9e)s sbservatian apenn se..
Mistakes and Misunderstandinn
Failure
sajest
mecgp_ta_checc Peableet_
Iioeg Search "caolnet toae)
Fund, iojectt
take the mag and try beat.
Code error ecoumed
Strap Seand 1184 aertl
istap best mugu
Feedbuck
sgant, auttt "aug
Analysis:
1tte a Take
Foodback: "Wothing tappers
Related Rules
Planner
mic receptacle interaeting
Prompi:
Prompe: segment
Overall Plan
decersstion sopetuee 2at (ltowwe).)))
abcerost sper appetuppeat seeat altroooee 11)
Direet Success
Indirect Suecess
Failure
Result
The Trajectory
Figure 2: The Planner Trajectory: Given the current task and rules, the Planner will interact with the environment through free-form code. Based on the trajectory result, the Planner will generate corresponding conclusion, which will be saved in the skill or reflection library.
As shown in Fig 2, according to the episodic reward, we categorize the result into Direct Success, Indirect Success (errors occurred but solved later), and Failure. In the case of Direct Indirect Success, the Planner will be prompted to organize its previous code into code block. For Indirect Success, additionally summarizes the mistakes and misunderstandinns that cause errors. For the Failure case, the Planner will be prompted to reflect on the reason for the failure carefully, suggest reasonable corrections, and specify the code segment that caused the error. We denote this response of the Planner as conclusion. Finally, we obtain a trajectory of the Planner:
= (Oo. 9, [thought, codee)) c1, ci...., (oooughtt.codde] cr, conclusion)
(2)
Skill Library and Reflection Library: Apart from rules, also manage and transmit conclusions from previous episodes, which provide essential details for generating planning code. the case of Direct or Indirect Success, we save the code block in conclusion
skill for that task type into the skill library [21, 24]. In the Failure case, we save its conclusion as a reflection for that task type into the reflection library. When new task comes, the code block of the most similar task is retrieved from the skill library. If there is no existing skill for the new task type, the reflection for that task type will be returned. As mentioned in the Introduction, compared with rules, these skills and reflections contain more programming details but are less generalizable to new scenarios, i.e., the Path Dependence problem.
Cooperation between Agents: our framework, rule management is not solely the responsibility of the Builder; the Planner also plays critical role by explicitly identifying the rules it engages in its response. This cooperation is facilitated by including the Planner's s thoughts within the trajectory which is provided to the Builder. This synergy enhances the identification and adjustment of problematic rules. In addition, conclusion from the Planner contains the detailed success process or reflections errors, which further assist the Builder in managing corresponding types of rules.
3.3 Builder and Consolidator Agents for Rule Management
Upon receiving the trajectory Tp. the Builder has to manage the rules through the rule system.
All tasks in ALFWorld are divided into 6 task types, e.g., pick_hear then tteetteeeeeeeeeeeeeeeeeeeeeeee place, look obj in light. For each task type, we store only one skill code.
5

---

Base
will writelupdate
Case 1 Prompt
"Sucuess Process rule
Builder
Direct Success
Base Prompt
I willl wnitllpdatee a Process" rule "Corrctted Emur"...
Imperfect Rules
Prompt
Builder
Cooddar
Indirect Success
coaclade uihor
pricces cireldlly and
"ingorfed Aggns
Imperfect Agent
Base Prompt
will add emplasis "Success Process" rule
Bellder reggoeer
Case Prompt
Builder
Planner
Trajectory
Fallure
Base Prompt
will waite "Unsolved Error rale
Chosly Pronpe:
Imperfect Rules
Base Prompt
Builder
Imperfect Agent
Bulider requene:
Base Prompt
will add an emphasis "Sucness Process" rule...
Prompt
Builder
Possible Results
Possible Cases
Figure 3: Case-Conditioned Prompts: Given the current trajectory, the Builder classifies the cause of the major eπror as "Imperfect Rules" or "Imperfect Agents". Then, the Builder will get the base prompt and corresponding prompt guide its rule management.
We intuitively identify rules as the kinds of knowledge that help task completion, including the analyses of the observed phenomenon ((ol o, a), the mechanism T(s'ls, a). and the correlation between the reward and Tp. i.e., the success process the occurred error. Therefore, unlike ExpeL [35] and AutoGuide [2]. which derive general insight from the trajectory, categorizes six specific rule types to extract environmental knowledge that targets different aspects of the trajectory. Furthermore, each rule in system is enhanced with Example attribute to illustrate its application and important details, making grounded and well-understood. Specifically, each rule in the rule system has these four attributes:
Rule System:
system
1. Rule Type: The type of the rule, chosen from ["Special Phenomenon", "Special Mechanism", "Useful Helper Method", "Success Process". "Corrected Error", "Unsolved Error"];
Rule Content: A description of the rule, beginning with the scope of its applicable scenarios;
3. Example: An example or code from the trajectory demonstrates this rule, where additional remarks, e.g. error-prone details, also be added it;
4. Validation Logs: Logs that track the rule's application and updates, including episode and rule IDs that trace the rule's s evolution, serving as a reference for the Builder and Consolidator.
The Builder manages the rules through the following functions of the rule system:
write_rules **lel_ atttributes:: Write down a new rule with its four attributes.
update_rule( rule(rule__ _id, **yule tryul_attributess Rewrite the attributes of a existing rule.
trop_generating(): When the trajectory is not needed or insufficient to derive any more new rules, the function should be called.
Similar to hierarchical reflections in Generative Agents [14), we allow the Builder utilize existing rules to induce more general or deeper rules and record their dependence in Rule Content or Validation Logs, more discussed in Appendix D.
To mitigate the risk of erroneous rule creation, such as deriving rules of success from a failed trajectory, we employ case-conditioned prompts. As illustrated in Fig the Builder first analyzes and determines if the major errors stem from "Imperfect Rules" or "Imperfect Agent". Based on this analysis and the trajectory results, targeted prompts guide the Builder in rule management 2 For example, in case of indirect success due to imperfect rules (Case 2), the prompts will guide the Builder extract or update the success process, helper methods, and error reflections
Cass-Conditioned Prompting:
Notice: These prompts for the Builder mavironment-indeeene and shared across all environments.

---

for all in corresponding rule types. Finally, the Builder responds with the potential rules detailing their relation with existing rules and uses the functions of the rule system to manage rules.
Rule Consolidation:
When the number of rules in the rule system exceeds Nmar. the Consolidator agent steps in consolidate related rules and delete redundant rules. It uses three functions of the rule update_rule(rule rule(rule_id)
system: get_ "ir__rrjecttrr(ppee (eprrndeeiddedeie Given the current rules, the Consolidator identifies potentially relevant or overlapped rules, uses get jrajerjjttory function to investigate the trajectories they depend on, and finally calls the remaining functions manage the rules. During the management, the Consolidator ensures that consolidation retains details of rules and examples.
and ceiieterrue(rriee
3.4 Manual Formulation
Once the building stage is complete, we can obtain set of rules targeted to different situations. whose applicability has been validated through online optimization. Our next goal is to enhance their readability and global understanding. To achieve this, we introduce the Formulator agent, designed to transform these rules into a user-friendly manual, analogous to teacher imparting a wealth of knowledge through easily digestible lessons. As depicted in Fig 1, the Formulator begins by categorizing all rules based on their target scenarios. This categorization aids in structuring the manual and ensures that related rules are discussed together, which enhances the logical flow and accessibility of the information. For each category, the Formulator drafts an introduction, summarizing the rules contains and highlighting the key points and overall principles that govern the specific scenarios. Finally, the Formulator compiles the rules and their introductions into comprehensive manual formatted in Markdown.
Experiments
In line with prior works [2, 21], we conduct the experiments on three interactive environments: (1) ALFWorld [17] is a text-based virtual household environment containing six distinct task types. We run the building stage on 36 tasks (6 tasks for each task type) sampled from the training set of the testing stage on the validation unseen set containing 134 tasks across these six types. (2) MiniWoB++ [8] is simulated web environment where agents complete diverse tasks the Internet by performing keyboard and mouse actions. Prior works [5. 21] selects 9 task types with environmental feedback and 44 task types without feedback from MiniWoB++ tasks. We perform experiments on 9 task types with feedback or on all 53 task types. At each stage, we randomly sample tasks for each task type. (3) WebArena [37] introduces realistic web environments by emulating the functionality and data of popular websites. This benchmark poses significant challenges for LLM agents due to its large observation and action space, along with tasks that require longer planning horizons. Following AutoGuide [2], our experiments focus on the Reddit domain within WebArena.
ALFWorld, and each task is run only once. Following previous works [16, 21, 33].
During the building and formulating stages, we GPT-4-turbo agents. At the testing stage, we equip the Planner agent with GPT-4-turbo or GPT-3.5-turbo (gpt-3. 1opt-3.5-turbo-11055 1106), to evaluate the effect of generated manuals on relatively smaller LLM.
Compared Methods: In the experiments, we compare AutoManual with the following methods of LLM Agent: (1) ReAct [33] prompts LLM to generate the reasoning trace using CoT [28] and
next-step action; (2) Reflexion [16] agents generate reflection on task feedback signals, which is saved in the memory for subsequent trials; (3) Expel. [35] extract insights and skills from the offline
trajectories of Reflexion agents; (4) RCI [5] agent recursively criticizes and improves its output for solving computer tasks; (5) AdaPlanner [21] allows the LLM agent to generate and adaptively refine a code-style plan; (6) Planner+Lib. represents our Planner agent equipped with skill and reflection libraries ($3.2) during building and testing stages without any rules. We re-implement prior methods with GPT-3.5 and GPT-4 versions the same as ours for fair comparisons.
ReAct, Reflexion, and ExpeL provide LLM agents with 12 human examples (2 examples per task type) of ALFWorld. For AdaPlanner, they provide human examples (1 example per task type) of ALFWorld as the start of skill discovery. For our methods, agents are provided only one human example of the simplest task (Put) on F.rlo... On MiniWob+++ our agents are provided one human example (search- (search-engine) for tasks with feedback and 4 examples for all tasks. On WebArena,
7

---

Table 1: Success rate (%) of LLM agent methods on ALFWorld tasks. For each method, the number of all human examples used is listed. "Planner+Lib." represents only using skill&reflection library during the building and testing stages. We run all experiments times and show the average.
Methods
Examples
Put
Clean
Heat
Cool
Examine
Put two
ALL
Testing LLM: GPT-3.5-turbo
ReAct [33]
24.7
37.7
36.4
44.4
11.8
41.9
Reflexion [16]
87.5
44.1
73.9
50.0
61.1
35.3
59.8
ExpeL [35]
62.5
61.3
30.4
61.9
55.5
52.2
AdaPlanner [21]
83.3
46.2
65.2
74.2
68.5
52.9
63.3
Pianner+Lib.
77.8
88.2
82,6
72.7
37.0
27.5
66.5
AutoManual
95.8
79.6
87.0
78.8
100.0
66.7
86.2
Testing LLM: GPT-4-nubbo
ReAct [33]
95.8
76.3
69.6
86.4
72.2
52.9
76.8
Reflexion [16]
100.0
95.7
78.3
86.4
77.8
70.6
85.9
ExpeL [35]
94.4
82.8
72.4
81.8
72.2
58.8
79.2
AdaPlanner [21]
88.9
90.3
85.5
75.8
64.8
41.2
76.4
Planner+Lib.
100.0
93.5
100.0
93.9
88.9
39.2
88.1
AutoManual
100.0
98.9
100.0
95.4
100.0
90.2
97.4
Table 2: Success rate (%) of LLM agent methods task types with feedback and all 53 task types of MiniWoB++. For each method, the number of human examples used listed.
Methods
Examples
With feedback sye>)
Examples
ALL (53
Testing L.M:: GPT-3.5-tubo
[33]
22
45.6
104
77.3
AdaPlanner [21]
13
71.6
38
89.4
Planner+Lib.
63.6
87.0
AutoManual
82.2
92.7
Testing LLM:: GPT-4-turbo
RCI [33]
22
60.4
104
88.6
AdaPlanner [21]
13
74.1
38
90.3
Planner+Lib.
80.2
4
94.4
AutoManual
94.5
4
98.3
agents are also provided with one human demonstration. To reduce randomness, we performed each experiment three times and reported the average. More details of the implementation and prompts for AutoManual can be found in the Appendix.
4.1
Main Results
Main Results on ALFWorld: As shown in Tab. 1, AutoMan- ual significantly outperforms the existing methods, evidenced by overall success rates 86.2% when using GPT-3.5-turbe for the testing stage and 97.4% when using GPT-4-turbo. No- ticeably, AutoManual requires little expert prior knowledge about the environment and is only provided with one human example to achieve excellent results. In comparison, the rules induced by ExpeL hardly improve performance, its offline trajectories are composed of individual actions rather than code. We find the performance of AdaPlan- is lower than reported. One reason is that AdaPlanner requires LLM to output specific formats complete its function-ford code, which is difficult for creative LLM, e.g., GPT-4-turbo. In addition, AdaPlanner and Planner+Lib. are inferior to AutoManual because they only store successful paths as skills and inevitably face the Path Dependence problem. Especially, tasks in Put Two have various scenarios, such as "two objects can occur different processes to solve (Appendix G shows an example). Furthermore, Planner+Lib often does not mark error-prone points in its skills, such as "target objects may appear unconventional locations".
Table 3: Test on WebArena (Reddit).
Methods
Examples
Suc(%)
ReAct [37]
2
6.0
AutoGuide [2]
19
43.7
SteP [19]
14
55.0
Planner
1
51.1
AutoManual
65.1
same receptacle or different receptacles", that require

---

106
Avg. Succeso Rate(
Avg Sulles Rallet%
20
Crosa boo bye I0FT4 surtet
fanitionnall (GPT-4uurbs)
single rast Tyee (arn bubal
Planrer (aFt-kartto)
Croselbet Type Seurtl Singeitatt Type 10702 Sttbbl
Autiklamnal turssl
50 The Nambbe of Building Tasks
36
22 The Number of Building Tasks
(a) Cross-Task Single-Task Type.
(b) AutoManual v.s. Skill Library.
Figure 4: (a) The success rate curve with standard deviation when testing GPT-4-turbo or GPT-3.5- turbo on ALFWorld. Building performed cross-task or single-task type. (b) The success rate curve with standard deviation using AutoManual or Planner+Lib. when testing with GPT-4-turbo or GPT-335-turbo on task types with feedback in MiniWob++.
Table 4: Ablation study of AutoManual on ALFWorld when testing with GPT-4-turbo.
Online Skill&Reflect Lib.
Case Prompt
Formulation
Avg. Eroo Steps (1)
Success Rate (%)
2.3
1.5
88.1
1.3
90.7
1.6
89.5
1.0
93.8
0.5
0.3
97.4
Main Results on MiniWoB++: As shown in Tab. 2, the performance of AutoManual exceeds the previous methods and Planner+Lib. by large margin. Especially in 9 task types with feedback, these tasks have higher diversity and require LLM agents to cope with various situations. For example, the tasks in login-usee login-user-popup type will interrupt the agent's plan at any time, requiring the agent to cope with unexpected situations. Therefore, solely imitating previous successful experiences without extracting targeted rules will lead to task failure. Additionally, due the flexibility of free-form codes, our method shows better adaptability while requiring fewer expert examples than prior methods.
Learning Curves. We show the success rate curves (testing with GPT-4-turbo or GPT-3.5-turbo) when gradually increasing the tasks of the building stage in Fig In the left image, we share rules across all task types (Cross-task Type), as in AutoManual, each task type builds separate set of rules (Single-task Type) during the building stage. Fig 4 (a) demonstrates that sharing rules across task types facilitate rule optimization. The rules for each task type deepen understanding of the environment, thereby boosting the planning of other tasks. In Fig 4 (b), we compare AutoManual and Planner+Lib. 9 tasks with feedback in MiniWob++.. We find that Planner+Lib. tends to get stuck with a lower success rate. In the face of highly diverse scenarios, Skill Library cannot express the rules behind the environment, thus falling into the Path Dependency problem.
4.2 Ablation Study
In this ablation study, we quantify the impact of each core component of the AutoManual framework on performance, specifically focusing on success rates and error reduction during task execution. Since we allowed the Planner to replan up to times, each task could have up to 4 error steps.
Online v.s. Offfine Rule Management: We perform offline AutoManual by collecting all trajectories and then managing rules from them. As Tab 4 shows, without online rule management, the generated mundane
manual can only slightly improve planning (from 88.1% to 90.7%). This is because mistakes and fewer direct successes will occur in the trajectories (the distributional shift problem), and the rules cannot be verified by feedback from the environment.
Skill& Reflection Libraries: Retrieving historical conclusions is essential for correct planning, as they record massive interacting details that can complement the rules. Without them, there will be

---

more errors in the details, and the success rate drops from 97.4% to 89.5%. However, as discussed previously, using plain experiences without inducing rules will lead Path Dependency.
Case-Condee Case-Conditional Prompts: the hallucination, as evidenced by increase in success rate from 93.8% to 97.4%. These prompts ensure that the Builder updates rules reasonably and grounded.
This strategy further improves the rule management process by reducing
Effect of Manual Formulation: The final formulation of rules into comprehensive manual contributed to the success rate of 97.4% and decreased average error steps, demonstrating the effectiveness of presenting rule-basee knowledge in organized and accessible format. It not only aids the Planner in mastering multiple rules but is also friendly for human reading.
Conclusion
In this paper, we introduce AutoManual, framework significantly advancing LLM agents by enabling adaptability and continual learning through online rule optimization. Utilizing the structured rulc system, Matu autonomously generates comprehensive manuals, achieving high success rates in benchmarks like ALFWorld and MiniWoB++. This approach reduces reliance on human-provioe provided examples and expert interventions, illustrating a robust method for enhancing agent generalization and addressing the Path Dependency problem in diverse environments.
6 Acknowledgements
Thanks to Dr. Zhe Zeng for her invaluable assistance with the OpenAl API and GPT Plus services. This work was supported in part by The National Nature Science Foundation of China (Grant No: 62273303), in part by Yongjiang Talent Introduction Programme (Grant No: 2022A-240-G)..
References
[1] Michael Ahn. Anthony Brohan. Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Daniel Ho, Jasminc Hsu, Julian Ibarz, Brian lehter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Jayant Joshi, Ryan C. Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huch Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego M Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, F. Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan Yan. Do as can, not i say: Grounding language in robotic affordances. Conference on Robot Learning, 2022. 1.2
[2] Yao Fu, Dong-Ki Ki Kim, Jackycom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lec. Autoguide: Automated generation and selection of state-aware guidelines for large language model agents. ArXiv, abs/2403.08977 2024. 3, 6. 7, 8, 13, 14
[3] Sirui Hong. Xiawu Zheng. Jonathan P. Chen, Yuheng Cheng, Ceyao Zhang. Zili Wang. Steven Ka Shing Yau, Zi Hen Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, and Chenglin Wu. Metagpt: Meta programming for multi-agent collaborative framework. ArXiv, 02..23388.033222 2023.
[4] Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou, Jayashree Karlekar, Sugiri Pranata, Akira Kinose, Koki Oguri, Felix Wick, and Yang You. Rap: Retrieval- Retrievallaugmenneed planning with contextual memory for multimodal I1 agents. ArXiv, Abs/2402.03610, 2024. 3
[5] Geunwoo Kim, Pierre Baldi, and Stephen Marcus McAleer. Language models can solve computer tasks. In Neural Information Processing Systems, 2023. 7. 15,
Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ArXiv, 0bs/2005.006433 2020.
Jacky Liang, Wenlong Huang, F. Xia, Peng Xu, Karol Hausman, Brian Ichter, Peter R. Florence, and Andy Zeng. Code as policies: Language model programs for embodicd control. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493-9500, 2022. 1, 2, 4, 14
[8] Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement leam- ing on web interfaces using workflow- -giiedd exploration. In International Conference on Learning Representations (ICLR), 2018. 7
10

---

[9] Bodhisattwa Prasad Majumder. Bhavana Dalvi, Peter Alexander Jansen, Oyvind Tafjord. Niket Tandon, Li Zhang, Chris Gallison-Burche and Peter Clark. Clin: A continually learning language agent for rapid task adaptation and generalization. ArXiv, 1bs/2310.10134, 2023.
[10] Relichiro Nakano, Jacoh Hilton, Suchir Balaji, Jeff Wu, Ouyang Long, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobhe, Tyna Floundou, Gretchen Krueger, Kevin Button, Matthew Kuight, Beajamin Chess, and John Schulman. Webgpt: Browser-assisted question-answerigg with human feedback. ArXiv, abs/21 2021.
[11] OpenAL. Gpt-4 technical report. ArXiv, abs/2303,00000 08774, 2023. 2
[12] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddic Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models follow instructions with human foodback. In Neural Information Processing Systems, 2022. 2
[13] Charles Packer, Vivian Fang. Shishir G. Patil, Kevin Lin. Sarah Wooders, and Joseph E. Gonzalez. Memgpt: Towards iims as operating systems, ArXiv, 0tsss/330.08566 2023. 3
[14] Joon Sung Park, Joseph C. O' Brien, Carric J. Cai, Meredith Ringel Morris, Percy Liang, and Michael Bernstein. Generative agents: Interactive simulacra of human behavior, Proceedings of the 36th Annual ACM Symposiun on User Interface Software and Technology, 2023. 1, 1, 6
[15] Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, Maosong Sun, and Wei Liu. Communicative agents for software development. ArXiv, abs/2307.0790 2023. 2
[16] Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Neural Information Processing Systems, 2023, 1, 2, 3, 7, 8, 14, 15
[17] Mohit Shridhar, Xingdi Yuan, Mare-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. Proceedings of the International Conference on Learning Representations (ICLR), 2021. 7
[18] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated rohot task plans using large language models. 2023 IEFE International Conference on Robotics and Automation (ICRA), pages 1.2
11523- 11530, 2022.
[19] Paloma Sodhi, S. R. K. Branavan, Yoav Artzi, and Ryan McDonald. Step: Stacked Ilm policies for web actions. arXiv, abs/2310.03722 2024. 8
[20] Chan Hee Song. Jiaman Wu, Clay Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. Lim- planner: Few-shot grounded planning for embodied agents with large language models. 2023 JEEE/CVF International Conference on Computer Vision (ICCV). pages 2986-2997, 2022. 2
[21] Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang- Adaplanner: Adaptive planning from feedback with language models. In Neural Information Processing Systems, 2023. 1, 2, 3, 4, 7, 8,
15, 16
[22] Sai Vemprala, Rogerio Bonatti, Arthur Fender C. Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilitics. IEEE Access, 12:55682-5566 2023. 4
[23] Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi. Jun Takamatsu, and Katsushi Ikeuchi. Chatgpt empowered long-step robot control in various environments: A case application. IEEE Access, 11:95060- 95078, 2023. 1.2
[24] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi (Jim) Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. ArXiv, 0bs/2305.16299. 2023. 1, 2, 4, 5, 13
2023.
[25] Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Exccutable code actions elicit better Ilm agents. ArXiv, abs/2402.01030, 2024. 2, 14
[26] Xinyuan Wang, Chenxi Li, Zhen Wang. Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric Xing. and Zhiting Hu. Promptagent: Strategic planning with language models enables expert-level prompt optimization. ArXiv, abs/2310. 10.1427,, 2023.
11

---

[27] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Inter- active planning with large language models enables open-world multi-task agents. In Neural Information Processing Systems, 2023. 2
[28] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models, Neural Information Processing Systems, 2022. 2.7
[29] Ronald J Williams. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine learning, 8:229 256, 1992. 3
[30] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding. Boyang Hong. Ming Zhang. Junzhe Wang. Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Qin Liu, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng. Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huan, and Tao Gui. The rise and potential of large language model based agents: A survey, ArXiv, abs/2309 0.709477444 2023." 2
[31] Chengrun Yang, Xuezhi Wang. Yifeng Lu, Hanxiao I Liu, Quoe V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. ArXiv, abs/2309.034444 034309,,,,49,,.. 2023. 3
[32] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Neural Information Processing Systems, 2023. 3, 13
[33] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. Re- Act: Syncrgizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. 1, 2, 7, 15
[34] Wenqi Zhang, Tang, Hai Wu, Mengna Wang, Yongliang Shen, Guiyang Hou, Zeqi Tam, Peng Li, Yue Ting Zhuang, and Weiming Lu, Agent-pro: Learning to evolve via policy-level reflection and optimization. ArXiv, 175724 2024. 3
[35] Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Liu, and Gao Huang. Expel: Llm agents are experiential learners. In AAAI Conference on Artificial Intelligence (AAAI), 2024. 2, 3, 6, 7, 8, 14, 15
[36] Andy Zhou, Kai Yan, Michal tree search unifies reasoning acting and planning in language models. ArXiv, 10.00440, 2023, 3
Benpenttok-RRothman Haohan Wang, and Yu-Xiong Wang. Language agent
[37] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Rohert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Weharena: A realistic web environment for building autonomous agents. ArXiv, abs/2307. 1 13854, 2023. 7,8
[38] Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models human-level prompt engincers. ArXiv, abs/221 1.01910, 2022. 3
[39] Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijic Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang. Qiao, Zhaoxiang Zhang, and Jifeng Dai. Ghost in the minceraft: Generally capable agents for open- open-world world environments via large language models with text-based knowledge and memory. ArXiv, abs/2305. 17144, 2023. 1,
[40] Zhancheng 7hu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, and Hanjun Dai. Large language models can leam rules. ArXiv,
2023. 3
12

---

Appendix
Limitations
13
B Broader Impacts
14
C Difference with Prior Methods
14
Implementation Details
14
E Presentation of Generated Manuals
16
More Experiments
19
F.1 Ablation Study of Rule System
19
F.2 Sensitivity Analysis of Examples and Initial Rules
20
G Case Study
20
G.1 An Example of Path Dependency Problem
20
G.2 Trajectory of The Building Stage
22
H Prompts and Examples
26
H.1 Prompts for Planner Agent
26
H.2 Examples for ALFWorld and MiniWob++
28
H.3 Prompts for Builder Agent
32
H.4 Prompts for Consolidator Agent
35
H.5 Prompts for Formulator Agent
36
A
Limitations
Despite the significant contributions of the AutoManual framework, several limitations warrant further discussion. First, our method heavily relies the capabilities of GPT-4-turbo generate reliable rules, which may limit the framework's 's applicability with less advanced language models.
Secondly, the current implementation places all rules directly within the context of LLM, which, while effective smaller or moderately complex environments, may not scale well to larger, more dynamic settings. For such expansive environments, integrating the rule system with Retrieval- Augmented Generation (RAG) techniques, similar to the approach taken by AutoGuide [2], could enhance performance by dynamically selecting relevant rules based on the context, thereby managing the cognitive load on the LLM more efficiently.
Thirdly, for complex and challenging tasks, the agents of AutoManual are insufficient in exploring the environment, as they only attempt solutions based on current knowledge. To enhance agents' exploration of unfamiliar environments, it may be necessary to endow the agent with curiosity [24] combine it with tree search algorithms [32].
Lastly, there remains challenge in ensuring that the Planner agent consistently adheres to the rules outlined in the manual. In practice, the Planner may sometimes ignore these rules (or have hallucinations about observations) and cannot generate the ideal solution code that can be applied to different situations. This indicates need for additional mechanisms to enforce or verify rule compliance during operations. This issue underscores the potential necessity for developing more sophisticated methods to ensure rule adherence or introduce more robust validation steps within the planning process.
13

---

B Broader Impacts
The AutoManual framework, leveraging LLM agents, presents positive and negative impacts on safety. On the positive side, by autonomously generating reliable manuals, method enhances the predictability and reliability of LLM behaviors in dynamic environments, potentially reducing errors and increasing operational safety. However, relying on LLMs also introduces risks of unpredictable behaviors when agents encounter unanticipated scenarios or when rule adherence is not fully ensured.
Furthermore, the manuals generated by our method can be invaluable tools for human workers. They encapsulate a distilled form of interaction-base based learning that can aid in training, provide decision support, and improve task efficiency in various domains. This can not only enhance productivity but also ensure that human workers are better informed and prepared to manage the complex systems with which they interact.
Lastly, AutoManual's ability to generate structured, context-aware manuals from interactive experi- ences suggests a promising avenue for constructing comprehensive knowledge bases for AI. These manuals can contribute to a global knowledge base shared by LLMs of different sizes, promoting broader AI research and development. It offers a method to systematically organize and retrieve complex interaction data in a way that accessible and useful for both machines and humans.
C Difference with Prior Methods
We compare AutoManual with prior methods that extract rules from experiences, i.e., Expel. [35] and AutoGuide [2], and discuss all differences here:
1) Interactive Form. ExpeL and AutoGuide, following Reflexion [16], each time output the thought and next-step action to interact with the environment. Our Planner agent interacts using free-form code. As evidenced by previous works [7, 25], using code as the plan is more efficient because the code will automatically perform the in- ul-paan actions and require LLM responses far fewer times. Additionally, planning with code can enjoy the powerful programming capability of GPT. More importantly, compared with action sequences, code is easier generalize to similar scenarios, facilitating the management.
2) Online v.s. Offline. ExpeL. and AutoGuide extract rules from offline experiences of Reflexion agent. We update rules online by alternating rule practice and rule management. This online rule management can verify the reliability and applicability of the rules in timely manner, forbidding rules to be armchair general. Additionally, online-update rules can help planners continuously improve their trajectories, making it casier for higher-quality success processes to emerge.
3) Collaboration Between Agents. Building rules online also enables collaboration between the Planner and Builder agents. In our AutoManual, the Planner is prompted to describe the rules considered and analyze special phenomena, which ease the management of the Builder. contrast, the builder in Expel. and AutoGuide can only receive action sequences from the Planner.
4) Rule System. In ExpeL and AutoGuide, each rule has only two attributes: the content and score. In our rule system, each rule has four attributes: "Type", "Content". "Example", and "Logs". These attributes provide comprehensive representation of rules and facilitate the management and usage of rules. Moreover, we allow rules to build on top of other rules.
5) Handling excessive rules. ExpeL and AutoGuide utilize the rule scores to delete the low-score rules. However, we found that rule scores are unreliable in the experiments because the Builder tends to give overconfident scores to all rules. Instead, our AutoManual employs Consolidator agent to merge and delete redundant rules.
D Implementation Details
In the building stage of all experiments, the maximum number of rules was set to 12 to balance the context length and rule diversity. We use OpenAl's Assistant API for all agents to save their history and prevent duplicate inputs. We set all LLMs temperatures to 0 and maximum context length to 16000. Reflexion agents are allowed to try at most 3 trials for each task. For AdaPlanner and AutoManual, we allow the Planner agent to replan at most 3 times on ALFWorld and 6 times on
14

---

Table The GPT models used for each method in our implementation when using GPT-3.5-turbo or GPT-4-turbo as test-time LLM. Here GPT-3.5 denotes gpt-3. 10t.-3..-turb--te 1106 and GPT-4-turbo
denotes gpt-4-114 4-16 preview.
Test-Time LLM
Method
GPT-3.5-turbo
GPT-4-turbo
ReAct [33]
Actor: GPT-3.5-turbo
Actor: GPT-4-turbo
Actor: GPT-3.5-turbo
Actor: GPT-4-turbo
Reflexion [16]
Self-reflection: GPT-3.5-trbo
Self-reflection: GPT-4-turbo
Offline Trajectory:
Offline Trajectory:
Actor: GPT-3.5-turbo
Actor: GPT-4-turbo
Self-reflection: GPT-3.5-turbo
Self-reflection: GPT-4-turbo
ExpeL
Insight Extraction:
Insight Extraction:
Builder: GPT--4-turbo
Builder: GPT-4-turbo
Task Inference:
Task Inference:
Actor: GPT-3.5-turbo
Actor: GPT-4-turbo
Task/State/AAeett grounding:
Task/State/Agent grounding:
RCT [5]
GPT-3.5-turbo
GPT-4-turbo
Planner/Refiner: GPT-3.5-turbo
Planner/Refiner: GPT-4-turbo
Adaplanner [21]
ask_LLMC) GPT-3.5-tutt -tr.5---turb-iinstuut
ask LLM) G.PT-3.5-turo-insttue
Building Stage:
Building Stage:
Planner: GPT-4-turbo
Planner: GPT-4-turbo
Builder: GPT-4-turbo
Builder: GPT-4-turbo
AutoManual
Formulator: GPT-4-turbo
Formulator: GPT-4-turbo
Testing Stage:
Testing Stage: Planner: GPT-4-turbo
Planner: GPT-3.5-turbo
MiniWob++ in response the environmental feedback. In the Building stage ALFWorld, tasks for each task type, total of 36 tasks, by default. We shuffle all tasks, and when tasks in a task type succeed three times in a row during building, we consider that this task type has been solved and will no longer run The API call cost for building and formulating stages is about $14 in total.
We made slight modification to the text format of ALFWorld to make it more suitable for code: for each object, "object id" was changed to "object_id", and all preceding articles were removed. The maximum action step for task is set to 50.
For the 9 tasks with feedback Miniwob++, find that "email-inbox-nl-urre and "email-inbox- forward-nl-turks are repetitions of used 7 task types in the building stage, while in the testing stage, all types were evaluated. In the building stage, we run total of 42 tasks (6 tasks for each type).
and "email-inbor maabox-forward-nil Therefore, we only
For all 53 tasks Miniwob++, since rumning the building directly will lead to large number of rules, and tasks without feedback have low variability, we adopt a two-step building strategy: we first our building stage on 9 task types with feedback and then only update the skill and reflection libraries on 44 task types without feedback. The 4 examples for experiments on all task types are from the following task types: 'click-menu', "enter-date",
We chose these tasks mainly because they are difficult for GPT-4-turbo to try out due to the lack of HTMM feedback. For example, the HTML in 'enter-date' will not display the cursor but requires the cursor the correct position to enter.
Environmental feedback executing code in Mini +ob++ is as follows: Whenever of the actions executed, we will log whether the action was executed successfully or failed. Finally, the results of these actions concatenated together, and used as feedback of the code along with the HTML text.
For AdaPlanner [21] and RCI [5], we use their official code on GitHub to implement their methods. We fix their bugs on matching text patterns but still find their performance is much lower than they reported. This may be because they have not released the code used in their papers, or their methods greatly affected by the GPT version (newer versions of GPT will be more creative).
15

---

E Presentation of Generated Manuals
We present the generated manual' on ALFWorld by AutoManual in Fig 5. As shown in the manual, the Formulator agent categorizes the rules into four categories: "Navigation and Search", "Object Interaction and Location Management", "Task-Specifi Processes", and "Correctness and Validation" " with clear introductions. We find our agents have found and recorded lots of important rules that directly impact task completion. For instance, "the agent should include all possible locations its search" " in rule_0, "The agent can only hold one object at time" in rule_3. 33 and "When multiple items of the same lype are present at a location, the agent may have choose one to interuct with or examine." " in rule 5. For tasks that can be solved by fixed strategy, such "Cool", "Heat", "Clean" and "Examine", AutoManual provides clear Success Process type rules. For the complex task, "Put Two", AutoManual conducts classified discussions in its Success Process rule.
These demonstrate that AutoManual resolves the Path Dependency problem of skills by dig- ging deeper into mechanisms, updating and incorporating success processes, and annotating important details.
Moreover, we prompt the Builder to break down large phenomena into specific rules and derive deeper rules from them later. In the initial rules upon rules. In the manual 5, we find that the Builder can correctly utilize rules recording basic mechanisms build more complex rules. For example, rule 4 uses rule induce solution: "If all objects are found at the same location, handle them sequentially according to rule_
give simple example of how build rules
However, find the generated manual is still not perfect. There are some unnecessary duplicates between rules, such as rule_7 and rule_8, which are both Success Process type rules for "Examine" tasks, but they divided into two rules.
Table 6: The success rate (%) of different methods on task types with feedback of MiniWob+++
Task Type
RCI [5]
AdaPlanner [21]
Planner+Lib.
AutoManual
Examples
Test with GPT-3.5
search-engine
33.3
100.0
66.7
66.7
tic-tac-toe
16.7
33.3
terminal
55.6
100.0
100.0
100.0
login-user-popup
33.3
33.3
33.3
66.7
guess-number
11.1
22.2
11.1
94.4
email-inbox
88.9
100.0
email-inbox-nl-turk
94.4
77.8
amail-inbox-forwarddaa
61.1
83.3
94.4
100.0
mmai--nbbox-forrardd-----
55.6
94.4
88.9
77.8
Average
45.6
71.6
63.6
82.2
Test with GPT-4-turbo
search-engine
44.4
100.0
100.0
100.0
tic-tac-toe
33.3
22.2
22.2
66.7
terminal
88.9
100.0
100.0
100.0
-og---userppopup
38,9
33.3
72.2
100.0
guese-number
22.2
44.1
33.3
88.9
email-inbox
77.8
100,0
100.0
94.4
email-inbox-nl-turrk
72.2
100.0
100.0
100.0
amail-inbbox-forwardd-ll
88.9
100.0
100.0
100.0
amsil-inbor--oovvrdd---m
77.8
66.7
94.4
Average
60.4
80.2
94.5
The manual doesn't include Validation Logs of the rules, which visible for the Planner.
16

---

Housekeeper Agent Interaction Manual
Overview
This manual intended essist the housekeeper agent in the successful execution of tasks within simulated environment. The rules provide guidance on navigating, searching the enviranment, interacting with objects, and managing task speaific pracesses, as well as encuring correctness actions using assertions.
Navigation and Search
Introduction
These rules provide guidance how to search for objects, including the helper methods streamline the process ensure thoroughness.
Included Rules
rule 0 (type="Special Mechanism"): Objeccts can be found in unconventional locations, and the agent should include all possible
locations in its search. For example, in epoch 9, the agent found soapbar on the tollet, which is an unconventional location for storing such Items.
[type- (type="Useful Helper Method"): shown the example. For example.
multiple receptacles
agent
"find, oobject' method
Deffine helper method find def find (agat,,, Lo_ehk,, deject_naxe]; receptaale recep_to_chee oo_ohhok: observalion sgeni.go_ion (oeeooereep)
needed
it "closed" in observacion:
agent.opent recpptac)))
Chack it the object is in/on the receptacin.
object__name
objeot_ids ids pet_objjot__vtt wtt.htttthhhcceeettteteeeeeeeeee objoct_panee
objact__dde Ide, recsptacle
retuzn None, Wone
axsections validate each casezt object, not Hone, "Beroes Could not object."
Object Interaction and Location Management
Introduction
These rules inform the agent on how to interact with objects, from taking and placing items to handling multipee items the same type. Proper management crucial
task exerution.
Included Rules
rule,2 2 (type= "Specal Phenomena"). When using a microwave, the agent can Interact with heat object) even there another object inside, the agent holding something, and the microwave door not explicitly mentioned to be open.
For example, epoch_ 1, the agen: was able to heat the mug with the microwave even though there was an egg inside the microwave agent was holding mug.
rule. 3 ttypen'special Mechanism"): The agent only hold abject and down held abject before taking
anather.
For example,, epoch_2. 22 the agens was holding statue 4 and attempped to take statue without putting down statue first, resulting
NNothing happons' observation.
either collect abjects before attempting to place them or find and place them one by one, ensuring they revisit locations with multiple objects if necessary, If all objects are found at the same location, handle them sequentially eccording to rule_3.
rule. ttype=* (type="Success Success Process"): When tasked with placing multiple
receptacle,
For example, epoch_15 15, the agent should have revisited sidetable_1 1 to collect the second pencil before attempting place in required statues coffeetable_
coffectable_1. epoch_22, 23, agent failed by this rule.
revisit addressed
Figure 5: The Generated Manual for ALFWorld: Part 1.
17

---

which includes going the receptacie and opening it is closed, Before performing put take action, the agent must ensure it the correct location. When multiple items of the same type are present at location, the egent mey have to choose one to interact with or examine.
rule_5 (typen Mechanism"): The agent
with receptacle obsenve
example, epoch_ 16, the agent had to open several closed cabinets (e.g. cabinet cabinet 2) to find items such the mug. epoch_21, agent
multiple
desk_1
('elarmdockfff
Task-Specific Processes
Introduction
category outlines steps required complete specific tasks, heating. cooling. examining objects abject's assistance.
Included Rules
rule 6 (type="Succes, Process"), ") task involves cooling or heating an objert before placing it, the stops are:(1) search the object using 'Tind, object rule, 1, (2) the object. (3) coolheat it requied, (4) go target receptacle, and (5) put object. Ersure egent's
environment updated
example.
ecceple,
mog por.
(Step 1] Use "Cind_objeos" xethod to seazch all. receptacles
[Step 2]
mog
(Step 3] do to the fridge, open it necensary, and cool, the
[Soop 4]
coffecrachine and put the mag
rule_7 (type= Process"): " When
examining object desidamp,, agent desklamp the object, and then use the desklamp examine the object. For example.
and the object, the desidampa
Use 'rind_obe ooj_oos wethod search for the deck' object.
[Step 2] Make wane the desktuup
Take object
[Siop 4) Use the desklarp to exaxino the objjee..
rule (type="Succaa Process"): When to object under a desidamp, ensure the lamp before using examine object. example.
[Step 4]
wilwedy
ooservation agant.co, tol seceppapee vita cesictarp)
agent..cxe desklamp)
assert "curn on in observacion "slready on" In observation, 'Brror in (Step 4jx Falled to ase the desklamp." [Biop 5) similarly,
slaxn
(Step 6) With the desklang on, exanine the alazs olook using the desklanp.
Correctness and Validation
Introduction
Instructions asserting code to confirm and enhance relisbility the agent's actions.
Included Rules
rule.9 9 "oppee"Correced Error"): Assertions the agent's code should confirm state changes such as location or held objects, rather rely specific phrasas
exampie, Instead asserting YYuu observation, agent should assert the location and held object state changes, when handling multiple required objects at same location, the agent should manage them sequentially without unnecessary variables.
Figure 6: The Generated Manual for ALFWorld: Part 2.
18

---

Table 7: Ablation study of Rule System on ALFWorld.
Success Rate (%) Testing with
Method
GPT-3.5-turbo
GPT-4-turbo
AutoManual
86.2
97.4
AutoManual without "Type"
74.6
91.5
AutoManual without "Example"
76.8
92.7
AutoManual without "Validation Logs"
85.4
97.0
AutoManual without "Useful Helper Method"
79.6
94.3
AutuManual without Cooperation between Agents
78.8
93.8
Case-Conditiones Prompts without Classification
83.2
95.6
Table 8: Sensitivity Analysis of Examples and Initial Rules on task types with feedback of MiniWob++. AutoManual uses human example of search-enging enter-text task.
Success Rate (%) Testing with
Method
GPT-3.5-turbo GPT-4-turbo
AutoManual
82.2
94.5
AutoManual with enter-text Example
78.3
92.8
F
More Experiments
F1
Ablation Study of Rule System
The well-structured rule system is essential for the rule management and usage. We perform the ablation study of our rule system in Tab 7.
AutoManual without "Type": remove the "Type" attribute of each rule in the rule system and the instruction for the Builder to manage various types of rules. As shown in Tab 7, without the "Type" attribute, the performance drops significantly, from 86.2% 74.6% and 97.4% 91.5%, as the Builder cannot manage specific types of rules and loses specific instruction on each rule.
AutoManual without "Example" ": remove the "Example" attribute of each rule in the rule system. The performance also drops by a large margin, as the Builder cannot specify necessary details in rules, and rules with lower understandability can sometimes be misleading to the Planner.
AutoManual without "Validation Logs" has little impact on performance, but "Validation Logs" of rules are very useful during debugging.
We also conducted some detailed ablation studies of our AutoManual in Tab 7.
Helper methods defined by humans and the Planner can serve as the solution for subgoal of task that can be used in multiple tasks. AutoManual without "Useful Helper Method": we remove the "Useful Helper Method" in the "Type" attribute, the human example, and initial rules, and we no longer encourage the Planner to write helper functions when writing code. The performance drops from 86.2% to 79.6% and 97.4% 94.3%, demonstrating that extracting reusable helper methods can facilitate the programming.
AutoManual without Cooperation between Agents: remove the Planner's thoughts in the trajectory when providing it to the Builder and no longer require the Planner to output Relevant Rules. The performance decreases significantly, indicating the thoughts and conclusions of the Planner are helpful for the Builder to manage rules.
CosseConditioned Prompts without Classification: we remove the requirement to classify the error reason to "Imperfect Rules" or "Imperfect Agents" and only use the prompts for "Imperfect Rules" during case-conditioned prompting. As demonstrated in Tab 7, the results are inferior to using all 5 cases because the analysis and classification of error reason can boost the Builder to manage rules accurately.
19

---

mumber F.2 Sensitivity Analysis of Examples and Initial Rules
We analyze the sensitivity of our rule optimization to the initial condition, i.e., the human example and initial rules. In the experiments 9 task types with feedback of MiniWob++, of search-engimg task and the corresponding initial rules, which are shown in Listing 8 and In these initial rules, extract the successful process of search- search-enging task and helper method to turn to the next page. Here, we provide AutoManual with an example enter-text task, the simplest task type in MiniWob++, shown in Listing 10.
example
The results Tab demonstrate that even if given less initial knowledge, AutoManull can still auto- matically learn the required knowledge through online optimization, which shows that AutoManual has good robustness to the initial rules. In addition, we found that the knowledge it learned about search ch engine is highly similar to the standard knowledge provided by humans, such the need to turn pages for the search index larger than 3.
G
Case Study
G.1
An Example of Path Dependency Problem
In the following example, the existing skill can correct solve the "Put two" task when the two objects occur the same receptacles. However, in the new scenario, the Planner still replicates the strategy but fails to complete the task, as the target objects are at different receptacles.
Listing 1: An Example of Path Dependency Problem of the Planner ALFWorld
Here the code relevant skill:
You are in the middle of a room. Locking quickly around you, you see bed_1, desk_1, draver_E, draver_4, drauer_a, draver_2, draver_1, dresser_1, and garbagecan_
The task to: two cellphone in bed.
"'yython
# Helper nethod nanually parse the observation for cellphones def parsa_for: por_collpoonaabbbarrra
'cellphone' folloved by return [sord for word in observation. aplit() if 'cellphona' in word]
This simple parser
# Define the 1ist of receptacles to search for the cellphones. recoptacles_to_chestt 'drawer_1', 'draver_2', 'drazer_'', 'drawar_4", 'draser_5', 'dresser_1'
# Initialize list to keep track of found cellphones. found cellphones
Search each receptacle for the collphones.
for receptacle recaptacles.to_cheik. observation agent.go, to..tooeceetttcceeee
receptacle closed, open it.
"closed" in observation:
observation (gent.open(eeceptacll
Manually parse the observation for cellphones.
found_ bound_cellpann www.cccllel....cceeeeeeeoeeoeeee extend (parsa_far for..collhooeeeebseeeeeoo too callphones found, stop searching.
if Len celphooee)))) 3- 2:
break
Ensure tuo cellphones vere found. assert len (anmuff_ceelpphoeeee 2, 'Error: Less than cellphones found in the environment.)
cellphones in for cellphone in found_ celppeooeppppe111 a Take only two cellphones receptacle where the cellphone found.
observation [igett.go.ttrrecoopaclll. Take the cellphone.
observation agent.take_fran(ce faak_ffra(cellphono, receptacle) # Go to the bad and put the callphone in it.
tatt:goott">>ed_1))
observation agent.put_fks in_orr ptt.ppp___orrooollllleoo.ees 'bed_1")
<e1
Please pay close attention to the process and details of this succesaful code xhen uriting code. Also, be. aware of potential randemaess (the current environment may differ from this ane).
Initial
20

---

You the middle of Looking quickly around you, you arnchair_1, cabinet_4, cabinet_3, cabinet_2, cabinet_l, drazer_s, draser_4, draser_a, drover_ 2, draver_1, dresser_1, garbagecan_1. safe_1, shelf_12, shelf_11, shelf_10, shelf_9, ahelf_8, shelf_7, shelf_6, shelf_5, shelf.4, shelf_3, shelf_2, chelf_1, sidetable_1, and sefa_1. task to: put two pillow in sofa.
Planner:
Understanding
The task is to find two pillove in the environment and place then in or on the sofa. The environment containa various receptacles and surfaces where the pilloss night be located.
#*# Plan for the task:
Dafina Hit of all receptacles and surfaces check for pillous.
2. Search each receptacle for pillows.
pillows 4. Move to the sofa's location.
found.
Place each pillow on the sofs.
Hooee is the code for the task:
"python
Helper nothod manually parse
pillows
def parse_for, farse_for_pilloss(e pil.oooosecherrttttin
This ie a simple pareer that looks for the word 'pillow' followed by an underscore and nunber (word for word observation. split() if 'pillow' word]
Define
receptacles
pillows.
receptacles_toe t -
"cabinet_1', 'cabinet_2', 'cabinet_3', 'cabinet_4', 'drawer_1', 'draver_2', 'drawer_3', 'draver_4', 'drazer_5", 'dresser_!', 'shelf_1', 'shelf_2', 'shelf_3', 'shelf_4', 'ehelf_5', 'sheli_8', "', 'shelf_8', 'shelf_9', 'sbelf_10', 'shelf_il', 'sholf_12', 'sidetable_1"
Initialize list to keep track of found pillove. hinddppllows
Search each receptacle for the pillovs.
for receptacle receptacles.to.chetk. observation wgwettggo.toee totnt.oo.orrrecpaccll
receptacle closed, open it.
if 'closed' in observation:
observation agent. (reccpptaac)))
# Manually parse the observation for pilloza.
found_pillove htapo/pllooo..eee(pptette.. wtteedeparaaae_oo fir.piilooe
too pillous found, stop searching. if confoond_pplloo >- 2: break
too pilloss ware found. assert (anffond.pillos >- 2, 'Error: Leas than 2 pillous found in the environment."
place pillows
for pillow in found_pillowe # Take only tvo pillows pillow vas found.
receptacle
observation agant.go..torreoppaclloo Take pillov.
observation agent. tareettak__eeee froe__rromppllov, receptacle) Go the zefa and put the pillov in it.
agent.go_to "o__to("sffa_"" observation agent.put__e onr itrriinoooooiiiiieeeeeeeeeeeeeee on(pillov, 'sofs_1')
<e1
Feedback including errors:
obe_1: Act: toottgoo_o"""bbieee 1)), Obe: cabinet_1 is closed. You are at cabinet_1 and holding nothing.
obe_2: agent.open (ptaettaaiieet.
open cabinet_1. cabinet_1, you see tissuebox_1.
at cabinet_1 and holding nothing.
obs_3: Act: agent.go_tto"aattee go_tt(("cbinet__2))
cabinet_1 holding nothing. cabinet_3 and bolding nothing, obe_5: Act: agent.open( (pea(tccbinet_3") Obe: You open cabinet_3. In cabinet_3,, you see box_3, and tissuebox_2
Nothing happans. You
obs.4: Act:
agent.go_too ggenttgo_.to""abbiie ("o_ttt"eabiiet_3" Obe: cabinet,3 is closed. You
obe_6: Act: cabinet_3 and holding nothing.
4)), Obe: Nothing happens. You are at cabinet_3 and holding nothing.
obs_7: agent.go_toc
draver_1 holding nothing. abs_8: Act: agent. open d'draser_1'). Cbs: You open draver_1. In draser_ see nothing. You are at
go_tooorrtveeer ('éraver_ Obe: draver_1 closed.
draver_1
holding nothing.
draver. and holding nothing. obe_10: Aet: agent open( draser_2"), Obe: You open drawer_2. In draver_ you see nothing. You are at holding nothing.
obe_9: Act: agent.go__ to 3o("drave__2 22), Obe: drawer_2 2 closed. You
draver_2 agent.open(
obe_11: Act: agent.go_tor (aravadr
Obe: drazer_3 is closed. You are at draver_3 3 and holding nothing. open draver_3. 3. draver_3, 3, you see nothing.
obs agent, open open("drazezz "raaeer33)).
at Oba: draver_4 is closed. You are draver holding nothing.
draver_3 and holding nothing.
obs. obs_13: 13: Act: agent.go_tod wow.gooooooooooooo ('drraver_4")
21

---

obs_14: Act: agent. "r"aeer44)). 0be: You open drawer_4. _4. In draver_4, you nothing. You draver_4 4 and holding nothing.
obs_15: Act: tontt.goooo(orreeee
draver_5 holding nothing. draver_5 agent.openn( open("drazer_61) "r"aerr"""). Obs: You open draver_b. 5. In draver_b, you see nothing. You are at holding nothing.
.)), Uba:
closed.
abs_ Act: agent,
obe Act: agent.go_too tontttgooooo to("dresser_1")
box_2, laptop.1, statue_1, 2atch_ and obs_ 18: Act: agent.go_to( gotttt(tttt(h1) Obs: On shelf_1, you nothing. You at shelf_1 and holding nothing.
Obe: On dreaser_1,
xatch_1. 1. You are at dreeser_1 1 and holding nothing.
obs_19: agent.go_too to_.ooottotehi__"" 0bs: shelr_2, you nothing. You shelf_2 and holding
nothing.
oba_20: Act: agent.go_.oo (o..tooothttt_."" Obs: Nothing happena. You are at abelf_2 2 holding nothing.
obe 20: Act: wot.tooootooeee_____ Obe: On shelf_4, you nothing. You are at chelf_4 and holding nothing.
shalf_6 and holding obs_22: Act: 3gont.go_tt(>shel13 Obs: On shelf_5, you nexspaper_2 You nothing.
agent..oo.to(eeeelll Obs: shelf_6, you nothing, are shelf,6 and holding
nothing.
obs oba_24: Act: gognt.go_.to">eee (eeeeee(eee))")) Obe: Oo shelf_7, you see vatch_ You are at shelr.7 and holding nothing.
abs obs_25: Act: agent.go.tt((theef...)) Obs: On sholf_a, 8, you see nothing. You are at shelf_8 and holding nothing.
obs_26: Act: agent.gg_.t("sheei...9 Obs: do shelf_9, , you see nothing. You are at shelf_9 and holding
nothing.
obs Act: agent.go.toossheit..u Ubs: On abelf_10, you aee nothing. You shelf_10 10 and holding
nothing.
obe_28. _a__88: Act: sgent.go_to(<shhlff.
Obe: On shelf_11, you nothing. You shelf_11 11 and holding you callphone_1.
nothing.
obs_29: Act: agent.go_taa
abelf_12 and eidetable_1, you see creditcard_1, keychain neuspaper_1. You are at sidetable_1 and holding nothing.
Uba:
holding nothing.
obe_30: Act: agent.go_tof geent.go_to""sidetaai (o__to("iietabbe__))
Execution error:
Error: Less than pillovs found in the environment.
Currant state: You are at sidetable_1 and holding nothing.
G.2 A Trajectory of The Building Stage
Here, we present trajectory of the Planner and the corresponding responses of the Builder for task ALFWorld. In this "Indirect Success" trajectory, the Planner made the mistake of heating the mug without having it next to the microwave. The Planner then corrected the mistake and provided organized code in conclusion. After observing this trajectory, the Builder keenly found the cause of the error: the existing "Success Process" rule did not indicate that the agent must go to the object before interacting with the object. The Builder updated the "Success Process" rule and added a "Corrected Error" rule to clarify the mistake.
Listing 2: A Trajectory of the Planner and Builder on ALFWorld
Initial observation and the task:
middle of room. Looking quickly you, you see cabinet_13, cabinet_11,, cabinet_11, cabinet_10, cabinet_9, cabinet_8, cabinet_7, cabinet_s, cabinet_5, cabinet_4, cabinet_3, cabinet_2, cabinet_i, soffeamschine_1. countertop_1, diningtable_1, draver_1, fridge_i, garbagecan_1, microvave_1, shelf_3, shalf_2, shelf_l, sinkbarin_l, stoveburner_4, stoveburner_3, stoveburner_2, stoveburnar_1, and toaster_1.
task
some mug put in cabinet.
Planner:
#*a Undderatanding of the observation:
The agent is in a room with nunerous cabinete, appliances, and surfaces. The taak is to heat mug and cabinet. This requires finding mug, heating then placing any
place
#** Rules consider:
serule_ee:: Ensure all receptacles and surfaces are considered when searching for the mug. mug.
**rule 1se;
the 'bjec_t method to search *erule_2<x: After finding the mug, heat it and then put it in cabinet.
**rule_3*x; Open closed receptacles to search for objects.
#*# Plan for the task:
"find_obfect' nethod to search 2. Dace the mug is found, take it.
mug in all receptacles surfacee.
mug with an appropriate appliance (e.g., nicrovave). 4. Find an empty cabinet or any cabinet to place the heated mag in. Place the mug in the cabinet.
Hore in the code for the task: "python
22

---

[Step 1] Define the helper nethod 'fini_object' provided in rule_1 def find_object (agent, pecep.to_check. objoct_nane):
for receptacle recep,to, check:
observation agent.go_to(reett (rett((rccp)acll)
need to open receptacle. do, open it.
if "closed' in observation:
observation agent.open seett.oponrecceptaclee
Check if the object in/en the receptacle. if object_name nane in observation:
object..ide get_,bjject witject__it__idloooee wtbjjeetiib.idddbaaaatttoo id(obaervation, return objact_ida, receptacle return None, None
object..ann)
[Step 2] Get a sorted list all receptacles and surfaces check for a mug recep_to_check ['cabinet_l', 'cabinet_2', 'cabinst_3', 'cabinet_4", 'cabinet_5', 'cabinet_8' 'cabinat_7 ;, "cabinet_8", 'cabinet_9", 'cabinet__01 'cabinet_ll', sabbinet_12", 'cabinet_ 'comttrrop_11, diningtable__1, 'draver_1', "fridge_1', 'garbagesan.1', 'microvave.1", 'sbelf_1',, 'shelf_2', "shelf_3', sinkbasin, "atovaburner_1", "stoveburner_2" _2', 'stovaburner_3) "stovebarnar_4" 4", "teaster_1",
coffeemachine_11
object_ids, receptacle_vithe Assort that ze have found mug
object (agent, tocep_to_check, 'mug')
assart object__ids
None, f'Error [Stap There is mag in/on trcca_.to.check).
(Step 3]
mug
found mug object_idd[() [0]
observation agent, take_ from(found mug, seceptacle_vith.nge
Assert that the agent holding the mug
assert agont. holding found_mg, f'Error in [Step 3] cannot take stound_mug) fron receptacle_etth.mgge
[Step agent. heat__ith(forre observation
mug
appropriste appliance (e.g., 'microvave_1")
Assert that the ung has been heated assert 'heat! observation and found_mug observation, f'Error in [Step 4] cannot heat (found_rug) with microsave_1."
# (Step 5] Find an eupty cabinet or any cabinet to place the beated mug in does specify chich
to use, can
one
observation agent-go_tee tor.tooooooooooo to("cabinet__1")
Check if the cabinet is closed. If so, open it.
'closed' in observation:
observation (ppnttopan(ccabtnat__] observation agent. put_in_or orooooro(ooudd__u 'cabinet_1") Assert that the mug is put in the cabinet
assert 'put'
found_mug observation,
[Stop cannot put (found_mug)
cabinat_11"
Foedback including errors:
abs_1: Act: agent go_ttoocabinet......mmnm 17). Obs: cabinet_1 1 is closed. You are cabinet_1 and holding nothing. obe 2: Act: agent.opent oppppppppaaaaaaaaaeee "cabinot__") Obe: You open cabinet_1. In cabinet_1, you see nothing. You are at cabinet_: and holding nothing.
toouttgoooo""ahir_.. Obs: shelf_1, you mug.1.
abelf_1 holding
nothing.
obs_3et agent.take_reeeee.oooooeee f'mg_." 'shelf_1'). You mng_1 chelf_1. holding mug.1.
obe_35: Aet: agent, with 'nicrovave_1") Obs: Nothing happens. You are at shelf_1 and holding mag_1.
Execution arror:
Error in [Step 4]: I cannot heat mug_1 1 with microvave_1.
holding mag.1.
Planner's replanning:
#* Understanding of the current observations:
The agent successfully found mag sbelf_1 and took the nicrovave, the action failed with "lothing happens." This indicated that the agent tried use the microsave not being attempting heat the
attempting heat mig
microxave's
agent must go microxave
#M Kistakes in the previous code and the for The arror occurred because the agent attempted to heat the mug without being at the microvave's lecation. pracondition heating object with receptacle ss the receptacle.
agent be the
### Rules consider:
**rule_ees 0se: Ensure all receptacles surfaces are considered when searching for the mag.
**rule_ Use the "find_objest' method search for the mug.
*erule_2sx; Aftar finding the mag, heat it and than put it in cabinet. objects.
**rule_3<s; Open closed receptaclea
23

---

#** Plan for the task:
1. Since the mag has already been found and is being held by the agent, proceed to the nicrosave.
Opan nicrovave if
closed.
3. Heat the mug with the microrave.
empty cabinat: any cabinat place the mag Place the mug in the cabinet.
Here the updated code for the task: "python
[Step 4]
open if it's closed
observation agent. goo.tgootooooooo to"'microwavee. "closed" in observation:
observation agent.open ceatttooe(("""eeeee "pen("nierovave__1"
Assert that the agent is at the nicrovave and it's open
observation and 'closed' in observation, #'Error in [Step 4] Microvave open or not at nicrovave_1."
'microsave_11
# [Stap 5] Heat the with the microuava
observation
"microwave_1"
Assert that the mng been heated assert 'heat! observation and 'm in observation, f'Eroor [Stap 5]: cannot heat mag_1 with microsave_1."
[Step 6] Find eupty
any
place bested mug
* Since the task does not specify which cabinet to can choose the first one that is enpty or available
We already know that cabinets through 10 enpty contain other itens, choose cabinet_1 for cimplicity
observation agent.go_toe aggat.gp.tt((cabtnet.1 ("("cabinet__1)) if 'closed' in obeervation:
observation Assert that the agent is at cabinet_1 and it's open
poont.open("cabiieee
aseert 'cabinet_1' in observation and 'closed' not in observation, f'Error in [Step 6): Cabinet_1 is not cabinet_1.'
[Step 7] obeervation agent. put__n_ ii_oiinoo___eeee the mag is put assert *put? in observation and 'mug_1' in observation, f'Error in [Stop 7]: I cannot put mug.1 1 in cabinet_1.' ...
mug LotorronnMMe on('mug,1', 'cabinet_1")
cabinet
Feedback:
abs Act: poontt.g.tt("mtcrrooav..a Ubst In nicrovave_1, see ogg. You are microwave_ and holding mug.1.
abs Act: agent.heat utth- 'nicrosava_1"). You heat mug_t using microvave_1. microsave_1 and holding mug.1.
obs_ agent.go_tor poont.go_to("eatii ('cabinet__1) holding mig.1.
at cabinet_1 _bs_ agent.put._ put__n_to__oooo (fmug__., "cabinat_ IP). You put mug_1 in/on cabinet_1. You are at cabinet_1 and holding nothing. This spoch done. Succeed: True
cabinet_1, you nothing.
Current etate: You are at cabinet_1 and holding nothing.
Planner's conclusion:
Sunnary of aisunderstandinga and mistakes:
Initially, atteapted to heat the vithout being at the microwave's location, which is why the action failed with "Hothing happens. The correct procedure is to first go the microwave before attenpting to any object. After correcting this nistake, the agent succansfully heated the mug and placed in cabinet_1.
#2a Organized code block:
"pythonn
[ielper Nethod] Define the helper methed "iind_object' provided in rule_1 def find_objecct recep_to_check. object_nane): for raceptacla recep_to, check:
observation - connt.go_toreeeepttt
need to open receptacle. do, open it.
if "closed' in observation:
observation agent.open oppetttopp
Check if the object in/en the receptacle. if object_nane nane in obsorvation:
object_ids get__object_Ht wttee_itthhidccaerattttto. object_nane)
return object_ids, recoptacle
return None,
(Step 1] Get a sorted liet of all receptacles and surfaces check for a mug recep_to_check ['cabinet_1', ! 'cabinet_2', 'cabinet_3', 'cabinet_4", 'cabinet_5', 'cabinet_6', 'cabinet_] ,, "cabinat_8" 'cabinet_9", 'cabinet_10', 'cabinet_11', 'cabinet_12: "a' 'comttrropp_"", diningtable__1? 'draver_1', "fridge_1', 'garbagecan_1", "microvave_1", 'sbelf_1', 'shelr_2', "shelr_3', einkbasinn,', "stoveburner_1', "stoveburner_2", 'stoveburner.31 "etoveburner_4", "toaster_1,, coffeenachine.1" odject_ids, receptacle, with obje_t (agent, tecep_ 'mug")

---

Assert that have found mug
assert objact_ids is not None, 2'Error in [Stap 11: There is no mug in/on
# [Stop 2] Take the mug
found mig objact_ida[[] observation agent, tgeent.tkk__ie Lakk__Irontooudd Assert that the agent is holding the mug
receptacle, with___e__ii__ng
ansert holding found_mg, f'Error in [Step 2]: cannot take (found_mig) fron secoptacle.stth.mpg
# [Stap 3] Go to the nicrorave and open it if it's closed observation gjent-go.to"miirrge if "closed" in obeervation:
observation igent.open('mieee) open(""icroovvee_"
Assert the agent the nicrovave and it's open
assert 'nicrosave_1" open
observation and "closed' not in observation, f'Error in [Stap 3]: Microvave is not nicrovsve_1."
[Step 4] Beat the mug with the nicrovave obeervation beaat.beat_vithttee Assert that the mug has been heated 'microvave_1")
'heat! observation and found_mug observation, f'Error in [Stap 4] cannot heat (found_rag) with microzave_1."
* [Step 5] Find empty cabinet or cabinet to place the heated in
Since the task does not epecify which cabinet to use, we can choose the firet one that is eapty or available
Ne already knos that cabinets through 10 are empty or contain other itans, so we can choose cabinat_1 simplicity
observation agent. go_tiigooooo to.tt'"abbiee__1"
'closed' in observation:
observation agent. poont.ope("ccbinet..
Assert that the agent is at cabinet_1 and it's open
assert 'cabinet_1' observation and 'closed' observation, f'Error [Step 5) Cabinet_1 is not open or not at cabinat__1.'
# (Step 6] Place the mug in the cabinet observation agent.put,, ponnt.put__n_oo itr.iinooonoouuuuteeeeeoooeeeooe (rr_on(found__my mug, 'cabinet_1!)
# Assert that the mag is put in the cabinet aseert 'put' in observation and found_nug in observation, d'Eroor in (Step 6] cannot put (found_nug) in cabinat_1.'
Builder's classification:
Analyzing the acenario, ve can identify the reot cause the obeerved nietakes and relate then the axisting
agent initially atteapted in the action "Bothing happens." This mistake was due to the agent not folloving the precondition for 'itht_ action, requires agent to case, the microsave).
mug without being at nicrovave's location,
location receptacle
existing related serule_iss; This rule provides a 'find_object' nethod to search for objects, which the agent used the mug.
correctly
rinding and nanipulating it. Hosever, doea explicitly state object in recoptaclo after agent maat be location of the appliance to use it, which is relevant for the heating step.
werule_2ee. This 'Success Process' rule outlines the steps to
=erule_See, See; This rule reminds the agent to open closed receptacles to search for objects, which the agent correctly.
rule applicable cover the pracondition for heating an object, which to be at the same location the appliance.
type
(rule_2),
not explicitly
The fault in this scenario can be attributed to:
Inperfect Rulese: The existing 'Success Process' rule (rule_2) does fully document the
precondition for using appliances, such as the microvave. It does not state that the agent must ba at the location of the appliance to use for heating an object.
Based on the analysis, the conclusion is that the fault stens from *Iaperfect Rulest, The rules need to be updated include the precondition for using appliances, which is critical step the Success Process' for taaks involving heating objecta.
Builder:
Potential Rules:
appliance (a.g., nicrovave), agent location as the applisnce. This rule is related to the existing *Buccees Procese' ruls (rule_2), but it previouely
#Corrected
attempting
specific precondition using appliances
25

---

+Scccess Process*: existing 'Success Process' rule (rule_2) should be updated include the procondition of being at the location of the appliance before using it. This is critical step in tasks involving heating objects.
Difference:
The potential "Corrected Error' rule largeta the speciric precondition for ueing appliances, which nev addition and not covered by existing rules.
update the 'Success Precess' rule expands the existing stepa include the precondition for using appliances, which ia a rafinement of the existing rule.
Check Existing Rules:
Rule_2 needa updated include the precondition using appliances, as this was the key overeight that led to the agent's initial
Codas:
"python
# Update the existing 'Succest Process' rule to include the precondition for using appliances
rule_id='rule
rule="If
object put receptacle, agent object, ensure they are at the location of the heating appliance before using it, and then place the target receptacla.",
is
find
object
exemple=""ng For exanple, best mug and it cabinet:
[Step 1] Use 'find_object' nethod to search all receptacles
[Stop 2] Take the mug
[Stap 3] Go to the heating appliance (e.g., microuave) and enaure you are at its location
[Step 4]
nug
the appliance
# [Stap 5] Find an empty cabinet or cabinat to place the heated in
[Step 6] Place the mug the cabinet""",
to include the precondition for using appliances based the and subsoquent success in opoch_1."
Write new 'Corrected Error' rule for the precondition of using appliances
grysten.NTitt_rvee
rule="Before using an appliance (a.g., microvava), the agent must go to the location of the appliance
type="Gorracted Error",
example=""ng Correctiive code:
Go the microwave before attempting beat the
obeervation ggent.go_to(fee (o.tto(tiirroare_"")
# Heat the with the nicrovave obsarvation biat witt(t"gg_''', validation rearaiioonrerorddeeeeeeeeee to heat the mug due to not being the microuava's location."
from 'microvave_ 1?)"",,
correction epoch,1 agent initially failed
male_eysten.sto_ggem sysstem.too.ggmeratingg.. ( cer
H Prompts and Examples
H.1 Prompts for Planner Agent
Listing 3: System Prompts for Planner Agent
[Role]
You are an excellent household agent in simulated environment. Given household environment and task, you write Python code accomplish the task.
[Actions]
For each task, agent is created in environment, and the initial observation and global memory printed.
The agent is an instance of Agent class, which includes the state of the agent its location, what it's holding) and the actions it can take.
class Agent:
def _iintt__seell env: InteractiveEnvEnginne)) self location env env.agent www.agent_locattie self holdin = "nothing"
26

---

Here the admissible actions the agent take; all action functions return an observation string the result of the action. If the precondition of the action is not met, its observation will include "Nothing happens".
Go receptacle and update the agent's location. For example, 'On countertop_1, you candle_1, cloth_2, and soapbar._. 11. go_to ("countertop_1)
For example, 'On sidetable_2, aee nothing.' go_to (o_to('sidee ('sidetablle_2) def go_to(self, receptacle):
Open a receptacle and observe its contents.
# For example, 'You open cabinet_1. In cabinet_1, see cloth_1." 1. open(' cabinet_1*
def open(self, receptacle);
Close an opened receptacle. For example, 'You close cabinet_1." close ('cabinet__) 1*) def close receptacle):
Take object from receptacle if the agent is not holding anything. For example, 'You take soapbar_1 from towelholder_1." lake_fromt ('soapbar__1 "towelholder__") def take_ fake_frommself, object, receptacle):
Put an object or receptacle if the agent holding it. For example, 'You put soapbar_1 in/on cabinet_1.² put_in_or_on (Psoppoansaaa_111 "cbbine__1" def put_in_or on (self, object, receptacle):
Use a lamp.
# For example, 'You turn on desklamp_11 1. use(deeslammp_1") def (self,,f, object):
Clean an object with a receptacle. # For example, 'You clean soapbar_1 using sinkbasin_1, clean_with('soappp _ 'sinkbasin_1')
def alean_with(sell object, receptacle):
Heat object with receptacle. For example, 'You heat tomato_1 using microwave_1.' heat_with wwt.wwiihhhttttoo_..... ('tomato_1", mi microvave_
def heat_with object, receptacle):
Cool object with receptacle. For example, 'You cool pan_2 using fridge_1.' fool_with('pann_2 'fridge_1") def cool with(eff,,,,,, object, receptacle):
useful function that you can use. Extract a list of object_ids ids with the specified object_name from the observation. def
object_name):
[Reaponse Instructions]
You will be given rules discovered by colleagues, summarizing game mechanisms and experiences of success and failure. Use these rules to guide your coding efforts.
27

---

Aim to understand and apply the principles behind the examples of these rules, adapting them to fit your specific scenarios within the simulated environment.
Output Format Instructions:
1. First, explain your understanding of the task and the current observations, then describe rules (with their IDs) that need be considered and plan for the task. Then, write your code between "'''pyhhon" and mescn, No text should follow after the code block.
2. After receiving feedback, you should also explain your understanding of the current observations, including special (unexpected) formats or phenomena, mistakes in your previous code, and the reasons for errors. Then, describe rules (with their IDs) that need be considered, plan for the task, then write code.
Follow these instructions:
1. DO NOT USE undefined function attribute of the agent. Your code must be directly executable in the given environment. won't implement any placeholders in your code for you.
2. Your code should be consistent with the code examples of the rules (please copy the code if there is no better modification!), making it easier for the builder agent refine and develop
rules effectively.
your code, you are encouraged define helpful functions, which should general reusable different scenes. The helper methods the rules are already defined in the coding environment, so you can directly use them. If you don 't need to modify them, don't redefine them.
4. After defining helper methods, your code should be divided into steps (marked with [[Step]" in comments) that each step achieves its subgoal.
write assertions the end of each step to ensure
Follow these instructions. Don't give up or change the task.
Listing 4: Conclusion Prompt for Indirect Success
Please summarize the misunderstandinga and mistakes you made, and then organize your code into code block. You should copy used parts from your previous code, including helper methods steps. You can only modify the previously wrong step, and make sure you don't miss any detail! So that I can use it with similar but not the same scenarios.
Listing 5: Conelusion Prompt for Failure
You failed complete the task. Now carefully review the trajectory of the executed identify the reasons
actions and the corresponding observations, failure. This reason is often your mistake or misunderstanding rather than the environment's a error. By carefully comparing with applicable rules and examples, pinpoint where your code deviated from expected standards.
If there exist coding errors, specify elucidate how they contributed to the errors. If you want to write a correction for an error, you must double-check its plausibility! Additionally, determine whether your codes were in strict adherence to the relevant rules and examples. This reflection and documentation will
segments the problematic code and
reminder for completing future tasks.
H.2 Examples for ALFWorld and MiniWob++
Listing 6: The Put Example for ALFWorld
You are in the middle of a room. Looking quickly around you, you see cabinet_4, cabinet_3, cabinet_2, cabinet_1, countertop_1, garbagecan_1, sinkbasin_2, sinkbasin_1, toilet_2, toilet_1.
Your task is to: find some spraybottle.
### Understanding of the observation:
### Rules to consider:
28

---

### Plan for the task: I need to get list of receptacles, and go to search each receptacle until seeing spraybottle.
Here is a code of my plan: "python
Define helper method to search receptacles for the target object def find_object (agent, recep_to, to_cckk,,, object_name): for receptacle in secep_to_checee observation = agent.go_ (receeptacle)) # Check if we need to open the receptacle. If we do, open it. if 'closed' in observation:
observation Check if the object in/on the receptacle. if object_name in observation:
object_ids get_object_ ierecttwith_dddbbseeeeee idobeeevva,oon, (oject_namm) return object_ids, receptacle return None, None
# (Step 1] Get sorted list of receptacles and surfaces to check for spraybottle. And use 'find_objec'' method to search recep_to_cheee check ['cabinet_1', 'cabinet_2', 'cabinet_31 'cabinet_4', 'countertop_1 'toilet_1', 'toilet_2', "sinkbasin_1', 'sinkbasin_2' 'garbagecan_1") object_ids. ids, receptacle_ teappaacee_vtth__ee withe_witth______peeee apraybttee = obje_t (agent, recep_to to to_ckkck,, spraybottle')
assert object_ide ids is None, f'Error in [Step 1]: There is spraybottle in/on { recep_to_checkk,
# [Step 2] Take the spraybottle Cound_spraybottll object_ids[0] observation agent. take_ frtee_rrooffoddppppppee spraydoott,, receptacle seahcllewwthh,_ee spraybottle) assert agent. Cound_spraybottlle from (receptacle)."
foradotpryy f'Error [Step 2] cannot take
[Step 3] Go toilet and put the spraybottle it observation peont.go_tootooilee
# check if toilet_1 is closed. If so, open it.
if cloosdd in observation:
observation agent. agent.open
observation agent.put_iie ii__i_n_________ or_ or_oonne (found aa spraybottle, 'toilet_1')
<ce
Listing 7: Initial Rules for ALFWorld Induced from Put Example
"rule_0":
("rule": "At the initial observation of the environment, the agent can only observe receptacles, such as cabinet_1, countertop_1. The agent needs to go to the receptacle to view objects in it, even for open receptacles.", "type": "Special Mechanism", "example";
"validation_record"" "Provided by User."), "),
"rule_1":
("rule"; there are multiple receptacles to searched, the agent can write use the "find_obe "bjjcct method as shown in the example.", "type": "Useful Helper Method", "example": ".,
Define helper method to find the object def find_object (agec (agent, recep_to_checkl object_name): for receptacle in recep_to_checks
needed
observation agent.go__te (receptacle)
do, open it. Check if the object is in/on the receptacle. if object_nams in observation:
Check if we need to open the receptacle.
'closed' observation:
observation agent. opentt.openneeeeeeeee (peeecppacc))))))
29

---

object_ids return object_ids, receptacle
object_name)
return None, None
"", "validation_record"" "Essential helper method provided by User "),
"rule_2":
("rule":
task to put some object on some receptacle, first use find_object' method in rule_1 to search all receptacles, take the object, then to the target receptacle and put the object.", "type": "Success Process", "example"; ".
For example, to put some spraybottle on toilet: # [Step 1] Use 'find_object' method in rule_1 to search all receptacles recep_to. to_check object_ids, receptacle, with spraybottle')
(bje recep_to, theck,kk, If the toilet is closed, open it first.
[Step 2] Take the spraybottle
# [Step 3] Go to toilet and put the spraybottle on it
"yalidation_record"" "Provided by User
For MineWob++, the Planner is allowed to complete the task in steps: initially output segments of planned steps and pause observe their outcomes inform subsequent steps.
Listing 8: The search-engine Example for MineWob++
Your task is Use the textbox click the search result. ....HTT.)....
"Jerald" and press "Search", then find and
### Understanding of the observation: ### Rules to consider:
Plan for the task: need click <input> textbox and type "Jerald". Then need to click "Search" button and get the searching result. Finally, navigate to and click on the 4th search result.
Here code my plan: "'yytoon
# [Step 1] Find <input> textbox and type "Jerald". textbox_xpar _pattoxxxppahhheee agent.ccick__ppee xpath ttteeeexxxxpttttt agent.type (""eeraal")
search--sarcchteeett]
# [Step 2] Click the search button htricchhuutto__eeeee stml_string wwt.cccccttttssrrraraaaaa bptto___ahhtt # Pause here*, waiting for the search to execute and for the results to be displayed.
"//=@ei-'searchh]]
Feedbacks ###
(some feedbacks)... (resulting HTML)...
#*# Understanding
observation: ... Because one page only displays search results, need to turn to the next page for the 4th search result.
### Rules to consider:
### Plan for the task:
cc "python
Define a helper method to turn to next page. def turn_to_next tattttooee__tteeeee__ee (aggep).g((aeeet) (agent):
next_page_xpant f"//* (@id=' poadddaaaaaaaiion1Ll "eclass='pge-ine next "]"
html_string agent. click xpath (next Becleex___age___e )age_xppthh)
30

---

return html_string
[Step 3] ttur__to_utt _o_next_paaee method to turn to the next page for 4th search result and click the result sarlistring turn_to ton_to_neee _e_.eeet_ae data-result start from 0 reault_xpatie f"//* /ataa/deeppaarotttttaeeaaaaaaaaa
agent. wtt.lcccccckpceeee (paacccl xxptth)
Listing 9: Initial Rules for MineWob++ Induced from search- Example
'rule_0':
of the target search result exceeds (the number of results per page), use toun__toneet_page method as shown in the example.", "type": "Useful Helper Method", "example": ".,
("rule":
Turn the next page.
def turn_to turn_to_next__a taa_tto__et___eeee_e___ page (agent):
next_page_zpar f"//* [ [#class= daclass='page-itens next']" html_string agent click_xpath _eapxtt__aee__ppp____ html_string
"validation_record" "Essential helper method provided by User. "),
'rule_1': {
"rule": "If the task needs to
search engine, first click <input> textbox, type the target string, and then click \"Search\" button. Next, use turn_to_next_paat in rule_1 to turn the page if the target idx exceed 3 and click the target result.", "type": "Success Process", "example": ..>
For example, to search Jerald click the 4th result: [Step 1] Find <input> textbox and type "Jerald".
# [Step 2] Click the search button.
[Step 3] Use "turn_to_nent to_next__aaee method in rule_1 to turn the page and click the target result.
,",
"validation__rccrdd" "Provided by User "}
Listing 10: The enter-text Example for Mine Wob++
Your task is Enter "Ronda" into the field and press Submit. ...(HTM))...
### Outputs ###
### Understanding of the observation: to consider:
### Plan for the task: I should click the input box, type "Ronda" and click the submit button.
Here code of my plan: '' "'python [Step 1] Click the input box haputbox_xpath = "//input "//input(@id="/ [tid='tt?] agent. wwwck_liicccccocc _pttthxppthhheeeceeeeeeeeeeaaeea (inputiox_xpattttt
[Step 2] Type "Ronda" into the input box agent. type ("Ronda")
[Step 3] Click the submit button submit bettbbuuttttpaaeeeeeeeee "//button [ agent. click_xpathe (submitt_ (uutton_xpath)
31

---

### Feedbacks ###
(some feedbacks)... ...
obs_3: Act: agent. aalick_ppaat""//btttooo http:/buutttn(iiidddbbbbbbbaa
Obs: Action done.
The epoch
is Done. Succeed: True.
H.3 Prompts for Builder Agent
Listing 11: System Prompts for The Builder Agent
[Role]
You are observing housekeeper agent aa codes and acts within simulated environment (game). , Your Role is to construct a manual of rules to not only assist the agent in completing tasks but also to do so with the least amount of attempts/ This requires recording and analyzing the experiences of the agent's successes and failures, and updating previous discoveries.
[Functions]
You will be presented with the current epoch's trajectory. The interaction in the trajectory includes the agent's analysis, execution code, and the reaulting feedback.
You should use the following methods of rule_systen to build and improve rules.
sule_system.. write write_rule(rule type*"", example="" validatiion Write down a new rule the game you discovered.
# Parameters:
# rule: rule of the game you discovered.
type: the type of the rule, chosen from ["Special Phenomena//echaniiam"
Corrected Error", "Unresolved Error", "Useful Helper Method", "Success Process"], The "Corrected Error" can include misunderstandings and mistakes that have been corrected.
example: example (or code) from the trajectory demonstrates this rule. You can detailed information the comment.
validation_recaee record: your validation record on this rule, including the epoch IDs and rule IDs from which this rule is induced.
sall_system.. update_ caattee_rueerule__ttee rule="", type="", example="", ialidatiio_rreordd"l)
# Rewrite the attributes of an existing rule when you come with a better understanding.
# Input only the attributes you want to rearite.
sutee__ysemmssee stops-toeeeeeeeee (top_generating()
# Description: stop generating rules from the current epoch.
# Use Case: When you believe that the trajectory of the current epoch is not needed insufficient to derive any should also this function when you have updated all rules for the current epoch.
rules, you call this function and wait
next epoch's
[Actions]
At each epoch, agent created in environment, and the initial observation target task are printed. The agent can only use the following action functions:
agent toatttoo_toorrcctccccll # Go to a receptacle and update the agent's location. agent. egenttopenneeee (receptacle) # Open receptacle and observe its contents. agent.close (receptacle) Close opened receptacle.
agent.tan take_akeeeeeeete from(object,, receptacle) # Take an object from receptacle if the agent is not holding anything.
agent.ppt__att ceat_in_or_ond(e (object, receptacle) Put object in or receptacle if the agent is holding it.
agent, use (object) Use lamp.
agent. clean_withhttaaaeeeeeec (objjcec,,, receptacle) Clean an object with a receptacle. agent.heat heattthhtt_tttte with (object, receptacle) # Heat an object with receptacle. wwt.coooowiithhheeeeeeeoe receptacle) # Cool object with receptacle.
32

---

set_objeci_witth_d (observation object_name) # Extracts list of object_ids with the specified object_mant name from the observation.
[Response Instructions]
Upon receiving the current trajectory, you should first identify the case of the current trajectory's result and then build rules following the corresponding instructions.
Listing 12:
Case Classify Prompts for Builder
Please analyze the scenario to identify the root cause of the observed mistakes and describe the existing rules related to the mistakes. Then check whether there exists "Success Process" rule applicable to this type of task. Finally, determine whether the fault stems from:
-Impefeect Rules*: the agent encounters unexpected phenomena that are not fully documented in the current rules, or the rules have not included the "Success Process" of this task type.
*Imperfect Agent*: the rules fully document the "Success Process" and the agent fails to follow these rules meticulously
reminders such scemarios,
Consider each step of the process carefully and conclude with either IIperfect Rules* +Imperfect Agent* based your analysis.
Listing 13: Base Prompts for Builder
[Output Format Instructions]
Based on the current trajectory and your analysis, you should output the following
things:
Potential Rules: Describe your thoughts about potential rules based the
trajectory. Depending on the results, you may need to check *Success Process*, Helper Method*, *Corrected Error*, #Unresolved Error*, and other findings in sequence. Each potential rule needs to be clarified whether it is related to existing rules.
Check Difference: Describe whether the potential rules target different phenomena. Check Existing Rules: Describe whether existing rules are conflicted or need updating.
Codes: Finally, sequentially call the rule_system's functions within "<< "python"" and "'rcn,
[Detailed instructions]
**aintain a maximum of 12 rules** Try to make each rule useful and non-repetitive, and insert new rules into closely related ones.
**Add Rules** Extract "Special Phenomena/Mechanistr rules when interactions appear counterintuitive, anviroument-specffic or when the agent expresses uncertainty about the environment mechanics (e.g., using "assume..." in the comment). Refrain from making speculative suggestions guesses. Instead, conservatively document phenomena and the agent's valuable insights.
Keee new rules targeted strategy into targeted units as individual rules. These can later be upgraded or merged into a more general larger rule.
precise. Break down large phenomenon or general
**Write rules' scope** The time when the rule is triggered, the task scope of the rule should be mentioned the beginning of the rule. **Avoiding overconfident for new ruleg*s Please acknovledge the need for further verification in your note.
**Update Rules** If existing rule needs be updated include you should try preserve insert categorial discussion or just insert new content into it (or its example). Especially, the rules of "Success Process" and "Useful Helper Method" type should retain their details. Then update the rule's validation_recant recordd after further verification revision.
phenomenon
details of existing content and preferably
Listing 14: Case I Prompts for Builder
33

---

**Rulea for Success** You should "Useful Helper Method" and "Succesa Process
For each useful helper function identified: If it is not already included in a rule, create rule of type "Useful Helper Method" and record its code unchanged in the rule's example section. If method with similar functionality already exists the rule, consider whether the rule needs to updated.
If the success process does not fall within the scope of an existing "Success Process" rule, faithfully document all steps (marked as "[Step]") in the successful code within rule of type "Success Process", and document necessary codes and reminders in the rule's example; if the success process of the current task falls within the scope the existing "Success Process" rule, consider whether the rule needs to be updated to incorporate the current roadmap.
Listing 15: Case 2 Prompts for Builder
**Rules for Success** You should extract "Useful Helper Method" and "Success Process
For each useful helper function identified: If is already included rule, create rule of type "Useful Helper Method" and record its code unchanged in the rule's example section. If method with similar functionality already exists in the rule, consider whether the rule needs to be updated.
If the Process" rule, faithfully document code within a rule of type "Success Process", and document necessary codes and reminders in the rule's example; if the success process of the current task falls within the scope of the existing "Success Process" rule, consider whether the rule needs updated incorporate the
process does fall within the acope of existing "Succeas steps (marked as "[Step]") in the successful
roadmap.
**Rules for Misstep** You should reflect on the main misstep to improve efficiency and log it into the "Corrected Error" type rule, including corrective code validated by the feedback (with the help of the agent's analysis and code, but its conclusion may not correct and should be checked carefully).
Listing 16:
Case 3 Prompts for Builder
**Rules for Succesa** You might need to update "Useful Helper Method" and "Success Process".
For each useful helper method identified: If method with similar functionality already exists the rule, consider whether the rule needs to be updated.
If the success process of the current task falls within the scope of the existing "Success Process" rule, consider whether you need to update the rule to include some tips include important and specific code in its examples.
**Rules for Misstep** Identify existing rules that agents failed to follow resulted in major mistakes. You should update the rule to emphasize some important points (you can add **...* ** at the part of the rule you want to emphasize) to add error-por pponne points (perhaps added to the comments of the example code).
Listing 17: Case 4 Prompts for Builder
**Rules for Final Error** Based on your previous analysis and conclusion, summarize the final error that led to failure. You should write an "Unresolved Error" rule to record the error: in what situation, what the agent did, and what results were produced. So that they serve reminders for the agent in the future. Please don't rush propose any definitive reasons or suggestions record it.
error; just
The final error is unresolved and cannot be included in rules of other types than Unresolved Error". the task failed, you cannot write down any "Success Process" "Useful Helper Method" rules.
Listing 18: Case 5 Prompts for Builder
34

---

**Rulea for Misstep** Identify existing rules that agenta failed follow and resulted major misstep. should update the rule emphasize some important points (you can add **...* at the part of the rule you want to emphasize) or to add error-promt points (perhaps added to the comments of the example code).
Remember that the rules "Success Process" and "Useful Helper Method" type should retain their details.
H.4 Prompts for Consolidator Agent
Listing System Prompts for Consolidator
[Role]
You are observing housekeeper agent
codes and within simulated environment (game). Your goal is to construct a manual of rules to assist the agent in completing various tasks in the environment. Your Role is to merge delete previously found rules analyzing the experiences of the agent.
[Functions]
You will be presented with the current found rules. The rules are extracted from many epochs' trajectories, in which each interaction includes the agent's analysis, execution code, and the resulting feedback.
rule represented with 'rule_id' and has the following attributes:
rule: the description of the rule, which begins with its use case or scope.
type: the type of the rule.
example: example (or code) from the trajectory demonstrates this rule. You can add detailed information
the comment.
calidation_recorce your validation record on this rule, including the epoch IDs and rule IDs from which this rule is induced.
should use following methods rule_systen to delete and merge rules.
rule_ maile_ystem..mm woyaatem.uddae...m uedateerrleeeteeeeeeeeeee www.eeullleeeteeeeeeeee rule="", type="", example="", validation record="")
# Rewrite the attributes of existing rule when you come up with better understanding.
Input only attributes you want to rewrite.
wwwtem.deeeeeeeeeeee.. rule (rule__e)
# Delete existing rule with rule_id.
**How merge** To merge existing rules, you can rule_system system. update_ rule for one rule and then call rule_system.dell delete_ delete another rule.
rule_system. seatem.gett_aee interaaciions (epoch_ids)
# Get the interaction history of previous epochs their IDs.
get the interactions from previous epochs epoch starts from 0). You may need to check the validation_record of an existing rule to know which epochs to get.
Use Case: You can use this
# Parameters:
epoch_ids: string containing the epoch IDs from previous epochs, separated by commas, e.g., op
rule_system. stop_generating(()
# Description: stop generating rules from the current epoch.
Use Case: You should call this function when you have finished updating all rules for the current epoch.
[Actions]
At each epoch, agent created in environment. The agent can only the following action functions its code to interact with environment:
agent.go gontt.go_tooo to (receptacce) # Go to a receptacle and update the agent's location. agent.open
agent.open(reccetaccce (receptacle)
Open receptacle and observe its contents.
35

---

agent. close (receptacle) # Close opened receptacle. agent. take_aaeeeeeee frommebject,, receptacle) # Take object from receptacle if the agent not holding anything.
receptacle) Put an object in or on receptacle if the
agent is holding it.
agent. use (object) Use lamp.
agent. claaenlllnensseeeee clean_withhobjectt agenttheat_wvie aahat_wwthhhe (object, receptacle) # Heat an object with receptacle. cool_
receptacle) Clean object with receptacle.
receptacle) # Cool an object with receptacle. get_object_wite (observation, object_namp # Extracts 1ist of obfect_ids with the specified object_man from the observation.
[Response Instructions] Output Process:
After receiving the current rules, you should select potential rules to investigate and then delete merge rules.
Detailed instructions:
**Maintain a maximum of 12 rules**
Merge if addressed** If "Success Process" rule address the "Corrected Error" or "Unresolved Error" rule, you can consider merging these rules while retaining their details.
**Retain important details** The rules of "Success Process" and "Useful Helper Method" type should retain their details, and should not be deleted easily refreshed by new updates. You cannot merge two rules of type "Success Process" or Useful Helper Method"!
IIssrrtion is preferable** If rule is updated to include the content of other rules, you should try to preserve the details of the existing content and preferably insert categorial discussion insert new content to (or its example).
H.5 Prompts for Formulator Agent
Listing System Prompts for Formulator
[Role]
You are observing housekeeper agent as codes and acts within simulated environment (game). Your goal is to construct a manual of rules to assist the agent in completing various tasks in the environment. Your role is to formulate a manual based the found rules, including categorizing and summarizing related rules.
[Functions]
You will be presented with the current found rules. The rules are extracted from many epochs' trajectories, in which each interaction includes the agent's analysis, execution code, and the resulting feedback.
A rule is represented with 'rule_id' and has the following attributes:
rule: the deacription of the rule, which begins with its use case or scope.
type:
the type of the rule.
example: example (or code) from the trajectory demonstrates this rule. You can add detailed information
the comment.
validation_reecee record:: your validation record on this rule, including the epoch IDs and rule IDs from which this rule is induced.
[Actions]
At each epoch, an agent is created in an environment. The agent can only use the following action functions in its code to interact with the environment:
agent.go go_toootttooo to (receptacle)
receptacle and update the agent's ' location. agent. ogen (receptacle) Open receptacle agent. close (receptacle) Close a opened receptacle. agent, take fake_fron (object, receptacle) # Take an object from receptacle if the agent not holding anything.
observe its contents.
36

---

agent.put_ing wut_pputt__ooataee aan_or_oo((ttt (object,, receptacle) Put object in receptacle if the agent is holding it.
agent, use (object) Use lamp.
agent.clean_wieeee clean_sith(objees receptacle) Clean an object with receptacle.
agent. hat___ithbbbbect receptacle) # Heat an object with receptacle. agent. wggnttccol_withhhoboooooooo get_objectt_iiee pithect_wtt____ abjjct_witchiiddbaaarr iobbeeevvtioon, object_ mame)) Extracts list of object_ids ids with the specified object_nane name from the observation.
receptacle) # Cool object with receptacle.
(Response Instructions]
Output Process:
After receiving the current rules, you should output the following things:
General Understandings: Describe your overall understanding of all rules and some specific rules.
Category of Rules: Methodically analyze the connections between related rules, then cluster these rules, and propose category each rule must belong to one only one category!
for the clusters. Make
The Manual: Fimally, sequentially write a structured manual within 'mrkkooow and sere, In the manual, you first describe the overview of all rules and then introduce each category of rules. In each category, you should list the rules and write rule_id within and
Detailed instructions:
1. Categorize rules based their use cases and topics they target, not based on their "type".
two "Success Process" rules follow the same critical success points or process you can consider categorizing them into one category and propose a general strategy with the critical success points in the Introduction section of the category. But you don't have to do this if they don't 't follow the same critical success points.
3. make the manual more accessible, please make the categories and rules appear in order from easy to difficult and from basic to complex.
37

---

NeurIPS Paper Checklist
1352
1353
1.
Claims
1354
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
1355
1358
Answer: [Yes]
1357
Justification: As shown in the Section of Experiments, our methods require little human expert knowledge of the new environment and can resolve the path dependency problem.
1358
1359
Guidelines:
1380
The answer NA means that the abstract and introduction do not include the claims made the paper.
1361
1362
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
1383
1384
NA answer to this question will not be perceived well by the reviewers.
1365
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
1366
1387
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
1388
1389
2. Limitations
1370
Question: Does the paper discuss the limitations of the work performed by the authors?
1371
Answer: [Yes]
1372
Justification: See Limitation Section.
1373
Guidelines:
1374
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
1375
1376
The authors are encouraged to create a separate "Limitations" section their paper.
1377
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
1378
1379
1380
1381
1382
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
1383
1384
1385
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
1386
1387
1388
1389
1380
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
1381
1392
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
1393
1394
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
1395
1396
1397
1398
1399
1400
3.
Theory Assumptions and Proofs
1401
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
1402
1403
Answer: [NA]
37

---

1404
Justification:
The paper includes no theoretical results.
1405
Guidelines:
1406
The answer NA means that the paper does not include theoretical results.
1407
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
1408
1409
All assumptions should be clearly stated or referenced in the statement of any theorems.
1410
The proof' can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
1411
1412
1413
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
1414
1415
Theorems and Lemmas that the proof relies upon should be properly referenced.
1416
4. Experimental Result Reproducibiliyy
1417
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
1418
1419
1420
Answer: [Yes]
1421
Justification: We describe the implementation details in the section of Experiments and Appendix.
1422
1423
Guidelines:
1424
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
1425
1426
1427
1428
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
1429
1430
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
1431
1432
1433
1434
1435
1436
1437
1438
1439
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
1440
1441
1442
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
1443
1444
1445
1446
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
1447
1448
1449
1450
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
1451
1452
1453
1454
1455
5. Open access to data and code
1456
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
1457
1458
38

---

1459
Answer: [Yes]
1480
Justification: We have provided the prompts and examples in the Appendix. We have provided the source code.
1481
1462
Guidelines:
1463
The answer NA means that paper does not include experiments requiring code.
1484
Please see the NeurIPS code and data submission guidelines (https://nips cc/ http:/ideee/eeeuumiiisiiiii..eee
1485
for more details.
1486
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
1467
1468
1469
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //n /ttps.ccppppuuuu cttps//cccpbbcbccuucccecco wttp://giieeeeeeeeooe /eodee///mmdbmmii ssi aginnylccy for more details.
1471
1472
1473
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
1474
1475
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
1476
1477
1478
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
1479
1480
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
1481
1482
6. Experimental Setting/Details
1483
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
1484
1485
1486
Answer: [Yes]
1487
Justification:
We have described these in the experiments
1488
Guidelines:
1489
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
1491
1492
The full details can be provided either with the code, in appendix, or as supplemental material.
1493
1494
7. Experiment Statistical Significance
1495
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
1496
Answer: [Yes]
1498
Justification:
We performed each experiment 3 times and have shown the error bar (standard deviations).
1499
1500
Guidelines:
1501
The answer NA means that the paper does not include experiments.
1502
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
1503
1504
1505
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
1506
1507
1508
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
1509
1510
The assumptions made should be given (e.g., Normally distributed errors).
39

---

1511
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
1512
1513
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
1514
1515
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
1517
1518
1519
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
1520
1521
8.
Experiments Compute Resources
1522
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
1523
1524
1525
Answer: [Yes]
1626
Justification:
We use the OpenAl API for all experiments and report the cost.
1527
Guidelines:
1528
The answer NA means that the paper does not include experiments.
1529
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
1530
1531
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
1532
1533
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that tidn't make it into the paper).
1534
1535
1536
9.
Code Of Ethics
1537
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttpas:://///ss/.eeeee http:///rrpp.p.cpppte///eeeee//e
1538
ines?
1539
Answer: [Yes]
1540
Justification:
We have reviewed the NeurlPS Code of Ethics.
1541
Guidelines:
1542
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
1543
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
15M
1545
The authors should make sure to preserve anonymity (e.g.. if there a special consid- eration due to laws or regulations in their jurisdiction).
1546
1547
10.
Broader Impacts
1548
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
1549
1550
Answer: [Yes]
1551
Justification:
See Broader Impacts Section.
1552
Guidelines:
1553
The answer NA means that there is no societal impact of the work performed.
1554
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
1555
1556
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
1557
1558
1559
40

---

1580
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
1581
1562
1563
1564
1565
1566
1567
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
1588
1589
1570
1571
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
1572
1579
1574
1575
11. Safeguards
1576
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
1577
1578
1579
Answer: [NA]
1580
Justification: The paper does not provide pretrained language models, image generators, or scraped datasets.
1581
1582
Guidelines:
1583
The answer NA means that the paper poses such risks.
1584
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
1585
1586
1587
1588
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
1589
1580
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
1591
1592
1593
12. Licenses for existing assets
1594
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
1585
1596
1597
Answer: [Yes]
1598
Justification:
We have stated and cited the asset used in the paper.
1599
Guidelines:
1600
The answer NA means that the paper does not use existing assets.
1601
The authors should cite the original paper that produced the code package or dataset.
1602
The authors should state which version of the asset is used and, if possible, include URL.
1603
1604
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
1605
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
1606
1607
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com cattaatsesss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
1608
1609
1610
1611
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
1612
41

---

1613
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
1614
1615
13.
New Assets
1616
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
1617
1618
Answer: [NA]
1619
Justification: The paper does not release new assets.
1620
Guidelines:
1621
The answer NA means that the paper does not release new assets.
1622
Researchers should communicate the details of the dataset/cooe/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
1623
1624
1625
The paper should discuss whether and how consent was obtained from people whose asset is used.
1626
1627
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
1628
1629
14.
Crowdsourcing and Research with Human Subjects
1630
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
1631
1633
Answer: [NA]
1634
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
1635
Guidelines:
1636
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
1637
1638
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
1639
1640
1641
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
1642
1643
1644
15.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
1645
1646
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
1647
1648
1649
1850
Answer: [NA]
1651
Justification: The paper does not involve crowdsourcing nor research with human subjects.
1652
Guidelines:
1653
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
1854
1655
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
1696
1657
1658
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
1659
1661
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
1662
42


Reviews:
[{"review_id": "aGHmjkHaoJ", "reviewer": "Reviewer_zSsL", "summary": "The authors propose AutoManual, a method for improving LLM agents by collecting and updating rules online via rule-assisted interactions in the environment, where code form planning is used for the interactions. In addition to the management of rules, AutoManual also maintains skill and reflection libraries per task type, where successful trajectories form the skill libraries and failed trajectories form the reflection libraries. The authors empirically show that AutoManual outperforms the compared baselines in ALFWorld and MiniWoB++ environments.\n\n**Post-Rebuttal/Discussion**: While I appreciate the detailed author response, I don't think it fully addresses the major concerns I had. For example, by \"maintaining the skill and reflection libraries per task type (L166)\", I meant there exist explicitly and discretely isolated skill and reflection libraries, one for each task type. This design is made by the humans, not the LLMs. It may be doable for ALFWorld, which is comprised of clearly separable task types, but it would require more advancement for more complex environments. Therefore, I maintain my original score.", "questions": "* Are the \"initial rules\" made or selected by humans or fully automatically derived by Builder?\n* Please take a look at the Weaknesses section above.", "limitations": "The authors discussed the fair limitations in the Limitations section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed method, AutoManual, alternates between exploiting and updating the set of rules, which is sound as the rules also get updated online based on the relevant experience.\n- The authors show that the proposed method exhibits strong performance in ALFWorld and MiniWoB++, by comparing it empirically with multiple baselines. They also provide a fair amount of analyses including the ablation studies, which help to understand the proposed method in empirical aspects.", "weaknesses": "- While the presented empirical evaluations mainly compare the number of examples used as a measure of the amount of domain or human knowledge provided to the methods, such knowledge can be injected in different forms, e.g., input prompts or the exposure of LLMs such as GPT to the benchmarks and corresponding execution codes during their training. But another point I notice is that, this work injects such knowledge in a more explicit or discrete form by maintaining the skill and reflection libraries per task type (L166), which contributes most to the performance according to Table 3.\n- Over-claiming statements need to be clarified or toned down. For instance, the Abstract claims that LLM agents \"typically require elaborate design and expert prompts\" in specific domains and suggests AutoManual as a solution. However, it appears that AutoManual leverages human-designed prompts for each domain as well as domain knowledge for design (as mentioned above).\n- The experimental settings for the GPT-3.5-turbo results may not be entirely fair. According to Table 1 from Appendix D, the \"GPT-3.5-turbo\" runs with all the baselines are actually based on GPT-3.5-turbo except for the Builder component for ExpeL. It means that the Actor and Self-reflection for offline trajectories for ExpeL are also equipped with GPT-3.5-turbo, unlike Planner and Builder for the building stage for AutoManual.\n- The manuscript can be improved in terms of presentation. especially the choice of technical vs non-technical terms. For instance, \"free-form code\" appears multiple times, but I couldn't find clarifications on what exactly \"*free-form* code\" is and whether (and how) it is different from just \"code\", which I assume is not. I have a similar concern about \"natural programming capabilities\" (L141).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1- **Clarification on Environment-Specific Information:**\n\n   First, the environment-specific information provided by humans can be categorized into four types:\n\n   - (i) Brief description of the environment and roles (about 25 words)\n   - (ii) Actions that the agent can perform\n   - (iii) Examples of humans completing the task\n   - (iv) Potential environmental knowledge embedded during LLM training\n\n   Among these, Types (i) and (ii) are fundamental information for the agent to perform tasks in the environment, and can easily and justifiably be included in the prompts. Type (iv) is challenging to control and eliminate, but in our experiments, all methods used the same LLM and the same types (i) and (ii) prompts to balance the influence of this knowledge. Type (iii), the examples, are the most labor-intensive, each exceeding 200 words (ReAct and Reflexion require 12 such examples). To further illustrate the significance of examples in previous methods, we demonstrate in the table below that the performance of Reflexion (GPT-4-turbo) with only two 'put' task examples drops from 85.9% to 73.1%. Our AutoManual system requires just one human-provided example (and derived initial rules).\n\n   Additionally, as stated in Section 3.2, all content in the skill and reflection libraries is proposed and updated by the LLM Agent itself, not manually injected, which differs from \"the examples.\"\n\n2- **Reduction of Human-designed Prompts:** \n\n   As previously mentioned, types (i) and (ii) of the LLM Agent's input prompt are fundamental environmental information, while type (iv) is inherently uncontrollable during training. The main human effort is concentrated on type (iii), with prior works requiring multiple human-written environment-specific examples (each over 200 words), whereas AutoManual needs just one from humans. Furthermore, some previous Agents, such as Generative Agents [14] and Voyager [23], were specifically designed for their respective gaming environments within their frameworks, but we do not tailor our framework to any particular application environment. Therefore, we claimed that AutoManual significantly reduces the need for elaborate design and domain knowledge. \n\n  We appreciate your feedback and will tone down the claims in the Abstract accordingly.\n\n3- **Experimental Fairness with Expel**: \n\nThank you for highlighting this. We followed the original settings of the Expel paper in our experiments with Expel (GPT-3.5-turbo), which indeed might not have been entirely fair compared to our methods (GPT-3.5-turbo). We have added the experiment settings you mentioned below, where both Builder and Offline Trajectory’s Actor and Reflector use GPT-4-turbo, while Task Inference uses GPT-3.5-turbo. The results show that although the new Expel setup with GPT-4-turbo improves performance, it still does not match AutoManual's results: 67.2% vs. 86.2%. As discussed in our paper, this is due to Expel’s reliance on offline learning rather than our online learning, its rule management is rudimentary, etc.\n\n4- **Clarification of Terminology**: \n\nThe term \"free-form code\" is defined in Section 3.2, paragraph two, as: \"This form simplifies planning by only generating code necessary for the current environmental situation and feedback without the overhead of integrating previously executed code.\" We use \"free-form code\" to distinguish from the \"function-form code\" used by Voyager and AdaPlanner. They require the output code to be a Python function, while our approach imposes no such restriction. \n\nThe term \"natural programming capabilities\" is described in the paper CodeAct [24] as \"Code data is widely used in pre-training today’s LLMs. These models are already familiar with structured programming languages.\" We appreciate your suggestion and will clarify these definitions further.\n\n5- **Initial Rules**: \n\n'Initial rules' are extracted by humans from the single provided example, demonstrating how rules should be extracted to the Agent. In Appendix F.2, we investigated whether the provided initial rules and that one example impact results, finding AutoManual robust to such variations.\n\n**Table: Additional Results**\n| Methods                    | Examples | Put  | Clean | Heat | Cool | Examine | Put two | ALL  |\n| -------------------------- | -------- | ---- | ----- | ---- | ---- | ------- | ------- | ---- |\n| Reflexion (gpt-4-turbo) | 2 (put)   | 88.9 | 90.8  | 66.7 | 88.9 | 55.6    | 33.3    | 73.1 |\n| ExpeL (gpt-3.5-turbo) (updated setting) | 12       | 80.6 | 62.4  | 71.0 | 71.2 | 64.8    | 51.0    | 67.2 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R8b3FTtxgv", "reviewer": "Reviewer_azPd", "summary": "The authors propose a new approach to interactive environment interactions by LLMs which they dub AutoManual. The approach takes an LLM and uses it in several different roles, differentiated by what context is included in their prompts, and some number of saved rules. The fundamental idea of the approach is to optimize over the saved rules in order to improve the performance. The authors demonstrate that AutoManual can perform well with a single human-example in both ALFWorld and MiniWob++ compared to baseline approaches and they demonstrate the importance of its various components through an ablation study.\n\nEdit: I have updated my review based on the authors' response.", "questions": "1. Why didn't the authors include a gpt-3 baseline for AdaPlanner?\n2. Do the authors feel AutoManual handles hallucinations better? If so, why?\n3. Can the authors given the value of N_max and other hyperparameters missing from the paper?", "limitations": "Yes the authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper proposes what appears to be a useful framework for applying LLMs to decision making tasks. Clear arguments are made for the various components of the framework, which are all fully described in the appendices. The ablation study meaningfully demonstrates the additive value of the various components, and the comparison to baselines demonstrates strong performance.", "weaknesses": "I have one concern with the paper and a few minor concerns.\n\nMy major concern has to do with the reported performance of AdaPlanner. The authors acknowledge that they could not replicate the performance but justified this that AdaPlanner's format requirements were too strict. The original authors, Sun et al. identified that the issue with 3.5-turbo (and one would assume later models) was their tendency to hallucinate. If AutoManual is robust to these hallucinations that would be excellent, but it would make sense to include an AdaPlanner implementation with gpt-3 given that is the stated preferred models by AdaPlanner's authors. Further, the results with gpt-3 appear to meet or exceed the performance of AutoManual. This is by far my largest concern with the current paper draft and I'd strongly encourage the authors to include an AdaPlanner gpt-3 baseline or explain their reasoning not to.\n\nIn terms of minor concerns, there are a number of claims that are somewhat overblown in the current paper text such as \"to enhance their readability and global understanding\" when no results demonstrate improved readability or global understanding. Similarly, the authors claim the approach is \"user-friendly\" and \"like a teacher\" in the same paragraph, given that the authors do not include any human subject study or other user tests these are unsupported in the current draft. \"making it grounded and well-understood\" would be yet another unsupported claim. Another minor concern is that some implementation details and associated design decisions are not given, such as not specifying the value of Nmax used for any of the experiments. Finally, the language is somewhat off in certain parts such as \"the function should be called\"->\"this function should be called\", \"like a teacher conveys\"->\"like how a teacher conveys\", and so on.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Regarding the Exclusion of AdaPlanner's GPT-3 Results:\n\n1- **Reproducibility Issues with AdaPlanner's GPT-3**: The high success rate claimed by AdaPlanner's GPT-3 is nearly impossible to replicate. We utilized the publicly available code from AdaPlanner and tested it multiple times using OpenAI's davinci-002 and gpt-3.5-turbo-instruct models, only achieving success rates between 50%-70%. During our trials, we identified and fixed several formatting bugs in the code, yet the highest success rate plateaued at 76%. Additionally, others have also reported difficulties in replicating these results in the issues section on AdaPlanner's GitHub [see our comments below]. We invite reviewers to personally test the code to see the reproducibility of the results.\n\n_Note: OpenAI discontinued text-davinci-002 and text-davinci-003 models as of January 4, 2024, leaving davinci-002 as a substitute and recommending users to switch to gpt-3.5-turbo-instruct which is purported to have equal or enhanced capabilities._\n\n2- **Lack of Implementation Details for \"Skill Filtering\"**: The AdaPlanner paper discusses a crucial \"Skill Filtering\" stage in Section 3.3, yet we found no implementation of this in the provided code, nor is it detailed in the paper. According to the paper, this filtering boosts success rates significantly, as shown in Figure 4. In addition, implementing such filtering would require testing a substantial subset, if not all, of the test set multiple times. This frequent validation using the test set could unfairly benefit AdaPlanner over other methods that do not use the test set in such a manner.\n\n3- **GPT-3 Tends to Copy Curated Examples**: Adaplanner meticulously designed its human examples to the extent that they only require replication to fulfill the task. Adaplanner provided one example for each of the six task types in ALFWorld, each example comprehensively offering the code needed to solve the respective task. This resulted in a scenario where, faced with a specific task type, one only needs to change the item names and target container names in the human example to produce a working code solution. To validate this point, we conducted an experiment:  For a new task, we replaced the item and target names in the human example with those from the new task description, and ran the resulting solution function. We discovered that this direct copying approach (simply replacing names) achieved a success rate as high as 70%. Furthermore, we found that 30% of the failures were mainly due to errors from the sub-function `ask_LLM()` rather than issues with the solution itself. The code and traces for this experiment are available at [our comments below].\n\nGiven such potent human examples, it becomes less about the LLM agent's planning abilities and more about who can best copy and make minor modifications. Compared to the high innovativeness of gpt-3.5-turbo, gpt-3 tends to replicate text from examples. This highlights the truth behind the authors' statement that \"gpt-3.5-turbo (and one would assume later models) was their tendency to hallucinate\": the newer models are more inclined to generate answers that differ in form from the perfect human examples. However, we believe that simply replicating these artificially designed solutions does not serve as a fair comparison. By contrast, our AutoManual system does not face this issue, as it was only provided with a single human example for the most basic task.\n\nFor these reasons, we have excluded AdaPlanner GPT-3's extraordinary result from our comparative analysis. We believe it is our duty to provide **fair and reproducible** experimental results as a standard for comparison for our readers and for future research.\n\n## On Exaggerated Claims:\nWe acknowledge that we did not conduct human subject testing on \"readability or global understanding\" for AutoManual. However, the enhanced readability of sorted and categorized Markdown Manual text over a list of unordered rules is both intuitive and supported by established research [1]. We have provided the final generated Markdown Manual in the appendix (Figures 1 and 2) so reviewers can assess its organization and readability themselves. Proper categorization and arrangement not only improve readability but are also known to enhance LLM performance, aligning with human experience and empirical evidence [2].\n\nWe appreciate your feedback, and we will adjust our language to moderate the claims made.\n\n## Additional Questions:\nIn Appendix D, we mentioned our choice of `N_max` as 12 to avoid overly long context issues. This section also includes detailed descriptions of our hyperparameters. We will make these more prominent.\n\n[1]: Beck, I. L., McKeown, M. G., Sinatra, G. M., & Loxterman, J. A. (1991). Revising social studies text from a text-processing perspective: Evidence of improved comprehensibility. Reading Research Quarterly, 27, 25 1-276. \n\n[2]: [How I Won Singapore’s GPT-4 Prompt Engineering Competition](https://towardsdatascience.com/how-i-won-singapores-gpt-4-prompt-engineering-competition-34c195a93d41)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lilkZkDHyK", "reviewer": "Reviewer_HWDR", "summary": "LLM agents show promise in many environments but require expert prompts in specific environments to perform well. This paper introduces the AutoManual framework which autonomously builds a 'manual' of rules through interactions to help adapt to new environments. The framework consists of 3 agents: the Planner which generates code to carry out a plan, the Builder which updates rules, and a Formulator which consolidates the rules into a manual. The Builder updates rules through a original case-conditioned prompting strategy to mitigate hallucinations. AutoManual is evaluated on 2 benchmarks: AlfWorld and MiniWoB++ and achieves high performance.", "questions": "Suggestions\n- Provide an explanation/justification for the incomplete ablation table in the paper\n- Provide a subset of results (e.g. a single column) to verify performance isn't brittle due to temperature 0\n\nQuestions\n-  Manuals are constructed by sampling specific task types in both environments. I am curious whether you experimented with constructing manuals with limited task types and testing on unseen task types. In other words, can the manuals be used to perform other tasks (e.g. for AlfWorld only constructing a manual out of Put but testing on Put Two)\n- Despite the case conditioned prompting to mitigate hallucinations for the Builder, the Planner can ignore these rules as noted in the limitations. Is this the most common failure case and, if so, what were the practical solutions that were taken to make the Planner pay attention to the Builder's rules?\n- As a follow-up, AlfWorld and MiniWoB++ are environments where there is likely overlap in the LLM's training data and the benchmarks (home environment and web-based environments); however, how would AutoManual perform in environments that may not align with the LLMs priors (e.g, Atari games or Sokoban)", "limitations": "The limitations section is transparent and the broader impacts section addresses societal impact.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "$\\textbf{Originality}$: AutoManual breaks apart the knowledge acquisition problem into 3 agents, one of which uses a case conditioned prompting strategy, which breaks down rule updates into a decision tree that simplifies LLM responses, mitigating hallucinations.\n\n$\\textbf{Quality}$: AutoManual is evaluated on 2 benchmarks against popular baselines and related knowledge acquisition works and provides insightful ablations into its components. High quality.\n\n$\\textbf{Clarity}$: Plenty of helpful figures and very clear approach.\n\n$\\textbf{Significance}$: Autonomously collecting knowledge to formulate an instruction manual simplifies human efforts when developing prompts for LLM policy approaches.", "weaknesses": "This paper is pretty strong already so there are only 2 things I wanted to point out that could make it stronger.\n\n1. The ablations are very helpful in justifying design decisions but the table looks odd with seemingly random kinds of ablations. I understand if there is a concern with resource-constraints running all 16 combinations of ablations but why this set of 7 ablations?\n2. Temperature 0 for OpenAI is typically a red flag to me that performance is brittle since there are no guarantees of determinism [1]. Despite experiments being run 3 times and averaged, I've experienced varying performance across different days. I'd be interested to see if a subset of the results remains relatively the same.\n\n[1] https://community.openai.com/t/why-the-api-output-is-inconsistent-even-after-the-temperature-is-set-to-0/329541", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1- **Significance of Selected Ablations**: \n\nEach row in the ablation study table in Section 4.2 was carefully chosen to demonstrate the contribution of different components of AutoManual to the overall results. Specifically:\n   - The first row, \"Planner only,\" serves as our baseline, testing only our Planner Agent without any building process.\n   - The second row, \"Planner + Lib.,\" involves building only the Skill & Reflect Library during the building process, highlighting that merely recording skills and reflections without constructing rules is insufficient.\n   - The third row shows the \"Offline Learning\" version of AutoManual, illustrating that using offline trajectories rather than online learning significantly reduces effectiveness, thereby supporting our claims made earlier in the paper.\n   - Rows four to six respectively demonstrate the impacts of removing the Skill & Reflect Lib., case-conditional prompting, and manual formulation components from AutoManual, underlining the necessity of these elements for optimal performance.\n   - The final row represents the full version of AutoManual, achieving the best results.\n\n   These meanings also correspond to the comparison of each paragraph in Section 4.2. Other combinations have no special meaning or comparative significance, so they are not included in the results. We will make the significance of each row more explicit in the revised manuscript.\n\n|                       Meaning                       | Online | Skill&Reflect Lib. | Case Prompt | Formulation | Success Rate (%) |\n| :-------------------------------------------------: | :----: | :----------------: | :---------: | :---------: | :--------------: |\n|                    Planner only                     |        |                    |             |             |       77.6       |\n|                    Planner+Lib.                     |        |         √          |             |             |       88.1       |\n|                  Offline Learning                   |        |         √          |      √      |      √      |       90.7       |\n|                AutoManual w.o. Lib.                 |   √    |                    |      √      |      √      |       89.5       |\n| AutoManual w.o. case-conditional prompting strategy |   √    |         √          |             |      √      |       93.8       |\n|             AutoManual w.o. formulation             |   √    |         √          |      √      |             |       96.5       |\n|                     AutoManual                      |   √    |         √          |      √      |      √      |       97.4       |\n\n2- **GPT's Temperature**: \n\nWe also noted that setting the temperature to 0 for GPT does not always produce identical results, but the impact on our experiments is minimal. This is evident from the standard deviation, approximately 2, shown in Figure 4. In fact, the three runs in each result were not completed on the same day; the experiments often differed by many days, and then the average was taken. To confirm the stability of our AutoManual, we have conducted additional experiments with varying temperatures and 3 runs below.\n\n   Table: Additional Experimental Results of AutoManual on ALFWorld\n\n| Method                  | Temperature | Runs | Success Rate（%） |\n| ----------------------- | ----------- | ---- | --------------- |\n| AutoManual（gpt-4-turbo） | 0           | 1    | 96.3            |\n| AutoManual（gpt-4-turbo） | 0           | 2    | 98.1            |\n| AutoManual（gpt-4-turbo） | 0           | 3    | 96.7            |\n| AutoManual（gpt-4-turbo） | 0.5         | 1    | 95.6            |\n| AutoManual（gpt-4-turbo） | 0.5         | 2    | 96.3            |\n| AutoManual（gpt-4-turbo） | 0.5         | 3    | 97.6            |\n\n3- **Task Generalization in AutoManual**: You raised an excellent point regarding task generalization. But we have to say that each task type has its own unique solution, so the Manual built on some tasks may not have a high test result on the remaining tasks. For example, AutoManual only constructs a manual out of Put, but when testing on Put Two, it will require specific strategies and an understanding of the environment: For Put Two, the agent needs to know that it cannot take two things at the same time, and should employ a strategy of taking one and putting one, etc. However, for some similar task types, AutoManual does have task generalization. For instance, removing one of the clean, heat, and cool task types during building and testing across all tasks showed a success rate nearly identical to 97.6%. This is because strategies for these tasks can be easily generalized from one to another.\n\n4- **The Planner Can Ignore Rules**: This issue, as noted in the Limitations of the Appendix, is indeed recurring and challenging. We may consider integrating an additional validator agent to re-confirm whether the Planner adheres to the set rules or commits errors, using techniques like self-refine, but this would significantly increase token consumption. We leave this challenge to future research.\n\n5- **Other Environments**: It is important to note that almost no environment is both practical and unrelated to GPT’s training data. For famous games like Atari or Sokoban, relevant knowledge is likely included in GPT's training dataset. Given that Atari's [Instruction Manual](https://atariage.com/system_items.php?SystemID=2600&itemTypeID=MANUAL) is a public document, GPT has likely seen and memorized the strategies for playing these games. In addition, even though home and web-based environments might overlap with GPT's training materials, our experiments show that relying solely on GPT's inherent knowledge (Planner Only) achieved only a 77.6% success rate, indicating the insufficiency of intrinsic knowledge for solving tasks in these environments. Our AutoManual method effectively supplements environmental knowledge through trial and error.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nMpg81SREu", "reviewer": "Reviewer_3jfh", "summary": "This paper introduces AutoManual, a framework enabling LLM agents to autonomously build understanding of new environments and generate instruction manuals through interactive learning. The system comprises three main components: a Planner agent generating code-based plans for environment interaction, a Builder agent updating a structured rule system based on observed trajectories, and a Formulator agent compiling rules into a comprehensive manual. To mitigate hallucinations in rule management, the authors implement a case-conditioned prompting strategy for the Builder. The framework is evaluated on the ALFWorld and MiniWoB++ benchmarks, demonstrating high success rates. AutoManual aims to address limitations of prior approaches such as path dependency and reliance on extensive human-provided instructions, potentially improving the adaptability and generalization of LLM agents in diverse environments.", "questions": "1. How is the initial rules and examples selected and how sensitive is the performance to the quality of the initial rules/examples provided? How this initialization needs to be changed when we switch to different environments?\n2. In Section 3.2 Skill and Reflection library construction, the most similar task is retrieved from the skill library when new task comes, how is the similarity between task been defined?\n3. In 3.3 Rule types definition, how does the 6 rule types come up?  \n4. Table 2 shows the success rate of LLM agents on 9 task types with feedback, what are those feedback, and is the feedback rely on an oracle who know the correct answer/solution for the given query?", "limitations": "The authors provide a reasonable discussion of limitations, including reliance on GPT-4 and potential scaling challenges. However, more concrete mitigation strategies are missing. They also touch on broader impacts, noting both positive potential (e.g., improved safety, worker training) and risks (e.g., unpredictable behaviors). \n\nSide note: should the limitation and broader impact discussions be included within the main 9-page scope rather than in the Appendix?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\n\n- Novel and creative framework for online rule learning and manual generation by LLM agents\n- Innovative case-conditioned prompting strategy for rule management\n\nQuality:\n\n- Detailed examples and case studies demonstrating the approach\n- Careful comparison to relevant baselines (e.g., ReAct, Reflexion, ExpeL)\n\nClarity:\n\n- Clear explanation of the overall framework and key components\n- Well-structured paper and easy to follow\n- Detailed appendix with prompts, examples, and implementation details\n\nSignificance:\n\n- Addresses an important challenge in building more general and adaptable LLM agents\n- Demonstrates a path towards more autonomous learning in new environments", "weaknesses": "1. Heavy reliance on very capable LLMs like GPT-4, and dependency on oracle feedback during rule construction process limits broader applicability, do we have results by fine-tuning smaller models on the generated data?\n2. Scalability to larger, more complex environments is unclear - the current approach puts all rules in LLM context. It’s not clear how easily the method can be generalized to other environments, for example, SWE-Bench.\n3. Limited evaluation of the quality and usefulness of the generated manuals for humans. Lack of theoretical analysis on the rule learning process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1- **Fine-tune Smaller Models:** We are unsure about the \"generated data\" referred to by the reviewer. Is this data generated through interactions between GPT-4 and the environment to create successful trajectories? This also seems to require the use of very capable LLMs. It is important to note that the goal of our work is to enable LLM Agents to quickly and autonomously adapt to unfamiliar new environments. These environments do not have sufficient high-quality human or generated data for fine-tuning smaller models. Our method requires only 36 turns of interactions with the environment to build a Manual, whereas accumulating enough data to fine-tune an LLM would require an order of magnitude more interactions.\n\n2- **Scalability to Larger, More Complex Environments:** We currently place all rules in the context to ensure that the Planner can access all the rules, thus eliminating the impact of rule retrieval on our experimental outcomes. However, for more complex and larger environments, leveraging technologies like Retrieval-Augmented Generation (RAG) is indeed necessary to effectively address the challenges. To demonstrate our method's potential in handling complex environments, we conducted experiments on the Reddit site of WebArena. WebArena [1] is a high-difficulty benchmark simulating real website interactions. In this environment, we set a maximum of 15 rules for AutoManual. We used AutoManual paired with GPT-4-turbo to build and formulate tasks in the Reddit site environment, subsequently testing on 106 tasks. The test results and comparisons with other works are shown in the table below. We provide the code and trajectories at [our comment below]. We will also include these experimental results in our paper.\n\nNotice: Although SWE-Bench is also a challenging benchmark, its interactivity is limited. Therefore, we chose a more interactive environment, WebArena, which aligns better with the \"Interactive Environment\" in our paper's title.\n\nTable: Test on WebArena (Reddit) using GPT-4-turbo\n| Methods       | Examples | Success Rate(%) |\n| ------------- | -------- | --------------- |\n| ReAct [1]     | 2        | 6.0             |\n| AutoGuide [2] | 19       | 43.7            |\n| SteP [3]      | 14       | 55.0            |\n| AutoManual    | 1        | 65.1            |\n\n3- **Sensitivity Analysis of AutoManual on Examples and Initial Rules:** In Appendix F.2, we analyze the sensitivity of AutoManual to the example and initial rules. We provided AutoManual with an example of the \"enter-text\" task from MiniWob++, which is the simplest task type in this suite, along with the corresponding initial rules. Our findings suggest that while simpler examples and initial rules do influence the outcomes, the impact is minimal.\n\n**Strategy for Generating Initial Rules and Examples:** Generally, for a new environment, we first select one task to serve as the example task, with no specific criteria for selection (In the ALFWorld experiments, we chose the simplest \"put\" task and achieved significant results). Then, a human operator generates a trajectory that completes the task, thus providing the necessary example. Initial rules typically comprise a \"special mechanism\" and a \"success process\" type rule, which involves considering any counterintuitive aspects of the environment and summarizing strategies to accomplish the example task.\n\n4- **Task Similarity:** We follow the vector similarity approach used by Voyager and Adaplanner for skill retrieval, utilizing OpenAI's embedding model to process task names into embeddings. These embeddings serve as queries and keys, with the vector similarity from the FAISS library used to retrieve the most similar skill.\n\n5- **How The Six Rule Types Were Created:** In \"Rule System\" of Section 3.3, we have mentioned the selection of the six rule types used in our rule system. Specifically, we believe that knowledge about the environment necessitates recording the transition functions, $T(o'|o, a)$ for observable states and $T(s'|s, a)$ for system states, as well as understanding how trajectories $\\tau_{\\rho}$ lead to outcomes of r=0 or r=1. These insights correspond to the task types: \"Special Phenomenon\" for $T(o'|o, a)$, \"Special Mechanism\" for $T(s'|s, a)$, and \"Success Process\" for $r(\\tau_{\\rho})=1$, \"Unsolved Error\" $r(\\tau_{\\rho})=0$. Additionally, the analysis of task segments within trajectories helps in achieving overall task success by identifying successful or failed sub-goals. This leads to the rule types \"Useful Helper Method\" and \"Corrected Error.\"\n\n6- The terminology \"9 task types with feedback\" and \"44 task types without feedback\" used in our paper follow previous works, Adaplanner. The \"44 task types without feedback\" denotes tasks where the HTML of the webpage remains static and does not change in response to the actions performed. For example, in the \"click-button-sequence\" task type, the user is required to click buttons One and Two, yet no observable changes in the HTML occur during these actions; the reward is provided only at the end of the task. Conversely, the \"9 task types with feedback\" refer to tasks where the webpage reacts to the actions taken. An illustrative example is the \"tic-tac-toe\" task, where after the agent makes a move, the webpage (the opponent) responds by making its move based on the agent's actions. This makes the \"9 task types with feedback\" more challenging as they represent more complex tasks, and the agent does not have full visibility of the webpage's content at the start.\n\n[1] Zhou, Shuyan, et al. \"Webarena: A realistic web environment for building autonomous agents.\" arXiv preprint arXiv:2307.13854 (2023).\n\n[2] Fu, Yao, et al. \"Autoguide: Automated generation and selection of state-aware guidelines for large language model agents.\" arXiv preprint arXiv:2403.08978 (2024).\n\n[3] Sodhi, Paloma et al. \"Step: Stacked llm policies for web actions.\" arXiv preprint arXiv:2310.03720 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We have provided detailed rebuttals for each reviewer's Weaknesses and Questions. We kindly ask reviewers to review the respective rebuttals, and if you find that our rebuttals have addressed your concerns, please consider raising your scores. Thank you very much for your patience.

Meta-review: This paper introduces AutoManual, which is a framework built upon LLMs to facilitate planning using three agents; Planner, Formulator, and Builder. This planning framework automatically improves with experience by codifying (Formulator) and improving/revising (Builder) a set of planning rules which can then be used by the Planner to accomplish a goal. AutoManual is validated on two benchmarks against various baselines.

The reviewers were mixed but not negative. Reviewers shared concerns that the > 50 tasks evaluated were 'homogeneous' in nature, which called into question how generalizable the approach was. There were also remaining questions about how much sensitivity there was to prompt engineering due to the temperature = 0.

The reviewers agreed that there was originality, positive results (including ablations), and sound results. While concerns remain about generalizability, I am inclined to view this paper on the positive side due to the careful way in which the approach was evaluated, the positive results, and the contribution that this paper makes towards open-ended learning with LLMs for planning, which is a difficult problem for LLMs.
Decision: Accept (poster)