AdaPlanner: Adaptive Planning from Feedback with Language Models
Authors: Haotian Sun
Yuchen Zhuang
Lingkai Kong
Bo Dai
Chao Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Large language models
decision making
closed-loop planning

Abstract: Large language models (LLMs) have recently demonstrated the potential in acting as autonomous agents for sequential decision-making tasks. However, most existing methods either take actions greedily without planning or rely on static plans that are not adaptable to environmental feedback. Consequently, the sequential decision-making performance of LLM agents degenerates with problem complexity and plan horizons increase. We propose a closed-loop approach, AdaPlanner, which allows the LLM agent to refine its self-generated plan adaptively in response to environmental feedback. In AdaPlanner, the LLM agent adaptively refines its plan from feedback with both in-plan and out-of-plan refinement strategies. To mitigate hallucination, we develop a code-style LLM prompt structure that facilitates plan generation across a variety of tasks, environments, and agent capabilities. Furthermore, we propose a skill discovery mechanism that leverages successful plans as few-shot exemplars, enabling the agent to plan and refine with fewer task demonstrations. Our experiments in the ALFWorld and MiniWoB++ environments demonstrate that AdaPlanner outperforms state-of-the-art baselines by 3.73% and 4.11% while utilizing 2x and 600x fewer samples, respectively. The implementation of AdaPlanner is available at https://github.com/haotiansun14/AdaPlanner.

AdaPlanner: Adaptive Planning from Feedback with Language Models
Haotian Sun'; Yuchen Zhuang
Lingkai Kong',
Bo Dail,
Chao Zhang'
Georgia Institute of Technology
(haotian. sun, yozhuang, lkkong,
bodai0cc vodai@cc.gatech.co edu
Abstract
Large language models (LLMs) have recently demonstrated the potential in acting as autonomous agents for sequential decision-making tasks. However, most exist- ing methods either take actions greedily without planning or rely on static plans that are not adaptable to environmental feedback. Consequently, the sequential decision- making performance of LLM agents degenerates with problem complexity and plan horizons increase. We propose closed-loop approach, AdaPlanner, which allows the L.LM agent to refine self-generated plan adaptively in response to environ- mental feedback. In AdaPlanner, the LLM agent adaptively refines its plan from feedback with both ilap and out out-of -oa------aa refinement strategies. To mitigate hallu- cination, develop code-style LLM prompt structure that facilitates plan gener- ation across a variety of tasks, environments, and agent capabilities. Furthermore, we propose skill discovery mechanism that leverages successful plans few-shor exemplars, enabling the agent to plan and refine with fewer task demonstrations. Our experiments in the ALFWorld and MiniWoB++ environments demonstrate that AdaPlanner outperforms state-of-the-ard baselines by 3.73% and 4.1 1% while uti- lizing 2x and 600x fewer samples, respectively. The implementation of AdaPlanner is available at httpp:://githut /
Introduction
Large language models (LLMs) have recently emerged as versatile autonomous agents for sequen- decision-making making in grounded environments. Traditional cecision-making methodologies like tial decision-
Reinforcement Learning (RL) require extensive task-specific training data and often lack the ability to generalize across tasks and environments. In contrast, LLMs are pre- pre-trained on massive and diverse textual data, which gives them extensive world knowledge and the ability to reason over the knowledge. This makes them highly versatile and able to handle complex, real-world scenarios that may involve multiple steps of planning and recision-making
Existing methods that leverage LLMs as autonomous agents for decision-making can be briefly categorized into two groups (Table Dp: open-loop systems and closed-loop systems. Open-loop methods 27 5 18 12
17 [6] rely on pre-determined plans to accomplish the desired task without any feedback adaptation mechanism. On the other hand, closed-loop systems 25 6 g 22 19 23
incorporate environment feedback to continuously monitor system behaviors and make refinements and adjustments of the plans accordingly, which therefore more flexible.
However, both existing open-loop and closed-loop LLM agents have inherent drawbacks. Open-loop systems computationally cheap and simple; however, they do not consider feedback from the environment and stick to the initial plan, which lack of adaptability, and, thus, can easily generate suboptimal plans. On the other hand, most existing closed-loop methods generate fixed plan and only update their executing actions upon environment feedback. This causes them to make
'Thee authors contributed equally to this work.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Methods
Feedback Utilization
Instruction Type
Prompting
Decomposition Experience Refinement
Open-Loop Methods
CoT 24
Prompting
Language
Least-To-Mae Most
2R
Prompting
Language
Sub-Goals
Zero-Shot Planner
Prompting
Language
HuggingGPT
18
Prompting
Language
Sub-Gools Sub-Goals
Chameleon
12
Prompting
Language
Implicir Closed-Loop Methods with Fixed Plan
ReAct [25
Taking Action
Prompting Prompting
Language
Inner Monologue
Tuking Action
Language
RCI (8)
Taking Action
Prompting
Language
ProgPrompt 22
Taking Action
Prompting
Code
Code as Policies
Taking Action
Prompting
Code
Reflexion 19
Taking Action
Prompting
Language
Past Failure
Explicit Closed-Loop Methods with Plan Refinement
DEPS 23
Modifying Plan
Prompting & Training
Language
Sub-Gools
Past Failure
AdaPlanner
Action & Plan
Prompting
Code
Sub-Goals
Past Failure Success
Table A comparison of methods that leverage LLMs for decision making. Each method's features
are reported across five categories: 1) Environment Feedback Utilization: The method can use feedback to decide the next action (Taking Action), revise the entire plan (Modifying Plan), or do both (Action & Plan). 2) Instruction Type: The method may require prompting, training, or both. 3) Prompting Style: The method can employ either natural language or code for its planning backend. 4) Task Decomposition; The method might decompose the task into sub-goals not. 5) Experience Refinement: The method can learn from past failure, past success, or both. The AdaPlanner proposed in this paper is highlighted in gray.
sub-optimal decisions that adapt to the environment in the short term but could have detrimental implications for future steps. DEPS [23] is the only exception, a method that modifies its entire plan based on feedback from the environment. However, it requires training a plan selector to choose the most successful plan, which requires a significant amount of task-specific data. As a result, applying this method to different tasks can be challenging.
To address the limitations of existing L.LM agents, we propose AdaPlanner, a closed-loop planning method with LLM playing two roles planner and refiner. The planner decomposes the task into manageable sub-goals and predicts environmental feedback for each. During execution, the refiner distinguishes and responds to two types of environment feedback - in-plan feedback feedbbaak is the environmental observation that aligns with the prediction, and out-of-plan feedback is one that deviates from the prediction. For in-plan feedback, the refiner can dynamically query the LLM to perform reasoning and extract key information from in-plan feedback expressed in natural language. This is achieved through specific action called ask LLMO), in which the LLM separately parses the observation and obtains information pertinent to subsequent actions. For out-of- plan feedback, the refiner proactively revises the entire plan and resumes to solve the current task from an intermediate point. AdaPlanner's s adaptive closed-loop framework alleviates the need for prior knowledge about the feedback structure and permits the agent instantly adopt refined plan rather than restarting from scratch in a reset episode. This leads to a more efficient and adaptive decision making process.
AdaPlanner operates solely via prompting, eliminating the need for dedicated training phase and reducing its computational cost. Furthermore, AdaPlanner leverages a code-based prompting for precise planning and refinement. The use of code prompts facilitates task decomposition into sub- goals and mitigates LLM hallucination during the decision makinn process. AdaPlanner also features skill discovery process, which accumulates successful experiences to guide future planning. This feature further enhances its long-term planning ability and sample efficiency.
We formally define the planning problem with LLM,, and introduce open-loop vs. closed-loop control system, which will motivate our method, Section 2 Each component of the proposed AdaPlanner is specified in Section 3, including code-based prompting in Section 3.1 closed-loop adaptation Section 3.2 and skill discovery in Section 3.3 and empirically justified Section The superior performance of AdaPlanner on both ALFWorld and MiniWoB++ demonstrates our proposed adaptive closed-loop framework can effectively enhance planning performance, even when faced with a limited number of samples.

---

[0, 10,02 " dy)
(0,0 m0.00
Planner
Environment
Planner
Environment
(o,a,, oc+1)
Open-Loop
Implicit Closed-Loop
(0 "n, By. By Pre1)
Skill Memory
extract info from
Oea: aligns with prediction
In-Plan Refiner
(a), 0g. Bp Beeer "Rp .uy) appa.." "n, of)
Planner
Environment
Out-of-Plan Refiner
AdaPlanner (0p9 "0,00 Pees)
0.41 prediction Explicit Closed-Loop
Figure I: A comparison between open-loop, implicit closed-loop, and explicit closed-loop systems.
2
Preliminaries
Problem Formulation. We consider adopting an LLM as an autonomous agent to solve different tasks in text-based environments. For initialization, the agent is provided with allowed actions A in the environment, as well as a text-grounded task definition g E G from the task space G. Besides, the initial state of the environment is also observed as E O from the observation space O. With such inputs, the LLM agent needs first to generate an initial planning policy for solving the task Po(Pols g. 01) G O + ((A"), where T' is the total length of steps in the generated plan and A--) is probability simplex function. Also, the agent can interact with the environment for feedback: When the agent interacts with the environment at the f-th step. the agent receives an observation or O from the environment and generates a trajectory-like context C2 = (01, 01. O2, ag2 " where a,, a²2, ... a²_ are the executed actions within the environment. As the agent may modify the actions according to the feedback, the executed actions d' a2. .....," aa- can be different from the actions a1 .2. at-1 in the initial plan. We denote p(-1g, C,, Pt) as the high-level planning policy that generates on given plan P.. Given the context ez and the entire plan at the last step Pt-1, the agent refines future decisions. In the end, the LLM agent should model both the initial planning policy and the environment secdback-condttioned policy to complete the given task successfully.
44-1, 0z).
entire plan and T--1g, Ct2 Pt) the
policy conditioned
Open-Loop System. An open-loop system is a non-feedback system (Figure [ where the output is solely dependent on the input, without any consideration of the environmental feedback. feedbabak. Thus, in an open-loop system, the entire initial plan over the time horizon T is predetermined and static by the initial planning policy Po(-Ig, 01), without any feedback- refinement. Despite their simplicity, open-loop systems are notably vulnerable to environmental changes, as they lack the capacity to adapt or adjust their plans based on environmental feedback.
Closed-Loop Systems. On the contrary, a closed-loop system (Figure I refers to a planning process that incorporates environment feedback to adjust and refine future decisions, involving both initial planning po(:Ig, 01) and two levels of feedback-based refinements, p(-1g, cr, P:-1) and (-Ig, C2, Pt-1), in the system.
Implicit Closed-Loop Systems. After each step of interaction with the environment, implicit closed- loop systems will maintain the initial plan (i.e., P. Pb) and only modify single action based on the feedback. Therefore, the feedback- the modified action from action space, while the remaining actions dzst. for future steps remain the same as the initial plan. Although locally-optimal actions are adopted at each step, inaccuracies in initial planning can result in task failure or non-completion.
refinement is defined as π <(a, g, C,2 Po). where af E A
Explicit Closed- Loop Systems. Explicit closed-loop systems refine the entire plan based on environ- ment feedback following the policy p( PPlg ct, Pt-1), where P. E ((AAT-t is the refined plan at time step containing the modified future actions a'z, to execute and P is the old plan modified in the previous time step. Allowing for constant refinement and improvement of the plan, explicit closed-loop systems can help prevent costly mistakes or missed opportunities that might arise from adhering to outdated plans. Our proposed AdaPlanner is an explicit closed-loop system.
3

---

3 AdaPlanner
Model Architecture. Our AdaPlanner model, shown in Figure 1 consists of two main components:
LLM-based agent that functions dually as planner and plan refiner, and a skill memory module designed to enhance sample efficiency through skill discovery.
The LLM-based agent, in its planner role, generates a comprehensive plan and performs preliminary assessments determine its feasibility. The initial planning is modeled as Po(Po)g g. 01). As the plan unfolds. the agent also operates as a refiner, conducting feedback- beedback-based refinement in both in-plan and out-of-plan manners. In-plan and out-of-plan refinement processes primarily differ in how they impact future actions. In-plan refinement is one-step action that integrates useful information into the existing plan for better action grounding. After this in-plan phase, future actions will be generated using the updated context <(a'>19. Ig. CSt U {h+}, ). Po), where h2 represents the new information obtained from & via in-plan refinement at timestep l. Out-of-plan refinement, on the other hand, leverages environmental feedback to directly revise the entire plan, denoted as p(P g, Ct, Pr-1). This mechanism allows for comprehensive adjustments to be made to the plan in response to unexpected environmental feedback. Skill memory serves as a repository, archiving past successful plans and their respective interactions with the environment. If the agent encounters task resembling the skills stored in memory, these skills can serve as few-shot exemplars in the LLM agent's 's prompt. This feature improves not only sample efficiency but also reliability for future planning.
Environment Interaction. AdaPlanner employs adaptive closed-loop planning and active environ- ment interaction for task solving. It can anticipate environmental observations and proactively refine the plan only when there a discrepancy between expected and actual outcomes. This is achieved by decomposing the planning process into N manageable sub-goals. During the planning and action- taking process, the agent selects from a set of timestamps, (fi,...,t,, i }, to evaluate the success of each sub-goal. If the sub-goal does not align with the planned prediction at timestep t E (tl...., &y J. the environment actively sends the previous sub-trajectories (0), a11 .......... ... refiner for plan revision. This process allows the agent to check the success status only at N crucial points, thereby reducing computational costs (number of API calls) and enhancing efficiency.
Ot, ay 0t+11) back to the
3.1 Plan Generation via Code-Based LLM Prompting
AdaPlanner plans and refines by using Pythonic code prompts for LLMs. Consistent with previous observations B [2], we have found that using code prompts instead of natural language prompts for LLMs reduces ambiguity and misinterpretation, which significantly reduces LLM hallucination during plan generation and refinement. We design code prompts during different stages of decision- making, including adaptive planning, feedback generation, and in-episode refinement. We provide detailed description of the prompts used at each stage in Appendix [
To generate an initial plan for solving a given task, we input a task description, the permissible actions in the environment, and, when available, sample demonstrations of task resolution into LLM. These pieces of information are all formatted into Pythonic code format for LLM prompting. Figure 2 (a) shows an example programming-buin based plan generated by AdaPlanner for solving a put task in the ALFWorld environment. The generated solution function is provided with two input arguments: the first is the agent object, which encapsulates environmental information to be used by the agent. The second is the variable chart_from which is a parameter indicating the subgoal from which the agent will later resume its execution with a refined plan. By default, the start from is initialized as 1. The value of this variable can be further reassigned during the refinement.
When prompting L.LM to generate the code-based plan, we design the prompt to teach L.M to decompose a complex task into sub-goals. As shown in Figure a the generated code plan solution (agent, start_from-l consists of: 1) a general plan at the outset that decomposes the task into subgoals in the form of comments; and 2) a sequence of sub-plans, each consisting of admissible actions corresponding to a specific subgoal. Such a mechanism allows our method to handle complex, long- horizon tasks by hierarchically decomposing them into sequences of subgoals. Furthermore, each subgoal ends with an assertion statement to test its fulfillment, which allows our method to interact actively with the environment and later resume its execution with a refined plan.

---

(a) Initial Plan (Extracted)
Environment
(b) Out-of-plan Feedback [estracted]
solution(agent, start, from=1):
Execution
Halt
Error in [Step 3]; cannot clean lettuce using the sinkbasin 1. am toilet and holding None, The last three interactions
General plan
General plan: need to get a list of receptacles to find the lettuce, clean put diningtable.
SStp 3]
before the error were:
[Stop 1J Get iist receptacles where the lettuce likely to appear.
AdaPlanner
Act: go to sinkbasin
Sub-plan
Obs: On the sinkbasin 2, you see nothing.
from <= 1:
Out-of-plan refine
recap_ to to__o- literal _evalask [LM]
In In-plan
regenttrcceptackes) in descending order based on the
refine
likalihood finding lettuce:)) }}
(c) Refined Plan (Extracted, revisions highlighted)
assert recop. to_ _o, I' f'Error in [Stop 1]: recep_.to, _oopp__t__g____
def solution(agent, start_ from=3]:
should not be empty. ragent.reportO)"
General plan: need to get list of receptacles to find lettuce, toke the lettuce to the sinkbasin, clean and put in diningtoble.
[Step 2J lettuce
receptacle in until seeing
A (Step 1] Get list of receptocies where the lettuce likely to appear.
start_ from for receptacle secep_to_poo
A (Step 2] Go to each receptacle in the list until seeing a lettuce
observation ageettgoto(eecepaacle)
if llltttuce' observation:
(Step 3] identify the lettuce I found and take it
break
start_ from 3:
Predict
assert 'lettuce' observation, f'Error [Step is no lettuce in/on (rece__to_go). (agent.report))"
found lettuce 'lettuce' + ask Lik_LLM(fFrom the observation, get the identifier of the lettuce: (observation)- '
via assert
observation agent.takoqfound Jettuce, recaptaclo) assert aoodi =: found, lettuce, fError in [Stop 3]: cannot
[Stop 3] Identify the lettuce just found and clean it from
take (found lettuce) from (receptacle). (agent.report()
In-plan refine
found_ Jettuce- lettuce' + aask_ From the observation, get the identifier of the lettuce: (observation):]
(Step 4] sinkbasin If start from <= 4;
the lettuce.
observation agenttcleanffound lettuuue 'sinkbasin 1') assert 'clean' observation, l'Error [Step cannot clean [fourd_lettuce) using the sinkbasin 1. (agent.report))
observation agent.gotof siikkaainnn 1
observation agenn.cleanffound Jettuce, 'sinkbasin 1') assert 'clean' observation, FError in [Step 4]: cannot clean
(found lettuce) using sinkbasin (agent.reportj))
[Step 4J to a diningtable and put the lettuce
[Step 5] to diningtable put lettuce it.
Figure An illustrative example from ALFWorld to show the proposed adaptive closed-loop planning through code. The task is to put some clean lettuce on the diningtable. The in-plan feedback in (a) sentence like On the countertop 2, you see knife 1, a lettuce 1, saltshaker 2, and a soapbottle 1. This feedback is managed by the ask ask_LLM() action. The execution of the initial plan might yield misaligned observations, triggering an out- of-plan feedback and refinement process. For instance, the agent cannot clean the lettuce if not currently located at a sinkbasin. The out-of-plan feedback in (b) assists AdaPlanner in generating revised plan (c) so that the agent will move to a sinkbasin before cleaning the lettuce. AdaPlanner then determines to resume from step 3 within the same episode. The task can be successfully completed using the refined plan.
3.2 Adaptive Closed-Loop Plan Refinement
Once an initial plan is generated, AdaPlanner then prompts the LLM to correct any syntax errors. After this, the code undergoes execution through the environment interface. The interface is responsible for grounding the actions in the environment, and also for routing environmental observations back to the code as a return value. This bi-directional flow allows AdaPlanner to adapt and refine its plan in response to environmental observations in a closed-loop manner.
In-Plan Feedback and Refinement via ask_LLM Action. When AdaPlanner observes that the environment is aligned with the anticipated plan, it performs in-plan refinement. This allows it to extract useful information from the observation that can be used for upcoming actions. To achieve this, we provide the agent with an additional action called ask [ak_( which is used to formulate a plan alongside ssecifccc actions. The ask_ (s function enables AdaPlanner to self-query and perform reasoning based on specific information parsed from environmental observations. For instance, in [Step 3] in Figure 2 (a), the ask_LLMC action extracts the identifier of the found natural-language object lettuce from the natural- language observation. This information can then be fed into later actions. As an additional atomic action, this in-plan refinement is integrated into the plan at any point where the planner deems a reasoning process is necessary. Existing code-generation based methods 222 D face a challenge in this task, especially when there is no prior knowledge of the structure and organization of these feedback sentences. In contrast, our AdaPlanner method leverages I.LM to parse critical information from diverse feedback presented in natural- -anguag sentences to streamline plan execution.
5

---

Out-of-Plan Refinement with the Seeine-The--esstee Mechanism. After each sub-plan execu- tion, AdaPlanner actively checks an assertion condition ensure that the current plan is proceeding as expected. If the assertion fails, AdaPlanner performs out-of-plan refinement. For example, in Figure 2 (a), after [Step 3], the agent is expected to hold lettuce. If this condition is not met, AdaPlanner generates an error message that details the current progress of execution gathered by the report() () function. In ALFWorld tasks, this function provides a report of the agent's location, the object it is holding, and the last three interactions with the environment, shown in Figure (b).
AdaPlanner then utilizes this information to perform out-of-plan refinement.
During the out-of-plan refinement as in Figure 2 (c), AdaPlanner uses prompt similar to the one used during the initial planning stage, but with an additional feedback message that reflects the current state. Detailed prompts are provided in Appendix [ AdaPlanner then refines the plan based on the newly acquired information and also determines the value of start_ from by comparing the plan before and after the refinement. The newly refined solution() () is then executed from the breakpoint start_ ffom. This breakpoint contains all variable states that were saved prior to refinement. Consequently, the current episode can continue from an intermediate checkpoint without restarting from scratch. We call this mechanism temee-then-rsumree It significantly speeds up task completion and reduces the number of LLM calls required.
3.3 Skill Discovery
Acquiring expert demonstrations for task solving can be costly, particularly the number of tasks increases. To address this issue, we have equipped AdaPlanner with a skill discovery feature. This is memory scheme that discovers and archives successful trajectories, thereby improving planning performance when dealing with similar tasks. The skill discovery process consists of two stages, which can be conducted alternately over several rounds, based on the interaction costs and computation resources.
Skill Acquisition. In the first stage, AdaPlanner attempts to solve unseen tasks, leveraging a limited number of human demonstrations other simpler tasks, or even no demonstrations. The model capitalizes on adaptive closed-loop planning to iteratively explore and refine solutions via trial-and- error approach. Upon successful completion of given task, the latest solution and the corresponding interactions are treated as candidate discovered skills.
Skill Filtering. the second stage, we compare the planning performance with and without the integration of the discovered solution into the prompt. If the inclusion of this solution boosts the success rate, it is archived as a discovered skill. Conversely, if it does not improve performance, discarded. This filtering stage is crucial because the iterative closed-loop refinement may integrate cpisode-specific information into the revised solution, potentially compromising its generalizability.
4 Evaluation
We test AdaPlanner two text-based decision-making environments: 1) ALFWorld (2 is text-based virtual household environment encompassing six distinct task types set. We evaluate AdaPlanner total of 134 tasks across these six types. 2) MiniWoB++ [I]] is a simulation environ- ment that covers a large range of computer tasks. We select 9 MiniWoB++ tasks with environmental feedback, and also adopt and test the 53 tasks evaluated in RCI [.. Both environments aim to solve complicated challenges with long- long-horizon solutions and sparse rewards. We also carefully designed ablation studies to justify the significance of each component in AdaPlanner. The Setup details and prompts for AdaPlanner are depicted in Appendix A and D Detailed introductions to each baseline are presented in Appendix B Note that we evaluate different baselines for these two benchmarks. These methods utilize task-specific samples for prompting or training purposes, thus necessitating separate evaluations for each benchmark.
we use success rate (%) to evaluate the performance of tested methods. The success rate is defined the number of successful episodes over the total number of episodes. Note that in ALFWorld, failure of an episode occurs when the total number of actions attains 50, with the task still unsolved. In MiniWoB++, failures can occur in two scenarios: either due to the execution of invalid actions or if the task remains unfinished following the execution of the entire plan.
Metries. Consistent with previous works 20 25
Main Results. AdaPlanner consistently outperforms the existing haselines, achieving state-of-the-ar performance, i.c., an overall success rate of 91.79% in ALFWorld tasks (Table 2 and 91.11% in

---

Method
Pick
Clean
Heat
Cool
Examine
Pick two
All (134 tasks)
Truining Baxed Methody
BUTLER 20
46,00
39.00
74.00
100.00
22.00
24.00
37.00
Implicit Closed-Loop Methods with Fixed Plan
ReAct 25
(GPT-3)
66.67
41.94
91.03
80.95
55.56
35.29
61.94
ReAct
(PT-3355
37,50
64.52
69.57
42.86
38.89
17.65
47.76
Reflexion 19 (GPT:3 3.5)
75.00
90.32
91.30
90.48
88.89
94.12
88.06
Reflexion 19 (GPT-3.5)
50.00
41.94
65.22
52.38
66.67
47.06
52.99
Explicis Closed-Loop Methods with Plan Refinement
AdaPlanner (GPT:3)
100.00
96.77
95.65
100.00
100.00
47.06
91.79
AdaPlanner (GPT-3.5)
77.78
93.55
69.57 93.65
62.96
78.43
80.60
Table 2: Success rate (%) tested methods on six ALFWorld tasks. For ReAct and AdaPlanner, GPT-3.5 refers to gpt-3.5-tun 5-turbo, while GPT-3 represents text-davinci-000 For Refexion, GPT-3.5 indicates gpt-3. 3. 5-turbo. GPT-3+3.5 used in the original Reflexion implementation. which utilizes both GPT-3
and GPT-3.5 dext-davinet (eriiicc-03) 003) for action generation and failure reflection, respectively. AdaPlanner is prompted with one specific example per task, making up six demonstrations in total. This is half the number of samples used in React and Reflection. The best-performing results are marked in bold. The results of our method are colored in gray.
MiniWoB++ tasks with feedback (Table 3 Specifically, in ALFWorld, AdaPlanner equipped with GPT-3 achieves a remarkable success rate exceeding 95% in the majority of individual tasks. also surpasses all other baselines in the Pick, Clean, and Examine tasks. Notably, even in the task with the lowest performance (Pick two), AdaPlanner still outperforms BUTLER and ReAct. In the MiniWoB++ environment, AdaPlanner demonstrates superiority over all other methods on tasks that provide feedback. This superior performance suggests that AdaPlanner effectively leverages feedback refine its plans and enhance its performance. Furthermore, AdaPlanner maintains competitive performance on tasks without feedback, achieving a success rate of 93.22%. Note that AdaPlanner's s success rates of tasks without feedback are still comparable to CC-Net, the state-of-the-an art model requiring over 23,000 samples per task. This result highlights the efficacy of the grogramming-basee planning strategy employed by AdaPlanner. In both environments, AdaPlanner consistently delivers superior competitive performance when compared to not only training- based methods but also implicit closed- loop methods under the same LLM models. These results affirm the effectiveness of the proposed explicit closed-loop plan refinement in AdaPlanner.
Furthermore, summarize the rela- tionship between success rate (%) and the number of samples in Figure 3
100
AdaPlanner
100 AdaPlanner
cc-iET
Success rate (%6
80
Retlexton
Successs rete (%6
80
WGF
In ALFWorld, AdaPlanner yields the highest performance with the fewest number of samples. In Mini WoB++, our method outperforms most base- lines. Notably, our method achieves performance comparable to CC-Net but requires 600 times fewer sam- ples. This study highlights that Ada- Planner significantly reduces the need for extensive demonstrations or expert trajectories, thereby offering a more resource-efficient solution.
ReAct
60
60
Webre-5-30
40
BUTLER
40
101
10-
Samples per task
Samples per task
(a) ALFWorld
(b) MiniWoB++
Figure 3: Relationship between success rate (%) and the
number of expert demonstrations in ALFWorld and Mini- WoB++ environments. We adopt the same settings as in Table E (GPT-3 version) and Table 3 The top-left corner represents the pinnacle of sample efficiency.
Adaptive Closed-Loop Architecture Enhances Planning Performance. Figure 4a shows the performance v.s. the number of closed-loop refinements, under settings with different numbers of demo samples. The detailed example selection for this study is provided in Appendix A We observe a significant trend of increased success rates corresponding to each subsequent closed-loop plan refinement. This indicates the AdaPlanner's s ability to consistently leverage real-time feedback for performance enhancement, regardless of the number of samples used. Remarkably, AdaPlanner
7

---

Method
With feedback tasks)
No feedback tasks)
All(sa tasks)
Truining-Basee Based Methods
CC-Net 7
87.00
95.66
94.00
WGE
67.60
87.93
86.00
Finetuming Based Methods
WebN-T5-3B 4
38.50
54.67
52.00
Implicit Closed-Loop Methods with Fixed Plan
RCT
8
81.56
92.68
91.00
Explicir Closed-Loop Methods with Plan Refinement
AdaPlanner
91.11
93.22
92.87
Table 3: Success rate (%) of tested methods on two subsets of tasks in the MiniWoB++ environment.
RCI and AdaPlanner harness GPT-3.5 (gpt-3.5- torbb and text sext-davinci-003) Our AdaPlanner method is provided with 38 human-writter demonstrations: then, it automatically discovers 21 additional examples via skill discovery, which makes up the final set of 59 examples for 53 tasks. This is around half the number of samples used in RCI and over one six hundredths of the number of samples used in CC-Net. The rest-performing results are marked in bold. The results of our AdaPlanner are colored in gray. Per-task success rates are provided in Appendix
backends.
maintains this trend of success rate enhancement even when the total number of demonstrations across all six tasks is as low as two. In addition, as displayed in Table AdaPlanner consistently outperforms both ReAct and Reflexion in producing the shortest trajectory lengths across all tasks in ALFWorld. Moreover, comparison with Reflexion, depicted in Figure [4]] shows AdaPlanner's consistently superior performance across all iterations of closed-loop corrections. These observations highlight AdaPlanner's sample efficiency and its potential for real-world applications where the number of available demonstrations is limited.
Method
Pick
Clean
Hcat
Cool
Examine
Pick two
All (134 tasks)
22Act[25
19.55
25.79
19.7
27.86
29.72
36.29
25.81
Reflexion
18.77
25.25
19.13
25.76
27.11
35.76
18.90
AdaPlanner
10.79
13.45
17.61
13.33
21.00
20.71
15.60
Table 4: Average trajectory length (# steps) per task for ReAct, Reflexion, and AdaPlanner. We adopt
the same settings in Table ( version). Following the setting in [25] be terminated after reaching 50 steps.
19 each episode will
Code Interface Mitigates Hallucination. The latest gpt-3. 3-..---tuuro reported to be the most capable GPT-3.5 model while reducing the cost by a tenth compared to other prevailing GPT-3 M and text-davinci-003
and 3.5 models [4] (e.g., text-davinci--nn from Table indicate that gpt- 3.pt-3.5-turbb. underperforms in decision-making tasks relative to its predecessors, i.e., text-davinci -002, in all LLM-agents. Upon examination of trajectories from both models, we observed a noticeable hallucination with GPT-3.5 (gpt-3. 5-turbo), shown in Appendix B We hypothesize that gpt-3. gpt-3.5-turbe 5-turbo might be a smaller-scale model that is more prone to hallucination. We also found similar hypothesis drawn based various experiments in 26). Furthermore, gpt-3.5-turbo 5-turbo is primarily optimized for human conversation tasks, which could potentially compromise its performance on tasks such as code generation and I3) Despite this, AdaPlanner demonstrates a remarkable level of resilience against hallucination even with 3.5--urbtbb (Table 2). while ReAct and Reflexion are more sensitive the hallucination issue. AdaPlanner's i resilience against ballucination can be attributed to its use of code prompts, which provide more formal and constrained generation space for LLM. For comparison, we implement an ablation version of AdaPlanner without the code interface by translating solution examples directly into plans and actions using natural language. Without the code interface, AdaPlanner's performance substantially drops in both ALFWorld and MiniWoB++ environments (Figure 4c). from 81% to 46% and from 93% to 66%, respectively. This significant performance drop underscores the essential role of the code interface in AdaPlanner.
However, our findings
8

---

9316
100
77.3
80
Success rate (60
80
With
15.0
Successs rata
60
66%
60
60%
72.5
Samples
83
Ad3P anner IGPT 3)
Adal unner (cP7-5.55)
Succisss
38%
45%
70.0
Refiecos 10P1-3+33 2+35)
19%
67.5
Befescos (CPT-3.5)
With
Closesioop corrections
Closed loop corrections
ALFWord
MiriWe++
ALFWorl
HinWo8
(a)
(b)
(c)
(d)
Figure 4: Performance comparison 134 ALFWorld tasks in different cases. We adopt the same settings as Table 2 (a) and (b) presents the success rate (%) with different numbers closed-loop conrections: (a) compares AdaPlanner with different numbers of samples; (b) compares AdaPlanner and Reflexion with two LLMs. (c) shows the success rate (%) of AdaPlanner with and without code interface (CT). (d) shows the success rate (%) of AdaPlanner with and without skill discovery (SD). Note that for (a), the number signifies the total number of samples used across all six tasks.
Skill Discovery Improves Sample Efficiency. The skill discovery in AdaPlanner utilizes long-term memory mechanism that retains successful solutions, thus boosting planning performance when faced with similar tasks. An ablation study depicted in Figure 4d compares the performance of AdaPlanner with and without the implementation of skill discovery. In the skill acquisition stage, we provide a maximum of one demonstration. expert demonstration of the simplest task (put). We evaluate the average success rate of the method on the remaining five tasks, which are comparatively more challenging and require additional apply zero-shot prompting, omitting any examples in the
ALFWorld, AdaPlanner prompted with only
steps for completion. In MiniWoB++, skill acquisition phase. For both environments, we operate the method using GPT-3.5 in adaptive closed-loop mode, and round of skill discovery is conducted. As Figure 4d illustrates, the inclusion of skill discovery significantly enhances performance. the ALFWorld tasks, the success rate of AdaPlanner nearly doubles when skill discovery is employed. Similarly, in the MiniWoB++ tasks, the overall success rate increases by approximately 15% with skill discovery. Moreover, we conducted ablation study on MiniWoB++ evaluate the impact of sample numbers, as detailed in Table 5 AdaPlanner with skill discovery requires only 15 samples to outperform the variant without skill discovery, even though the latter used twice many samples. It is evident that skill discovery enhances sample efficiency.
# samples
38
30
With SD
92.87
84.06
79.17
75.17
60.38
Without
82.40
73.58
68.70
64.70
45.47
Table 5: Success rate (%) of AdaPlanner on MiniWob++ with and without skill discovery (SD) across varying numbers of expert samples per task. We adopt the same settings as in Table ( version) and Table 3
5 Related Work
Many works have studied how to leverage LLMs autonomous agents to accomplish decision- making tasks within text-based environments. Earlier studies, like Chain-of-TT -f-hughhs [24] and Zero-Shot Planner (5), utilize prompts to guide LLMMs in generating complete action sequences for elementary tasks. For more complex tasks, methods like HuggingGPT [18] and Chameleon [12 also generate the initial plan of using different tools and then call the corresponding APIs for execution. Meanwhile, some other works also prompt LLMs to compose plans in Planning Domain Definition Language (PDDL) 10 15). However, all these plans are created in open-loop fashion without adapting to feedback from external environments.
To address the limitations of open-loop systems, recent techniques have emerged that focus on establishing closed-loop systems. These systems are capable of leveraging environmental feedback. thereby facilitating more adaptive decision- decision-making. ReAct 25 and Inner Monologue [6] allow
LLM agents to take single-step actions according to the environmental feedback. Reflexion [19], as an extension of ReAct, tries to resolve this issue by enabling the ReAct agent to revise itself from past trials and errors. Moreover, RCI starts by formulating comprehensive plan, modifying the immediate action when the agent encounters failure at the current step. While all the aforementioned

---

methods can adapt their decisions based on environmental feedback, they assume the -LM--enerated initial plan correct and do not adjust it. Rather, they solely modify the immediate action being executed and are easy to fall into local sub-optimal actions without considering the long-term plans.
To further enhance the agents' both capabilities of planning and adapting environmental feedback, strict closed-loop architectures are proposed that can recursively refine the generated plans. DEPS [23] one of the examples that initially proposes an entire plan and then applies real-world feedback to recursively refine it during execution. However, this method requires training a selector to generate plan that is highly probable to succeed, which makes it difficult to generalize the plans and actions to other tasks. Besides, the required data for training the plan selector are often unavailable in practice and expensive to collect. In contrast, AdaPlanner generates and refines plans via LLM prompting, making it widely applicable to various secision-making problems. Another PDDL-based planning method is proposed in [21] with the ability to leverage feedback and modify the plans. However, the plan formulated on the training set remains static and lacks task-specific refinement during the execution of the evaluation task. In contrast, AdaPlanner dynamically refines the plan and adapts to various feedback throughout the task-solving process.
Conclusion and Limitations
We proposed AdaPlanner, a closed-loop approach enabling LLM agents to adaptively refine their generated plans according to enviromment feedback. We defined two different refinement strategies, in-plan and out-of-plan refinement, to fully leverage environment information. Furthermore, to mitigate the LLMs' hallucination issue and make them learn from the past experience, we proposed code-style prompting and skill discovery mechanisms. Through comprehensive experiments, we demonstrated that AdaPlanner outperforms the state-of-the-at baselines significantly and has better sample efficiency. Our ablation studies also showed the effectiveness of different components in AdaPlanner. One limitation of AdaPlanner is that it still require few-shot expert demonstrations for solving complex tasks. Although AdaPlanner has already achieved better sample efficiency than existing methods, it is interesting study how to further enhance AdaPlanner to solve complex tasks with no demonstrations in the future.
7 Broader Impacts
Our research approach focuses treating LLMs as autonomous agents and improving their ability to solve complex sequential decision-making tasks. However, this research line carries inherent risks, including security threats, potential misuse, and unintended consequences such as job displacement due to automation. To mitigate these risks, it is essential for researchers and policymakers to collabo- rate in creating and implementing effective regulations guide the development and deployment of these technologies towards positive outcomes. Additionally, we believe that the research community should coordinate efforts design principles and techniques that prioritize safety and human values before LLM agents are deployed in various industries. This will help ensure that LLMs are aligned with ethical and moral standards while promoting their positive impact on society.
Acknowledgments and Disclosure of Funding
This work was supported in part by NSF (ISS200334,, IIS-210696I, CAREER IIS-2 IIS-214338), ONR (MURI N00014-17-1-260 122566), IDEaS Cyberinfraastrucure Resources, and Microsoft Accelerate Foun- dation Models Research Program.
References
[1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1 33:1877-1901, 2020.
[2] W. Chen, X. Ma, X. Wang, and W. W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks, arXiv, page 12118.358 2022.
[3] I.. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Yang, Callan, and G. Neubig. Pal: Program- aided language models. arXiv, page 221 10435v2, 2022.
10

---

[4] I. Gur, O. Nachum, Y. Miao, M. Safdari, A. Huang, A. Chowdhery, S. Narang, N. Fiedel, and A. Faust. Understanding html with large language models. arXiv, page 2 2022.
[5] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv, page 2201.07207v,, 2022.
[6] W. Huang, F. Xia, T. Xiao, H. Chan, Liang, P. Florence, A. Zeng. J. Tompson, Mordatch, Y. Chebotar, P. Sermanet, N. Brown, T. Jackson, L. Luu, S. Levine, K. Hausman, and B. Ichter. Inner monologue: Embodied reasoning through planning with language models. arXiv,, page 2207.05608v1, 2022.
[7] C. Humphreys, D. Raposo, T. Pohlen, G. Thornton, R. Chhaparia, A. Muldal, Abramson, P. Georgiev, A. Goldin, A. Santoro, and T. Lillicrap. A data-driven approach for learning to control computers. arXivProceedings of the 39th International Conference on Machine Learning, Baltimore, Maryland, USA, PMLR 162, 2022. page 22 2022.
[8] G. Kim, P. Baldi, and S. McAleer. Language models can solve computer tasks. arXiv, page 1 2023.
[9] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. arXiv, page 2209.07753v3, 2022.
[10] B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang. J. Biswas, and P. Stone. Llm+p: Empowering large language models with optimal planning proficiency. arXiv, page 2304. 2023.
[11] E. Z. Liu, K. Guu, P. Pasupat, and P. Liang. Reinforcement learning on web interfaces using workflow-guic gugidedd exploration. In International Conference on Learning Representations, 2018.
[12] Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y. N. Wu, S.-C. Zhu, and J. Gao. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv preprint arXiv:2304.099 098442, 2023.
[13] OpenAl. Openai platform documentation: Gpt-3.5 models (retrieved on may 16, 2023).
https//plaaformmopenee wwt.:/oppeoianoooosssseeeeeeeeee htt..///////oodee///rrriiiee
[14] L. Ouyang. J. Wu, X. Jiang, D. Almeida, C. Wainwright, Mishkin, C. Zhang, Agarwal,
K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback.
Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[15] V. Pallagani, B. Muppasani, K. Murugesan, F. Rossi, L. Horesh, B. Srivastava, F. Fabiano, and
A. Loreggia. Plansformer: Generating symbolic plans using transformers, 2022.
[16] A. Parisi, Y. Zhao, and N. Fiedel. Talm: Tool augmented language models. arXiv preprint
arXiv:2205. 1255, 2022.
[17] T. Schick, J. Dwivedi- Yu, R. Dessi, R. Raileamu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and
T. Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint DXXiv:2302.047.1 2023.
[18] Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303. S1XXi::230.17588,, 2023.
[19] N. Shinn, Labash, and A. Gopinath. Reflexion: an autonomous agent with dynamic memory and self-refiection. arXiv preprint arXiv:2303.1 1366, 2023.
[20] M. Shridhar, X. Yuan, M.-A. Cote, Y. Bisk, A. Trischler, and M. Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021.
[21] T. Silver, Dan, K. Srinivas. J. B. Tenenbaum, L. Kaelbling, and M. Katz. Generalized planning in pddl domains with pretrained large language models, 2023.
[22] Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and
A. Garg. Progprompt: Generating situated robot task plans using large language models. In Second Workshop on Language and Reinforcement Learning. 2022.
[23] Z. Wang. S. Cai, A. Liu, X. Ma, and Y. Liang. Deseribe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv, page 2302.01560v 2023.
[24] J. Wei, X. Wang. D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E, Chi, Q. Le, and D. Zhou. Chain- of-thought prompting elicits reasoning in large language models. arXiv, page 2201. 11903v6 2022.
11

---

[25] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference Learning Representations, 2023.
[26] J. Ye, X. Chen, N. Xu, C. Zu, Z. Shao, S. Liu, Y. Cui, Z. Zhou, C. Gong, Y. Shen, J. Zhou, S. Chen, T. Gui, Q. Zhang, and X. Huang. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models, 2023.
[27] D. Zhou, N. Schärli, Hou, J. Wei, N. Scales, X. Wang. D. Schuurmans, C. Cui, O. Bousquet,
Q. Le, and E. Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv, page 2205.10625V. 1062v22 2022.
12

---

A Experimental Setup
ALFWorld 20] is a comprehensive suite of synthetic, text-based environments, encompassing six distinct task types set, Pick, Clean, Heat, Cool, Examine, and Pick two, within a virtual household. Each task possesses a unique high-level objective (e.g.. put some vase in safe, etc.) that necessitates agent navigation and interaction with various objects shelf 6, clean apple, etc.). To fulfill the stipulated task, the agent is required to implement a sequence of actions aimed accomplishing the predetermined goal. However, given that an object may potentially reside in any one of over 50 possible locations in a task instance, the agent must sequentially explore each of these. Consequently, the entire action trajectory could involve more than 50 individual actions, presenting a significant challenge to the agent.
receptacles (e.g.. go to
MiniWoB++ M is task suite of simulation environments that covers a large range of com- puter tasks for net agents. The computer tasks start from simple button-clicking to more challenging ones with longer time horizons (e.g., click- www.c--hhhcbooxe--rrreee
reasoning (e.g., wog.n-osee--ooooeeeeeeeoeeeee -ue---uuuur-ppppe ), and stochastically multi-layouts), These challenges are suitable for
seees-hhhccooeeesoftbbee unexpected pop-ups (e.g.. login-
varying layouts (e.g., multi-orderingg evaluating our proposed closed-loop framework. Each task interacts with a 160px x 210px web environment, where the state space is purely the HTML code of the web. Following RCI [8. we define the actions space as two sets of operations, i.e., clicking and typing ac- tions. The clicks allow the agent to interact with clickable HTML elements (e.g.. webpage but- tons). The typings are conducted with keyboard-based actions, such as inputting characters into the input box and stroking functional keys (e.g., ENTER, BACKSPACE). We select nine Mini- WoB++ tasks where environment observations (i.e., the HTML code) change after certain actions: search-engine, tic-tac-
guess-numben number, email-inbox. email-inbox forbbr--rrrarddnl. and email-inbox forward -nl-turk. Take the task terminal as an example. Search results appear on the wehpage after inputting the keyword and pressing the search button. Therefore, environment feedback can be interpreted from the change HTML code and further leveraged by closed-loop planning. We also adopt and test the 53 tasks evaluated in RCI (8).
omail-inbox-nl toe, terminal, logen
B Baseline Details
ALFWorld. Following set of previous works 20 25 19]. we evaluate AdaPlanner on 134 different environments. By default, we include one sample as an exemplar per task to prompt AdaPlanner. For the study presented in Figure 44a we adopt the setting of prompted samples as in Table 6 For the study in Figure 4d we use one sample of the simplest task put to prompt the rest of the five tasks, which are more complex and require more steps to solve. For baselines, we compare AdaPlanner with BUTLER (0). ReAct [25). and Reflexion [1). BUTLER (20 is an imitation learning method trained with 100k samples per task. ReAct and Reflexion, as outlined in Table [, are prompting- based methodologies utilizing an implicit closed- framework. They employ a total of 6 and 8 samples, respectively, across all six tasks. BUTLER results are sourced from [20]. We evaluate ReAct, Reflexion, and AdaPlanner empowered by both GPT-3 (text-davinci-002 and GPT-3.5 (gpt-3.5 101-3.5-ture tururb and -ext-davvici-003) models.
# samples
Pick
Clean
Heat
Cool
Examine
Pick two
2
Clean Clean
Clean
Clean
Examine
Examine
4
Pick Clean
Clean
Clean
Examine
Pick two
6
Pick
Clean
Heat
Cool
Examine
Pick two
Table 6: The specific allocation of samples for prompting each task is divided into three cases based on the total number of samples (2, 4, and 6) used across the six types of tasks. For instance, when total of 2 samples utilized to prompt four tasks (Pick, Clean, Heat, and Cool). Similarly, a sample from the Examine task is used to prompt two tasks (Examine and Pick two).
used for all tasks, single expert trajectory sample for the Clean task is
Min. Overall, we report the evaluation results of RCI ( and the proposed AdaPlanner in GPT-3.5 court-davvncii-O00 -003), along with three training or finetuning- bassud baselines: Computer
13

---

Control Agent Architecture(CC-Nel) 77]. Workflow-Guider Exploration (WGE) [ and WebN-TS- 3B [4. CC-Net and WGE employ supervised learning and reinforcement learning with over 23K and 10 demonstrations per task, respectively. WebN-T5-3B uses 12K demonstrations to finetune pre-trained language model. RCI a prompting-basee approach that is categorized as the implicit closed-loop method in Table which utilizes 93 samples across the 53 tasks. For these 53 tasks, we first provide AdaPlanner with 38 numan-written demonstrations and perform skill discovery to obtain another 21 additional examples, i.e., 59 examples are used for 53 tasks, Evaluation results of RCI, respectively.
CC-Net, WGE, and WebN-T5-3B are sourced from the works of 8
C Proportion of two refinements
Environment
In-plan refinement Out-of-plan Proportion (in/out)
ALFWorld(134 tasks)
2.83
6.40
0.44
MiniWoB ++(9 tasks with feedlbuck)
0.33
0.19
Table 7: Average number and ratio of in- plan and out-of plan refinement invoked per task.
Table illustrates that out-of-plan refinement invoked more frequently in both environments, emphasizing its important role in AdaPlanner's planning process. The average occurrence for both types of refinements varies between ALFWorld and MiniWoB++. This be attributed to the differences in the average lengths of the trajectories in ALFWorld and MiniWoB++ (15.60 and 5.40 steps, respectively).
D Prompts
D.I ALFWorld
Basic Information. The <basic_ Obasic_info> defines the agent and admissible actions for AdaPlanner. Note that the actual definitions of action functions are not specified in the prompt. Instead, only formalized definition with several examples is provided, such that the planner acquire how to compose a plan based on these actions. As can be seen in the following part, this <basic_info> prompt used in both <initial _aaiiiing and <refinement> prompts.
<basic_info> Prompt
# You are a household agent. Here is some Python code defining a household environment:
literal_eval convert the from ast import literal_eval
ask() list.
# In the environment, you can ask questions to assistant by ask(): from large_ large__lnnuage_moe _model import ask_llm as ask # example: You have list of receptacles, and you want to sort them by the likelihood of a soapbar appearing in them. You can do this by asking the assistant:
receptacles ['countertop 1', 'garbagecan 1', 'sinkbasin 2', siinbbssin 1',
'toilet 1', ttoiletpaperhanger 1', 'towelholder 1']
answer ask Sort the list receptacles, starting from the one a soapbar is most likely to appear: (receptacles). You should return a Python list.') # answer ('sinkbasin 1', "sinkbasin 2', 'countertop 1', 'towelholder 1', 'toiletpaperhanger 1', 'garbagecan 1', 'toilet 1']
# Agent class represents the state of the agent, including its location, # what it's holding as well as the actions it can take. class Agent:
def __init__ receptaclea):
self locatici None
self holding None
self receptacce - receptacles
14

---

# Here are the admissible actions the agent can take:
# Go receptacle
update the agent's location.
# For example, 'On the countertop 1, you see candle 1, a cloth 2, and soapbar 1. goto (' c'cuuntertopp 1')
# For example, 'On the sidetable 2, you nothing. goto ('sidetable 2 def (self,, receptacle):
# Take object from receptacle if the agent is not holding anything. # For example, 'You pick up the soapbar 1 from the towelholder 1." take ('soapbar 1', 'towelholder 1') def (self, object, receptacle):
receptacle the agent is holding # For example, 'You put the soapbar in/on the cabinet 1. put ('soapbar 1', 'cabinet 1') def (self, object, receptacle):
# Put object in
# Open receptacle and observe its contents. # For example, 'You open the cabinet 1. The cabinet 1 is open. In it, you cloth 1. receptacc ("abbntt cabinet 1')
def open peepprecettall receptacle):
# Clean an object with a receptacle.
# For example, 'You clean the soapbar 1 using the sinkbasin 1.' clean('sa s'soppaar 1', 'sinkbasin 1') clean,self, object, receptacle):
# Heat object with receptacle.
# example, 'You heat tomato using microwave 1. heat tomato 1', 'microwave 1') def heat, object, receptacle):
# Cool an object with receptacle.
# For example, 'You cool the pan 2 using the fridge 1. cool('po 2',
'ridge 1') def cool (self, object, receptacle):
# Turn on an object.
# For example, 'You turn the desklamp 1. turn_on ('desklamp 1') def (aelf, object):
# Report agent's 's current state, including its location, what it's holding, and last action and observation. # This function should only be used assertion. (self):
def (self:
Initial Planning. The <initial.planning prompt is employed to generate the preliminary plan. bhasic_ino>>
In this context, <basic_ info> is substituted by the content of the <basic_info> prompt. The <sample> is replaced with expert trajectory, while <receptacle, list> is substituted by the list of interactive receptacles provided by the task environment. Finally, <task> is substituted by the task description, expressed in natural language.
15

---

<initial_plaming> Prompt
<basic_info>
# Now complete the function solution() below solve the by composing the agent's methods to interact with the environment.
# For each step you plan to take, 1) mark with [Step xx]', 2) give a reason why you think it is good step to take 3) write an assertion to check if the step is successful.
# Here is an example of solution to the task:
<sample>
# Here the actual task.
# define environment and agent receptacles <receptacle_listt agent = Agent (receptacles)
# <task>
# You should complete your solution function below:
def solution start_from=1);
Samples. In ALFWorld, there six types of tasks: Pick, Clean, Heat, Cool, Examine, and Pick two. For each type, we gather one expert sample of solutions that the planner can refer to. These six expert samples presented follows:
The expert sample for the task Pick:
<sample_pin Prompt
# define environment and agent receptacles = ('diningtable 1', 'drawer 2', 'drawer 1', 'sinkbasin 1', 'toilet 1', "sidetable 2', 'sidetable 1', 'cabinet 1', 'countertop 1', 'microwave 1', 'fridge agent (recepatap)))
# Your task is to: put soapbar countertop.
# here is solution:
solution(agent, start_from-l)
# General Plan: need to get a list of receptacles where the soapbar is likely to appear, and then go to each receptacle in the list until seeing soapbar. Then can put get the identifier of the soapbar and take it. Finally can go the countertop and put the soapbar. start_from print("(Ste) 1] get list of receptacles where the soapbar is likely to appear")
# I ask the assistant do that.
answer ask(f Given a list receptacles, please sort descending order based on the likelihood of finding a soapbar in each of them. The list of receptacles is: regepttxecept.clee)) You should directly return Python list.') recep_ to tt_ttt_cchhheee check expectation: returned recep_to_chesd should not be empty. assert vecep_to_check, f EEError in [Step 1]: recep_to_ checkkk should not be empty. (agent.report
aideral_evalaneeet
start_from
("[nep 2] go to each receptacle in the list until seeing a soapbar")
for receptacle in recep_to_check:
observation agent. (receptacle)
check if receptacle closed. so, open it.
if 'closed' in observation:
observation agent, receptacee (receptacle)
check if soapbar is in/on the receptacle.
16

---

if 'soapbar' in observation:
break
expectation: I should able find a receptacle where a soapbar is in/on it. assert 'soapbar' in observation, f f'Error in [Step 2] There is no soapbar in/on Precep_to_checki (agent.report())
Start_from.
print(" (Step 3] identify the soapbar juts found and take it") # I need to get the identifier of the soapbar. I can ask the assistant to do that.
answer ask From the observation, get the identifier of an object. For exemple, the cabinet 1, you see cloth 2, and a toiletpaper 2. The identifier of cloth is 2. Now, (observation) The identifier of the soap? Only Output single number without any other words. soapd_s f'soapbar (answer)'
observation agent. take (found_soapbar, receptacle)
expectation: I should be able to take the aoapbar from the receptacle. assert agent.holding =- found_soapbar f' E'Error in [Step 3] I cannot take (found_soapbar) from the (receptacle). (agent.reeoot()))
start_ start_from from
print ("[Step 4] go to countertop and put the soapbar on it") # There are multiple countertops, and I only need to go to of them. observation aggnt.ggtann "ooutotoottco 1')
check if the countertop closed. If so, open it.
if 'closed' in observation:
observation agent agent.open__t cetettppee__eeeeee repee_eeceptacce( reccppacleeccommtetooo 11)
observation agent-put puttt.putfoouuue (found_soapbar, 'countertop 1')
expectation: I should be able put the soapbar on countertop. assert Yoo put the soaud_soapbar) in/on the countertop 1. in observation, f ErEror in [Step 4] cannot put the (found_soapbar) on the countertop 1.
The expert sample for the task Clean:
<sample_ Prompt
# define environment and agent receptacles ('diningtable 1', 'drawer 2', 'drawer 1', 'sinkbasin 1', 'toilet 1', 'sidetable 2', 'sidetable 1', 'cabinet 1', 'countertop 1', 'microwave 1', 'fridge 1'] agent Agent (receptacles)
# Your task is put clean lettuce in diningtable / clean lettuce and put diningtable.
# here a solution:
def solution (agent, start startffrom=1):
# General plan: need to get list of receptacles to find the lettuce, take the lettuce the sinkbasin, clean it and put it diningtable.
start_from 1] get list of receptacles where the lettuce is likely to print(" appear.")
# I ask the assistant do that.
answer ask(f Given a list receptacles, please sort them descending order based on the likelihood finding a lettuce in each of them. The list of receptacles is: Sagent..rceptaccie You should directly return Python list.') recep_to_chest check expectation: returned recep_to_ches should not be empty. assert tecep_to_check, f 'EError in [Step 1]: recep_to, theck should not be empty.
exterallevallansvee
start_from ( [Step go to each receptacle the list until seeing a lettuce")
17

---

for receptacle in recep_to_check:
observation agent. (receptacle)
check if receptacle closed. If so, open it.
if 'closed' in observation:
observation agent. agent.open_ open__eceetacleeceoo (receptacle) check if lettuce is in/on the receptacle.
if 'lettuce' in observation: break
# expectation: I should be able to find a receptacle where a lettuce is in/on it. assert 'lettuce' in observation, f'Error in [Step 2] There is lettuce in/on trecep_to_cheece ragent.
if start_from <= 3:
print [SSee 3] identify the lettuce juts found and take it")
need get the identifier of the lettuce. to that.
ask the assistant
answer = ask(f From the observation, get the identifier of an object. For example, On the cabinet 1, you see cloth 2, and toiletpaper 2. The identifier of cloth is 2. Now, (observation) The identifier of the lettuce? Only Output a single number without any other words. found_ lettuce f'lettuce {answer)'
observation agent. take foouddlettuuee receptacle)
# expectation: I should be able take the lettuce from the receptacle. assert agent.holding take (found_lettuce) from the (receptacle). ragent.report())
found_lettuce, f' Error in [Step 3] cannot
if start_froe <= 4:
print ("teep 4] go sinkbasin clean the lettuce. ") should go to sinkbasin first if I ceent.goto('sinneee # check if the sinkbasin is closed. If so, open it.
clean the lettuce.
observation
1')
if 'closed' in observation:
observation agent. recoa__rcppaaclee'seeeeao observation agent. agent.clean (foundleetuue, 'sinkbasin expectation: I should be able to clean the lettuce. assert Yoo clean the lettuce] using the sinkbasin 1. in observation, f'Eroor in [Step 4] I cannot clean the [found_ [etttuce] using the sinkbasin 1. (agent.report()) I should have been sinkbasin 1 and holding [ooudd_eetuce))
1')
if start_froa <= 5:
print (" 5] go diningtable and put the lettuce on it. There are multiple diningtables, and only need go one of them. observation agent.goton ('dnniniingttb 1') # check if the diningtable is closed. If so, open it.
if 'closed' in observation:
observation agent..penn_reeeeeeec open__eceptacce(( d'dinngtabbe 1')
observation agent.ppu (found_lettuce 'diningtable 1') expectation: I should be able to put the lettuce on the diningtable. assert You put the observation, f' Erroo in [Step 5]: cannot put the [ettud_) the diningtable 1. (agent.report())
in/on the diningtable 1. in
The expert sample for the task Heat:
<sample_heat> Prompt
# define environment and agent
receptacles ['diningtable 1', 'dawwer 2', 'drawer 1', 'sinkbasin 1', 'toilet 1', "sidetable 2', 'sidetable 1', 'cabinet 1', 'countertop 1', 'microwave 1', 'fridge 1']
agent = Agent (receptacles)
Your task is to: put a hot lettuce in diningtable / heat some lettuce and put it in diningtable.
18

---

# here is solution:
def solution(agent, start_fad :aar_frem=1);
# General plan: need to get list of receptacles to find the lettuce, take the lettuce to the microwave, heat it and put it in a diningtable. if start_from <= 1:
print("(Step 1] get list of receptacles where the lettuce is likely appear.")
can ask the assistant to do that.
answer = ask(f Given a list of receptacles, please sort them in descending order based on the likelihood of finding a lettuce in each of them. The list of receptacles is: recepal.ecepptalees).. You should directly return Python list.') recep_to, st_check literal_evaal (answer)
# expectation: the returned recep_to_cod th_ckeck should not be empty. assert pacep_to_checte f f'Error in [Step 1]: recep_to to_p_t__chie check should not be empty. fagent.report
if start_ from <= 2:
("int("(Step 2] go to each receptacle in the list until seeing a lettuce")
for receptacle recep_to_
observation agent agent.goto (receptacle)
# check if the receptacle is closed. If ao, open it.
if 'closed' in observation:
observation agent.open_ open_receptacceeeeoo (receptacle)
check if a lettuce is in/on the receptacle. if 'lettuce' in observation: break
# expectation: I should be able find receptacle where lettuce is in/on
assert 'lettuce' in observation, f'Error [Step 2] There is no (epenttr)port())-
lettuce in/on trecep_to_checkk,
if start_from <- 3:
print("
identify the lettuce juts found
it")
I need to get the identifier of the lettuce. can ask the assistant to do that.
answer ask From the observation, get the identifier of object. For example, On the cabinet 1, you The identifier cloth 2. Now, {observation) The identifier of the lettuce? Only Output a single number without any other words. 1) f' f'lettuce (answer)'
cloth 2, and toiletpaper 2.
observation agent. ageet.take (foudd__eeeet receptacle) expectation: I should able take lettuce assert
receptacle. found_lettuce, f'Error in [Step 3] cannot take (found_lettuce from the (receptacle). (agent.reporten
if caartffrom <=
("int("SStep go to microwave to heat the lettuce") I should go to a microwave to heat the lettuce. observation agent. gottt.goo((() ('microuave 1')
check if the microwave is closed. If so, open it. if 'closed' in observation:
observation agent. oeweoopeeeeeeeeeee wwweppaccee observation agent. heatt (found_lettucee 'microvave 1') # expectation: I should be able to heat the lettuce. assert f You heat the (round_lettucee observation, f E'Error in [Step 4] I cannot heat the [found_lettuce) using the microwave 1. 1 and holding (found_lettuce,,
1
using the microwave 1. in
I should have been at microwave
if start_from <=
( [Step go to diningtable put the lettuce on it") # There are multiple diningtables, and only need to go to one of them. observation agent. goto diningtable 1')
19

---

# check if the diningtable is closed. If open it.
if 'closed' in observation:
observation agent. owt._peeeeetacleeee..ee d'diingttablee 1!)
observation agent. (found_lettuce, 'diningtable 1')
# expectation: I should be able to put the lettuce on the diningtable. assert I' You put the (found_lettuce) in/on the diningtable 1.' in observation, f' Error in [Step 5] cannot put the the diningtable 1. ragent.repoot().
The expert sample for the task Cool:
<sample_cool> Prompt
# define environment and agent receptacles ['diningtable 1', 'drawer 2', 'drawer 1', 'sinkbasin 1', 'toilet 1', "sidetable 2', 'sidetable 1', 'cabinet 1', 'countertop 1', 'microwave 'fridge 1']
agent Agent (receptacles)
# Your task is put cold lettuce diningtable cool lettuce and put diningtable.
# here is a solution:
def colutionaagen' start_frow ftom-_from-1);
# General plan: need get list of receptacles to find the lettuce, take the lettuce to the fridge, cool it put diningtable. start_from print(" 1] get list of receptacles where the lettuce is likely to appear.")
I ask the assistant do that.
answer ask(f Given a list receptacles, please sort them descending order based on the likelihood of finding a lettuce in each of them. The list of receptacles is: (agent. receptalecc. You should directly return Python list.') recep_
tecep_to_cco tt__tt_cccheeee check literal_evall (ansuer) expectation:
returned recep_to_chee should not be empty. assert recep_to_ce tesppttt_ceck,, f f'Error in [Step 1]: sacep_to_cheat should not be empty. (agent. regert..eport()
start_from, print (" [Sttep 2] go to each receptacle in the list until seeing a lettuce")
for receptacle in recep_to_cheet
observation check if receptacle closed. If so, open it.
(receptacle)
if 'closed' in observation:
observation agent. rece__ccee (receptacle)
check if lettuce is in/on the receptacle. if 'lettuce' observation:
break
# expectation: I should be able to find a receptacle where a lettuce is in/on it. 'lettuce' in observation, f'Error in [Step 2] There is lettuce in/on precep_to_checkk (agent.
if start_from <= 3:
print ("antt(""sa [Step 3] identify the lettuce juts found and take it") assistant
need get the identifier of lettuce. can to do that.
answer = ask(f From the observation, get the identifier of an object. For example, On the cabinet 1, you see cloth 2, and toiletpaper 2. The identifier cloth is 2. Now, {observation} The identifier of the lettuce? Only Output a single number without any other words. found_lettuce f'lettuce (answer)"
observation agent. take lettuce, receptacle) expectation: I should be able take the lettuce from the receptacle,
20

---

assert agent. holding == found_lettuce, f' E'Error in [Step 3]: I cannot take [etnu_]ettucc) from the (receptacle). (agent.report ()).
if start_froet <- 4:
print (" ("ttep 4] go to fridge to cool the lettuce") # I should go to a fridge cool the lettuce.
observation agent. gontttoooo 1')
check if the fridge closed. If so, open it. if 'closed' in observation:
observation agent. wawp__rrrppacceereee....
1')
observation agent. cool lettuc,, 'fridge 1') expectation: I should be able cool the lettuce. assert You cool the observation, f'Error in [Step 4] I cannot cool the (found lettuce) using the fridge 1. Bagent.reporte. I should have been at fridge and holding [ettuc_e
using the fridge 1.
if start_froe <= 5:
print(" 5] go to diningtable and put the lettuce on it") # There multiple diningtables, and only need to go of them. observation agent.goto g'innnnniingttble 1') check if the diningtable is closed. so, open it.
if 'closed' in observation:
observation agent. rpen_receptacle( d'diingtabbe 11)
observation agent. age (found_ 'diningtable 1') expectation: I should assert YoYou put the (found_lettuce) in/on the diningtable 1. in observation, f E'Error in [Step 5] I cannot put the (foun__lettuce) on the diningtable 1. (agent.report())
able to put the lettuce on the diningtable.
The expert sample for the task Examine:
<eample_examine> examine> Prompt
# define environment and agent receptacles = l'diningtable 1', drawer 2', 'drawer 1', 'sinkbasin 1', 'toilet 1', "sidetable 2', 'sidetable 1', 'cabinet 1', 'countertop 1', 'microwave 1', 'fridge 1']
agent Agent (receptacles)
# Your task is to: look at the bowl under the desklamp examine the bowl with the desklamp
# here solution:
polutionaagent, start_from=1).
# General plan: I need to get list of receptacles to find the bowl and take the bovl with me, then get another list of receptacles to find the desklamp and turn it
start_from
(rint("(Ste)) 1] get list of receptacles where a bowl is likely to appear.")
# I can ask the assistant to do that.
Givenn list receptacles, please them descending order based on the likelihood finding a boul each them. The list of receptacles is: regetttrecappaalees). You should directly return Python list.') recep_to, teoe__to_cheee check = literal_ever (answer)
expectation: returned recep_to_ teop__t__ceck should be empty. assert empty.
f 'Error in [Step 1]: recep_to_che should not be
if start_ <=
2] go to each receptacle the list until seeing a pen") for receptacle in recep_to_check: observation agent.goto (receptacle)
check if the receptacle is closed. If so, open it.
21

---

if 'closed' in observation:
observation agent. rec_tteeee (receptacle) check if a bowl in/on the receptacle. if 'pen' in observation:
break
# expectation: I should be able find receptacle where bowl is in/on it.
assert 'pen' observation, f'Error in [Step 2]: There is no bowl in/on trecck. (agent.report(() report()))
if start_from <=
print(" take bowl from the receptacle")
need to get the identifier of the bowl so that can take it. can ask the assistant to do that.
answer = ask(f From the observation, get the identifier of object. For example, On the cabinet 1, you The identifier pen? Only Output a single number without any other words. !) found_pen t'pen (answer)'
cloth 2, and toiletpaper 2. cloth 2. Now, {observation) The identifier of the
observation agent. take expectation: I should
receptacle)
take boul from the receptacle. found_pen, f'Error in [Step 3] cannot take
assert agent.holding [found_pen) from the (receptacle). (agent.xeport())
if start_frod <= 4:
print("(Step get list of receptacles where a desklamp likely to appear.")
# I can ask the assistant to do that.
answer = ask(f Given list of receptacles, please sort them in descending order based on the likelihood of finding desklamp each them. The list of receptacles is: ragent..eccetacless) You should directly return Python list.')
recep_to. to__ttt__cecc = aageral_evallaseete # expectation: the returned recep_to_check should be empty.
assert recep_to_check f EError in [Step 4] recep_to_cheet should not empty.
if start from <= 5:
print (" ("teep go each receptacle in the list until seeing desklamp"))
for receptacle in recep_to_chall theckkkkk
observation agent. goto (receptacle)
# check if the receptacle is closed. If open it. if 'closed' observation:
observation agent.open_recepee reen_receptalle (receptacc)))))
# check if a desklamp is in/on the receptacle.
if 'desklamp' in observation: break
expectation: I should be able find a receptacle where a desklamp in/ on it.
assert 'desklamp' in observation, f'Error in [Step 5] There is no desklamp in/on saacep_to_checkke
start_from. print (" (Steep 6] turn on desklamp")
# There might be multiple desklamps in the environment, and I need to get the identifier of the desklamp. I ask the assistant to that. answer ask(f From the observation, get the identifier of an object. For example, On the cabinet 1, you see a cloth 2, and a toiletpaper 2. The identifier of cloth is 2. Now, (observation) The identifier of the desklamp? Only Output single number without any other words.") found_deaklamp
deaklamp (anawer).
can directly turn on the desklamp that just found. observation agent. turn_on (found, desklamp)
22

---

# expectation: the desklamp should be turned now. assert 'turn on' in observation, Error in [Step 61 cannot desklampp in/on (receptacle). (agent.report
The expert sample for the task Pick two:
<sample_picktvo Prompt
# define environment and agent receptacles = l'diningtable 1', 'drawer 2', 'drawer 1', 'sinkbasin 1', 'toilet 1', "sidetable 2', 'aidetable 1', 'cabinet 1', 'countertop 1', 'microwave 1', 'fridge 1']
agent Agent (receptacles)
# Your task is put cellphone in cabinet / find two cellphone and put them cabinet
# here is a solution:
def solution (agent, start_from-1)
if stom <= 1:
(rint("[Step get list of receptacles where cellphone is likely appear.")
# I can aak the assistant to do that.
answer - ask Given a list of receptacles, please sort them in descending order based the likelihood of finding cellphone in each of them. The list receptacles is: recett.receptacles). You should directly return Python list.')
recep_to to_ check = literal_eww (answer) # remove the destination from the list recep_to_com wwo__check.reeoo ('cabinet 1') expectation: returned recep_to_chee should not be empty. assert cerp__to_chek, f' E'Error in [Step 1]: recep_to_ tece__toccheck should not be empty. Eagent.reporte
start_from print(" ("[Seep go to each receptacle the list until seeing a cellphome")
for receptacle in recep_to_check:
observation agent (receptacle)
check if receptacle closed. If so, open it. if 'closed' in observation:
observation agent. agent. recepnacce (receppacc))) # check if cellphone is in/on the receptacle.
if "cellphone' in observation: break
# expectation: I should be able to find a receptacle where a cellphone is in/on it. assert 'elllhhone' in observation, f'Error in [Step 2] There is cellphone in/on trecep_to_chest) (agent.
if start_from <= 3:
print (" [Step 3] identify the first cellphone found and take it")
need get the identifier of the cellphone. to that.
ask the assistant
answer = ask(f From the observation, get the identifier of an object. For example, On the cabinet 1, you see cloth 2, and toiletpaper 2. The identifier of cloth is 2. Now, (observation). The identifier of the cellphone? Only Output a single number without any other words. ') celpdd_ne f' cellphone (answer)'
observation agent. eyent.take take louudd_cellphone receptacls) # expectation: I should be able to take the cellphone from the receptacle.
assert agent holding
celdd_oel,,, f'Error in [Step 3]: cannot take (found_cellphone)) from the (receptacle). (agent.report())
if ftom <= 4:
23

---

print (Step 4] go to cabinet and put the first cellphone found it.
There are multiple countertops, and I only need to go to one them. observation agent. goto cabinet 1')
# check if the cabinet is closed. If so, open it.
if 'closed' in observation:
observation agent. optoopppeeeeeeeeeeeeeee rpen__eccetaalee cabinet 1
observation agent. (found_cellppp)pp celphhnee,,, 'cabinet 1 # expectation: I should be able to put the cellphonei on the countertop. assert You put the (found_cell) celnd__ell)hoee) in/on the cabinet 1, in observation, f' E'Error in [Step 4] I cannot put the (found_celllphoee) the cabinet 1. ragent.report
if start_from <= 5:
print (" 5] go each of the remaining receptacle in the list until seeing second cellphone")
for receptacle recep_to_ches
observation agent goto (receptacle)
# check if the receptacle is closed. If so, open it.
if 'closed' in observation:
observation check if a cellphone is in/on the receptacle.
if 'cellphone' in observation: break
# expectation: I should be able find receptacle where cellphone is in/on it.
assert 'cellphone' in observation, f'Error in [Step 5] There is no second cellphone in/on {recep_to, to checkk).. (epentt)))or(())
start_from
print ("ttep 6] identify the second cellphone I just found and take it") # need to get the identifier of the cellphone. can ask the assistant to do that.
answer ask(f From the observation, get the identifier of object. example, the cabinet 1, you see cloth 2, and a toiletpaper 2. The identifier of cloth is 2. Now, (observation). The identifier of the cellphone? Only Output a single number without any other words. 1) found_ cellphone2 = delllhonee (answer)'
observation agent. take foondd_cellponee receptacle) expectation: should receptacle.
take the cellphone from the
assert agent. holddig == found_ celppoone, f'Error in [Step 6] I cannot take (found_cellphone2) from the (receptacle). (agent.report())
if start_froe <= 7: ("int("(Ste)) 7] go to cabinet and put the second cellphone found on it")
observation agent. ggent.goto( cabinet 1')
observation agent put put (found_ celpphone, 'cabinet 1')
expectation: I should be able to put the cellphone2 on the countertop. assert You put the (found_cell cellphone]] in/on the cabinet 1. in observation, f' E'Eroor in [Step 7] I cannot put the (founn_cellphonee) the cabinet 1. (agent.report
Code Check. After plan generation, we employ the following prompt to verify and rectify any syntax errors. The placeholder <solution fonc_ucccc___ is replaced by the generated solution function. The <heck> prompt prompts the model to return two questions. If the response to Question is Yes, the answer to Question is adopted as the corrected solution function. Otherwise, the solution function is kept unchanged.
<code_check> Prompt
You are given a Python code snippet to define a function called solution.
[Code]
24

---

<solution_func>
Question Are there any syntax errors present the code? Answer Yes/No. Question 2: Fix the syntax errors and output an error-free version the code. Only Output the revised code after [Revised code] without any other words.
Out-of-Plan Refinement. In the event of an assertion error, we use <refinement> to conduct the out-of-plan refinement. In this prompt, <basic_infol is replaced by the content of the <basic_ing info> prompt. The placeholder <sample> is substituted with an expert trajectory, while <receptacle_ sespptttlleelst>>>> is replaced by the list of interactive receptacles provided by the task environ- ment. <task> is replaced by the task description in natural language. Finally, <error_ __soo> is replaced by the assertion error message returned by the solution function. To adhere to the context length limit of the GPT-3/3.5 models, the previously generated solution function is not included in this prompt. Instead, incorporate comprehensive information in the assertion error message, enabling the refiner to generate a revised plan based on these details.
<refinement> Prompt
<basic_info>
# Here is example of successful solution for solving similar task: (Successful example] receptacles ['diningtable 1', 'drawer 2', 'drawer 1', 'sinkbasin 1', 'toilet 1', siidetable 2', 'sidetable 1', 'cabinet 1', 'countertop 1', 'microwave 1', 'fridge 1'] agent Agent (receptacles) <sample>
# Here is the actual task. # define environment and agent receptacles <receptacle_list> agent (xeeepaecc))))ees))
# <task>
You have generated code of solution() () solve the task. However, you executed solution() function and get an error message: <error_msg>
Let's think step step. Referring to the successful case and the error message, you should complete the solution function with the correct code. solution solutionaagent, start_from=1)-
Determining start_from After formulating a revised plan, we utilize the following prompt to ascertain from which step the new solution function should commence. In this con- text, the sprevious_solut prevised_solutionn is replaced by the updated one. Subsequently, the argument start_from=1 substituted with the step number that this prompt yields.
is replaced by the preceding solution function, while the
Estart_from from> Prompt
Previcusly, you generated
code defining solution function in [Previous solution]. The previous code is executed and outputs some error. Now you just revised the code as in [Revised solution]. Determine from which step these two version differs. You should only output the step number without saying any other words.
[Previous solution] <previous_solution
[Revised solution] <revised_solution>
25

---

3 6. You can ask the D.2 MiniWoB++
Basic Information. Similar the ALFWorld tasks, the abasic_infie MiniWoB++ defines the agent and admissible actions for AdaPlanner. Note that the actual definitions of action functions are not specified in the prompt. Instead, only formalized definition with several examples is provided, such that the planner can acquire how to compose a plan based on these actions. As can be seen the following part, this <basic_infor prompt used both <initial Sanitial_planinng planning? and <ref inement> prompts.
<basic_info>> Prompt
# Interact with the HTML webpage to finish the computer task. Here is some Python code defining computer task environment:
the environment, you can ask questions an assistant by ask() from large_ language Language_model model import ask_llm as ask # for example: You want to solve algebra problem assistant to solve it directly for you. answer ask ('Solve an algebra problem. You should directly output the value the unknown. For example, solve yy + 2' -> 1, Now solve x 3 - 6' ->!) # anawer '3'
# Agent class represents can take.
of the HTML webpage and the actions
class Agent: def (self, initial_state).
self.current_state = initial_stata
# Here are the admissible actions the agent can take: # Action: type a string into the input box # this function returns the string of the HTML code after taking the action # e.g., rrw_html_state agent. type ("hello") type(self, characters: str) str:
# Action: press key the keyboard, the input be of the following: # enter, space, arrow_left, arrow_right, arrov_up, arrow_dowe down, backspace # this function returns the string of the HTML code after taking the action "ee.kkeeeee"""
# e.g., new new_html_sae state = def prels_ key: str) -> str:
# Action: click a <select> element in a list with an XPath
# this function returns the string of the HTML code after taking the action # e.g., new www_html_state def click_optionnselff, opath_of_option str) str:
(@idd'ccrr [)option(1)")
# Action: click an HTML element with its XPath # this function returns the string of the HIML code after taking the action # e.g., new new_html_state
(aiiikkppath"//buuuuui [0id= buttoni '] def click_ xiik_xpath(seff,, xpath_of of__eement: str) -> str:
# Action: the cursor HTML element with XPath
# this function returns the string of the HIML code after taking the action # e.g., new new_html_state = agent. catree_ouse_ore hotp///uttto [Gid=' bbttonn' "] def move_mouse_on on (self, xpath_of _el_eennn:: str) >> str:
prompt is employed to generate the preliminary plan. In this context, <basic_inff info> is substituted by the content of the <basic <basic_info> prompt. The <sample> is replaced with an expert trajectory (or discarded in the skill discovery stage). The cinitial_statee state> is substituted by the initial HTML code of the task UI. Finally, <task> substituted by the task description, expressed in natural language.
Initial Planning, The <initial_plamnen
26

---

<initial_plaming> Prompt
<basic_info>
<sample>
# Now complete the function solution() below to solve the task by composing the agent' 's methods to interact with the environment.
# In the solution function, start with commented "# General plan: " For each step you plan to take, mark with [Step xx] write an assertion check the step is successful.
# Here is the actual task. # define environment and agent. The state the list of HTML elements the webpage.
initial_state \"
<initial_state>
\
agent Agent (Antitial__state
# Task: <task>
# Here is the solution:
def solution(agent: start_ start_from=1);
Samples. In MiniWoB++,, we select nine tasks where feedback is available. Here are the expert samples that we gathered for these tasks.
The expert sample for the task email-inbox:
<sample_ setilleemmallee www.e-ealllli.ee inbox> Prompt
# Here are three examples of solutions.
# Task: Find the email by Brittani and reply to them with the text "Aliquet. sollicitudin
lectus.".
solution (agent, start_from-l)
# General plan: should first click on the email by Brittani, then click on the zeply button, then type the text "Aliquet. Sollicitudin nam lectus." and finally click the send button.
if start_from <=
print(' [Step click on the email by Brittani') agent. attittraftttnntttacttinnn state_after clent.click_xppah ("//span [@class=' "eclaasssemmmi--eeppy1111
(etr:caassssmmmmlseeeeeeee email-sender' and text BrritBani
the reply content should be displayed on page.
assert 'reply-text' correctly. The reply button is not displayed on the page."
state_after_ine after__iteeraction 'I cannot do [Step
if start_from <-
print(' [Step type text "Aliquet. Sollicitudin nam lectus,"")
"eppydt-rrrrplytett""""""" agent. t'p 'Aliquet. Sollicitudin nam lectus.') agent.type
state_after_ inniterriittaaction
ceatttllcccxxpxxttte ("//# "eid='send-reply "))
# Task: Find the email by Blanca forvard that email to Agathe. def solution (agetion start_from-l]
# General plan: I should first click on the email by Blanca, then click the forward button, then type "Agathe" and finally click the send button.
if start_from
print(' [Step click on the email by Blanca')
wta.tcccccc_xpaccc//aaaaa [0class=' email-sender" "emllass="mmilleendder and text ()='Blanca' state_after_interacttii
crent.clickket cltckkxpptt("///ppaaaa (aclassssaaaaasssaaaaaaaaaaaaaa email-forvard']""
forward content should be displayed on page. assert 'forward-sender' in
'I cannot do [Step
1] correctly. The forvard button is not displayed on the page.'
27

---

if start_from <= 2:
print(' [Step 2] type "Agathe"')
ayent. Agathe') (yent.type("Agathe") Stat__afte_inntore
agent. wtt.cccccccppac//// (0id='sem) fordworw
# Taak: Find the email by Salli and click the trash icon delete it. solution (ageti,, start_frome ftomt_from=t):
# Ceneral plan: I should firat click on the email by Salli, then click on the trash icon.
if start_from <=
print( [Step click on the email by Salli')
xoant..ciickxpat(""//a xpath("//divlleleelll aear://aaaasemmmaaeeeeaeeeaa email-sended and text Salli'] (extl)"aa1l agent.click,
[0class= ttrnse
The expert sample for the task email-inboox-oovrrddnll
saampleeeemall-nox--fooooo
Prompt
# Here is an example of solution.
# task: Send Alice the email from Beth navigate the message from Beth and send to Alice.
def solution (agent, start_ f:omm_
# Ceneral plan: I should firat click on the email from Beth, then click on the "Forward" button, then type "Alice" in the "To" inputbox, finally click on the "Send" button.
start_from
print(' [Step 1] click on the email from Beth') (taos/pppttt///llllsssesaeee state_after_interacttiin
and text()=
egent.click_nyg haapecccccccxxaa(///// (t/ppptaaattttaalaaeeaa / "emilaas="mmll--ffrrrrr"""
the "To" inputbox should be displayed on page.
assert 'forward-sender' in
f'I cannot do [Step
1] correctly. The "To" inputbox is not displayed the page. Current state: aetate_atte____ afterr_itteeaction..
if start_from <- 2:
print(' [Step 2] type "Alice" in the "To" inputbox') agent. hatta(///anppttlclaa==="oooorrr
state_after_interes
agent.click, xxpckk h/span petpecppppptpppppppaaaadaaeeeeee senda-noarr )) the email should be sent successfully.
assert 'email-sender' in correctly. The email is not sent successfully. Current state: (state_ wwwaaaattttrteeee interrattact)
I'I cannot do [Step 2]
The expert sample for the task email--nbbex-foraaddd-----
<sample email <eample_email-inbooxe
Prompt
Here an example solution.
# taak: Send Alice the email from Beth / navigate to the message from Beth and send it to Alice / I want to forward the email from Beth over to Alice def Dalution(agent, start_from-1):
# General plan: should first click on email from Beth, then click on "Forward" button, then type "Alice" the "To" inputbox, finally click on the "Send" button.
if start_from <= 1:
print(' [Step click the email from Beth') wtc.ickkppaaaa//aaaeeaaa
and text()="Btth"))
state_after trtttaaaattrnnnerattiioo
# the "To" inputbox should be displayed page.
assert 'forward-sender' state_ itt.rrrnntteactttoor. cannot [Step correctly. The "To" inputbox not displayed on the page. Current
state: attate_after_interactie
28

---

if start_from <=
print(' [Step type "Alice" in the "To" inputbox') ceont.click_ppahhh "fras agent. Captetttpp((Acce
state_atter_ntorcctiio agent.click_xpear xxick__ppahh(llll Baathh///span//eee (@id- -enddddddssn--orroeeeee send--orwardd") the email should be sent succesafully. assert 'email-sender' correctly. The email is not sent auccessfully. Current state: ceaate_after_iteracctie
state_after- after_interaction f'I cannot [Step 2]
The expert sample for the task email-inbox-nl-turk.
<sample, <sample_email-inee email-inbox-nl-tueeee Prompt
Here are examples of solution.
# Task: "Aliquet. Sollicitudin nam lectus." is my reply to Brittani 's most recent email / Find the email by Brittani and reply to them with the text "Aliquet. Sollicitudin nam lectus.". def solution (agent, start_free :eart_fro==1): # General plan: should first click on the email by Brittani, then click on the reply button, then type the text "Aliquet. Sollicitudin nam lectus." and finally click on the send button.
if start_from <=
print(' [Step click on the email by Brittani') http:cliicckxatt/////vvvrr/aeeee
eeeemmllleddeee and text Brittani']")
state_after,iittracctint
sgent.click_xpath ("//ppp [#class= [ecpa_/mmmmmmmaaeeelpp1]]]]]]]]
the reply content should be displayed page. assert 'reply-text' correctly. The reply button is not displayed on the page."
state_after_interactt 'I cannot do [Step
if start_from <=
print(' [Step type text "Aliquet. Sollicitudin nam lectus."") "aaiic_ppat((//teettraaaaaa "repyi--trpply-eet_]] agent. tynet.tpp(()) S'Aeqq'Ai. Sollicitudin nam lectus.')
state_after_inee ttrresffttttittractioo agent. click aliick_ppathh///4
# Task: Find the last email by Blanca and send it to Agathe Find the email by Blanca and forward that email to Agathe. def solution (agent, start start_from-t from=))o # General plan: should first click on the email by Blanca, then click the forward button, then type "Agathe" and finally click the send button.
start_from
print(' [Step 1] click on the email by Blanca') agent. wtt.cc.ppattt///iiiiiiaaaaa ( email-sender and text text()=Bllacee) Blanca'] ctate_aator__intorecito.u
[6class-' www.eeemmllllllle.eee forvard')")
the forward content should be displayed on page.
assert 'forward-sender' in state_afted itterrritteraatin, 'I cannot do [Step
1] correctly. The forward button is not displayed on the page.
start_frod
print( (" [Step 2] type "Agathe"') agent.clickk.e http::cccccxxaahc///eeeaea [0class= http://appptoccllss=aaaaaaarrdrd forward-sender agent.typed ('Agathee)
ttate_afterjinterett
agent. cww.cclccccccceeeeee (liick_ppath"//// ("/th" [0id= send-forward
# Taak: Delete this email from Salli / Please find Salli's email in the inbox and delete it.
def (alution ctart_fromsl
# General plan: should first click on the email by Salli, click on the trash icon.
if start_from <= 1: print(' [Step 1] click the email by Salli')
29

---

sen and text ()= Salli'] texll'='
agent. haap::/......ppapaaaaaaaaaaaaaa
The expert sample for the task guess
<sample_guee -ueeleeguss--ee Ses---guuussuuuurereeee number>> Prompt
# Here an example solution.
# Taak: Guess the number betueen 0-9 and press Submit. Use the feedback below to find the right number.
# Here the solution:
solution (agent, start_frem # Ceneral plan: Given a list of possible_numben I will try the number in the middle of the list and get feedback from the html state.
# Now the given list of maasible_uumbere is [0, 2, 3, 4, 5, 6, 7, 9].
if start_from print(' [Step maintain a list the possible numbers left and try the middle one')
# before making guess, I should store the html state for future comparison.
state_befor seafore_interactaae
agent. agent.current. wwrrentustatte
I will choose the number 5, which is in the middle of possible_numbers guess_ ___mm_uurrrrr. - 5
# click the input box, type the number guessed and click submit. agent.click beeenttclic__ppahhhaeee (paatt///inutt(0iiddtttt agent.presst (ey"ccckeppace"" agent. type (guess (gsuess_number)) state_after__ stt.aafet_tttttattionnt
agent. click_ www.tppa////aapp (/ddd//""((
after input and submit the guessed number, the html_state should be changed and contain the feedback. Otherwise this step is not successful. assert state_after iiteeriittraction != 1] but the html state did not change.
'I did [Step
if start_from from
print(' [Step 2] get the feedback information from the new html state') If the guess is successful, the keyword "higher" or "lover" should not be present in the observation. Otherwise should use assert to jump out pass the feedback.
observation ask(f' Answer question based the html code belos: lesste__fter_inteeeee
Question: Which one is displayed? "The number is lower than" or "The number is higher than"? ? You must only output the displayed sentence without saying any other words.')
assert "higher" not observation, f' Yoo tried the number {guess_number)
[Step 1], and the correct number greater than this number. I need to revise solution function according to the new plan: Now the given list of possible_number numbers is [6, 7, 8, 9].' assert "lower"
in observation, 'Yoo tried the number (guess_numbor number) in [Step 1], and the correct number is smaller than this number. I need to revise solution function according to the new plan: Now the given list of seaaible_uubber is [o, 1, 2, 3, 4]."
The expert sample for the task login-user-pooup:
(sample_login-usee
Prompt
# Here is example of solution. Task: Enter the username "kanesha" and the password "oT" into the fields and press login.
def solution (aguti, start_frone foom=ffro==1):
# General plan: I should first click on the username field, then type in the username, then click the password field, then type in the pasaword, then click on the login button.
if start_from <= 1: print [Step 1] Click on the username field') tntte_aftee__iterctttt agent. cwwck_lkcppaaeeec (" /ii////u///////////ee [@id=\ 'username) ")")
30

---

# during interaction, some popupe may appear. If so, I need to jump out to handle the popups.
assert 'popup' not in iiatte_aate___ttrrcttt popups appeared, you need to close the popup at the beginning of [Step 1]. The current html is: {state_afe after aatee_iiteraction] You need to add some code the beginning of [Step 1] cancel the popup before any other actions.'
f'After [Step i], some
if start_from <= 2:
print(' [Step 2] Type in the username') State_after_intercttin
agent.type( (ypee'knnshha)
during interaction, some popups may appear. so, need to jump out to handle the popups.
assert 'popup' not in state_after, interaction, f' A'Atter [Step 2], some popups appeared, you need to close the popup at the beginning of [Step interactien) You need to add some
2]. The current html the beginning [Step 2] cancel the popup before any other actions."
if start_from <- 3:
print(' [Step Click on the password field') state_ atlaaaafeeinttrttiiin
agent. click_xpathh (aiikk_patt""///pppiii h/in//ttnniiattieee ppasswordy "]")
# during interaction, some popupa may appear. If so, I need to jump out to handle the popupa.
assert 'popup' not in state_after_intee interactionn, f' f'After [Step 3] some popups appeared, you need to close the popup at the beginning of [Step You need to add some
3]. The current html is: saatte_afteriittereee code at the beginning of [Step 3] to cancel the popup before any other actions.'
start_froe print(' [Step 4] Type in the pasaword') state_ate intterritteraction agent. typee''oT)) # during interaction, to handle the popups.
popups may appear. If so, need to jump out
assert 'popup' not in state_ after attaaaafetterraaeeeaeaa popups appeared, you need to close the popup at the beginning of [Step 4] The current html is: actate_after cicate_after_inteaccttii interaction] You need to add some code the beginning of [Step 4] cancel the popup before any other actions."
interaction, f'After [Step 4], some
if start_from <= 5:
print [Step 5] Click the login button') state_afee aterrartrttttttttttooo
agent. www.ccccicccaeeeeeeeeeeee spptt((//buttonn) subbtn ") during interaction, some popups may appear. If so, need to jump out to handle the popups.
assert 'popup' not in state_after_ stateeaftteriinterattionn, interaction, f' A'Atter [Step 5], some popupa appeared, you need close the popup the beginning of [Step 5] The current html is: (state_after, interaction] You need to some siaaat__fteerniteaactiioo code at the beginning of [Step 5] to cancel the popup before any other actions."
The expert sample for the task search-
ksample_search-eng Prompt
Here an example solution.
# Task: Use the textbox to enter "Alice" and press "Search", then find and click the 7th search result.
def solution (agent, start_from=t): # General plan: should first click the inputbox, then type "Alice", then click on the "Search" button, finally look through pages and click on the 7th result.
if start_from <= 1: print(' [Step 1] click the inputbox and type "Alice"!)
31

---

hallicc_ppaat////(iiiddaaaaa agent.type(" type('Alice')
state_afe arrre_after_intrattatn
the search content should be displayed on page.
assert "search-des" in state_after, interaction, 'I cannot do [Step 1] correctly. The search content is not displayed the page.'
start_
print(' [Step 2] calculate the page number of the 7th result and click on the page')
# should count the number of results each page, iteratively turn next page until seeing the result. can use the following code to count the number results on each page.
caasults_displayeee count search-desc'' wttp_aafterritteereeeeeee
num_page (7 1) // aww.reeaule..diipllyyye.pee.r.e state_after_inee inttrrratttttttaaciinnn
agent. caattccccccxpxac(CC //'
# I should click the 7th result.
jum_to_clite mww.reeults__ueppaayeeeee despalls_ddsplayyd_pr....g num_page
num_ree cummrresults_dee
state_after_intereee
ceent.click_ppathheee //* [eid=""pagecoontaatt]]] [ammmlfmmmtttt to_ bo_click)]/a"
The expert sample for the task terminal:
<sample_ terminalt Prompt
# Here is example of solution. Task:
terminal below to delete def solution (agent, start_fes from=_r
ending with extension .gif
# General plan: I should first type "Is" to list all files in the terminal, then identify the filename ending with ".gif" and type "rm [filename] gif" to delete the identified file.
start_from
print(' [Step 1] type "Is" to list all files in the terminal') agent.type State_after__te inteerriitteaction the file list should be displayed terminal.
agent.press (e ('ener'')
tereeaafferrnteeattiiiee interaction, f'I cannot do [Step assert 'gif' state_ after
correctly. The file list is not displayed on the terminal. Current state:
start_from print(' [Step 2] identify the filename ending with ".gif" and type "rm [filename] .gi"" to delete the identified file') # I should identify the filename ending with " ".gif". can ask assistant to do that.
filename ask You are given some html code as follows: tatate_after, interaction) What is the file ending with the extension .gf?? You must directly output the full file name, including the extension.")
agent. type (rm rm (filename)') State_after, aafter_intercctinn
agent.press press_reyseeeee ('enter')
assert 'not found' not in state_after, state_after_interaaeet interaction, I'I cannot do [Step 2] correctly. The file ending with the extension gif is not deleted. Current state: (state (state_after_e after_interaction))
The expert sample for the task <sample Prompt
# Here is an example of solution. # Task: Playing as 'X', win game of tic-tac-toe. def solution (agent, start_from-1);
32

---

# The board layout and corresponding html xpath: htpppp//pppppo///ddddddddttttt10 hppp-eetttr("/---((odddtttttttt
Note that "mark-o" indicates the '0' placed board, "mark-x" indicates the 'x' placed board.
# General plan: Currently, no grid is occupied. The plan 1) put an 'X' the middlleeeenteee///AAlddddtttt-
2) put an 'X' in the ogp-lett((//t(lieeettttt011))1) 3) put an 'X' in the bottaarrriaa((///-lliittttt1111"
place_to_put___ ['4'', '0', '8']
for idx, place_id in enumerate to_aace_t__pt_X) print(f [Step (idx}] put an X in {place_id)) (
# before interaction, need to store the current state so that can compare it with the state after interaction. wttpouffoor_eieerctttteeeeeeeeee
agent. state citee_ffee_iiie effeer_ineraaction = agent.current_sstaee
'd}' # if the current state does not change after interaction, that means I cannot put 'X' in the desired location, and that location is already occupied assert state state_before, ceffore_interacciin. do [Step [idx}] put an X in the "//=[0id= "ootdd"ttt-pplace_id]] "d''' because it is occupied. I need to revise solution function according to the new plan. ... ask(f" PPlaying layout and corresponding html xpath: top-leftt(//MIiiid=tttt11111""" tp--iaat((//<(hhi-tttt2111]"
the plan will not work.
!= state_affi state_after inteerrinteraation, f cannot
'X', win a game of tic-tac-toe. The board ppppccatttt"//*((((d""tttt11"
|ottommrrgttt"//111eettttt11111 Note that "mark-o" indicates the '0' placed on board, "mark-x" indicates the ' placed on board. The game in progress is represented in html code: fagent. current_statel Report current board situation and generate plan that the 'X' player should follow continue this game. Use the format like "Currently,
been placed at (ppostiionn"/AAiiidtttttt and '0' has been placed at the plan is to: 1) put an 'X' in the (Maittio>>""//400ee (ttttddd==tttt''"ii""e"" itt-x']") 2) put an 'x' in the ..."!!!)
Therefore,
Code Check. We use the same (code_check> prompt for MiniWoB++ tasks ALFWorld.
Out-of-Plan Refinement. the event of an assertion error, use <refinement> to conduct the out- of-plan refinement. In this prompt, <basic_ing info> is replaced by the content of the <basic_info> prompt. The placeholder <solution_funs func> is replaced by the generated solution function, while <task> is replaced by the task description in natural language. Finally, <feedback> is replaced by the assertion eтrоr message returned by the solution function. Note that the <ref infeenteett prompt used here differs slightly from the one used for ALFWorld. In this context, we include the previously generated solution function the prompt, enabling the refiner to adjust the solution based on the enror messages accordingly.
<refinement> Prompt
<basic_info>
# Task: <task>
You have generated code of aolution() to solve the taak: iollutionffunc"
However, you executed the solution() function get an error message:
[Error message]
<feedback>
Let's think step by step. You must output revised solution function based on the error message. You must only complete the revised solution function without any other worda.
33

---

# In the solution function, start with commented "# General plan: " For each step you plan to take, mark with [Stepp xx] and write assertion check if the step successful. def solution start_from-l):
Determining start_from We use the same <start_from prompt for MiniWoB++ tasks as ALFWorld.
E
Hallucination with GPT-3.5 Model (gpt-3.5-tury)
We present the trajectories of three tested methods (ReAct, Reflexion, and AdaPlanner) for two tasks. Here, each model utilizes gpt-3.5 3-r-burbo as the backend.
E.1 Case 1: examine the book with the desklamp.
ReAct's trajectory for this case is detailed below. Despite the presence of two expert demonstrations, ReAct encounters significant difficulties, as it extensively hallucinates inadmissible actions sce- narios. For example, search for a book titled "The Great Gatsby" and ask household members they know where the book is are not feasible within the given environment.
ReAct Trajectory of Case (Hallucinated actions are marked in pink.)
You are in the middle of a room. Looking quickly around you, you see a bed 1, a desk 2, a desk 1, a drawer 6, a drawer 5, drawer 4, a drawer 3, a drawer 2, drawer 1, a garbagecan 1, laundryhamper 1, safe 1, shelf 6, a shelf 5, shelf 4, shelf 3, shelf 2, and shelf examine the book with the desklamp.
Your task is
> think: To solve the task, I need to find and take book, then find and desklamp. First, need to find book. book is more likely appear in desk (1-2), shelf (1-6), drawer (1-6). I can check one by one, starting with desk 1.
OK.
> go to desk 1
On the desk 1, you see a alarmelock 1, boul 2, a 1, a cellphone 3, desklamp
1, mug 1, a pen 1, and a pencil take book from deak Nothing happens.
> think: I need to be specific about which book I looking for. Let me try looking for specific title author.
OK.
search for a book titled "The Great Gataby"
Nothing happens.
> think: Let try looking for book by specific author. How about "To Kill Mockingbird" by Harper Lee?
OK.
search for a book by Harper Lee
Nothing happens.
> think: Maybe I should try looking in different location. Let check shelf 1.
OK.
> go to shelf
On the shelf 1, you see a bowl 1.
> go to shelf
On the shelf 2, you see a creditcard 1, a pencil 2.
> go shelf
On the shelf you see alarmclock 1, boul 2, and desklamp
> take book from shelf 3
Nothing happens.
> think: Let me try looking in different location. How about draver 1?
OK.
> go to drawer 1
The drawer 1 is closed.
> open drawer
You open the drawer 1. The drawer 1 open. it, you see nothing.
go drawer 2
On the draver 2, you cellphone 2.
go drawer
34

---

Nothing happens.
> go to draser 4 The drazer 4 is closed.
> open drawer
You open the drawer 4. The drawer 4 is open. In it, you see nothing.
> go to drawer 5
On the drawer 5, you see nothing.
go drawer 6
The drawer 6 is closed.
>
open draver
You open the drawer 6. The drawer 6 is open. In it, you see nothing,
> think: Let me try looking in different location. How about desk 2?
OK.
> go to desk 2
On the desk 2, you see a alarmclock 3, boul 3, a 3, a mug 2, and pencil 3.
take book from desk
Nothing happens.
think: Let try looking different location. How about shelf 4? OK.
> go to shelf
On the shelf 4, you see a alarmclock 2.
> think: Let me try looking in different location. How about shelf 5?
OK.
> go to shelf
On the shelf 5, you see nothing.
think: me try looking different location. How about shelf 6?
OK
go shelf
On the shelf you nothing.
> think: It seems like the book I am looking for is not in of the locations I have checked so far. Let me ask the household members if they know where it is. OK.
household members they know the book is
Nothing happens.
> think: Maybe I should try asking specific household member. How about asking John?
0K.
ask John he knows where the book is Nothing happens.
think: It seems like unable find the book. may need ask for information try different approach the task.
OK.
> ask the household members if they have any suggestions on where to look for book Nothing happens.
> think: It looks like may need to give up this task for and back it later with more information different approach.
0K.
> end task Nothing happens. > exit room
Nothing happens.
> end interaction Nothing happens.
leave the household
Nothing happens.
> end interaction and try again later Nothing happens.
> end interaction and back later with approach Nothing happens.
end interaction and move on different task or activity Nothing happens.
> end interaction and leave the room/householdd/lcctiio
Nothing happens.
end interaction and try again later with different approach or more information
Nothing happens.
end interaction and seek help from someone outside of the household
35

---

Nothing happens. > end interaction and try to find book in a different room or location in the house Nothing happens.
The trajectory of Reflexion, after two rounds of iterative trials and errors, is detailed below, Despite multiple closed-loop corrections, Reflexion fails to formulate feasible plan. Instead, it persists in generating unfeasible actions in its reflection part. For instance, it plans to ask the household members for help, or to look for a ladder or step stool outside of the room. These actions fall outside the scope of admissible actions in the ALFWorld en- vironment.
Reflexion Trajectory of Case (Hallucinated actions are marked in pink.)
Your memory for the task below: Trial 0:
In this environment, my plan to find book then find and use desklamp. However, I spent too much time checking every possible location without thoroughly examining each one. I should have focused on the most likely locations for a book (desk, shelf, drawer) and thoroughly searched each one before moving on to the next. Additionally, members for help earlier in the process. In the next trial, I will prioritize searching the most likely locations and ask for help if am unable to find the book after thorough search. Trial 1:
should have asked the household
In the next trial, I will prioritize searching the most likely locations for book (desk, shelf, drawer) and thoroughly examine each before moving to the next. If am unable to find the book after thorough search, will ask the household members for help. If the book is out of reach, I will look for ladder or step stool outside of the room that I can bring in to reach it. Here is the task:
You are in the middle of a room. Looking quickly around you, you see a bed 1, a deak 2, a desk 1, a drawer 6, drawer 5, drawer 4, a drawer 3, a drawer 2, drawer 1, a garbagecan 1, laundryhamper 1, safe 1, shelf 6, a shelf 5, shelf 4, shelf 3, shelf 2, and shelf 1. examine the book with the desklamp.
Your task is
> think: To solve the task, I need to find and take book, then find and use desklamp. First, need to find book. book is more likely to appear in desk (1-2), shelf (1-6), draver (1-6). I can check one by one, starting with deak 1.
0K.
> go to desk 1
On the desk 1, you see a alarmclock 1, boul 2, a 1, a cellphone 3, desklamp 1, mug 1, pen 1, and pencil 1. examine books desk Nothing happens. > examine shelf Nothing happens.
> go to shelf 2
On the shelf 2, you see a creditcard 1, and a pencil 2. > go to shelf
On the shelf 3, you see a alarmclock 1, a bowl 2, a desklamp
take book from shelf
Nothing happens.
> examine book shelf
Nothing happens.
> ask household member for help
Nothing happens.
think: I need keep searching for the book. It could be in deak 2, shelf 4-6, draver 1-6. will check draver 1 next.
OK.
> go to drawer 1
The drawer 1 is closed.
> open drawer
You open the draver 1. The draver 1 is open. In it, you see nothing.
> go to drawer 2
On the drawer 2, you see a cellphone 2. book from drawer Nothing happens.
examine book in drawer
36

---

Nothing happens.
> think: The book is not in drawer 2. I will check drawer 3 next. αK.
> go to drawer 3
Nothing happens.
> examine books drawer
Nothing happens.
in drawer 3. I will check drawer next. The drawer 4 is closed.
think: The book OK.
> go to drawer 4
>
open drawer 4
You open the drawer 4. The drawer 4 is open. In it, you see nothing.
> think: The book is not in drawer 4. I will check draver next.
OK.
go drawer 5
On drawer 5, you see nothing.
think: The book is in drawer 5. I will check drawer next. OK.
> go to drawer 6
The drawer 6 is closed.
>
open draver
You open the drawer 6. The drawer 6 is open. In it, you see nothing.
> think: The book is not in drawer 6. I will check desk 2 next.
> go desk 2
On the deak 2, you
think: The book is alarmclock boul desk 2. I will check shelf next.
3, mug and pencil 3.
0K.
> go to shelf 4
On the shelf 4, you see a alarmclock 2.
> think: The book is not on shelf 4. will check shelf next.
0K.
> go to shelf
On the shelf 5, you see nothing.
think: OK
book
on shelf will check shelf next.
go shelf
On the shelf you nothing.
> think: The book is not on shelf 6. have searched all the most likely locations for a book and have not found it. will ask the household members for help. 0K.
ask household member help
Nothing happens.
> think: The household member did not respond. I will look for ladder step stool outside of the room that I can bring in to reach higher shelves or areas. 0K.
go outside of the
Nothing happens.
> look for a ladder or step stool
Nothing happens.
> think: I was unable find ladder step stool. will continue searching the room for the book and ask the household members for help again if necessary.
OK.
> go to desk 1
On the desk 1, you see a alarmclock 1, boul 2, a 1, a cellphone 3, a desklamp
1, a mug 1, a pen 1, and a pencil 1.
examine books on desk Nothing happens.
think: I have searched the thoroughly and have found the book. will ask the household members for help again. OK.
> ask household member for help
Nothing happens.
> think: The household member not responding. will try think of any other possible locations for the book and search those areas.
OK.
> think: The book could be in the laundry hamper safe. will check those areas next..
oK.
37

---

> go to laundry hamper 1
Nothing happens.
examine laundry hamper
Nothing happens.
> go to safe 1 The safe 1 is closed.
try open safe Nothing happens.
The trajectory of AdaPlanner is detailed below. For better readability, only present the major sub- goals generated by the planner. Evidently, AdaPlanner successfully completes this task without the need for any revisions. This indicates that the code interface can effectively mitigate the hallucination in
AdaPlanner Trajectory of Case
You in the middle of room. Looking quickly around you, you bed drawer 4, drawer 3, a drawer 2,
desk 2, desk 1, drawer 6, drawer drawer 1, garbagecan 1, laundryhamper 1, safe 1, shelf 6, a shelf 5, shelf 4, shelf 3, a shelf 2, and a shelf 1.
Your task is to: examine the book with the desklamp. [Step 1] get a list of receptacles where book is likely to appear.
[Step 2] go to each receptacle in the 1ist until seeing a book > go to shelf
On the shelf 6, you see nothing.
go shelf
On the shelf you nothing.
> go shelf
On the shelf 4, you see mug 2, and pencil 1.
> go to shelf 3
On the shelf 3, you see a creditcard 1, a desklamp 1, and mug 1.
> go to shelf
On the shelf 2, you see a pen 1.
> go to shelf
On the shelf 1, you see a bowl 1.
go
2
On the desk 2, you
cellphone 2, creditcard 2, mug 3, and pencil 2. alarmclock creditcard 1, desklamp 1, laptop 3,
> go to desk 1
On the desk 1, you
and a mug 1.
> go to bed
On the bed 1, you see a book 1, a cellphone 1, a laptop 2, a laptop 1, a pillow 2, and a pillow 1.
[Step 3] take the book from the receptacle
book from You pick up the book from the bed
[Step 4] get a list of receptacles where desklamp is likely appear.
[Step 5] go to each receptacle in the list until seeing desklamp
> go to desk 1
On the desk 1, you see a alarmclock 1, creditcard 1, a desklamp 1, a laptop 3, and a mug 1. [Step 6] turn on desklamp
use desklamp 1 You turn on the desklamp 1.
E.2 Case 2: put some watch safe.
In the second trajectory, ReAct also hallucinates infeasible actions (e.g., look for a manual or instructions and ask someone in the household), ultimately leading to task failure.
ReAct Trajectory of Case (Hallucinated actions are marked in pink.)
You are in the middle of a room. Looking quickly around you, you see a armchair 1, cabinet 4, cabinet 3, a cabinet 2, a cabinet 1, drawer 5, a drawer 4, drawer 3, a drawer 2, a drawer 1, dresser 1, garbagecan 1, safe 1, shelf 6,
shelf 12, shelf 11, a shelf 10, shelf 9, shelf 8, shelf shelf 1, sidetable 1, and
shelf 5, shelf 4, shelf 3, shelf sofa 1.
38

---

Your task is to: put some watch safe.
> think: To solve the task, I need to find and take watch, then put it on the safe.
OK.
> think: First I need to find a watch. watch is more likely to appear in a drawer (1-5), shelf (1-12), cabinet (1-4), dresser (1), sidetable (1), or sofa (1). can check one by one, starting with drawer
OK.
> go draser 1
The drawer 1 is closed.
> open drawer
You open the drawer 1. The draver 1 is open. In it, you see nothing.
> go to drawer 2
The drawer 2 is closed.
> open drawer
You open the drawer 2. The drawer 2 open. it, you see nothing.
go drawer 3
The drawer 3 closed.
> open draser
You open the drawer 3. The drawer 3 is open. In it, you see creditcard 2.
> go to drawer 4
The drager 4 is closed.
> open drawer
You open the drawer 4. The drawer 4 is open. In it, you see nothing.
> go to drawer 5
The drawer is closed.
open drawer
You open the drawer 5. The drawer 5 is open. In it, you nothing.
> go shelf
On the shelf 1, you see nothing.
> go to shelf 2
On the shelf 2, you see nothing.
> go to shelf
Nothing happens. > go shelf
On the shelf you see nothing. go shelf
On the shelf you nothing.
> go shelf
On the shelf 6, you see nothing.
> go to shelf 7
On the shelf 7, you see a newspaper 3.
> go to shelf
On the shelf 8, you see nothing.
go shelf
On the shelf you see nothing.
go shelf 10
On the shelf 10, you nothing.
> go to shelf 11
On the shelf 11, you nothing.
> go to shelf 12
On the shelf 12, you see nothing.
> go to cabinet
The cabinet
closed.
open cabinet You open the cabinet The cabinet 1 is open. it, you
go to cabinet
Nothing happens.
> go to cabinet
The cabinet is closed.
> open cabinet 3
You open the cabinet 3. The cabinet 3 is open. In it, you see tissuebox 2.
> go to cabinet:
Nothing happens.
go dresser
laptop 1, remotecontrol 1, tissuebox vase 3, > take watch 1 from dresser 1
On the dresser you
2, and watch
You pick up the watch from the dresser 1.
> think: Now I have watch (1), Next, need to put it on the safe.
OK.
go to safe 1
39

---

The safe 1 is closed.
> open safe
You open the safe 1. The safe 1 is open. In it, you aee cellphone 3, and a
cellphone 2.
> put watch on safe Nothing happens.
think: need be more specific about where to put the watch. Maybe on top of the cellphone OK.
> put watch on cellphone 3 in safe
Nothing happens.
think: Maybe I need place the watch inside the safe instead of top of something inside it.
OK.
> put watch in safe
Nothing happens.
>
think: Maybe I need to enter code to properly open the safe and put the watch inside.
OK.
> look for code key to open the safe
Nothing happena.
> think: Maybe I need to ask someone in the household for the code or key to open the safe.
OK.
>
ask someone in the household for the code key open the safe
Nothing happens.
>
think: Maybe I need to search for a manual or instructions on how to open the safe.
> look for manual instructions how to open the safe Nothing happens.
think: Maybe I need to try a combination numbers to open the safe. 0X.
> try the combination 1234 on the safe
Nothing happens.
> think: Maybe I need to try a different combination numbers to open the safe. oK.
Similarly, Reflexion experiences issues with hallucination during this task, such as seek additional resources ask for help. This contributes to the eventual failure of the task.
Reflexion Trajectory of Case 2 (Hallucinated actions are marked in pink.)
Your memory for the task below:
Trial 0:
In this environment, my plan was to find watch and put on safe. However, I did not consider that the safe may require a code or key to lock it. should have looked for code before attempting to put the watch inside. When I could not find way secure the watch inside the safe, I should have looked for another container safe in the room. In the next trial, will first look for a code or lock mechanism on the safe or key to the safe. If I cannot find one, I will look for another container or safe in the room.
lock mechanism
the safe
key the safe
Trial 1:
To solve the task, need to find a watch and a way to secure it inside the safe. First, will look for watch drawers (1-5), cabinets (1-4), dressers (1), shelves (1-12), sidetables (1), and sofas (1). Once find watch, I will examine the safe for any clues look for key in other locations. If I cannot find way secure the watch inside the safe, will look for another container safe in the room. If still cannot find way to secure the watch, I will seek additional resources or ask for help. Here is the task:
You are in the middle of a room. Looking quickly around you, you see a armchair 1, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a draver 5, a draver 4, a drawer 3, a drawer 2, a drawer 1, dresser 1, a garbagecan 1, safe 1, shelf 12, a shelf 11, a shelf 10, shelf 9, a shelf 8, a shelf 7, shelf 6, shelf 1, sidetable 1, and
shelf 5, shelf 4, shelf 3, shelf sofa 1.
Your task is put watch safe.
40

---

think: To solve the task, I need to find watch and way to secure it inside the safe. First, will look for a watch in drawers (1-5), cabinets (1-4), dressers (1), shelves (1-12), sidetables (1), and sofas (1).
OK.
> go to drawer 1
The drawer 1 is closed.
open drawer
You open the drawer The drawer 1 is open. it, you nothing.
> go to drawer 2
The drawer 2 is closed.
> open drawer
You open the drawer 2. The draver 2 is open. In it, you see nothing.
> go to drawer 3
The drawer 3 is closed.
> open drawer
You open the drawer 3. The drawer open. it, you see creditcard 2.
go drawer 4
The drawer 4 is closed.
> open draser
You open the drawer 4. The drawer 4 is open. In it, you see nothing.
> go to drawer 5
The drazer 5 is closed.
> open drawer
You open the drawer 5. The drawer 5 is open. In it, you see nothing.
> go to cabinet
The cabinet
closed.
open cabinet
You open the cabinet The cabinet 1 is open. it, you
> go cabinet
Nothing happens.
> think: The cabinet 2 may be locked or empty. I should try another location. OK.
> go to cabinet
The cabinet is closed.
open cabinet 3
You open cabinet The cabinet 3 open. it, you see tissuebox 2.
go cabinet
Nothing happens.
> think: The cabinet 4 may be locked empty. I should try another location. OK.
> go to dresser
On the dresser 1, you see a laptop 1, a remotecontrol 1, a tissuebox 1, a vase 3, a vase 2, and a watch 1.
think: Now I find watch (1). Next, need to examine the safe for any clues or look for key in other locations.
OK.
go safe 1
The safe 1 is closed.
> think: Before attempting to put the watch inside the safe, I should look for code or lock mechanism the safe key to the safe.
OK.
> examine safe 1 The safe 1 is closed.
look for key or code on safe
Nothing happens.
think: I cannot find key code the safe. ahould look for another container safe in the
0K.
> look for another container safe in the room
Nothing happens.
> think: I still cannot find another container or safe in the room. I should seek additional resources or ask for help.
Ox.
> seek additional resources ask for help
Nothing happens.
> think: I may need to think of a different solution to the task. Perhaps I can place the watch near the safe instead of inside it.
> put watch near safe
Nothing happens.
41

---

think: The task is incomplete. need to put the watch the safe. I should try to find a way to secure the watch on top of the safe or find container to put the watch in that can be placed on top of the safe.
OK.
look for container put the watch in that be placed top of the safe Nothing happens.
> think: I should try to find a way to secure the watch on top of the safe. OK.
look for way
the watch top of the safe
Nothing happens.
> think: I may need to think of a different solution to the task. Perhaps I can find way to attach the watch to the safe using tape or string.
0K.
look for tape string
Nothing happens.
> think: I cannot find tape or string. should seek additional resources or ask for help.
OK.
seek additional
ask for help
Nothing happens.
> think: The task is incomplete. need to put the watch on the safe. I should try to find a way to secure the watch on top of the safe or find a container to put the watch
that can be placed on top the safe. Since I cannot find any resources or help, will have abandon the task for now and come back it later with new ideas resources.
OK.
> end task Nothing happens.
exit room
Nothing happens.
think: I cannot exit the room until task complete have abandoned will continue look for solution the task.
0K.
continue searching for solution to the task
Nothing happens.
think: I have exhausted all possible options and take break and come back the task later with fresh perspective seek help from someone else.
in the should
0K.
> end task Nothing happens.
> exit room
Nothing happens.
> think: I cannot exit the room until the task is complete have abandoned it. I will take a break and come back to the task later with fresh perspective or seek help from someone else.
0K.
end task Nothing happens. > exit room Nothing happens.
Initially, AdaPlanner also experiences hallucination, presuming that the watch is located in the safe. This incorrect assumption triggers an assertion error. AdaPlanner then revises the plan better adapt to the actual environment. Following this refinement, the task is successfully completed.
AdaPlanner Trajectory of Case 2 (Hallucinated actions are marked in pink.)
You are in the middle of room. Looking quickly around you, a cabinet 4, cabinet 3, a cabinet 2, a cabinet 1, a drawer 5, a draver 4, a drawer 3, a drawer 2, a drawer 1, a dresser 1, a garbagecan 1, a safe 1, a shelf 12, a shelf 11, a shelf 10, a shelf 9, a shelf 8, a shelf 7, shelf 6,
see armchair 1,
shelf 5, shelf 4, a shelf 3, a shelf 2, shelf 1, sidetable 1, sofa 1.
Your task is put watch safe.
[Step 1] get list of receptacles where the safe is likely appear
42

---

(Step 2] go to each receptacle in the list until seeing safe
> go to safe 1
The safe 1 is closed.
> open safe You open the safe 1. The safe is open. In it, you see cellphone 3, and a
cellphone 2.
[Step 3] identify the watch just found and take it take watch 1 from safe Nothing happens.
[Step 4] go to the safe and put the watch in it > go to safe 1 Nothing happens. > put watch in/on safe 1 Nothing happens.
Error [Step 4] cannot put the watch safe 1. The last three interactions before error were: take watch 1 from safe Nothing happens.
AssertionError:
> go to safe 1
Nothing happens.
> put watch in/on safe 1
Nothing happens.
I am at safe 1 and holding None.
Refinement done with start_fron 1. [Step 1] get a list receptacles where the watch likely appear [Step 2] go each receptacle in the list until seeing watch > go safe 1 Nothing happens.
> go to cabinet
The cabinet is closed.
> open cabinet 1
You open the cabinet 1. The cabinet 1 is open. it, you see vase
> go cabinet
Nothing happens.
go cabinet
The cabinet is closed.
> open cabinet 3
You open the cabinet 3. The cabinet 3 is open. In it, you see tissuebox 2.
> go to cabinet
Nothing happens.
> go to dresser
On the dresser 1, you see a laptop 1, a remotecontrol 1, a tissuebox 1, vase 3, a a watch [Step 3] identify the watch just found and take it
vase 2,
take watch 1 from dresser
You pick up the watch from the dresser 1. [Step 4] go to the safe and put the watch in it
> go to safe 1
The safe 1 is open. In it, you see cellphone 3, and cellphone 2.
> put watch in/on safe 1
You put the watch in/on the safe 1.
F Detailed Evaluation Results on MiniWoB++
Task Name
AdaPlanner
CC-Net
WGE
WebN-T5-3B
RCI
g
choose-list
100
26
click-button
100
100
100
100
100
dlick-button-sqqeene
100
100
100
100
100
click-checkboxes
100
98
100
96
100
click-checkboxes--lrgee
100
71
84
22
94
cfick-checkhoxes-soft
80
95
94
54
72
click-chectboxee--ranfttrr
98
99
64
63
100
Continued on next page.
43

---

click-collapsible
100
100
100
100
2ick-collapsible-2
84
99
click-color
100
100
100
27
100
click-dialog
100
100
click-dialog-2
100
100
100
24
100
click-link
98
100
100
100
click-menu
78
94
n/a
37
100
click-option
100
99
100
87
100
click-scroll-list
100
n/a
100
click-shades
100
100
99
100
chick-shape
click-tab
100
100
100
74
100
click-tab-2
85
18
click-tab-2-hard
78
98
n/a
12
76
click-test
100
click-test-2
100
100
100
100
100
click-widget
100
100
93
100
98
count-shape
50
85
76
41
40
email-inbox
98
100
99
38
98
mnar---nboox-oorarr--l
100
100
n/a
60
100
momil-nbox--orvvad-lll---
100
100
n/a
33
94
email-inbox-nl-turk
90
100
93
23
98
enter-date
100
100
96
96
enter-password
98
100
100
97
100
enter-text
98
100
100
89
100
enter-text-dynnmii
96
100
100
enter-time
96
97
90
100
focus-text
100
focus-text-2
94
100
100
100
100
grid-coordinate
100
100
100
49
100
guess-number
88
100
0
20
identify-shape
96
100
100
88
76
login-user
100
100
100
82
100
login-user-popuy
98
100
n/a
72
68
multi-layouts
84
100
100
83
multi-orderings
100
100
100
88
100
navigate-tree
82
search-engine
100
100
99
34
100
simple-algehra
n/a
100
social-media
82
90
100
21
98
social-media-Be
100
1
100
social-media-soon some
90
85
42
90
terminal
98
0
n/a
n/a
100
tic-tac-toe
48
83
47
48
56
use-autocomplete
88
100
98
22
58
use-spinner
90
100
4
88
Table 8: Per-task success rate (%) of AdaPlanner, CC-Net (7]. WGE (I], WebN-T5-3B 44, and
RCI [8] "n/a" signifies that the corresponding success rate is not reported in the original paper of the method. The nine tasks with feedback marked in gray.
44


Reviews:
[{"review_id": "806CjWOQ63", "reviewer": "Reviewer_4ZaS", "summary": "This paper looks at explicit closed-loop systems with LLMs for adaptive planning utilizing environmental feedback. They showcase better planning performance on ALFWorld and MiniWOB++ environments over existing state-of-the-art works like ReAct and Reflexion.", "questions": "1. Given that the LLM seems to verify the plan to determine its feasibility, what was its efficiency in those assessments? Are there any results pertaining to that?\n\n2. Is there any classification of the tasks with respect to their hardness?\n\n3. For how many of these tasks did the human expert demonstration solve the task? \n", "limitations": "The authors have addressed some of the limitations. I have provided some limitations in the weaknesses section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and the experiments are thorough. They present an interesting improvement over the current works like ReAct and Reflexion.\n", "weaknesses": "1. The kind of tasks in these domains don’t seem to have interaction resolution where there are multiple conflicting causal links from the initial to the goal state which have to be resolved (including negative interactions between subgoals). This could also lead to the human demonstrations helping significantly with the  It would be useful to analyze the performance of AdaPlanner specifically in such cases. \n\n2. I think non-ergodic environments could clearly pose danger to such agents. It would be interesting to see how AdaPlanner can perform against ReAct or Reflexion in such environments. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Weakness 1]**\n\nThank you for your insightful comments. We assume “conflicting causal link” is referring to the existence of the non-revocable actions and states.” In MiniWoB++, a significant number of tasks indeed present multiple conflicting causal links. For example, a mistaken click in the task ```login-user-popup``` can directly lead to task failure.  As shown in our evaluation (Appendix 8.5), when provided with a single human demonstration for this task, AdaPlanner achieves a success rate of 98%, which is comparable to the state-of-the-art training-based baselines and outperforms all LLM baselines. We can update our paper to include specific case studies that elaborate on such cases.\n\n**[Weakness 2]**\n\nWe assume the “non-ergodic environments” refer to environments that present irrevocable actions. Within this context, both ALFWorld and MiniWoB++ are non-ergodic. In ALFWorld, several actions, such as ```clean```, ```cool```, and ```heat``` will pose an irreversible effect on the object. Also, most MiniWoB++ tasks encompass non-ergodic properties as several actions, such as clicking a button, are irreversible. Within this context, AdaPlanner has shown good overall performance in comparison with ReAct, Reflexion, and RCI. Our future work will focus on evaluating these approaches in other representative non-ergodic environments. \n\n\n**[Question 1]**\n\nIn AdaPlanner, the proposed code check and out-of-plan refinement determine and improve the feasibility of the generated plan. The following table shows the success rate (%) of AdaPlanner in ALFWorld and MiniWoB++ (in 9 tasks with feedback), ablating the out-of-plan refinement (assertion component) and code check. \n\n| Environments | Baselines | w/o Code Check | w/o assertion|\n|:----------------:|:-----------:|:--------------------:|:---------:|\n|ALFWorld (134 tasks)|80.60|79.85|75.12|\n|MiniWoB++ (9 tasks)|91.11|89.78|77.78|\n\nAs shown in the table, both the code check and out-of-plan refinement contribute to enhancing the success rate, with the latter demonstrating a particularly significant impact on AdaPlanner’s overall performance. This result shows the importance of the proposed assertion mechanism in efficiently validating the current plan's feasibility, thereby facilitating adaptive and robust planning through continuous feedback and refinement.\n\n**[Question 2]**\n\nIn ALFWorld, the tasks are categorized based on the length of the solution trajectories.[1]:\n| Hardness                                                  | Easy                          | Medium                                  | Medium                                 | Medium                                 | Hard                         | Hard                                                |\n| --------------------------------------------------------- | ----------------------------- | --------------------------------------- | -------------------------------------- | -------------------------------------- | ---------------------------- | --------------------------------------------------- |\n| Task                                                      | Pick                          | Clean                                   | Heat                                   | Cool                                   | Examine                      | Pick two                                            |\n| general solution chain                                    | find-> pick -> goto -> put | find->pick->goto->clean->goto->put | find->pick->goto->heat->goto->put | find->pick->goto->cool->goto->put | find->pick->find->examine | find->pick->goto->put->find->pick->goto->put |\n| Average length of AdaPlanner solution trajectories (step) | 10.79                         | 13.45                                   | 17.61                                  | 13.33                                  | 21                           | 20.71                                               |\n\nIn MiniWoB++, we follow the difficulty classification outlined in [2], arranging the 53 tasks into three tiers (easy, medium, and hard) based on the range of success rates. This categorization is illustrated in the following table:\n| Hardness           |     Easy     |    medium   |     hard    |\n|--------------------|:------------:|:-----------:|:-----------:|\n| Success rate range |   [1, 0.9]   |  (0.9, 0.6] |   (0.6, 0]  |\n| Example task       | click-widget | click-tab-2 | count-shape |\n\nWe will update our paper with this specification of hardness classification in the evaluation studies.\n\n**[Question 3]**\n\nIn ALFWorld, each of the six task types is provided with a single human expert demonstration, resulting in a total of 6 expert demonstrations across all 134 tasks. In MiniWoB++, we use 38 human demonstrations spread over 53 tasks. All LLM-based approaches we tested, including Adaplanner, employ different task settings when creating these expert demonstrations, which precludes their direct application to solving tasks in the test set. For instance, an expert demonstration in ALFWorld might involve receptacles and objects that do not align with those in the actual test cases.\n\n"}, {"review_id": "u3CB3RzKz9", "reviewer": "Reviewer_GDYQ", "summary": "The paper presents AdaPlanner, a closed-loop planning method that uses a large language model (LLM) to solve tasks in text-based environments. AdaPlanner operates by decomposing a complex task into manageable sub-goals and predicting environmental feedback for each. During execution, it refines its actions based on the feedback received from the environment. AdaPlanner operates solely via prompting, eliminating the need for a dedicated training phase and reducing its computational cost. The paper demonstrates that AdaPlanner consistently outperforms existing baselines, achieving state-of-the-art performance in ALFWorld tasks and MiniWoB++ tasks.", "questions": "- In the experiment, what is the proportion of in-plan and out-of-plan occurrences? How will this proportion change over time? This should be a necessary indicator for understanding the two refiners.\n- On MiniWoB++, will there be better performance from GPT-3 than GPT-3.5?\n- Is there still a necessity for AdaPlanner in larger-scale LLMs, such as models like GPT4 with better self-refining capabilities?", "limitations": "- As mentioned above, this paper still needs more experiments and analysis to further validate the rationality of its methods, as well as the observed phenomena and corresponding hypotheses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- AdaPlanner introduces a novel approach to task-solving in text-based environments using a large language model. It stands out for its closed-loop planning method and its ability to decompose tasks into manageable sub-goals.\n- The paper is well-written and clear. The authors have done a good job of explaining complex concepts and methodologies in an understandable manner.\n- The work presents a new way of leveraging large language models for task-solving in text-based environments. The results show that AdaPlanner can effectively leverage feedback to refine its plans and enhance its performance.", "weaknesses": "- The part about skill discovery is not described very clearly, and I still cannot understand the details of the skill discovery module well.\n- The author compared the version without a code interface in the experiment, but it seems that they did not specifically show the prompt after removing the code interface. At the same time, as an ablation experiment, it is also necessary to analyze the effects of specific components in the code interface.\n- The phenomenon that GPT-3 performs better than GPT-3.5 is interesting, but it seems that the paper only compares GPT-3 and GPT-3.5 in Alfworld, without conducting the same experiments in MiniWoB++ to further support the conclusion. And the author's hypotheses about this phenomenon (the smaller scale of GPT3.5) lacks specific analysis or literature references to support it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Weakness 1]**\n\nThank you for your valuable feedback. In the skill acquisition stage, AdaPlanner harnesses adaptive closed-loop planning to solve unseen tasks using limited or no demonstrations. The successful solutions found in this trial-and-error process are called the candidate discovered skills and are gathered in a pool.\n\nDuring the following skill filtering stage, we assess the effect of using each discovered skill from the pool as the demonstration in the prompt to solve similar tasks. If a candidate solution boosts the success rate, it is added to the memory of discovered skills; if not, it is discarded.\n\n**[Weakness 2]**\n\nFor the *prompts without code interface (CI)*:\nThe initial planning phase is prompted as follows:\n\n```\nPlay as a planner. Make a general plan to finish the task by referring to the paradigm that solved a similar task.\n\nHere is an example:\n\n<example>\n\nYou need to generate a new plan by transferring the given paradigm to the new task.\n\n[GIVEN PARADIGM]\n\n<paradigm>\n\n[NEW TASK]\n\n<task>\n\n```\n\nThe <example> contains human demonstrations of general plans as follows. Note that the question marks like (id?) indicate the information that will be completed through feedback. \n\n```\nYour task is to: heat some egg and put it in diningtable.\n\n1. Search_and_find egg (id?).\n2. Take egg (id?) from the place it was found (name? id?).\n3. Go_and_heat egg (id?) to microwave 1 to heat the egg (id?).\n4. Go_and_put egg (id?) in/on the diningtable 1.\n```\n\nFor *Ablation study on code interface components*:\n\nWe conducted the ablation studies on several components of the code interface, such as the assertion that allows for the out-of-plan refinement and the breakpoint start_from that enables the refine-then-resume mechanism.\n\n| Environment | Baseline | w/o Code Check | w/o assertion|\n|:----------------:|:-----------:|:--------------------:|:-------------:|\n|ALFWorld |80.60|79.85|75.12|\n|MiniWoB++ |91.11|89.78|77.78|\n\nThe table above shows the success rate (%) ablating ablating assertion and the breakpoint ```start_from```. In the table, removing any component results in a drop in performance, which signifies the effectiveness of these components in AdaPlanner's design. Notably, when the code interface is fully substituted with natural language, as previously detailed, there is a significant reduction in the performance across both environments. This significant performance drop underscores the essential role of the code interface in AdaPlanner.\n\n**[Weakness 3]**\n\nWe evaluated AdaPlanner and RCI on gpt-3.5-turbo and compared the results with those obtained on text-davinci-003:\n\n| Method                       | With Feedback (9 tasks) | No Feedback (44 tasks) | All (53 tasks) |\n|-------------------------------|-------------------------|------------------------|----------------|\n| RCI (gpt-3.5-turbo)           | 70.89                   | 76.36                  | 75.43          |\n| RCI (text-davinci-003)        | 81.56                   | 92.68                  | 91.00             |\n| AdaPlanner (gpt-3.5-turbo)    | 75.56                   | 78.05                  | 77.63          |\n| AdaPlanner (text-davinci-003) | 91.11                   | 93.22                  | 92.87          |\n\nThe table above shows the success rate (%)  on two subsets of tasks in MiniWoB++. We observe a noticeable performance drop of both methods on gpt-3.5-turbo, which coincides with the results obtained on ALFWorld (Table 2). \n\nWe also found a similar hypothesis drawn based on various experiments in [7]. Given that OpenAI does not publicly disclose the technical details or source code for its GPT models, a definitive validation of our hypothesis remains challenging. As indicated in OpenAI documentation, the gpt-3.5-turbo model is primarily optimized for human conversation tasks, which could potentially compromise its performance on tasks such as code generation and reasoning [8]. \n\n\n**[Question 1]**\n\nWe investigated the occurrence (average times per task) of both in-plan and out-of-plan refinements with our analysis as shown below: \n| Environment                      | In-plan refinement | Out-of-plan refinement | Proportion (in/out) |\n|-----------------------------------|--------------------|-------------|---------------------|\n| ALFWorld         \t\t\t\t\t| 2.83               | 6.40        | 0.44                |\n| MiniWoB++ \t\t\t\t\t | 0.33               | 1.78        | 0.19                |\n\nThe table illustrates that out-of-plan refinement is invoked more frequently in both environments, emphasizing its important role in AdaPlanner's planning process. The average occurrence for both types of refinements varies between ALFWorld and MiniWoB++. This can be attributed to the differences in the average lengths of the trajectories in ALFWorld and MiniWoB++, which stand at 15.60 and 5.40 steps, respectively.\n\nThe proportion of in-plan to out-of-plan refinements is task-specific. In tasks with extensive environmental observations in natural language, in-plan refinement might be more prevalent. On the other hand, analyzing this proportion 'over time' within a task may be challenging because neither type of refinement occurs densely enough to make such a temporal analysis meaningful. \n\nWe will include these findings and discussions in the updated version.\n\n\n**[Question 2]**\n\nYes. Please refer to our response above on *Weakness 3*.\n\n**[Question 3]**\n\nWe would like to emphasize that the proposed closed-loop structure and adaptive refinement in AdaPlanner are fundamentally compatible with a wide range of LLMs. AdaPlanner provides such a robust closed-loop planning framework and is supposed to assist larger-scale LLMs in further enhancing their performance.\n\nHowever, we haven't been granted access to the GPT-4 API at this time. Once it becomes available to us, we will carry out additional evaluations of AdaPlanner on GPT-4 and will update our paper to reflect any significant findings or discussions that arise from this analysis."}, {"review_id": "qJj3kKZyl2", "reviewer": "Reviewer_oy34", "summary": "Briefly summarize the paper and its contributions. This is not the place to critique the paper; the authors should generally agree with a well-written summary.\n\nThe paper proposes AdaPlanner, an LLM-based adaptive planner for text-based sequential decision-making tasks. The planner is adaptive in the sense that it can refine the generated plan/policy based on feedback. \n\nThe contributions made in this paper include the following\n1. interacting with the environment with LLM in the loop\n2. a code-style prompt is engineered for LLMs to output a policy \n3. refining the LLM policy for the current task based on feedback\n4. prompt tuning for new tasks based on previous interaction (termed skill discovery)\n\nThe proposed AdaPlanner is evaluated on two text-based sequential decision-making environments ALFWorld and MiniWoB++. Their experiments indicate that with feedback, LLMs can adapt the plan.\n", "questions": "\n**Major**\n\n1. How is the programmatic response from LLM converted to action responses? Did the conversion require manual intervention? For instance, Figure 2 has an indentation error which would result in a wrong plan. Were such indentation errors evaluated manually? Can authors provide a list of errors made by LLMs? \n1. In line 167, what does an alignment between ‘anticipated plan’ and environment mean? How does the AdaPlanner observe the alignment? \n1. Can authors provide details about the size of the task used in the prompt (for samples) vs the size of the task that was successfully solved by AdaPlanner? To establish the claim of sample efficiency, it is important to understand if the planner is able to efficiently plan for tasks that are significantly different from the prompts.\n1. The X-axis in Figure 3 indicates `# Samples per task`. Is this the number of samples provided for each trajectory? Or sum?  \n1. What was the length of plans or length of trajectories generated by AdaPlanner vs other approaches? To claim the effectiveness of the AdaPlanner, it is important to compare the length of successful trajectories.\n1. For skill discovery, how is the solution converted to the skill? How are skills represented? How large is the skill memory?  Were the discovered skills included in the count of samples used for training as they are training samples for the next set of trajectories?\n1. It is not clear how skills are filtered and what criteria are used for the evaluation and ranking of skills.\n1. What is the connection between skill discovery and prompt tuning?\n1. The success rate of \"With SD\" in Figure 4d looks significantly reduced from  Figure 4a. Were different settings used for theses experiments?\n1. At various places, the paper mentions \"environment feedback\". In my opinion, this is a misnomer. The feedback is not from the environment. The environment just provides the next observation, the feedback is generated by the agent itself. And the use of observation to refine a plan or next action is quite standard practice in RL. I would highly recommend dropping the term feedback from the title. \n1. The use of term plan and policy is a little confusing. A plan is a sequence of actions. A policy is a mapping from states to actions. By this definition, the `solution()` function is as a policy. In preliminaries, the planning policy ($\\rho$) is conditioned on a previous plan $P_t$. However, the appendix describes the refinement prompt using the assertion error (instead of `solution()`). Isn't the assertion error providing information about the policy (the `solution()` function)? So I am confused by the terminologies. Is the $\\rho$ refined conditioned on the policy or the plan? The usage of these terms is also confusing in the Preliminary section. Request authors to precisely define the mathematical notations and highlight what they represent in the examples.\n\n**Minor**\n\n12. In line 387, there are extra curly braces.\n12. The notation $\\rho$ is used in line 73 but introduced much later.\n12. As the context $c_t$ is defined as a sequence of action and observations from time step $0$ to $t$, it is not clear what $c_{>t}$ means (in line 116).  \n12. Open-Loop system in Figure 1 should have an arrow going from env to planner with $o_1$.\n12. Statement in Line 144 \"To generate a plan ..\" looks like a repetition of Line 141 \"To generate an initial plan...\"\n12. In line 116, if $h_t$ is obtained from $c_t$ then would it not be captured in $c_{>t}$? An example of $h_t$ would help better understand the proposed update.\n12. In line 73, as $\\rho$ is defined using $\\Delta(A^{T})$. But the length $T$ is not fixed. \n12. In line 73 $\\rho$ is defined where a plan is conditioned only on observation and goal. However, later it is conditioned on the context, plan, and goal. \n\n\n\n", "limitations": "\n* The evaluations are restricted to text-based sequential decision-making problems and task where the inadmissible actions do not cause drastic changes in the environment. On the contrary, inadmissible actions are like no-ops. Further, the paper does not present analysis of plan length. Hence, the analysis is limited to zero risk environments. \n* The claim made in the abstract about skill discovery mechanism enabling agent to plan with fewer task demonstration is not substantiated in the evaluations. Evaluation in Fig. 4d only established improvement in success rate, not sample efficiency. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "\n* The paper is well written.\n* The paper focuses on extremely relevant and signifcant problems. \n", "weaknesses": "* I find the paper lacks significant details. Please see the next section for the list of questions.\n* The paper employs sloppy mathematical notations.\n* The paper lacks the rigor of scientific evaluation. \n* Paper misses all references to LLM-based approaches for planning with PDDL. The one that I find most relevant for code generation is \"Generalized Planning in PDDL Domains with Pretrained Large Language Models, Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B. Tenenbaum, Leslie Pack Kaelbling, Michael Katz”\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Due to the 6000 characters limit, we only address the important questions. For detailed clarification, we will provide them during the discussion period.\n\n**[Weakness 1]** Due to the page limit, we primarily discussed the main components of AdaPlanner: closed-loop structure, code-styled prompting, and skill discovery in the main body. We have included additional details in the supplementary section to provide a more comprehensive understanding.\n\nIn response to the reviewer's list of questions regarding details and notations, we will address each of them separately in the following section. For some of the questions, we can incorporate clarifications in the updated version of our paper.\n\n**[Weakness 2]** In Section 2, we have provided a thorough explanation of the mathematical notations for understanding our method. We have defined planning policies for both open- and closed-loop systems in detail.\n\nFor the comment \"sloppy mathematical notations,\" we would appreciate it if the reviewer could provide specific examples. This will allow us to address the concern accurately and make any necessary revisions.\n\n**[Weakness 3]** We strongly disagree with the statement that \"the paper lacks the rigor of scientific evaluation\". Our evaluation is rigorous in the following aspects:\n- Benchmarks: We provided thorough evaluations spanning two representative and widely-accepted decision-making environments; \n- Baselines: We've contrasted AdaPlanner with prevailing methods and showed our approach outperforms existing methods in success rate and sample efficiency.\n- Ablation Studies: We validated contribution of each component within AdaPlanner. \n- Reproducibility: We've included extensive technical details to ensure strong reproducibility.\n\nWe firmly believe our research upholds rigorous scientific evaluation standards.\n\n**[Weakness 4]** Most of the existing work along this line of research, including [4, 5, 6] is still open-loop with PDDL. The only exception is [3], as the reviewer suggested. But it was impossible for us to cite in our submission because the paper was uploaded to ArXiv on May 18, 2023 --- after the submission deadline.\n\nMeanwhile, there are significant differences between [3] and our work. In [3], the plan formulated on the training set lacks task-specific refinement during the execution of the evaluation task. In contrast, our AdaPlanner dynamically refines the plan and adapts to various feedback. \n\nOur mechanisms in AdaPlanner could be applied to PDDL, but this extension is out of the current paper's scope and could be explored in the future.\n\n**[Q1.1]** As described in line 161-164, we formulated an environment interface that 1) grounds the generated actions from the solution() function to the admissible actions in the environment, and 2) routes the observation from the environment back to the code as a return value. \n\n**[Q1.2]** No. The interface described above automatically carries out this conversion without manual intervention.\n \n**[Q1.3]** Thanks for pointing out the indentation problem in Figure 2. We found that it was a typo made by the authors when making this figure, not the LLM. We will correct this typo in the updated version.\n\n\n**[Q2]** The alignment means the execution of the plan proceeds as expected and no assertion is triggered. This is consistent with our definition of in-plan feedback introduced in line 44-46. Note that there is no in-plan alignment, in-plan refinement is only used for extracting key information from feedback.\n\n**[Q3]** We added evaluation results in MiniWoB++ as follows:\n\n| Task group                                      | With Feedback  | No Feedback  | All |\n|-------------------------------------------------|----------------------------------|---------------------------------|---------------------------|\n| # Tasks solved  | 410.00                           | 2,050.84                        | 2,460.84                  |\n| # Samples in the prompt           | 13                               | 25                              | 38                        |\n| # Sample / # solved tasks (%)             | 3.17                             | 1.22                            | 1.54                      |\n\nIn MiniWoB++, the ratio # Sample / # solved tasks decreases to approximately 3%. This discussion and Figure 3 jointly show AdaPlanner’s sample efficiency. \n\n**[Q4]** ``# Samples per task`` refers to the number of expert demonstrations provided for each task type. \n\n**[Q5]** The comparison of the average trajectory lengths in ALFWorld is presented as follows:\n|   Method   |  Trajectory length (step) |\n|:----------:|:---------------:|\n|    ReAct   |    25.81      |\n|  Reflexion |     18.90      |\n| AdaPlanner |   15.60      |\n\nAdaPlanner generally requires fewer steps to complete the tasks compared to ReAct and Reflexion.\n\n**[Limitation 1]** We conducted an analysis of the trajectory length, detailed in our response to *Question 5*, which underscores the effectiveness of AdaPlanner. \n\nThe generalization to non-zero-risk environments for LLM agents is an interesting future direction. However, this is beyond the scope of our paper, and there is no widely-accepted benchmark for these tasks yet. \n\nWhile we have focused on text-based environments, our method is not limited to them. For example, integrating vision-language models like CLIP could potentially allow AdaPlanner to interact with more visually complex environments. \n\n**[Limitation 2]** We added ablation studies in MiniWoB++. The method with skill discovery requires only 15 samples to outperform the variant without skill discovery, even though the latter used twice as many samples. It is evident that skill discovery enhances sample efficiency.\n| \\# samples                  | 38    | 30    | 20    | 15    | 0     |\n| --------------------------- | ----- | ----- | ----- | ----- | ----- |\n| With SD (%)    | 92.87 | 84.06 | 79.17 | 75.17 | 60.38 |\n| Without SD (%) | 82.40 | 73.58 | 68.70 | 64.70 | 45.47 |"}, {"review_id": "hxcJDpZQqV", "reviewer": "Reviewer_aJpk", "summary": "LLMs have shown success as autonomous agents that make and execute plans in sequential decision problems. Existing methods either make open-loop plans, limiting adaptability to the environment, or closed-loop plans. Existing closed-loop methods, apart from DEPS, keep the plan static but simply modify immediate actions according to environment feedback, leading to potentially sub-optimal policies. The authors introduce AdaPlanner, a closed-loop LLM planner that additionally allows for *plan* refinement during the episode. The success of their method not only relies on this, but additionally code-style prompts and a skill-discovery mechanism for few-shot exemplars. AdaPlanner outperforms existing works while relying on far fewer demonstration examples from similar tasks. ", "questions": "- Line 80, could you define the output of pi, in the same way that you did for the planner?\n- Line 81, shouldn’t it be P_t rather than P_{t - 1}?\n- Lines 114 - 144 I think you’ve repeated the sentence twice.\n- Line 216, what are the 6 task types?\n- Line 132, how is N chosen and what’s its effect on performance?\n", "limitations": "AdaPlanner still requires demonstrations for learning. Would be worthwhile comparing with RL agents trained directly on the task, without any expert demonstrations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Empirically the authors show strong results with respect to sample efficiency and asymptotic performance.\n\n- Many ablations make it easy to understand which components of the model lead to overall success. \n\n- Conceptually simple approach.\n", "weaknesses": "- In the evaluation section, the baselines are glossed over. This makes it hard to comprehend the distinction between their approach and the baselines. \n   - I’d recommend adding some of the Appendix descriptions to the evaluation section, and potentially referencing Table 1 more often.\n\n- The authors use the term ‘hallucination’ a lot but do not define it.\n\n- The authors discuss in- and out-of- plan refiners a lot before providing intuitive examples for when either would be necessary. Could the authors provide more examples earlier on in the paper?\n\n- DEPS appears to be a relevant baseline. Could the authors include it or at least delve deeper into its limitations and why it is not appropriate?\n\n- It appears that the largest contributor to the success of AdaPlanner, over existing approaches, is code style prompts and skill prompts. Wouldn’t it be worthwhile to apply those modifications to existing approaches, like Reflextion (Fig 4), and contrast?\n\n- AdaPlanner prompts the LLM to correct any syntax errors. How important is this? Would be nice to include this ablation.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Weakness 1]**\n\nWe appreciate the reviewer's comments. Due to the space limitation, we postponed the detailed introduction of the baseline to the Appendix 8.2. We evaluated AdaPlanner against a selection of representative baselines, both training-based and LLM-based. \n- In ALFWorld, we compared AdaPlanner to\n  - BUTLER: an imitation-learning baseline for ALFWorld[1];\n  - ReAct [9] and Reflexion [10]: two prominent LLM-based methods classified as *Implicit Closed-Loop Methods*. \n- In MiniWoB++, we evaluated AdaPlanner against \n  - CC-Net [11], WGE [12]: two training-based methods integrating supervised learning with reinforcement learning;\n  - WebN-T5-3B [13]:  a fine-tuned language model;\n  - RCI [2], a *Implicit Closed-Loop Method* for solving MiniWoB++ tasks.\n\nThese methods reflect different perspectives (training-based and LLM-based) and offer a good reference frame for assessing how AdaPlanner enhances adaptive and sample-efficient planning.\n\nWe will update our paper to show the distinction of these baseline and build a more detailed connection to Table 1.\n\n\n**[Weakness 2]**\n\nIn the scope of LLM, “hallucination” commonly refers to *the produced content that is nonsensical or unfaithful to certain sources* [14,15,16,17]. When applied to the context of LLM for decision-making tasks, “hallucination” is specifically defined as two cases: 1) *the generated actions that are inadmissible in the environment*, or 2) *the unfaithful presumptions made by LLM about the environment settings*. \n\nAn example of the first case is provided in *ReAct and Reflexion trajectory of Case 1*, Appendix 8.4.1, where the agent generates illegal actions; the one for the second case is in the *AdaPlanner Trajectory of Case 2*, Appendix 8.4.2, where the agent unfoundedly presumes the location of the watch.\n\n**[Weakness 3]**\n\nWe appreciate the reviewer's comment on clarity. We will add an intuitive example from ALFWorld before the detailed introduction to these two refiners. We will also present Figure 2 earlier in the paper for better illustration.\n\n**[Weakness 4]**\n\nAs shown in Table 1, DEPS relies on a training-based selector and requires additional data collection. It only utilizes past failures to refine its plans. In contrast, AdaPlanner can leverage both past successes and failures. Moreover, unlike AdaPlanner’s in-plan refinement, DEPS does not extract key information from observations, which may make it less adaptive than AdaPlanner. \n\nDEPS was constructed around the OpenAI Codex API, which has been deprecated. DEPS is also primarily designed for Minecraft and poses difficulties for transfer to ALFWorld and MiniWoB++. \n\n**[Weakness 5]**\n\nReAct and Reflexion are notable for their use of natural language-based open-loop/in-plan reasoning and planning. The code interface and skill discovery are two of our major contributions over these methods. Although these two techniques are generally compatible with the existing methods, once these features were incorporated into the existing approaches, the modified variants would convert to new methods towards our method, which no longer accurately reflect the performance of the original methods as described in the papers. This would diminish the significance and novelty of the proposed AdaPlanner with our own techniques.\n\n**[Weakness 6]**\n\nWe added an ablation study on this component as follows:\n| Environment                      | Baseline | w/o Code Check |\n|-----------------------------------|---------------|----------------|\n| ALFWorld            | 80.60          | 79.85          |\n| MiniWoB++ | 92.87         | 91.92          |\n\nThe table above shows the success rate (%) ablating the code check. As shown, the code check contributes to enhancing the overall performance. This improvement is relatively marginal, because of the capability of the LLMs (GPT-3 and GPT-3.5) to generate code almost free of syntax errors. Upon further investigation, we found that only 1.49% of the code generated for ALFWorld contained syntax errors, thus making the code check not the primary contribution to the performance improvement. Instead, as detailed in Figure 4, the main contributions of our study are the proposed closed-loop structure, code-styled prompting, and skill discovery. These mechanisms collaboratively foster adaptive and sample-efficient decision-making.\n\n**[Question 1]**\n\n$\\pi(\\cdot|g, c_t, P_t)$ generates the action at $t+1$ step conditioned on a given plan  $P_t$.\n\n**[Question 2]**\n\nIt should be $P_{t-1}$ as we were discussing the $t$-th step. \n\n**[Question 3]**\n\nThank you for this comment. We will revise the sentence in our updated paper.\n\n**[Question 4]**\n\nThe 6 task types are ```Pick, Clean, Heat, Cool, Examine, Pick two```. We provided details about these tasks in Appendix 8.1 (line 407-415).\n\n**[Question 5]**\n\nThe LLM agent automatically determines the value of $N$ during plan generation. It's not a manually set parameter but rather task-specific.\n\n\n**[Limitations]**\n\nThe skill discovery mechanism can greatly alleviate the need for demonstration. In the ablation studies (Figure 4d), we applied a zero-shot prompting for MiniWoB++ by *omitting any demonstrations*. AdaPlanner then successfully finds feasible solutions over 21 tasks and enhances the overall success rate by 15%. As indicated in Figure 3, Incorporating skill discovery can effectively reduce the number of demonstrations required to achieve satisfactory performance. \n\nGenerally, RL agents using expert demonstrations outperform those without them. In our evaluation, we compared AdaPlanner with several RL-refined imitators, such as BUTLER, CC-Net, and WGE, which employ 100k, 23k, and 10 demonstrations per task, respectively. Comparing AdaPlanner with these baselines underscores AdaPlanner's superior performance and suggests its potential superiority over baselines that do not use demonstrations. \n\nOur future work will focus on enhancing AdaPlanner to perform well even without demonstrations. "}]

Author rebuttal:
**References**

[1] M. Shridhar, X. Yuan, M.-A. Cote, Y. Bisk, A. Trischler, and M. Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021.

[2] G. Kim, P. Baldi, and S. McAleer. Language models can solve computer tasks. arXiv, page
2303.17491v1, 2023.

[3] Generalized Planning in PDDL Domains with Pretrained Large Language Models, Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B. Tenenbaum, Leslie Pack Kaelbling, Michael Katz

[4] Liu, B., Jiang, Y., Zhang, X., et al. 2023. LLM+P: Empowering Large Language Models with Optimal Planning Proficiency. arXiv.

[5] Tom Silver, Varun Hariprasad, Reece S Shuttleworth, Nishanth Kumar, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. Pddl planning with pretrained large language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022.

[6] Vishal Pallagani, Bharath Muppasani, Keerthiram Murugesan, Francesca Rossi, Lior Horesh, Biplav Srivastava, Francesco Fabiano, and Andrea Loreggia. Plansformer: Generating symbolic plans using transformers. arXiv preprint arXiv:2212.08681, 2022.

[7] Ye, Junjie, et al. "A comprehensive capability analysis of gpt-3 and gpt-3.5 series models." arXiv preprint arXiv:2303.10420 (2023).

[8] OpenAI. OpenAI Documentation: GPT-3.5 models. Retrieved May 16, 2023, from https://platform.openai.com/docs/models/gpt-3-5

[9] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023.

[10] N. Shinn, B. Labash, and A. Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023.

[11] P. C. Humphreys, D. Raposo, T. Pohlen, G. Thornton, R. Chhaparia, A. Muldal, J. Abramson, P. Georgiev, A. Goldin, A. Santoro, and T. Lillicrap. A data-driven approach for learning to control computers. arXivProceedings of the 39th International Conference on Machine Learning, Baltimore, Maryland, USA, PMLR 162, 2022, page 2202.08137v2, 2022.

[12] E. Z. Liu, K. Guu, P. Pasupat, and P. Liang. Reinforcement learning on web interfaces using
workflow-guided exploration. In International Conference on Learning Representations, 2018.

[13] I. Gur, O. Nachum, Y. Miao, M. Safdari, A. Huang, A. Chowdhery, S. Narang, N. Fiedel, and A. Faust. Understanding html with large language models. arXiv, page 2210.03945v1, 2022.

[14] Katja Filippova. 2020. Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings. 864–870.

[15] Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On Faithfulness and Factuality in Abstractive Summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 1906–1919.

[16] Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A Controlled Table-To-Text Generation Dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 1173–1186.

[17] OpenAI. 2023. GPT-4 Technical Report. arXiv preprint arXiv: 2303.08774. 


Meta-review: The authors introduce a closed loop planner for textbased environments (miniwob and alfworld) with state of the art results.  The underlying model uses code based prompting.  Crucially the process is both iterative in that it's closed loop and that it performs refinement/corrections of its own plans. There is also a skill acquisition process by which a limited number of human demonstrations initialize the model to explore the action space and determine whether inclusion of discovered paths into the prompt improves success. The proposed approach outperforms previous algorithms that leverage the same underlying model.
Decision: Accept (poster)