Title: AdaPlanner: Adaptive Planning from Feedback with Language Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Large language models

decision making

closed-loop planning
Abstract: Large language models (LLMs) have recently demonstrated the potential in acting as autonomous agents for sequential decision-making tasks. However, most existing methods either take actions greedily without planning or rely on static plans that are not adaptable to environmental feedback. Consequently, the sequential decision-making performance of LLM agents degenerates with problem complexity and plan horizons increase. We propose a closed-loop approach, AdaPlanner, which allows the LLM agent to refine its self-generated plan adaptively in response to environmental feedback. In AdaPlanner, the LLM agent adaptively refines its plan from feedback with both in-plan and out-of-plan refinement strategies. To mitigate hallucination, we develop a code-style LLM prompt structure that facilitates plan generation across a variety of tasks, environments, and agent capabilities. Furthermore, we propose a skill discovery mechanism that leverages successful plans as few-shot exemplars, enabling the agent to plan and refine with fewer task demonstrations. Our experiments in the ALFWorld and MiniWoB++ environments demonstrate that AdaPlanner outperforms state-of-the-art baselines by 3.73% and 4.11% while utilizing 2x and 600x fewer samples, respectively. The implementation of AdaPlanner is available at https://github.com/haotiansun14/AdaPlanner.

AdaPlanner: Adaptive Planning from Feedback with Language Models

Abstract

Large language models (LLMs) have recently demonstrated the potential in acting as autonomous agents for sequential decision-making tasks. However, most existing methods either take actions greedily without planning or rely on static plans that are not adaptable to environmental feedback. Consequently, the sequential decisionmaking performance of LLM agents degenerates with problem complexity and plan horizons increase. We propose a closed-loop approach, AdaPlanner , which allows the LLM agent to refine its self-generated plan adaptively in response to environmental feedback. In AdaPlanner, the LLM agent adaptively refines its plan from feedback with both in-plan and out-of-plan refinement strategies. To mitigate hallucination, we develop a code-style LLM prompt structure that facilitates plan generation across a variety of tasks, environments, and agent capabilities. Furthermore, we propose a skill discovery mechanism that leverages successful plans as few-shot exemplars, enabling the agent to plan and refine with fewer task demonstrations. Our experiments in the ALFWorld and MiniWoB++ environments demonstrate that AdaPlanner outperforms state-of-the-art baselines by 3.73% and 4.11% while utilizing 2x and 600x fewer samples, respectively. The implementation of AdaPlanner is available at https://github.com/haotiansun14/AdaPlanner .

1 Introduction

Large language models (LLMs) have recently emerged as versatile autonomous agents for sequential decision-making in grounded environments. Traditional decision-making methodologies like Reinforcement Learning (RL) require extensive task-specific training data and often lack the ability to generalize across tasks and environments. In contrast, LLMs are pre-trained on massive and diverse textual data, which gives them extensive world knowledge and the ability to reason over the knowledge. This makes them highly versatile and able to handle complex, real-world scenarios that may involve multiple steps of planning and decision-making.

Existing methods that leverage LLMs as autonomous agents for decision-making can be briefly categorized into two groups (Table 1): open-loop systems and closed-loop systems. Open-loop methods [24, 27, 5, 18, 12, 17, 16] rely on pre-determined plans to accomplish the desired task without any feedback adaptation mechanism. On the other hand, closed-loop systems [25, 6, 8, 22, 9, 19, 23] incorporate environment feedback to continuously monitor system behaviors and make refinements and adjustments of the plans accordingly, which therefore is more flexible.

However, both existing open-loop and closed-loop LLM agents have inherent drawbacks. Open-loop systems are computationally cheap and simple; however, they do not consider feedback from the environment and stick to the initial plan, which lack of adaptability, and, thus, can easily generate suboptimal plans. On the other hand, most existing closed-loop methods generate a fixed plan and only update their executing actions upon environment feedback. This causes them to make

sub-optimal decisions that adapt to the environment in the short term but could have detrimental implications for future steps. DEPS [23] is the only exception, a method that modifies its entire plan based on feedback from the environment. However, it requires training a plan selector to choose the most successful plan, which requires a significant amount of task-specific data. As a result, applying this method to different tasks can be challenging.

To address the limitations of existing LLM agents, we propose AdaPlanner, a closed-loop planning method with LLM playing two roles - planner and refiner. The planner decomposes the task into manageable sub-goals and predicts environmental feedback for each. During execution, the refiner distinguishes and responds to two types of environment feedback in-plan feedback is the environmental observation that aligns with the prediction, and out-of-plan feedback is one that deviates from the prediction. For in-plan feedback, the refiner can dynamically query the LLM to perform reasoning and extract key information from in-plan feedback expressed in natural language. This is achieved through a specific action called ask_LLM() , in which the LLM separately parses the observation and obtains information pertinent to subsequent actions. For out-of-plan feedback, the refiner proactively revises the entire plan and resumes to solve the current task from an intermediate point. AdaPlanner's adaptive closed-loop framework alleviates the need for prior knowledge about the feedback structure and permits the agent to instantly adopt a refined plan rather than restarting from scratch in a reset episode. This leads to a more efficient and adaptive decision-making process.

AdaPlanner operates solely via prompting, eliminating the need for a dedicated training phase and reducing its computational cost. Furthermore, AdaPlanner leverages a code-based prompting for precise planning and refinement. The use of code prompts facilitates task decomposition into subgoals and mitigates LLM hallucination during the decision-making process. AdaPlanner also features a skill discovery process, which accumulates successful experiences to guide future planning. This feature further enhances its long-term planning ability and sample efficiency.

We formally define the planning problem with LLM, and introduce open-loop vs. closed-loop control system, which will motivate our method, in Section 2. Each component of the proposed AdaPlanner is specified in Section 3, including code-based prompting in Section 3.1, closed-loop adaptation in Section 3.2, and skill discovery in Section 3.3, and empirically justified in Section 4. The superior performance of AdaPlanner on both ALFWorld and MiniWoB++ demonstrates our proposed adaptive closed-loop framework can effectively enhance planning performance, even when faced with a limited number of samples.

Planner

Environment

Environment

Skill

Memory

Planner

Explicit

Closed-Loop

AdaPlanner

Open-Loop

Planner

Environment

Implicit

Closed-Loop

(

𝑎

1

,

𝑎

2

,

⋯

,

𝑎𝑇

)

(

𝑜

1

,

𝑎

1

,

⋯

,

𝑜𝑡

,

𝑎𝑡

)

(

𝑜

1

,

𝑎

1

,

⋯

,

𝑜𝑡

,

𝑎𝑡

,

𝒐𝒕

+

𝟏

)

(

𝑜

1

,

𝑎

1

,

⋯

,

𝑜𝑡

,

𝑎𝑡

,

𝑜𝑡

+1)

𝑜

1

,

𝑎

1

,

⋯

,

𝑜𝑇

,

𝑎𝑇

,

𝑜𝑇

+1

(

𝑎

1

,

𝑎

2

,

⋯

,

𝑎𝑡

,

𝑎𝑡

+1,

⋯

,

𝑎𝑇

),

𝑎𝑡

+1

′

,

⋯

,

𝑎𝑇

′

)

In-Plan

Refiner

Out-of-Plan

Refiner

𝑜𝑡

+1

aligns

with

prediction

𝑜𝑡

+1

violates

prediction

extract

info

from

𝑜𝑡

+1

𝑜

1

,

𝑎

1

,

⋯

,

𝑜𝑇

,

𝑎𝑇

,

𝑜𝑇

+1

2 Preliminaries

Problem Formulation. We consider adopting an LLM as an autonomous agent to solve different tasks in text-based environments. For initialization, the agent is provided with allowed actions A in the environment, as well as a text-grounded task definition g ∈ G from the task space G . Besides, the initial state of the environment is also observed as o 1 ∈ O from the observation space O . With such inputs, the LLM agent needs first to generate an initial planning policy for solving the task ρ 0 ( P 0 | g, o 1 ) : G × O → ∆( A T ) , where T is the total length of steps in the generated plan and ∆( · ) is probability simplex function. Also, the agent can interact with the environment for feedback: When the agent interacts with the environment at the t -th step, the agent receives an observation o t ∈ O from the environment and generates a trajectory-like context c t = ( o 1 , a ′ 1 , o 2 , a ′ 2 , · · · , a ′ t -1 , o t ) , where a ′ 1 , a ′ 2 , · · · , a ′ t -1 are the executed actions within the environment. As the agent may modify the actions according to the feedback, the executed actions a ′ 1 , a ′ 2 , · · · , a ′ t -1 can be different from the actions a 1 , a 2 , · · · , a t -1 in the initial plan. We denote ρ ( ·| g, c t , P t ) as the high-level planning policy that generates an entire plan and π ( ·| g, c t , P t ) as the action-generation policy conditioned on a given plan P t . Given the context c t and the entire plan at the last step P t -1 , the agent refines future decisions. In the end, the LLM agent should model both the initial planning policy and the environment feedback-conditioned policy to complete the given task successfully.

Open-Loop System. An open-loop system is a non-feedback system (Figure 1), where the output is solely dependent on the input, without any consideration of the environmental feedback. Thus, in an open-loop system, the entire initial plan over the time horizon T is predetermined and static by the initial planning policy ρ 0 ( ·| g, o 1 ) , without any feedback-based refinement. Despite their simplicity, open-loop systems are notably vulnerable to environmental changes, as they lack the capacity to adapt or adjust their plans based on environmental feedback.

Closed-Loop Systems. On the contrary, a closed-loop system (Figure 1) refers to a planning process that incorporates environment feedback to adjust and refine future decisions, involving both initial planning ρ 0 ( ·| g, o 1 ) and two levels of feedback-based refinements, ρ ( ·| g, c t , P t -1 ) and π ( ·| g, c t , P t -1 ) , in the system.

Implicit Closed-Loop Systems. After each step of interaction with the environment, implicit closedloop systems will maintain the initial plan (i.e., P t = P 0 ) and only modify a single action based on the feedback. Therefore, the feedback-based refinement is defined as π ( a ′ t | g, c t , P 0 ) , where a ′ t ∈ A is the modified action from action space, while the remaining actions a >t for future steps remain the same as the initial plan. Although locally-optimal actions are adopted at each step, inaccuracies in initial planning can result in task failure or non-completion.

Explicit Closed-Loop Systems. Explicit closed-loop systems refine the entire plan based on environment feedback following the policy ρ ( P t | g, c t , P t -1 ) , where P t ∈ ∆( A T -t ) is the refined plan at time step t containing the modified future actions a ′ ≥ t to execute and P t -1 is the old plan modified in the previous time step. Allowing for constant refinement and improvement of the plan, explicit closed-loop systems can help prevent costly mistakes or missed opportunities that might arise from adhering to outdated plans. Our proposed AdaPlanner is an explicit closed-loop system.

3 AdaPlanner

Model Architecture. Our AdaPlanner model, shown in Figure 1, consists of two main components:

an LLM-based agent that functions dually as a planner and a plan refiner, and

a skill memory module designed to enhance sample efficiency through skill discovery.

The LLM-based agent, in its planner role, generates a comprehensive plan and performs preliminary assessments to determine its feasibility. The initial planning is modeled as ρ 0 ( P 0 | g, o 1 ) . As the plan unfolds, the agent also operates as a refiner, conducting feedback-based refinement in both in-plan and out-of-plan manners. In-plan and out-of-plan refinement processes primarily differ in how they impact future actions. In-plan refinement is a one-step action that integrates useful information into the existing plan for better action grounding. After this in-plan phase, future actions will be generated using the updated context π ( a ′ >t | g, c >t ∪ { h t } , P 0 ) , where h t represents the new information obtained from c t via in-plan refinement at timestep t . Out-of-plan refinement, on the other hand, leverages environmental feedback to directly revise the entire plan, denoted as ρ ( P t | g, c t , P t -1 ) . This mechanism allows for comprehensive adjustments to be made to the plan in response to unexpected environmental feedback. Skill memory serves as a repository, archiving past successful plans and their respective interactions with the environment. If the agent encounters a task resembling the skills stored in memory, these skills can serve as few-shot exemplars in the LLM agent's prompt. This feature improves not only sample efficiency but also reliability for future planning.

Environment Interaction. AdaPlanner employs adaptive closed-loop planning and active environment interaction for task solving. It can anticipate environmental observations and proactively refine the plan only when there is a discrepancy between expected and actual outcomes. This is achieved by decomposing the planning process into N manageable sub-goals. During the planning and actiontaking process, the agent selects from a set of timestamps, { t 1 , . . . , t N } , to evaluate the success of each sub-goal. If the sub-goal does not align with the planned prediction at timestep t ∈ { t 1 , . . . , t N } , the environment actively sends the previous sub-trajectories ( o 1 , a ′ 1 , · · · , o t , a ′ t , o t +1 ) back to the refiner for plan revision. This process allows the agent to check the success status only at N crucial points, thereby reducing computational costs (number of API calls) and enhancing efficiency.

3.1 Plan Generation via Code-Based LLM Prompting

AdaPlanner plans and refines by using Pythonic code prompts for LLMs. Consistent with previous observations [3, 2], we have found that using code prompts instead of natural language prompts for LLMs reduces ambiguity and misinterpretation, which significantly reduces LLM hallucination during plan generation and refinement. We design code prompts during different stages of decisionmaking, including adaptive planning, feedback generation, and in-episode refinement. We provide a detailed description of the prompts used at each stage in Appendix D.

To generate an initial plan for solving a given task, we input a task description, the permissible actions in the environment, and, when available, sample demonstrations of task resolution into LLM. These pieces of information are all formatted into Pythonic code format for LLM prompting. Figure 2 (a) shows an example programming-based plan generated by AdaPlanner for solving a put task in the ALFWorld environment. The generated solution function is provided with two input arguments: the first is the agent object, which encapsulates environmental information to be used by the agent. The second is the variable start_from , which is a parameter indicating the subgoal from which the agent will later resume its execution with a refined plan. By default, the start_from is initialized as 1. The value of this variable can be further reassigned during the refinement.

When prompting LLM to generate the code-based plan, we design the prompt to teach LLM to decompose a complex task into sub-goals. As shown in Figure 2(a), the generated code plan solution(agent, start_from=1) consists of: 1) a general plan at the outset that decomposes the task into subgoals in the form of comments; and 2) a sequence of sub-plans, each consisting of admissible actions corresponding to a specific subgoal. Such a mechanism allows our method to handle complex, long-horizon tasks by hierarchically decomposing them into sequences of subgoals. Furthermore, each subgoal ends with an assertion statement to test its fulfillment, which allows our method to interact actively with the environment and later resume its execution with a refined plan.

General

plan

(a) Initial Plan

(Extracted)

def

solution

(agent, start_from=3):

# General plan: I need to get a list of receptacles to find the lettuce,

take the lettuce to the sinkbasin, clean it, and put it in a diningtable.

#

[Step 1] Get a list of receptacles where the lettuce is likely to appear.

…

#

[Step 2] Go to each receptacle in the list until seeing a lettuce

…

#

[Step 3] Identify the lettuce I just found and take it

if

start_from <= 3:

found_lettuce = 'lettuce' + ask_LLM(f'From the observation, get

the identifier of the lettuce: {observation}. ')

observation = agent.take(found_lettuce, receptacle)

assert

agent.holding == found_lettuce, f'Error in [Step 3]: I cannot

take {found_lettuce} from the {receptacle}. {agent.report()}'

#

[Step 4] Go to a sinkbasin to clean the lettuce.

if

start_from <= 4:

observation = agent.goto('sinkbasin 1')

observation = agent.clean(found_lettuce, 'sinkbasin 1')

assert

'clean' in observation, f'Error in [Step 4]: I cannot clean

{found_lettuce} using the sinkbasin 1. {agent.report()}'

#

[Step 5] Go to a diningtable and put the lettuce on it.

…

(c) Refined Plan

(Extracted, revisions highlighted)

Error in [Step 3]: I cannot clean lettuce 2

using the sinkbasin 1. I am at toilet 1 and

holding None.  The last three interactions

before the error were:

Act: go to sinkbasin 2

Obs: On the sinkbasin 2, you see nothing.

…

(b) Out-of-plan Feedback

(Extracted)

AdaPlanner

Environment

Execution

Out-of-plan

refine

Halt at

[Step 3]

Sub-plan

Predict

via assert

def

solution

(agent, start_from=1):

#

General plan: I need to get a list of receptacles to find the

lettuce, clean it, and put it in a diningtable.

#

[Step 1] Get a list of receptacles where the lettuce is likely

to appear.

if

start_from <= 1:

recep_to_go= literal_eval(ask_LLM(f'Sort

{agent.receptacles} in descending order based on the

likelihood of finding a lettuce.'))

assert

recep_to_go, f'Error in [Step 1]: recep_to_go

should not be empty. {agent.report()}'

#

[Step 2] Go to each receptacle in the list until seeing a

lettuce

if

start_from <= 2:

for

receptacle

in

recep_to_go:

observation = agent.goto(receptacle)

if

'lettuce' in observation:

break

assert

'lettuce' in observation, f'Error in [Step 2]: There

is no lettuce in/on {recep_to_go}. {agent.report()}'

#

[Step 3] Identify the lettuce I just found and clean it

if

start_from <= 3:

found_lettuce = ' lettuce' + ask_LLM(f'From the

observation, get the identifier of the lettuce:

{observation}.')

observation = agent.clean(found_lettuce, 'sinkbasin 1')

assert

'clean' in observation, f'Error in [Step 3]: I cannot

clean {found_lettuce} using the sinkbasin 1.

{agent.report()}'

#

[Step 4] Go to a diningtable and put the lettuce on it.

…

In-plan

refine

In-plan

refine

Figure 2: An illustrative example from ALFWorld to show the proposed adaptive closed-loop planning through code. The task is to put some clean lettuce on the diningtable. The in-plan feedback in (a) is a sentence like On the countertop 2, you see a knife 1, a lettuce 1, a saltshaker 2, and a soapbottle 1. This feedback is managed by the ask_LLM() action. The execution of the initial plan might yield misaligned observations, triggering an outof-plan feedback and refinement process. For instance, the agent cannot clean the lettuce if it is not currently located at a sinkbasin . The out-of-plan feedback in (b) assists AdaPlanner in generating a revised plan (c) so that the agent will move to a sinkbasin before cleaning the lettuce . AdaPlanner then determines to resume from step 3 within the same episode. The task can be successfully completed using the refined plan.

3.2 Adaptive Closed-Loop Plan Refinement

Once an initial plan is generated, AdaPlanner then prompts the LLM to correct any syntax errors. After this, the code undergoes execution through the environment interface. The interface is responsible for grounding the actions in the environment, and also for routing environmental observations back to the code as a return value. This bi-directional flow allows AdaPlanner to adapt and refine its plan in response to environmental observations in a closed-loop manner.

In-Plan Feedback and Refinement via ask_LLM() Action. When AdaPlanner observes that the environment is aligned with the anticipated plan, it performs in-plan refinement. This allows it to extract useful information from the observation that can be used for upcoming actions. To achieve this, we provide the agent with an additional action called ask_LLM() , which is used to formulate a plan alongside task-specific actions. The ask_LLM() function enables AdaPlanner to self-query and perform reasoning based on specific information parsed from environmental observations. For instance, in [Step 3] in Figure 2 (a), the ask_LLM() action extracts the identifier of the found object lettuce from the natural-language observation. This information can then be fed into later actions. As an additional atomic action, this in-plan refinement is integrated into the plan at any point where the planner deems a reasoning process is necessary. Existing code-generation-based methods [22, 9, 3] face a challenge in this task, especially when there is no prior knowledge of the structure and organization of these feedback sentences. In contrast, our AdaPlanner method leverages LLM to parse critical information from diverse feedback presented in natural-language sentences to streamline plan execution.

Out-of-Plan Refinement with the Refine-Then-Resume Mechanism. After each sub-plan execution, AdaPlanner actively checks an assertion condition to ensure that the current plan is proceeding as expected. If the assertion fails, AdaPlanner performs out-of-plan refinement. For example, in Figure 2 (a), after [Step 3] , the agent is expected to hold lettuce . If this condition is not met, AdaPlanner generates an error message that details the current progress of execution gathered by the report() function. In ALFWorld tasks, this function provides a report of the agent's location, the object it is holding, and the last three interactions with the environment, as shown in Figure 2 (b). AdaPlanner then utilizes this information to perform out-of-plan refinement.

During the out-of-plan refinement as in Figure 2 (c), AdaPlanner uses a prompt similar to the one used during the initial planning stage, but with an additional feedback message that reflects the current state. Detailed prompts are provided in Appendix D. AdaPlanner then refines the plan based on the newly acquired information and also determines the value of start_from by comparing the plan before and after the refinement. The newly refined solution() is then executed from the breakpoint start_from . This breakpoint contains all variable states that were saved prior to refinement. Consequently, the current episode can continue from an intermediate checkpoint without restarting from scratch. We call this mechanism refine-then-resume . It significantly speeds up task completion and reduces the number of LLM calls required.

3.3 Skill Discovery

Acquiring expert demonstrations for task solving can be costly, particularly as the number of tasks increases. To address this issue, we have equipped AdaPlanner with a skill discovery feature. This is a memory scheme that discovers and archives successful trajectories, thereby improving planning performance when dealing with similar tasks. The skill discovery process consists of two stages, which can be conducted alternately over several rounds, based on the interaction costs and computation resources.

Skill Acquisition. In the first stage, AdaPlanner attempts to solve unseen tasks, leveraging a limited number of human demonstrations of other simpler tasks, or even no demonstrations. The model capitalizes on adaptive closed-loop planning to iteratively explore and refine solutions via a trial-anderror approach. Upon successful completion of a given task, the latest solution and the corresponding interactions are treated as candidate discovered skills.

Skill Filtering. In the second stage, we compare the planning performance with and without the integration of the discovered solution into the prompt. If the inclusion of this solution boosts the success rate, it is archived as a discovered skill. Conversely, if it does not improve performance, it is discarded. This filtering stage is crucial because the iterative closed-loop refinement may integrate episode-specific information into the revised solution, potentially compromising its generalizability.

4 Evaluation

We test AdaPlanner on two text-based decision-making environments: 1) ALFWorld [20] is a text-based virtual household environment encompassing six distinct task types set. We evaluate AdaPlanner on a total of 134 tasks across these six types. 2) MiniWoB++ [11] is a simulation environment that covers a large range of computer tasks. We select 9 MiniWoB++ tasks with environmental feedback, and we also adopt and test the 53 tasks evaluated in RCI [8]. Both environments aim to solve complicated challenges with long-horizon solutions and sparse rewards. We also carefully designed ablation studies to justify the significance of each component in AdaPlanner. The Setup details and prompts for AdaPlanner are depicted in Appendix A and D. Detailed introductions to each baseline are presented in Appendix B Note that we evaluate different baselines for these two benchmarks. These methods utilize task-specific samples for prompting or training purposes, thus necessitating separate evaluations for each benchmark.

Metrics. Consistent with previous works [20, 25, 19, 7, 4, 11, 8], we use success rate (%) to evaluate the performance of tested methods. The success rate is defined as the number of successful episodes over the total number of episodes. Note that in ALFWorld, failure of an episode occurs when the total number of actions attains 50, with the task still unsolved. In MiniWoB++, failures can occur in two scenarios: either due to the execution of invalid actions or if the task remains unfinished following the execution of the entire plan.

Main Results. AdaPlanner consistently outperforms the existing baselines, achieving state-of-the-art performance, i.e. , an overall success rate of 91.79% in ALFWorld tasks (Table 2) and 91.11% in

Method

Pick

Clean

Heat

Cool

Examine

Pick two

All (134 tasks)

Training-Based Methods

BUTLER [20]

46.00

39.00

74.00

100.00

22.00

24.00

37.00

Implicit Closed-Loop Methods with Fixed Plan

ReAct [25] (GPT-3)

66.67

41.94

91.03

80.95

55.56

35.29

61.94

ReAct [25] (GPT-3.5)

37.50

64.52

69.57

42.86

38.89

17.65

47.76

Reflexion [19] (GPT-3 + 3.5)

75.00

90.32

91.30

90.48

88.89

94.12

88.06

Reflexion [19] (GPT-3.5)

50.00

41.94

65.22

52.38

66.67

47.06

52.99

Explicit Closed-Loop Methods with Plan Refinement

AdaPlanner (GPT-3)

100.00

96.77

95.65

100.00

100.00

47.06

91.79

AdaPlanner (GPT-3.5)

77.78

93.55

69.57

93.65

62.96

78.43

80.60

MiniWoB++ tasks with feedback (Table 3). Specifically, in ALFWorld, AdaPlanner equipped with GPT-3 achieves a remarkable success rate exceeding 95% in the majority of individual tasks. It also surpasses all other baselines in the Pick , Clean , and Examine tasks. Notably, even in the task with the lowest performance ( Pick two ), AdaPlanner still outperforms BUTLER and ReAct. In the MiniWoB++ environment, AdaPlanner demonstrates superiority over all other methods on tasks that provide feedback. This superior performance suggests that AdaPlanner effectively leverages feedback to refine its plans and enhance its performance. Furthermore, AdaPlanner maintains competitive performance on tasks without feedback, achieving a success rate of 93.22%. Note that AdaPlanner's success rates of tasks without feedback are still comparable to CC-Net, the state-of-the-art model requiring over 23,000 samples per task. This result highlights the efficacy of the programming-based planning strategy employed by AdaPlanner. In both environments, AdaPlanner consistently delivers superior or competitive performance when compared to not only training-based methods but also implicit closed-loop methods under the same LLM models. These results affirm the effectiveness of the proposed explicit closed-loop plan refinement in AdaPlanner.

Furthermore, we summarize the relationship between success rate (%) and the number of samples in Figure 3. In ALFWorld, AdaPlanner yields the highest performance with the fewest number of samples. In MiniWoB++, our method outperforms most baselines. Notably, our method achieves performance comparable to CC-Net but requires 600 times fewer samples. This study highlights that AdaPlanner significantly reduces the need for extensive demonstrations or expert trajectories, thereby offering a more resource-efficient solution.

10

1

10 1

10 3

10 5

40

60

80

100

Success rate (%)

BUTLER

ReAct

Reflexion

AdaPlanner

(a) ALFWorld

10

1

10 1

10 3

10 5

40

60

80

100

Success rate (%)

CC-NET

WGE

RCI

AdaPlanner

(b) MiniWoB++

Adaptive Closed-Loop Architecture Enhances Planning Performance. Figure 4a shows the performance v.s. the number of closed-loop refinements, under settings with different numbers of demo samples. The detailed example selection for this study is provided in Appendix A. We observe a significant trend of increased success rates corresponding to each subsequent closed-loop plan refinement. This indicates the AdaPlanner's ability to consistently leverage real-time feedback for performance enhancement, regardless of the number of samples used. Remarkably, AdaPlanner

maintains this trend of success rate enhancement even when the total number of demonstrations across all six tasks is as low as two. In addition, as displayed in Table 4, AdaPlanner consistently outperforms both ReAct and Reflexion in producing the shortest trajectory lengths across all tasks in ALFWorld. Moreover, a comparison with Reflexion, depicted in Figure 4b, shows AdaPlanner's consistently superior performance across all iterations of closed-loop corrections. These observations highlight AdaPlanner's sample efficiency and its potential for real-world applications where the number of available demonstrations is limited.

Code Interface Mitigates Hallucination. The latest gpt-3.5-turbo is reported to be the most capable GPT-3.5 model while reducing the cost by a tenth compared to other prevailing GPT-3 [1] and 3.5 models [14] ( e.g. , text-davinci-002 and text-davinci-003 .) However, our findings from Table 2 indicate that gpt-3.5-turbo underperforms in decision-making tasks relative to its predecessors, i.e., text-davinci-002 , in all LLM-agents. Upon examination of trajectories from both models, we observed a noticeable hallucination with GPT-3.5 ( gpt-3.5-turbo ), as shown in Appendix E. We hypothesize that gpt-3.5-turbo might be a smaller-scale model that is more prone to hallucination. We also found a similar hypothesis drawn based on various experiments in [26]. Furthermore, gpt-3.5-turbo is primarily optimized for human conversation tasks, which could potentially compromise its performance on tasks such as code generation and reasoning[13]. Despite this, AdaPlanner demonstrates a remarkable level of resilience against hallucination even with gpt-3.5-turbo (Table 2), while ReAct and Reflexion are more sensitive to the hallucination issue. AdaPlanner's resilience against hallucination can be attributed to its use of code prompts, which provide a more formal and constrained generation space for LLM. For comparison, we implement an ablation version of AdaPlanner without the code interface by translating solution examples directly into plans and actions using natural language. Without the code interface, AdaPlanner's performance substantially drops in both ALFWorld and MiniWoB++ environments (Figure 4c), from 81% to 46% and from 93% to 66%, respectively. This significant performance drop underscores the essential role of the code interface in AdaPlanner.

0

1

2

3

4

5

# Closed-loop corrections

67.5

70.0

72.5

75.0

77.5

80.0

Success rate (%)

2

4

6

(a)

0

1

2

3

4

5

# Closed-loop corrections

40

50

60

70

80

90

Success rate (%)

AdaPlanner (GPT-3)

AdaPlanner (GPT-3.5)

Reflexion (GPT-3+3.5)

Reflexion (GPT-3.5)

(b)

ALFWorld

MiniWoB++

0

20

40

60

80

100

Success rate (%)

46%

66%

81%

93%

Without CI

With CI

(c)

ALFWorld

MiniWoB++

0

20

40

60

80

100

Success rate (%)

19%

45%

38%

60%

Without SD

With SD

(d)

Skill Discovery Improves Sample Efficiency. The skill discovery in AdaPlanner utilizes a long-term memory mechanism that retains successful solutions, thus boosting planning performance when faced with similar tasks. An ablation study depicted in Figure 4d compares the performance of AdaPlanner with and without the implementation of skill discovery. In the skill acquisition stage, we provide a maximum of one demonstration. In ALFWorld, AdaPlanner is prompted with only one expert demonstration of the simplest task ( put ). We evaluate the average success rate of the method on the remaining five tasks, which are comparatively more challenging and require additional steps for completion. In MiniWoB++, we apply zero-shot prompting, omitting any examples in the skill acquisition phase. For both environments, we operate the method using GPT-3.5 in adaptive closed-loop mode, and one round of skill discovery is conducted. As Figure 4d illustrates, the inclusion of skill discovery significantly enhances performance. In the ALFWorld tasks, the success rate of AdaPlanner nearly doubles when skill discovery is employed. Similarly, in the MiniWoB++ tasks, the overall success rate increases by approximately 15% with skill discovery. Moreover, we conducted an ablation study on MiniWoB++ to evaluate the impact of sample numbers, as detailed in Table 5. AdaPlanner with skill discovery requires only 15 samples to outperform the variant without skill discovery, even though the latter used twice as many samples. It is evident that skill discovery enhances sample efficiency.

5 Related Work

Many works have studied how to leverage LLMs as autonomous agents to accomplish decisionmaking tasks within text-based environments. Earlier studies, like Chain-of-Thoughts [24] and Zero-Shot Planner [5], utilize prompts to guide LLMs in generating complete action sequences for elementary tasks. For more complex tasks, methods like HuggingGPT [18] and Chameleon [12] also generate the initial plan of using different tools and then call the corresponding APIs for execution. Meanwhile, some other works also prompt LLMs to compose plans in Planning Domain Definition Language (PDDL) [10, 15]. However, all these plans are created in an open-loop fashion without adapting to feedback from external environments.

To address the limitations of open-loop systems, recent techniques have emerged that focus on establishing closed-loop systems. These systems are capable of leveraging environmental feedback, thereby facilitating more adaptive decision-making. ReAct [25] and Inner Monologue [6] allow LLM agents to take single-step actions according to the environmental feedback. Reflexion [19], as an extension of ReAct, tries to resolve this issue by enabling the ReAct agent to revise itself from past trials and errors. Moreover, RCI [8] starts by formulating a comprehensive plan, modifying the immediate action when the agent encounters a failure at the current step. While all the aforementioned

methods can adapt their decisions based on environmental feedback, they assume the LLM-generated initial plan is correct and do not adjust it. Rather, they solely modify the immediate action being executed and are easy to fall into local sub-optimal actions without considering the long-term plans.

To further enhance the agents' both capabilities of planning and adapting to environmental feedback, strict closed-loop architectures are proposed that can recursively refine the generated plans. DEPS [23] is one of the examples that initially proposes an entire plan and then applies real-world feedback to recursively refine it during execution. However, this method requires training a selector to generate a plan that is highly probable to succeed, which makes it difficult to generalize the plans and actions to other tasks. Besides, the required data for training the plan selector are often unavailable in practice and expensive to collect. In contrast, AdaPlanner generates and refines plans via LLM prompting, making it widely applicable to various decision-making problems. Another PDDL-based planning method is proposed in [21] with the ability to leverage feedback and modify the plans. However, the plan formulated on the training set remains static and lacks task-specific refinement during the execution of the evaluation task. In contrast, AdaPlanner dynamically refines the plan and adapts to various feedback throughout the task-solving process.

6 Conclusion and Limitations

We proposed AdaPlanner, a closed-loop approach enabling LLM agents to adaptively refine their generated plans according to environment feedback. We defined two different refinement strategies, in-plan and out-of-plan refinement, to fully leverage environment information. Furthermore, to mitigate the LLMs' hallucination issue and make them learn from the past experience, we proposed code-style prompting and skill discovery mechanisms. Through comprehensive experiments, we demonstrated that AdaPlanner outperforms the state-of-the-art baselines significantly and has better sample efficiency. Our ablation studies also showed the effectiveness of different components in AdaPlanner. One limitation of AdaPlanner is that it still require few-shot expert demonstrations for solving complex tasks. Although AdaPlanner has already achieved better sample efficiency than existing methods, it is interesting to study how to further enhance AdaPlanner to solve complex tasks with no demonstrations in the future.

7 Broader Impacts

Our research approach focuses on treating LLMs as autonomous agents and improving their ability to solve complex sequential decision-making tasks. However, this research line carries inherent risks, including security threats, potential misuse, and unintended consequences such as job displacement due to automation. To mitigate these risks, it is essential for researchers and policymakers to collaborate in creating and implementing effective regulations to guide the development and deployment of these technologies towards positive outcomes. Additionally, we believe that the research community should coordinate efforts to design principles and techniques that prioritize safety and human values before LLM agents are deployed in various industries. This will help ensure that LLMs are aligned with ethical and moral standards while promoting their positive impact on society.

Acknowledgments and Disclosure of Funding

This work was supported in part by NSF (IIS2008334, IIS-2106961, CAREER IIS-2144338), ONR (MURI N00014-17-1-2656), IDEaS Cyberinfrastructure Resources, and Microsoft Accelerate Foundation Models Research Program.

References

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

W. Chen, X. Ma, X. Wang, and W. W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv , page 2211.12588v3, 2022.

L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. Pal: Programaided language models. arXiv , page 2211.10435v2, 2022.

I. Gur, O. Nachum, Y. Miao, M. Safdari, A. Huang, A. Chowdhery, S. Narang, N. Fiedel, and A. Faust. Understanding html with large language models. arXiv , page 2210.03945v1, 2022.

W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv , page 2201.07207v2, 2022.

W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, P. Sermanet, N. Brown, T. Jackson, L. Luu, S. Levine, K. Hausman, and B. Ichter. Inner monologue: Embodied reasoning through planning with language models. arXiv , page 2207.05608v1, 2022.

P. C. Humphreys, D. Raposo, T. Pohlen, G. Thornton, R. Chhaparia, A. Muldal, J. Abramson, P. Georgiev, A. Goldin, A. Santoro, and T. Lillicrap. A data-driven approach for learning to control computers. arXivProceedings of the 39th International Conference on Machine Learning, Baltimore, Maryland, USA, PMLR 162, 2022 , page 2202.08137v2, 2022.

G. Kim, P. Baldi, and S. McAleer. Language models can solve computer tasks. arXiv , page 2303.17491v1, 2023.

J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. arXiv , page 2209.07753v3, 2022.

B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone. Llm+p: Empowering large language models with optimal planning proficiency. arXiv , page 2304.11477v1, 2023.

E. Z. Liu, K. Guu, P. Pasupat, and P. Liang. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations , 2018.

P. Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y. N. Wu, S.-C. Zhu, and J. Gao. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv preprint arXiv:2304.09842 , 2023.

OpenAI. Openai platform documentation: Gpt-3.5 models (retrieved on may 16, 2023). https://platform.openai.com/docs/models/overview.

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

V. Pallagani, B. Muppasani, K. Murugesan, F. Rossi, L. Horesh, B. Srivastava, F. Fabiano, and A. Loreggia. Plansformer: Generating symbolic plans using transformers, 2022.

A. Parisi, Y. Zhao, and N. Fiedel. Talm: Tool augmented language models. arXiv preprint arXiv:2205.12255 , 2022.

T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761 , 2023.

Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580 , 2023.

N. Shinn, B. Labash, and A. Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366 , 2023.

M. Shridhar, X. Yuan, M.-A. Cote, Y. Bisk, A. Trischler, and M. Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations , 2021.

T. Silver, S. Dan, K. Srinivas, J. B. Tenenbaum, L. P. Kaelbling, and M. Katz. Generalized planning in pddl domains with pretrained large language models, 2023.

I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. In Second Workshop on Language and Reinforcement Learning , 2022.

Z. Wang, S. Cai, A. Liu, X. Ma, and Y. Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv , page 2302.01560v1, 2023.

J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chainof-thought prompting elicits reasoning in large language models. arXiv , page 2201.11903v6, 2022.

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations , 2023.

J. Ye, X. Chen, N. Xu, C. Zu, Z. Shao, S. Liu, Y. Cui, Z. Zhou, C. Gong, Y. Shen, J. Zhou, S. Chen, T. Gui, Q. Zhang, and X. Huang. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models, 2023.

D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. Le, and E. Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv , page 2205.10625v2, 2022.

A Experimental Setup

ALFWorld [20] is a comprehensive suite of synthetic, text-based environments, encompassing six distinct task types set, Pick , Clean , Heat , Cool , Examine , and Pick two , within a virtual household. Each task possesses a unique high-level objective (e.g., put some vase in safe , etc. ) that necessitates agent navigation and interaction with various objects or receptacles (e.g., go to shelf 6 , clean apple , etc. ). To fulfill the stipulated task, the agent is required to implement a sequence of actions aimed at accomplishing the predetermined goal. However, given that an object may potentially reside in any one of over 50 possible locations in a task instance, the agent must sequentially explore each of these. Consequently, the entire action trajectory could involve more than 50 individual actions, presenting a significant challenge to the agent.

MiniWoB++ [11] is a task suite of simulation environments that covers a large range of computer tasks for net agents. The computer tasks start from simple button-clicking to more challenging ones with longer time horizons ( e.g. , click-checkboxes-large ), reasoning ( e.g. , click-checkboxes-soft ), unexpected pop-ups ( e.g. , login-user-popup ), and stochastically varying layouts ( e.g. , multi-orderings , multi-layouts ). These challenges are suitable for evaluating our proposed closed-loop framework. Each task interacts with a 160px × 210px web environment, where the state space is purely the HTML code of the web. Following RCI [8], we define the actions space as two sets of operations, i.e., clicking and typing actions. The clicks allow the agent to interact with clickable HTML elements ( e.g. , webpage buttons). The typings are conducted with keyboard-based actions, such as inputting characters into the input box and stroking functional keys ( e.g. , ENTER, BACKSPACE). We select nine MiniWoB++ tasks where environment observations (i.e., the HTML code) change after certain actions: search-engine , tic-tac-toe , terminal , login-user-popup , guess-number , email-inbox , email-inbox-nl-turk , email-inbox-forward-nl , and email-inbox-forward-nl-turk . Take the task terminal as an example. Search results appear on the webpage after inputting the keyword and pressing the search button. Therefore, environment feedback can be interpreted from the change in HTML code and further leveraged by closed-loop planning. We also adopt and test the 53 tasks evaluated in RCI [8].

B Baseline Details

ALFWorld. Following a set of previous works [20, 25, 19], we evaluate AdaPlanner on 134 different environments. By default, we include one sample as an exemplar per task to prompt AdaPlanner. For the study presented in Figure 4a, we adopt the setting of prompted samples as in Table 6. For the study in Figure 4d, we use one sample of the simplest task put to prompt the rest of the five tasks, which are more complex and require more steps to solve. For baselines, we compare AdaPlanner with BUTLER [20], ReAct [25], and Reflexion [19]. BUTLER [20] is an imitation learning method trained with 100k samples per task. ReAct and Reflexion, as outlined in Table 1, are promptingbased methodologies utilizing an implicit closed-loop framework. They employ a total of 6 and 8 samples, respectively, across all six tasks. BUTLER results are sourced from [20]. We evaluate ReAct, Reflexion, and AdaPlanner empowered by both GPT-3 ( text-davinci-002 ) and GPT-3.5 ( gpt-3.5-turbo and text-davinci-003 ) models.

MiniWoB++. Overall, we report the evaluation results of RCI [8] and the proposed AdaPlanner in GPT-3.5 ( text-davinci-003 ), along with three training or finetuning-based baselines: Computer

Control Agent Architecture(CC-Net) [7], Workflow-Guided Exploration(WGE) [11], and WebN-T53B [4]. CC-Net and WGE employ supervised learning and reinforcement learning with over 23K and 10 demonstrations per task, respectively. WebN-T5-3B uses 12K demonstrations to finetune a pre-trained language model. RCI is a prompting-based approach that is categorized as the implicit closed-loop method in Table 1, which utilizes 93 samples across the 53 tasks. For these 53 tasks, we first provide AdaPlanner with 38 human-written demonstrations and perform skill discovery to obtain another 21 additional examples, i.e., 59 examples are used for 53 tasks. Evaluation results of RCI, CC-Net, WGE, and WebN-T5-3B are sourced from the works of [8, 7, 11, 4], respectively.

C Proportion of two refinements

Table 7 illustrates that out-of-plan refinement is invoked more frequently in both environments, emphasizing its important role in AdaPlanner's planning process. The average occurrence for both types of refinements varies between ALFWorld and MiniWoB++. This can be attributed to the differences in the average lengths of the trajectories in ALFWorld and MiniWoB++ (15.60 and 5.40 steps, respectively).

D Prompts

D.1 ALFWorld

Basic Information. The <basic_info> defines the agent and admissible actions for AdaPlanner. Note that the actual definitions of action functions are not specified in the prompt. Instead, only a formalized definition with several examples is provided, such that the planner can acquire how to compose a plan based on these actions. As can be seen in the following part, this <basic_info> prompt is used in both <initial_planning> and <refinement> prompts.

Initial Planning. The <initial_planning> prompt is employed to generate the preliminary plan. In this context, <basic_info> is substituted by the content of the <basic_info> prompt. The <sample> is replaced with an expert trajectory, while <receptacle_list> is substituted by the list of interactive receptacles provided by the task environment. Finally, <task> is substituted by the task description, expressed in natural language.

<initial_planning> Prompt

Samples. In ALFWorld, there are six types of tasks: Pick , Clean , Heat , Cool , Examine , and Pick two . For each type, we gather one expert sample of solutions that the planner can refer to. These six expert samples are presented as follows:

The expert sample for the task Pick :

The expert sample for the task Clean :

The expert sample for the task Heat :

The expert sample for the task Cool :

The expert sample for the task Examine :

The expert sample for the task Pick two :

Code Check. After plan generation, we employ the following prompt to verify and rectify any syntax errors. The placeholder <solution_func> is replaced by the generated solution function. The <code_check> prompt prompts the model to return two questions. If the response to Question 1 is Yes , the answer to Question 2 is adopted as the corrected solution function. Otherwise, the solution function is kept unchanged.

Out-of-Plan Refinement. In the event of an assertion error, we use <refinement> to conduct the out-of-plan refinement. In this prompt, <basic_info> is replaced by the content of the <basic_info> prompt. The placeholder <sample> is substituted with an expert trajectory, while <receptacle_list> is replaced by the list of interactive receptacles provided by the task environment. <task> is replaced by the task description in natural language. Finally, <error_msg> is replaced by the assertion error message returned by the solution function. To adhere to the context length limit of the GPT-3/3.5 models, the previously generated solution function is not included in this prompt. Instead, we incorporate comprehensive information in the assertion error message, enabling the refiner to generate a revised plan based on these details.

Determining start_from . After formulating a revised plan, we utilize the following prompt to ascertain from which step the new solution function should commence. In this context, the <previous_solution> is replaced by the preceding solution function, while the <revised_solution> is replaced by the updated one. Subsequently, the argument start_from=1 is substituted with the step number that this prompt yields.

D.2 MiniWoB++

Basic Information. Similar to the ALFWorld tasks, the <basic_info> of MiniWoB++ defines the agent and admissible actions for AdaPlanner. Note that the actual definitions of action functions are not specified in the prompt. Instead, only a formalized definition with several examples is provided, such that the planner can acquire how to compose a plan based on these actions. As can be seen in the following part, this <basic_info> prompt is used in both <initial_planning> and <refinement> prompts.

Initial Planning. The <initial_planning> prompt is employed to generate the preliminary plan. In this context, <basic_info> is substituted by the content of the <basic_info> prompt. The <sample> is replaced with an expert trajectory (or discarded in the skill discovery stage). The <initial_state> is substituted by the initial HTML code of the task UI. Finally, <task> is substituted by the task description, expressed in natural language.

Samples. In MiniWoB++, we select nine tasks where feedback is available. Here are the expert samples that we gathered for these tasks.

The expert sample for the task email-inbox :

The expert sample for the task email-inbox-forward-nl :

<sample_email-inbox-forward-nl> Prompt

The expert sample for the task email-inbox-forward-nl-turk :

The expert sample for the task email-inbox-nl-turk :

The expert sample for the task guess-number :

The expert sample for the task login-user-popup :

The expert sample for the task search-engine :

The expert sample for the task terminal :

The expert sample for the task tic-tac-toe :

Code Check. We use the same <code_check> prompt for MiniWoB++ tasks as ALFWorld.

Out-of-Plan Refinement. In the event of an assertion error, we use <refinement> to conduct the outof-plan refinement. In this prompt, <basic_info> is replaced by the content of the <basic_info> prompt. The placeholder <solution_func> is replaced by the generated solution function, while <task> is replaced by the task description in natural language. Finally, <feedback> is replaced by the assertion error message returned by the solution function. Note that the <refinement> prompt used here differs slightly from the one used for ALFWorld. In this context, we include the previously generated solution function in the prompt, enabling the refiner to adjust the solution based on the error messages accordingly.

Determining start_from . We use the same <start_from> prompt for MiniWoB++ tasks as ALFWorld.

E Hallucination with GPT-3.5 Model ( gpt-3.5-turbo )

We present the trajectories of three tested methods (ReAct, Reflexion, and AdaPlanner) for two tasks. Here, each model utilizes gpt-3.5-turbo as the backend.

E.1 Case 1: examine the book with the desklamp.

ReAct's trajectory for this case is detailed below. Despite the presence of two expert demonstrations, ReAct encounters significant difficulties, as it extensively hallucinates inadmissible actions or scenarios. For example, search for a book titled "The Great Gatsby" and ask household members if they know where the book is are not feasible within the given environment.

The trajectory of Reflexion, after two rounds of iterative trials and errors, is detailed below. Despite multiple closed-loop corrections, Reflexion fails to formulate a feasible plan. Instead, it persists in generating unfeasible actions in its reflection part. For instance, it plans to ask the household members for help , or to look for a ladder or step stool outside of the room . These actions fall outside the scope of admissible actions in the ALFWorld environment.

Reflexion Trajectory of Case 1 (Hallucinated actions are marked in pink.)

The trajectory of AdaPlanner is detailed below. For better readability, we only present the major subgoals generated by the planner. Evidently, AdaPlanner successfully completes this task without the need for any revisions. This indicates that the code interface can effectively mitigate the hallucination in gpt-3.5-turbo .

AdaPlanner Trajectory of Case 1

E.2 Case 2: put some watch on safe.

In the second trajectory, ReAct also hallucinates infeasible actions (e.g., look for a manual or instructions and ask someone in the household ), ultimately leading to task failure.

ReAct Trajectory of Case 2 (Hallucinated actions are marked in pink.)

Reflexion Trajectory of Case 2 (Hallucinated actions are marked in pink.)

Your memory for the task below: Trial 0: In this environment, my plan was to find a watch and put it on the safe. However, I did not consider that the safe may require a code or key to lock it. I should have looked for a code or lock mechanism on the safe or a key to the safe before attempting to put the watch inside. When I could not find a way to secure the watch inside the safe, I should have looked for another container or safe in the room. In the next trial, I will first look for a code or lock mechanism on the safe or a key to the safe. If I cannot find one, I will look for another container or safe in the room. Trial 1: To solve the task, I need to find a watch and a way to secure it inside the safe. First, I will look for a watch in drawers (1-5), cabinets (1-4), dressers (1), shelves (1-12), sidetables (1), and sofas (1). Once I find a watch, I will examine the safe for any clues or look for a key in other locations. If I cannot find a way to secure the watch inside the safe, I will look for another container or safe in the room. If I still cannot find a way to secure the watch, I will seek additional resources or ask for help. Here is the task: You are in the middle of a room. Looking quickly around you, you see a armchair 1, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a drawer 5, a drawer 4, a drawer 3, a drawer 2, a drawer 1, a dresser 1, a garbagecan 1, a safe 1, a shelf 12, a shelf 11, a shelf 10, a shelf 9, a shelf 8, a shelf 7, a shelf 6, a shelf 5, a shelf 4, a shelf 3, a shelf 2, a shelf 1, a sidetable 1, and a sofa 1. Your task is to: put some watch on safe.

Initially, AdaPlanner also experiences hallucination, presuming that the watch is located in the safe . This incorrect assumption triggers an assertion error. AdaPlanner then revises the plan to better adapt to the actual environment. Following this refinement, the task is successfully completed.

F Detailed Evaluation Results on MiniWoB++

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLLMs have shown success as autonomous agents that make and execute plans in sequential decision problems. Existing methods either make open-loop plans, limiting adaptability to the environment, or closed-loop plans. Existing closed-loop methods, apart from DEPS, keep the plan static but simply modify immediate actions according to environment feedback, leading to potentially sub-optimal policies. The authors introduce AdaPlanner, a closed-loop LLM planner that additionally allows for *plan* refinement during the episode. The success of their method not only relies on this, but additionally code-style prompts and a skill-discovery mechanism for few-shot exemplars. AdaPlanner outperforms existing works while relying on far fewer demonstration examples from similar tasks.\n\nSTRENGTHS:\n- Empirically the authors show strong results with respect to sample efficiency and asymptotic performance.\n\n- Many ablations make it easy to understand which components of the model lead to overall success. \n\n- Conceptually simple approach.\n\nWEAKNESSES:\n- In the evaluation section, the baselines are glossed over. This makes it hard to comprehend the distinction between their approach and the baselines. \n   - I’d recommend adding some of the Appendix descriptions to the evaluation section, and potentially referencing Table 1 more often.\n\n- The authors use the term ‘hallucination’ a lot but do not define it.\n\n- The authors discuss in- and out-of- plan refiners a lot before providing intuitive examples for when either would be necessary. Could the authors provide more examples earlier on in the paper?\n\n- DEPS appears to be a relevant baseline. Could the authors include it or at least delve deeper into its limitations and why it is not appropriate?\n\n- It appears that the largest contributor to the success of AdaPlanner, over existing approaches, is code style prompts and skill prompts. Wouldn’t it be worthwhile to apply those modifications to existing approaches, like Reflextion (Fig 4), and contrast?\n\n- AdaPlanner prompts the LLM to correct any syntax errors. How important is this? Would be nice to include this ablation.\n\nQUESTIONS:\n- Line 80, could you define the output of pi, in the same way that you did for the planner?\n- Line 81, shouldn’t it be P_t rather than P_{t - 1}?\n- Lines 114 - 144 I think you’ve repeated the sentence twice.\n- Line 216, what are the 6 task types?\n- Line 132, how is N chosen and what’s its effect on performance?\n\nLIMITATIONS:\nAdaPlanner still requires demonstrations for learning. Would be worthwhile comparing with RL agents trained directly on the task, without any expert demonstrations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBriefly summarize the paper and its contributions. This is not the place to critique the paper; the authors should generally agree with a well-written summary.\n\nThe paper proposes AdaPlanner, an LLM-based adaptive planner for text-based sequential decision-making tasks. The planner is adaptive in the sense that it can refine the generated plan/policy based on feedback. \n\nThe contributions made in this paper include the following\n1. interacting with the environment with LLM in the loop\n2. a code-style prompt is engineered for LLMs to output a policy \n3. refining the LLM policy for the current task based on feedback\n4. prompt tuning for new tasks based on previous interaction (termed skill discovery)\n\nThe proposed AdaPlanner is evaluated on two text-based sequential decision-making environments ALFWorld and MiniWoB++. Their experiments indicate that with feedback, LLMs can adapt the plan.\n\nSTRENGTHS:\n* The paper is well written.\n* The paper focuses on extremely relevant and signifcant problems.\n\nWEAKNESSES:\n* I find the paper lacks significant details. Please see the next section for the list of questions.\n* The paper employs sloppy mathematical notations.\n* The paper lacks the rigor of scientific evaluation. \n* Paper misses all references to LLM-based approaches for planning with PDDL. The one that I find most relevant for code generation is \"Generalized Planning in PDDL Domains with Pretrained Large Language Models, Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B. Tenenbaum, Leslie Pack Kaelbling, Michael Katz”\n\nQUESTIONS:\n**Major**\n\n1. How is the programmatic response from LLM converted to action responses? Did the conversion require manual intervention? For instance, Figure 2 has an indentation error which would result in a wrong plan. Were such indentation errors evaluated manually? Can authors provide a list of errors made by LLMs? \n1. In line 167, what does an alignment between ‘anticipated plan’ and environment mean? How does the AdaPlanner observe the alignment? \n1. Can authors provide details about the size of the task used in the prompt (for samples) vs the size of the task that was successfully solved by AdaPlanner? To establish the claim of sample efficiency, it is important to understand if the planner is able to efficiently plan for tasks that are significantly different from the prompts.\n1. The X-axis in Figure 3 indicates `# Samples per task`. Is this the number of samples provided for each trajectory? Or sum?  \n1. What was the length of plans or length of trajectories generated by AdaPlanner vs other approaches? To claim the effectiveness of the AdaPlanner, it is important to compare the length of successful trajectories.\n1. For skill discovery, how is the solution converted to the skill? How are skills represented? How large is the skill memory?  Were the discovered skills included in the count of samples used for training as they are training samples for the next set of trajectories?\n1. It is not clear how skills are filtered and what criteria are used for the evaluation and ranking of skills.\n1. What is the connection between skill discovery and prompt tuning?\n1. The success rate of \"With SD\" in Figure 4d looks significantly reduced from  Figure 4a. Were different settings used for theses experiments?\n1. At various places, the paper mentions \"environment feedback\". In my opinion, this is a misnomer. The feedback is not from the environment. The environment just provides the next observation, the feedback is generated by the agent itself. And the use of observation to refine a plan or next action is quite standard practice in RL. I would highly recommend dropping the term feedback from the title. \n1. The use of term plan and policy is a little confusing. A plan is a sequence of actions. A policy is a mapping from states to actions. By this definition, the `solution()` function is as a policy. In preliminaries, the planning policy ($\\rho$) is conditioned on a previous plan $P_t$. However, the appendix describes the refinement prompt using the assertion error (instead of `solution()`). Isn't the assertion error providing information about the policy (the `solution()` function)? So I am confused by the terminologies. Is the $\\rho$ refined conditioned on the policy or the plan? The usage of these terms is also confusing in the Preliminary section. Request authors to precisely define the mathematical notations and highlight what they represent in the examples.\n\n**Minor**\n\n12. In line 387, there are extra curly braces.\n12. The notation $\\rho$ is used in line 73 but introduced much later.\n12. As the context $c_t$ is defined as a sequence of action and observations from time step $0$ to $t$, it is not clear what $c_{>t}$ means (in line 116).  \n12. Open-Loop system in Figure 1 should have an arrow going from env to planner with $o_1$.\n12. Statement in Line 144 \"To generate a plan ..\" looks like a repetition of Line 141 \"To generate an initial plan...\"\n12. In line 116, if $h_t$ is obtained from $c_t$ then would it not be captured in $c_{>t}$? An example of $h_t$ would help better understand the proposed update.\n12. In line 73, as $\\rho$ is defined using $\\Delta(A^{T})$. But the length $T$ is not fixed. \n12. In line 73 $\\rho$ is defined where a plan is conditioned only on observation and goal. However, later it is conditioned on the context, plan, and goal.\n\nLIMITATIONS:\n* The evaluations are restricted to text-based sequential decision-making problems and task where the inadmissible actions do not cause drastic changes in the environment. On the contrary, inadmissible actions are like no-ops. Further, the paper does not present analysis of plan length. Hence, the analysis is limited to zero risk environments. \n* The claim made in the abstract about skill discovery mechanism enabling agent to plan with fewer task demonstration is not substantiated in the evaluations. Evaluation in Fig. 4d only established improvement in success rate, not sample efficiency.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents AdaPlanner, a closed-loop planning method that uses a large language model (LLM) to solve tasks in text-based environments. AdaPlanner operates by decomposing a complex task into manageable sub-goals and predicting environmental feedback for each. During execution, it refines its actions based on the feedback received from the environment. AdaPlanner operates solely via prompting, eliminating the need for a dedicated training phase and reducing its computational cost. The paper demonstrates that AdaPlanner consistently outperforms existing baselines, achieving state-of-the-art performance in ALFWorld tasks and MiniWoB++ tasks.\n\nSTRENGTHS:\n- AdaPlanner introduces a novel approach to task-solving in text-based environments using a large language model. It stands out for its closed-loop planning method and its ability to decompose tasks into manageable sub-goals.\n- The paper is well-written and clear. The authors have done a good job of explaining complex concepts and methodologies in an understandable manner.\n- The work presents a new way of leveraging large language models for task-solving in text-based environments. The results show that AdaPlanner can effectively leverage feedback to refine its plans and enhance its performance.\n\nWEAKNESSES:\n- The part about skill discovery is not described very clearly, and I still cannot understand the details of the skill discovery module well.\n- The author compared the version without a code interface in the experiment, but it seems that they did not specifically show the prompt after removing the code interface. At the same time, as an ablation experiment, it is also necessary to analyze the effects of specific components in the code interface.\n- The phenomenon that GPT-3 performs better than GPT-3.5 is interesting, but it seems that the paper only compares GPT-3 and GPT-3.5 in Alfworld, without conducting the same experiments in MiniWoB++ to further support the conclusion. And the author's hypotheses about this phenomenon (the smaller scale of GPT3.5) lacks specific analysis or literature references to support it.\n\nQUESTIONS:\n- In the experiment, what is the proportion of in-plan and out-of-plan occurrences? How will this proportion change over time? This should be a necessary indicator for understanding the two refiners.\n- On MiniWoB++, will there be better performance from GPT-3 than GPT-3.5?\n- Is there still a necessity for AdaPlanner in larger-scale LLMs, such as models like GPT4 with better self-refining capabilities?\n\nLIMITATIONS:\n- As mentioned above, this paper still needs more experiments and analysis to further validate the rationality of its methods, as well as the observed phenomena and corresponding hypotheses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper looks at explicit closed-loop systems with LLMs for adaptive planning utilizing environmental feedback. They showcase better planning performance on ALFWorld and MiniWOB++ environments over existing state-of-the-art works like ReAct and Reflexion.

STRENGTHS:
The paper is well written and the experiments are thorough. They present an interesting improvement over the current works like ReAct and Reflexion.

WEAKNESSES:
1. The kind of tasks in these domains don’t seem to have interaction resolution where there are multiple conflicting causal links from the initial to the goal state which have to be resolved (including negative interactions between subgoals). This could also lead to the human demonstrations helping significantly with the  It would be useful to analyze the performance of AdaPlanner specifically in such cases. 

2. I think non-ergodic environments could clearly pose danger to such agents. It would be interesting to see how AdaPlanner can perform against ReAct or Reflexion in such environments.

QUESTIONS:
1. Given that the LLM seems to verify the plan to determine its feasibility, what was its efficiency in those assessments? Are there any results pertaining to that?

2. Is there any classification of the tasks with respect to their hardness?

3. For how many of these tasks did the human expert demonstration solve the task?

LIMITATIONS:
The authors have addressed some of the limitations. I have provided some limitations in the weaknesses section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07