SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haotian Sun $^{1*}$ , Yuchen Zhuang $^{1*}$ , Lingkai Kong $^{1}$ , Bo Dai $^{1}$ , Chao Zhang $^{1}$
<sup>1</sup> Georgia Institute of Technology
{haotian.sun, yczhuang, lkkong, chaozhang}@gatech.edu, bodai@cc.gatech.edu
Large language models (LLMs) have recently demonstrated the potential in acting as autonomous agents for sequential decision-making tasks. However, most existing methods either take actions greedily without planning or rely on static plans that are not adaptable to environmental feedback. Consequently, the sequential decision-making performance of LLM agents degenerates with problem complexity and plan horizons increase. We propose a closed-loop approach, AdaPlanner, which allows the LLM agent to refine its self-generated plan adaptively in response to environmental feedback. In AdaPlanner, the LLM agent adaptively refines its plan from feedback with both in-plan and out-of-plan refinement strategies. To mitigate hallucination, we develop a code-style LLM prompt structure that facilitates plan generation across a variety of tasks, environments, and agent capabilities. Furthermore, we propose a skill discovery mechanism that leverages successful plans as few-shot exemplars, enabling the agent to plan and refine with fewer task demonstrations. Our experiments in the ALFWorld and MiniWoB++ environments demonstrate that AdaPlanner outperforms state-of-the-art baselines by  $3.73\%$  and  $4.11\%$  while utilizing 2x and 600x fewer samples, respectively. The implementation of AdaPlanner is available at https://github.com/haotiansun14/AdaPlanner.
Large language models (LLMs) have recently emerged as versatile autonomous agents for sequential decision-making in grounded environments. Traditional decision-making methodologies like Reinforcement Learning (RL) require extensive task-specific training data and often lack the ability to generalize across tasks and environments. In contrast, LLMs are pre-trained on massive and diverse textual data, which gives them extensive world knowledge and the ability to reason over the knowledge. This makes them highly versatile and able to handle complex, real-world scenarios that may involve multiple steps of planning and decision-making.
Existing methods that leverage LLMs as autonomous agents for decision-making can be briefly categorized into two groups (Table 1): open-loop systems and closed-loop systems. Open-loop methods [24, 27, 5, 18, 12, 17, 16] rely on pre-determined plans to accomplish the desired task without any feedback adaptation mechanism. On the other hand, closed-loop systems [25, 6, 8, 22, 9, 19, 23] incorporate environment feedback to continuously monitor system behaviors and make refinements and adjustments of the plans accordingly, which therefore is more flexible.
However, both existing open-loop and closed-loop LLM agents have inherent drawbacks. Open-loop systems are computationally cheap and simple; however, they do not consider feedback from the environment and stick to the initial plan, which lack of adaptability, and, thus, can easily generate suboptimal plans. On the other hand, most existing closed-loop methods generate a fixed plan and only update their executing actions upon environment feedback. This causes them to make
*These authors contributed equally to this work.
Methods	Feedback Utilization	Instruction Type	Prompting	Decomposition	Experience Refinement
Open-Loop Methods
CoT [24]	-	Prompting	Language	-	-
Least-To-Most [27]	-	Prompting	Language	Sub-Goals	-
Zero-Shot Planner [5]	-	Prompting	Language	-	-
HuggingGPT [18]	-	Prompting	Language	Sub-Goals	-
Chameleon [12]	-	Prompting	Language	Sub-Goals	-
Implicit Closed-Loop Methods with Fixed Plan
ReAct [25]	Taking Action	Prompting	Language	-	-
Inner Monologue [6]	Taking Action	Prompting	Language	-	-
RCI [8]	Taking Action	Prompting	Language	-	-
ProgPrompt [22]	Taking Action	Prompting	Code	-	-
Code as Policies [9]	Taking Action	Prompting	Code	-	-
Reflexion [19]	Taking Action	Prompting	Language	-	Past Failure
Explicit Closed-Loop Methods with Plan Refinement
DEPS [23]	Modifying Plan	Prompting & Training	Language	Sub-Goals	Past Failure
AdaPlanner	Action & Plan	Prompting	Code	Sub-Goals	Past Failure & Success
Table 1: A comparison of methods that leverage LLMs for decision making. Each method's features are reported across five categories: 1) Environment Feedback Utilization: The method can use feedback to decide the next action (Taking Action), revise the entire plan (Modifying Plan), or do both (Action & Plan). 2) Instruction Type: The method may require prompting, training, or both. 3) Prompting Style: The method can employ either natural language or code for its planning backend. 4) Task Decomposition: The method might decompose the task into sub-goals or not. 5) Experience Refinement: The method can learn from past failure, past success, or both. The AdaPlanner proposed in this paper is highlighted in gray.
sub-optimal decisions that adapt to the environment in the short term but could have detrimental implications for future steps. DEPS [23] is the only exception, a method that modifies its entire plan based on feedback from the environment. However, it requires training a plan selector to choose the most successful plan, which requires a significant amount of task-specific data. As a result, applying this method to different tasks can be challenging.
To address the limitations of existing LLM agents, we propose AdaPlanner, a closed-loop planning method with LLM playing two roles - planner and refiner. The planner decomposes the task into manageable sub-goals - planner and refiner. The refiner distinguishes and responds to two types of environment feedback - in-plan feedback is the environmental observation that aligns with the prediction, and out-of-plan feedback is one that deviates from the prediction. For in-plan feedback, the refiner can dynamically query the LLM to perform reasoning and extract key information from in-plan feedback expressed in natural language. This is achieved through a specific action called ask_LLM(), in which the LLM separately parses the observation and obtains information pertinent to subsequent actions. For out-of-plan feedback, the refiner proactively revises the entire plan and resumes to solve the current task from an intermediate point. AdaPlanner's adaptive closed-loop framework alleviates the need for prior knowledge about the feedback structure and permits the agent to instantly adopt a refined plan rather than restarting from scratch in a reset episode. This leads to a more efficient and adaptive decision-making process.
AdaPlanner operates solely via prompting, eliminating the need for a dedicated training phase and reducing its computational cost. Furthermore, AdaPlanner leverages a code-based prompting for precise planning and refinement. The use of code prompts facilitates task decomposition into sub-goals and mitigates LLM hallucination during the decision-making process. AdaPlanner also features a skill discovery process, which accumulates successful experiences to guide future planning. This feature further enhances its long-term planning ability and sample efficiency.
We formally define the planning problem with LLM, and introduce open-loop vs. closed-loop control system, which will motivate our method, in Section 2. Each component of the proposed AdaPlanner is specified in Section 3, including code-based prompting in Section 3.1, closed-loop adaptation in Section 3.2, and skill discovery in Section 3.3, and empirically justified in Section 4. The superior performance of AdaPlanner on both ALFWorld and MiniWoB++ demonstrates our proposed adaptive closed-loop framework can effectively enhance planning performance, even when faced with a limited number of samples.
Figure 1: A comparison between open-loop, implicit closed-loop, and explicit closed-loop systems.
Problem Formulation. We consider adopting an LLM as an autonomous agent to solve different tasks in text-based environments. For initialization, the agent is provided with allowed actions  $\mathcal{A}$  in the environment, as well as a text-grounded task definition  $g\in \mathcal{G}$  from the task space  $\mathcal{G}$ . Besides, the initial state of the environment is also observed as  $o_1\in \mathcal{O}$  from the observation space  $\mathcal{O}$ . With such inputs, the LLM agent needs first to generate an initial planning policy for solving the task  $\rho_0(P_0|g,o_1):\mathcal{G}\times \mathcal{O}\to \Delta (\mathcal{A}^T)$ , where  $T$  is the total length of steps in the generated plan and  $\Delta (\cdot)$  is probability simplex function. Also, the agent can interact with the environment for feedback: When the agent interacts with the environment at the  $t$ -th step, the agent receives an observation  $o_t\in \mathcal{O}$  from the environment and generates a trajectory-like context  $c_{t} = (o_{1},a_{1}^{\prime},o_{2},a_{2}^{\prime},\dots ,a_{t - 1}^{\prime},o_{t})$ , where  $a_1^\prime ,a_2^\prime ,\dots ,a_{t - 1}^\prime$  are the executed actions within the environment. As the agent may modify the actions according to the feedback, the executed actions  $a_1^\prime ,a_2^\prime ,\dots ,a_{t - 1}^\prime$  can be different from the actions  $a_1,a_2,\dots ,a_{t - 1}$  in the initial plan. We denote  $\rho (\cdot |g,c_t,P_t)$  as the high-level planning policy that generates an entire plan and  $\pi (\cdot |g,c_t,P_t)$  as the action-generation policy conditioned on a given plan  $P_{t}$ . Given the context  $c_{t}$  and the entire plan at the last step  $P_{t - 1}$ , the agent refines future decisions. In the end, the LLM agent should model both the initial planning policy and the environment feedback-conditioned policy to complete the given task successfully.
Open-Loop System. An open-loop system is a non-feedback system (Figure 1), where the output is solely dependent on the input, without any consideration of the environmental feedback. Thus, in an open-loop system, the entire initial plan over the time horizon  $T$  is predetermined and static by the initial planning policy  $\rho_0(\cdot | g, o_1)$ , without any feedback-based refinement. Despite their simplicity, open-loop systems are notably vulnerable to environmental changes, as they lack the capacity to adapt or adjust their plans based on environmental feedback.
Closed-Loop Systems. On the contrary, a closed-loop system (Figure 1) refers to a planning process that incorporates environment feedback to adjust and refine future decisions, involving both initial planning  $\rho_0(\cdot |g,o_1)$  and two levels of feedback-based refinements,  $\rho (\cdot |g,c_t,P_{t - 1})$  and  $\pi (\cdot |g,c_t,P_{t - 1})$ , in the system.
Implicit Closed-Loop Systems. After each step of interaction with the environment, implicit closed-loop systems will maintain the initial plan (i.e.,  $P_{t} = P_{0}$ ) and only modify a single action based on the feedback. Therefore, the feedback-based refinement is defined as  $\pi(a_{t}'|g, c_{t}, P_{0})$ , where  $a_{t}' \in \mathcal{A}$  is the modified action from action space, while the remaining actions  $a_{>t}$  for future steps remain the same as the initial plan. Although locally-optimal actions are adopted at each step, inaccuracies in initial planning can result in task failure or non-completion.
Explicit Closed-Loop Systems. Explicit closed-loop systems refine the entire plan based on environment feedback following the policy  $\rho(P_t|g, c_t, P_{t-1})$ , where  $P_t \in \Delta(\mathcal{A}^{T-t})$  is the refined plan at time step  $t$  containing the modified future actions  $a_{>t}'$  to execute and  $P_{t-1}$  is the old plan modified in the previous time step. Allowing for constant refinement and improvement of the plan, explicit closed-loop systems can help prevent costly mistakes or missed opportunities that might arise from adhering to outdated plans. Our proposed AdaPlanner is an explicit closed-loop system.
3 AdaPlanner
Model Architecture. Our AdaPlanner model, shown in Figure 1, consists of two main components:
- an LLM-based agent that functions dually as a planner and a plan refiner, and
- a skill memory module designed to enhance sample efficiency through skill discovery.
The LLM-based agent, in its planner role, generates a comprehensive plan and performs preliminary assessments to determine its feasibility. The initial planning is modeled as  $\rho_0(P_0|g,o_1)$ . As the plan unfolds, the agent also operates as a refiner, conducting feedback-based refinement in both in-plan and out-of-plan manners. In-plan and out-of-plan refinement processes primarily differ in how they impact future actions. In-plan refinement is a one-step action that integrates useful information into the existing plan for better action grounding. After this in-plan phase, future actions will be generated using the updated context  $\pi (a_{>t}^{\prime}|g,c_{>t}\cup \{h_t\},P_0)$ , where  $h_t$  represents the new information obtained from  $c_{t}$  via in-plan refinement at timestep  $t$ . Out-of-plan refinement, on the other hand, leverages environmental feedback to directly revise the entire plan, denoted as  $\rho (P_t|g,c_t,P_{t - 1})$ . This mechanism allows for comprehensive adjustments to be made to the plan in response to unexpected environmental feedback. Skill memory serves as a repository, archiving past successful plans and their respective interactions with the environment. If the agent encounters a task resembling the skills stored in memory, these skills can serve as few-shot exemplars in the LLM agent's prompt. This feature improves not only sample efficiency but also reliability for future planning.
Environment Interaction. AdaPlanner employs adaptive closed-loop planning and active environment interaction for task solving. It can anticipate environmental observations and proactively refine the plan only when there is a discrepancy between expected and actual outcomes. This is achieved by decomposing the planning process into  $N$  manageable sub-goals. During the planning and action-taking process, the agent selects from a set of timestamps,  $\{t_1, \dots, t_N\}$ , to evaluate the success of each sub-goal. If the sub-goal does not align with the planned prediction at timestep  $t \in \{t_1, \dots, t_N\}$ , the environment actively sends the previous sub-trajectories  $(o_1, a_1', \dots, o_t, a_t', o_{t+1})$  back to the refiner for plan revision. This process allows the agent to check the success status only at  $N$  crucial points, thereby reducing computational costs (number of API calls) and enhancing efficiency.
3.1 Plan Generation via Code-Based LLM Prompting
AdaPlanner plans and refines by using Pythonic code prompts for LLMs. Consistent with previous observations [3, 2], we have found that using code prompts instead of natural language prompts for LLMs reduces ambiguity and misinterpretation, which significantly reduces LLM hallucination during plan generation and refinement. We design code prompts during different stages of decision-making, including adaptive planning, feedback generation, and in-episode refinement. We provide a detailed description of the prompts used at each stage in Appendix D.
To generate an initial plan for solving a given task, we input a task description, the permissible actions in the environment, and, when available, sample demonstrations of task resolution into LLM. These pieces of information are all formatted into Pythonic code format for LLM prompting. Figure 2 (a) shows an example programming-based plan generated by AdaPlanner for solving a put task in the ALFWorld environment. The generated solution function is provided with two input arguments: the first is the agent object, which encapsulates environmental information to be used by the agent. The second is the variable start_from, which is a parameter indicating the subgoal from which the agent will later resume its execution with a refined plan. By default, the start_from is initialized as 1. The value of this variable can be further reassigned during the refinement.
When prompting LLM to generate the code-based plan, we design the prompt to teach LLM to decompose a complex task into sub-goals. As shown in Figure 2(a), the generated code plan solution(agent, start_from=1) consists of: 1) a general plan at the outset that decomposes the task into subgoals in the form of comments; and 2) a sequence of sub-plans, each consisting of admissible actions corresponding to a specific subgoal. Such a mechanism allows our method to handle complex, long-horizon tasks by hierarchically decomposing them into sequences of subgoals. Furthermore, each subgoal ends with an assertion statement to test its fulfillment, which allows our method to interact actively with the environment and later resume its execution with a refined plan.
Figure 2: An illustrative example from ALFWorld to show the proposed adaptive closed-loop planning through code. The task is to put some clean lettuce on the diningtable. The in-plan feedback in (a) is a sentence like On the countertop 2, you see a knife 1, a lettuce 1, a saltshaker 2, and a soapbottle 1. This feedback is managed by the ask_LLM() action. The execution of the initial plan might yield misaligned observations, triggering an out-of-plan feedback and refinement process. For instance, the agent cannot clean the lettuce if it is not currently located at a sinkbasin. The out-of-plan feedback in (b) assists AdaPlanner in generating a revised plan (c) so that the agent will move to a sinkbasin before cleaning the lettuce. AdaPlanner then determines to resume from step 3 within the same episode. The task can be successfully completed using the refined plan.
3.2 Adaptive Closed-Loop Plan Refinement
Once an initial plan is generated, AdaPlanner then prompts the LLM to correct any syntax errors. After this, the code undergoes execution through the environment interface. The interface is responsible for grounding the actions in the environment, and also for routing environmental observations back to the code as a return value. This bi-directional flow allows AdaPlanner to adapt and refine its plan in response to environmental observations in a closed-loop manner.
In-Plan Feedback and Refinement via ask_LLM() Action. When AdaPlanner observes that the environment is aligned with the anticipated plan, it performs in-plan refinement. This allows it to extract useful information from the observation that can be used for upcoming actions. To achieve this, we provide the agent with an additional action called ask_LLM(), which is used to formulate a plan alongside task-specific actions. The ask_LLM() function enables AdaPlanner to self-query and perform reasoning based on specific information parsed from environmental observations. For instance, in [Step 3] in Figure 2 (a), the ask_LLM() action extracts the identifier of the found object lettuce from the natural-language observation. This information can then be fed into later actions. As an additional atomic action, this in-plan refinement is integrated into the plan at any point where the planner deems a reasoning process is necessary. Existing code-generation-based methods [22, 9, 3] face a challenge in this task, especially when there is no prior knowledge of the structure and organization of these feedback sentences. In contrast, our AdaPlanner method leverages LLM to parse critical information from diverse feedback presented in natural-language sentences to streamline plan execution.
Out-of-Plan Refinement with the Refine-Then-Resume Mechanism. After each sub-plan execution, AdaPlanner actively checks an assertion condition to ensure that the current plan is proceeding as expected. If the assertion fails, AdaPlanner performs out-of-plan refinement. For example, in Figure 2 (a), after [Step 3], the agent is expected to hold 1ettuce. If this condition is not met, AdaPlanner generates an error message that details the current progress of execution gathered by the report() function. In ALFWorld tasks, this function provides a report of the agent's location, the object it is holding, and the last three interactions with the environment, as shown in Figure 2 (b). AdaPlanner then utilizes this information to perform out-of-plan refinement.
During the out-of-plan refinement as in Figure 2 (c), AdaPlanner uses a prompt similar to the one used during the initial planning stage, but with an additional feedback message that reflects the current state. Detailed prompts are provided in Appendix D. AdaPlanner then refines the plan based on the newly acquired information and also determines the value of start_from by comparing the plan before and after the refinement. The newly refined solution() is then executed from the breakpoint start_from. This breakpoint contains all variable states that were saved prior to refinement. Consequently, the current episode can continue from an intermediate checkpoint without restarting from scratch. We call this mechanism refine THEN-resume. It significantly speeds up task completion and reduces the number of LLM calls required.
3.3 Skill Discovery
Acquiring expert demonstrations for task solving can be costly, particularly as the number of tasks increases. To address this issue, we have equipped AdaPlanner with a skill discovery feature. This is a memory scheme that discovers and archives successful trajectories, thereby improving planning performance when dealing with similar tasks. The skill discovery process consists of two stages, which can be conducted alternately over several rounds, based on the interaction costs and computation resources.
Skill Acquisition. In the first stage, AdaPlanner attempts to solve unseen tasks, leveraging a limited number of human demonstrations of other simpler tasks, or even no demonstrations. The model capitalizes on adaptive closed-loop planning to iteratively explore and refine solutions via a trial-and-error approach. Upon successful completion of a given task, the latest solution and the corresponding interactions are treated as candidate discovered skills.
Skill Filtering. In the second stage, we compare the planning performance with and without the integration of the discovered solution into the prompt. If the inclusion of this solution boosts the success rate, it is archived as a discovered skill. Conversely, if it does not improve performance, it is discarded. This filtering stage is crucial because the iterative closed-loop refinement may integrate episode-specific information into the revised solution, potentially compromising its generalizability.
4 Evaluation
We test AdaPlanner on two text-based decision-making environments: 1) ALFWorld [20] is a text-based virtual household environment encompassing six distinct task types set. We evaluate AdaPlanner on a total of 134 tasks across these six types. 2) MiniWoB++ [11] is a simulation environment that covers a large range of computer tasks. We select 9 MiniWoB++ tasks with environmental feedback, and we also adopt and test the 53 tasks evaluated in RCI [8]. Both environments aim to solve complicated challenges with long-horizon solutions and sparse rewards. We also carefully designed ablation studies to justify the significance of each component in AdaPlanner. The Setup details and prompts for AdaPlanner are depicted in Appendix A and D. Detailed introductions to each baseline are presented in Appendix B Note that we evaluate different baselines for these two benchmarks. These methods utilize task-specific samples for prompting or training purposes, thus necessitating separate evaluations for each benchmark.
Metrics. Consistent with previous works [20, 25, 19, 7, 4, 11, 8], we use success rate  $(\%)$  to evaluate the performance of tested methods. The success rate is defined as the number of successful episodes over the total number of episodes. Note that in ALFWorld, failure of an episode occurs when the total number of actions attains 50, with the task still unsolved. In MiniWoB++, failures can occur in two scenarios: either due to the execution of invalid actions or if the task remains unfinished following the execution of the entire plan.
Main Results. AdaPlanner consistently outperforms the existing baselines, achieving state-of-the-art performance, i.e., an overall success rate of  $91.79\%$  in ALFWorld tasks (Table 2) and  $91.11\%$  in