SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Method	Pick	Clean	Heat	Cool	Examine	Pick two	All (134 tasks)
Training-Based Methods
BUTLER [20]	46.00	39.00	74.00	100.00	22.00	24.00	37.00
Implicit Closed-Loop Methods with Fixed Plan
ReAct [25] (GPT-3)	66.67	41.94	91.03	80.95	55.56	35.29	61.94
ReAct [25] (GPT-3.5)	37.50	64.52	69.57	42.86	38.89	17.65	47.76
Reflexion [19] (GPT-3 + 3.5)	75.00	90.32	91.30	90.48	88.89	94.12	88.06
Reflexion [19] (GPT-3.5)	50.00	41.94	65.22	52.38	66.67	47.06	52.99
Explicit Closed-Loop Methods with Plan Refinement
AdaPlanner (GPT-3)	100.00	96.77	95.65	100.00	100.00	47.06	91.79
AdaPlanner (GPT-3.5)	77.78	93.55	69.57	93.65	62.96	78.43	80.60
Table 2: Success rate (\%) of tested methods on six ALFWorld tasks. For ReAct and AdaPlanner, GPT-3.5 refers to gpt-3.5-turbo, while GPT-3 represents text-davinci-002. For Reflexion, GPT-3.5 indicates gpt-3.5-turbo. GPT-3+3.5 is used in the original Reflexion implementation, which utilizes both GPT-3 (text-davinci-002) and GPT-3.5 (text-davinci-003) for action generation and failure reflection, respectively. AdaPlanner is prompted with one specific example per task, making up six demonstrations in total. This is half the number of samples used in React and Reflection. The best-performing results are marked in bold. The results of our method are colored in gray.
MiniWoB++ tasks with feedback (Table 3). Specifically, in ALFWorld, AdaPlanner equipped with GPT-3 achieves a remarkable success rate exceeding  $95\%$  in the majority of individual tasks. It also surpasses all other baselines in the Pick, Clean, and Examine tasks. Notably, even in the task with the lowest performance (Pick two), AdaPlanner still outperforms BUTLER and ReAct. In the MiniWoB++ environment, AdaPlanner demonstrates superiority over all other methods on tasks that provide feedback. This superior performance suggests that AdaPlanner effectively leverages feedback to refine its plans and enhance its performance. Furthermore, AdaPlanner maintains competitive performance on tasks without feedback, achieving a success rate of  $93.22\%$ . Note that AdaPlanner's success rates of tasks without feedback are still comparable to CC-Net, the state-of-the-art model requiring over 23,000 samples per task. This result highlights the efficacy of the programming-based planning strategy employed by AdaPlanner. In both environments, AdaPlanner consistently delivers superior or competitive performance when compared to not only training-based methods but also implicit closed-loop methods under the same LLM models. These results affirm the effectiveness of the proposed explicit closed-loop plan refinement in AdaPlanner.
Furthermore, we summarize the relationship between success rate  $(\%)$  and the number of samples in Figure 3. In ALFWorld, AdaPlanner yields the highest performance with the fewest number of samples. In MiniWoB++, our method outperforms most baselines. Notably, our method achieves performance comparable to CC-Net but requires 600 times fewer samples. This study highlights that AdaPlanner significantly reduces the need for extensive demonstrations or expert trajectories, thereby offering a more resource-efficient solution.
(a) ALFWorld
(b) MiniWoB++
Figure 3: Relationship between success rate (\%) and the number of expert demonstrations in ALFWorld and Mini-WoB++ environments. We adopt the same settings as in Table 2 (GPT-3 version) and Table 3. The top-left corner represents the pinnacle of sample efficiency.
Adaptive Closed-Loop Architecture Enhances Planning Performance. Figure 4a shows the performance v.s. the number of closed-loop refinements, under settings with different numbers of demo samples. The detailed example selection for this study is provided in Appendix A. We observe a significant trend of increased success rates corresponding to each subsequent closed-loop plan refinement. This indicates the AdaPlanner's ability to consistently leverage real-time feedback for performance enhancement, regardless of the number of samples used. Remarkably, AdaPlanner
Method	With feedback (9 tasks)	No feedback (44 tasks)	All(53 tasks)
Training-Based Methods
CC-Net [7]	87.00	95.66	94.00
WGE [11]	67.60	87.93	86.00
Finetuning-Based Methods
WebN-T5-3B [4]	38.50	54.67	52.00
Implicit Closed-Loop Methods with Fixed Plan
RCI [8]	81.56	92.68	91.00
Explicit Closed-Loop Methods with Plan Refinement
AdaPlanner	91.11	93.22	92.87
maintains this trend of success rate enhancement even when the total number of demonstrations across all six tasks is as low as two. In addition, as displayed in Table 4, AdaPlanner consistently outperforms both ReAct and Reflexion in producing the shortest trajectory lengths across all tasks in ALFWorld. Moreover, a comparison with Reflexion, depicted in Figure 4b, shows AdaPlanner's consistently superior performance across all iterations of closed-loop corrections. These observations highlight AdaPlanner's sample efficiency and its potential for real-world applications where the number of available demonstrations is limited.
Table 3: Success rate (\%) of tested methods on two subsets of tasks in the MiniWoB++ environment. RCI and AdaPlanner harness GPT-3.5 (gpt-3.5-turbo and text-davinci-003) as backends. Our AdaPlanner method is provided with 38 human-written demonstrations; then, it automatically discovers 21 additional examples via skill discovery, which makes up the final set of 59 examples for 53 tasks. This is around half the number of samples used in RCI and over one six hundredths of the number of samples used in CC-Net. The best-performing results are marked in bold. The results of our AdaPlanner are colored in gray. Per-task success rates are provided in Appendix F.
Method	Pick	Clean	Heat	Cool	Examine	Pick two	All (134 tasks)
ReAct[25]	19.55	25.79	19.7	27.86	29.72	36.29	25.81
Reflexion[19]	18.77	25.25	19.13	25.76	27.11	35.76	18.90
AdaPlanner	10.79	13.45	17.61	13.33	21.00	20.71	15.60
Table 4: Average trajectory length (# steps) per task for ReAct, Reflexion, and AdaPlanner. We adopt the same settings as in Table 2 (GPT-3 version). Following the setting in [25, 19], each episode will be terminated after reaching 50 steps.
Code Interface Mitigates Hallucination. The latest gpt-3.5-turbo is reported to be the most capable GPT-3.5 model while reducing the cost by a tenth compared to other prevailing GPT-3 [1] and 3.5 models [14] (e.g., text-davinci-002 and text-davinci-003.) However, our findings from Table 2 indicate that gpt-3.5-turbo underperforms in decision-making tasks relative to its predecessors, i.e., text-davinci-002, in all LLM-agents. Upon examination of trajectories from both models, we observed a noticeable hallucination with GPT-3.5 (gpt-3.5-turbo), as shown in Appendix E. We hypothesize that gpt-3.5-turbo might be a smaller-scale model that is more prone to hallucination. We also found a similar hypothesis drawn based on various experiments in [26]. Furthermore, gpt-3.5-turbo is primarily optimized for human conversation tasks, which could potentially compromise its performance on tasks such as code generation and reasoning[13]. Despite this, AdaPlanner demonstrates a remarkable level of resilience against hallucination even with gpt-3.5-turbo (Table 2), while ReAct and Reflexion are more sensitive to the hallucination issue. AdaPlanner's resilience against hallucination can be attributed to its use of code prompts, which provide a more formal and constrained generation space for LLM. For comparison, we implement an ablation version of AdaPlanner without the code interface by translating solution examples directly into plans and actions using natural language. Without the code interface, AdaPlanner's performance substantially drops in both ALFWorld and MiniWoB++ environments (Figure 4c), from  $81\%$  to  $46\%$  and from  $93\%$  to  $66\%$ , respectively. This significant performance drop underscores the essential role of the code interface in AdaPlanner.
(b)
Figure 4: Performance comparison on 134 ALFWorld tasks in different cases. We adopt the same settings as in Table 2. (a) and (b) presents the success rate (\%) with different numbers of closed-loop corrections: (a) compares AdaPlanner with different numbers of samples; (b) compares AdaPlanner and Reflexion with two LLMs. (c) shows the success rate (\%) of AdaPlanner with and without code interface (CI). (d) shows the success rate (\%) of AdaPlanner with and without skill discovery (SD). Note that for (a), the number signifies the total number of samples used across all six tasks.
Skill Discovery Improves Sample Efficiency. The skill discovery in AdaPlanner utilizes a long-term memory mechanism that retains successful solutions, thus boosting planning performance when faced with similar tasks. An ablation study depicted in Figure 4d compares the performance of AdaPlanner with and without the implementation of skill discovery. In the skill acquisition stage, we provide a maximum of one demonstration. In ALFWorld, AdaPlanner is prompted with only one expert demonstration of the simplest task (put). We evaluate the average success rate of the method on the remaining five tasks, which are comparatively more challenging and require additional steps for completion. In MiniWoB++, we apply zero-shot prompting, omitting any examples in the skill acquisition phase. For both environments, we operate the method using GPT-3.5 in adaptive closed-loop mode, and one round of skill discovery is conducted. As Figure 4d illustrates, the inclusion of skill discovery significantly enhances performance. In the ALFWorld tasks, the success rate of AdaPlanner nearly doubles when skill discovery is employed. Similarly, in the MiniWoB++ tasks, the overall success rate increases by approximately  $15\%$  with skill discovery. Moreover, we conducted an ablation study on MiniWoB++ to evaluate the impact of sample numbers, as detailed in Table 5. AdaPlanner with skill discovery requires only 15 samples to outperform the variant without skill discovery, even though the latter used twice as many samples. It is evident that skill discovery enhances sample efficiency.
# samples	38	30	20	15	0
With SD	92.87	84.06	79.17	75.17	60.38
Without SD	82.40	73.58	68.70	64.70	45.47
Table 5: Success rate  $(\%)$  of AdaPlanner on MiniWob++ with and without skill discovery (SD) across varying numbers of expert samples per task. We adopt the same settings as in Table 2 (GPT-3 version) and Table 3.
Many works have studied how to leverage LLMs as autonomous agents to accomplish decision-making tasks within text-based environments. Earlier studies, like Chain-of-Thoughts [24] and Zero-Shot Planner [5], utilize prompts to guide LLMs in generating complete action sequences for elementary tasks. For more complex tasks, methods like HuggingGPT [18] and Chameleon [12] also generate the initial plan of using different tools and then call the corresponding APIs for execution. Meanwhile, some other works also prompt LLMs to compose plans in Planning Domain Definition Language (PDDL) [10, 15]. However, all these plans are created in an open-loop fashion without adapting to feedback from external environments.
To address the limitations of open-loop systems, recent techniques have emerged that focus on establishing closed-loop systems. These systems are capable of leveraging environmental feedback, thereby facilitating more adaptive decision-making. ReAct [25] and Inner Monologue [6] allow LLM agents to take single-step actions according to the environmental feedback. Reflexion [19], as an extension of ReAct, tries to resolve this issue by enabling the ReAct agent to revise itself from past trials and errors. Moreover, RCI [8] starts by formulating a comprehensive plan, modifying the immediate action when the agent encounters a failure at the current step. While all the aforementioned
methods can adapt their decisions based on environmental feedback, they assume the LLM-generated initial plan is correct and do not adjust it. Rather, they solely modify the immediate action being executed and are easy to fall into local sub-optimal actions without considering the long-term plans.
To further enhance the agents' both capabilities of planning and adapting to environmental feedback, strict closed-loop architectures are proposed that can recursively refine the generated plans. DEPS [23] is one of the examples that initially proposes an entire plan and then applies real-world feedback to recursively refine it during execution. However, this method requires training a selector to generate a plan that is highly probable to succeed, which makes it difficult to generalize the plans and actions to other tasks. Besides, the required data for training the plan selector are often unavailable in practice and expensive to collect. In contrast, AdaPlanner generates and refines plans via LLM prompting, making it widely applicable to various decision-making problems. Another PDDL-based planning method is proposed in [21] with the ability to leverage feedback and modify the plans. However, the plan formulated on the training set remains static and lacks task-specific refinement during the execution of the evaluation task. In contrast, AdaPlanner dynamically refines the plan and adapts to various feedback throughout the task-solving process.
6 Conclusion and Limitations
We proposed AdaPlanner, a closed-loop approach enabling LLM agents to adaptively refine their generated plans according to environment feedback. We defined two different refinement strategies, in-plan and out-of-plan refinement, to fully leverage environment information. Furthermore, to mitigate the LLMs' hallucination issue and make them learn from the past experience, we proposed code-style prompting and skill discovery mechanisms. Through comprehensive experiments, we demonstrated that AdaPlanner outperforms the state-of-the-art baselines significantly and has better sample efficiency. Our ablation studies also showed the effectiveness of different components in AdaPlanner. One limitation of AdaPlanner is that it still requires few-shot expert demonstrations for solving complex tasks. Although AdaPlanner has already achieved better sample efficiency than existing methods, it is interesting to study how to further enhance AdaPlanner to solve complex tasks with no demonstrations in the future.
7 Broader Impacts
Our research approach focuses on treating LLMs as autonomous agents and improving their ability to solve complex sequential decision-making tasks. However, this research line carries inherent risks, including security threats, potential misuse, and unintended consequences such as job displacement due to automation. To mitigate these risks, it is essential for researchers and policymakers to collaborate in creating and implementing effective regulations to guide the development and deployment of these technologies towards positive outcomes. Additionally, we believe that the research community should coordinate efforts to design principles and techniques that prioritize safety and human values before LLM agents are deployed in various industries. This will help ensure that LLMs are aligned with ethical and moral standards while promoting their positive impact on society.
This work was supported in part by NSF (IIS2008334, IIS-2106961, CAREER IIS-2144338), ONR (MURI N00014-17-1-2656), IDEaS Cyberinfrastructure Resources, and Microsoft Accelerate Foundation Models Research Program.
[1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[2] W. Chen, X. Ma, X. Wang, and W. W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv, page 2211.12588v3, 2022.
[3] L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. Pal: Program-aided language models. arXiv, page 2211.10435v2, 2022.
[25] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023.
[26] J. Ye, X. Chen, N. Xu, C. Zu, Z. Shao, S. Liu, Y. Cui, Z. Zhou, C. Gong, Y. Shen, J. Zhou, S. Chen, T. Gui, Q. Zhang, and X. Huang. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models, 2023.
[27] D. Zhou, N. Scharli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. Le, and E. Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv, page 2205.10625v2, 2022.
A Experimental Setup
ALFWorld [20] is a comprehensive suite of synthetic, text-based environments, encompassing six distinct task types set, Pick, Clean, Heat, Cool, Examine, and Pick two, within a virtual household. Each task possesses a unique high-level objective (e.g., put some vase in safe, etc.) that necessitates agent navigation and interaction with various objects or receptacles (e.g., go to shelf 6, clean apple, etc.). To fulfill the stipulated task, the agent is required to implement a sequence of actions aimed at accomplishing the predetermined goal. However, given that an object may potentially reside in any one of over 50 possible locations in a task instance, the agent must sequentially explore each of these. Consequently, the entire action trajectory could involve more than 50 individual actions, presenting a significant challenge to the agent.
MiniWoB++ [11] is a task suite of simulation environments that covers a large range of computer tasks for net agents. The computer tasks start from simple button-clicking to more challenging ones with longer time horizons (e.g., click-checkboxes-large), reasoning (e.g., click-checkboxes-soft), unexpected pop-ups (e.g., login-user-popup), and stochastically varying layouts (e.g., multi-orderings, multi-Layouts). These challenges are suitable for evaluating our proposed closed-loop framework. Each task interacts with a  $160\mathrm{px} \times 210\mathrm{px}$  web environment, where the state space is purely the HTML code of the web. Following RCI [8], we define the actions space as two sets of operations, i.e., clicking and typing actions. The clicks allow the agent to interact with clickable HTML elements (e.g., webpage buttons). The typings are conducted with keyboard-based actions, such as inputting characters into the input box and stroking functional keys (e.g., ENTER, BACKSPACE). We select nine MiniWoB++ tasks where environment observations (i.e., the HTML code) change after certain actions: search-engine, tic-tac-toe, terminal, login-user-popup, guess-number, email-inbox, email-inbox-nl-turk, email-inbox-forward-nl, and email-inbox-forward-nl-turk. Take the task terminal as an example. Search results appear on the webpage after inputting the keyword and pressing the search button. Therefore, environment feedback can be interpreted from the change in HTML code and further leveraged by closed-loop planning. We also adopt and test the 53 tasks evaluated in RCI [8].
B Baseline Details
ALFWorld. Following a set of previous works [20, 25, 19], we evaluate AdaPlanner on 134 different environments. By default, we include one sample as an exemplar per task to prompt AdaPlanner. For the study presented in Figure 4a, we adopt the setting of prompted samples as in Table 6. For the study in Figure 4d, we use one sample of the simplest task put to prompt the rest of the five tasks, which are more complex and require more steps to solve. For baselines, we compare AdaPlanner with BUTLER [20], ReAct [25], and Reflexion [19]. BUTLER [20] is an imitation learning method trained with 100k samples per task. ReAct and Reflexion, as outlined in Table 1, are prompting-based methodologies utilizing an implicit closed-loop framework. They employ a total of 6 and 8 samples, respectively, across all six tasks. BUTLER results are sourced from [20]. We evaluate ReAct, Reflexion, and AdaPlanner empowered by both GPT-3 (text-davinci-002) and GPT-3.5 (gpt-3.5-turbo and text-davinci-003) models.
# samples	Pick	Clean	Heat	Cool	Examine	Pick two
2	Clean	Clean	Clean	Clean	Examine	Examine
4	Pick	Clean	Clean	Clean	Examine	Pick two
6	Pick	Clean	Heat	Cool	Examine	Pick two
Table 6: The specific allocation of samples for prompting each task is divided into three cases based on the total number of samples (2, 4, and 6) used across the six types of tasks. For instance, when a total of 2 samples are used for all tasks, a single expert trajectory sample for the Clean task is utilized to prompt four tasks (Pick, Clean, Heat, and Cool). Similarly, a sample from the Examine task is used to prompt two tasks (Examine and Pick two).
MiniWoB++. Overall, we report the evaluation results of RCI [8] and the proposed AdaPlanner in GPT-3.5 (text-davinci-003), along with three training or finetuning-based baselines: Computer
Control Agent Architecture(CC-Net) [7], Workflow-Guided Exploration(WGE) [11], and WebN-T5-3B [4]. CC-Net and WGE employ supervised learning and reinforcement learning with over 23K and 10 demonstrations per task, respectively. WebN-T5-3B uses 12K demonstrations to finetune a pre-trained language model. RCI is a prompting-based approach that is categorized as the implicit closed-loop method in Table 1, which utilizes 93 samples across the 53 tasks. For these 53 tasks, we first provide AdaPlanner with 38 human-written demonstrations and perform skill discovery to obtain another 21 additional examples, i.e., 59 examples are used for 53 tasks. Evaluation results of RCI, CC-Net, WGE, and WebN-T5-3B are sourced from the works of [8, 7, 11, 4], respectively.
C Proportion of two refinements
Environment	# In-plan refinement	# Out-of-plan	Proportion (in/out)
ALFWorld(134 tasks)	2.83	6.40	0.44
MiniWoB++(9 tasks with feedback)	0.33	1.78	0.19
Table 7: Average number and ratio of in-plan and out-of-plan refinement invoked per task.
Table 7 illustrates that out-of-plan refinement is invoked more frequently in both environments, emphasizing its important role in AdaPlanner's planning process. The average occurrence for both types of refinements varies between ALFWorld and MiniWoB++. This can be attributed to the differences in the average lengths of the trajectories in ALFWorld and MiniWoB++. (15.60 and 5.40 steps, respectively).
D Prompts
D.1 ALFWorld
Basic Information. The <basic_info> defines the agent and admissible actions for AdaPlanner. Note that the actual definitions of action functions are not specified in the prompt. Instead, only a formalized definition with several examples is provided, such that the planner can acquire how to compose a plan based on these actions. As can be seen in the following part, this <basic_info> prompt is used in both <initial_planning> and <refinement> prompts.
<basic_info> Prompt
You are a household agent. Here is some Python code defining a household environment:
Use literal.eval to convert the answer from ask() to a list. from ast import literal.eval
In the environment, you can ask questions to an assistant by ask(): from large_language_model import ask_llm as ask # for example: You have a list of receptacles, and you want to sort them by the likelihood of a soapbar appearing in them. You can do this by asking the assistant: receptacles = ['countertop 1', 'garbagecan 1', 'sinkbasin 2', 'sinkbasin 1', 'toilet 1', 'toiletpaperhanger 1', 'towelholder 1'] answer = ask(f'Sort the list of receptacles, starting from the one a soapbar is most likely to appear: {receptacles}. You should return a Python list.') # answer = ['sinkbasin 1', 'sinkbasin 2', 'countertop 1', 'towelholder 1', 'toiletpaperhanger 1', 'garbagecan 1', 'toilet 1']
Agent class represents the state of the agent, including its location, # what it's holding as well as the actions it can take.   
class Agent: def __init__(self, receptacles): self.location  $=$  None self.holding  $\equiv$  None self.receptacles  $\equiv$  receptacles