AutoPlan: Automatic Planning of Interactive Decision-Making Tasks With Large Language Models
Authors: Siqi Ouyang
Lei Li
Venue: EMNLP 2023 Findings (2023)
Keywords: Planning
Decision Making
Large Language Model

Abstract: Recent large language models (LLMs) are promising for making decisions in grounded environments. However, LLMs frequently fail in complex decision-making tasks due to the misalignment between the pre-trained knowledge in LLMs and the actual rules in the environment. Existing methods require either costly gradient computation or lengthy in-context demonstrations. In this paper, we propose AutoPlan, an approach to guide LLM-based agents to accomplish interactive decision-making tasks. AutoPlan augments the LLM prompt with a task-solving plan and optimizes it through iterative experience collection and reflection. Our experiments show that AutoPlan, though using no in-context demonstrations, achieves success rates on par with the baselines using human-written demonstrations on ALFWorld and even outperforms them by 8% on HotpotQA. The code is available at https://github.com/owaski/AutoPlan.

AutoPlan: Automatic Planning of Interactive Decision-Making Tasks With Large Language Models
Siqi Ouyang
Lei Li
Computer Science Department University of California Santa Barbara siqiouyang@ucsb.c edu
Language Technology Institute Carnegie Mellon University leilieee cmu. edu
Abstract
Texk Texcription:
Aetise take hos cibtublle
wa real
Recent large language models (LLMs) are promising for making decisions in grounded environments. However, LLMs frequently fail in complex decision-makin making tasks due to the misalignment between the pre-trained knowl- edge in LLMs and the actual rules in the environment. Existing methods require ci- ther costly gradient computation lengthy in-context demonstrations. In this paper, we propose AutoPlan, an approach to guide LLM- based agents to accomplish interactive decision- making tasks. AutoPlan augments the L.LM prompt with task-solving plan and optimizes it through iterative experience collection and reflection. Our experiments show that Auto- Plan, though using in-context demonstra- tions, achieves success rates par with the baselines using human-wrilten demonstrations on ALFWorld and even outperforms them by 8% on HotpoiQA. The code is available at https://gi thub. cwm/owaski/ /AukoPlAn.....n.
to snlue pir iotentwaing
disemattonn
fol Colloding actions:
actior (2r moccerr
take sxject fror racoptacle
hett objact aich recoptacee
accoor (wat bad a ait coatte 1]
Taok Inviunce: pur hat aad in sodktabble
tecteat))
This starts:
stulte marr
countactip stertatla toareer abdetable
Thoud str11 accrouss actson (wet
bed.))
ThanKc shoul atontable bed..)
Conrvatinn: You hext bac ricroww
actioro mached, Texk eatls.
The 1PriT nent take birl.]
X
Figure 1: Problem illustration: Plamning for decision- making tasks. Given the description of an environment, legit actions, and task instance, the goal is to guide an LIMM to generate a sequence of thoughts and ac- tions (highlighted in yellow) reacting to observations provided by the environment (highlighted in blue) to solve the task instance. An LLM may fail in complex tasks due the lack of prior knowledge.
in ALFWorld (Shridhar et al., 2021)), LLM agents struggle to find the correct action sequence within the maximum number of actions (Figure 1). The primary reason behind such failures is the misalign- ment between LLM's pre-trained knowledge (e.g.. generating fluent sentences) and the concrete rule of the grounded environment (c.g.. houschold item functionality in ALFworld). In the ALFWorld en- vironment, the agent can only heat an object with a microwave instead of a toaster. However, the LLM does not learn such knowledge during pretraining, eventually failing the task.
1 Introduction
The ability to make decisions lies at the core hu- man intelligence, enabling us to navigate through a multitude of choices and select the best possi- ble actions based on available information. Recent large language models, trained with trillions of tokens, have gained impressive reasoning ability and now have the potential to act autonomous agents for decision decision-making tasks in grounded en- vironments (Zhang et al., 2022; Chowdhery et al., 2022; OpenAI, 2023; Touvron et al., 2023).
Existing methods aligning LLMs desired en- vironments either employ reinforcement learning (RL) and imitation learning (IL) methods (Ouyang et al., 2022: Carta et al., 2023), or provide a few demonstrations to conduct in-context learning (ICL) (Yao et al., 2023). On the one hand, RL and IL methods require computationally costly gradient computation for existing LLMs. On the other hand, the performance of ICL methods highly depends on the selection of demonstrations. In this work, we propose AutoPlan, a purely
Decision-making tasks in grounded environ- ments can be as simple as calculating mathematical problems with an external calculator or as complex doing housework. Current LLM can easily use external calculator by decomposing the formula into atomic function calls (Bubeck al., 2023). However, LLMs frequently fail in more complex tasks in an environment with many objects and pre- requisite dependencies. Considering the Heat task

---

Experience Collection
Prospt
Oiservation
Taot Deser iption:
Thooghtff (1 need to Lake the bosl frus the sidvtable.1 Artion]take howl from sidetante I-]
Task Plan X;
You need to interace with staulated tousebald to salve 5on my intenleneing Trink]], Arttml] and
devervation You plck up bool 1.]
the location
[scorvation]] You
following actions:
target object
froe receptacle
Actioonlo torster
toaster.]
the object the receptacle
object tazon receptacle
deserusti on(on the toaster you notrinng]]
mext juct with racaptacle
5. Go ta the Enaster
Thought (Row need to test the boal with the teaster.] Action] heat howl with tosster 1]
4. Heat the target object with
Tase Instance: pat hot boul in sidetable
the the target receptacle
deservationAcctton falles. Teastor cornet hest bowt.]
Plan: X,
Place heated object
Loaktig
ThoughlLL mead check ricrosae insteed.] Aettonn
in/on the target receptacle
you,
diservationthee
span.]
sidecable and sidetable
Thought (111) try excrnone heat the object.)
Think[I should check steetaste for the bowl.]
Actionfheat bowl with nicrowave i] deservattion: You hest boul I wtth escrovave J. Naxirun number of actions reached. lask fails.
actionigo to sidecable 1J
@ssarnationjOn the countersop 1, you sue bool 1.]
Task Plan Xi+1
Correct
Plan Update
Reflection
1. Go to the location of
2. Take the object from the
target object
Task Description:
Current Task Plan:
Revision 2:
Summerizu interactions, Lduillfy flaved acttors, reutise the currect
recoptacle
Task Instance
sumsary: found the beal I on stdetable 1, I teted to haxt it with the taester 1 hat fatled. I tinally hest it with Eicrosave but falled the task by excewting the maximu allowed namber
the target object with the litcoowvee
the Microwave
Suamary 1:
Flav
Task Instance 8 Sumary B:
actions. abservation satd, need to best the bool with sicrowave instead toaster in this task. Aevision: change "tosste"" step 3-4 Into "Mtcrowove"
laae Actions B:
Fimm:
5. Go to the target receptacle
5. Place the heated object
Tast Tostance Summary 2:
in/on the target receptacle
Flav 2:-
updated Plan:
Figure 2: One optimization iteration of AutoPlan on Heat task of ALFWorld. Given the current plan X; (with wrong steps highlighted in red), the LLM agent collects interaction experiences from a batch of task instances (prompts and LLM outputs are highlighted in grey and yellow, respectively). Then, the agent reflects on the experiences and outcomes through summarization, flaw identification, and plan revision. Finally, the agent aggregates the current batch of task instances together with their reflections and updates the task plan X4+1 (with correct steps highlighted in green).
prompt-based method, to guide an LLM to solve such decision-making tasks without costly gradi- ent computation or in-context demonstrations. In high-level speaking, AutoPlan solves the task by iteratively interacting with the given environment conditioning on a task plan described in natural language. Figure 2 illustrates how AutoPlan finds an optimal plan to guide the LLM to heat an object correctly and put it at the target location. Auto- Plan starts with an empty plan and uses the LLM agent collect interaction experiences condition- ing on an initial incorrect plan. Then AutoPlan instructs the LLM agent to reflect on the collected experiences and revise the task plan based on the reflection. It further deploys the new task plan to collect more experiences with the LLM agent.
perience. We evaluate AutoPlan and other methods on two distinct benchmarks.
Our contributions are:
We propose AutoPlan, a novel prompting method to align LLMs with the need for grounded decision-making making tasks without computing gradi- ents or using human-writter demonstrations. Our experiments show that AutoPlan achieves on-par success rates with baselines involv- ing human-written demonstrations on ALF- world (Shridhar et al., 2021) and even 8% higher accuracy on HotpotQA (Yang et al., 2018).
We verify that larger batch size leads to more stable learning, and the explicit SIR reflection ensures the plan update is practical and progres- sive.
The primary technical challenge of this approach is to ensure stable and progressive plan optimiza- tion since the plan expressed in natural language can be highly slapdash and versatile. We propose two techniques in AutoPlan: (1) experience batch- ing and (2) SIR reflection. We batch multiple ex- periences before updating the plan to help reduce variance. We introduce an explicit SIR reflection (Summarization, flaw Identification, plan Revision) to elicit helpful information from the interaction ex-
Related Works
Finetuned LLM Agent Reinforcement Learning has been widely used to train LLMS to master in- teractive tasks. ChatGPT (OpenAI, 2023) applics Reinforcement with Human Feedback (RLHF) finetune a pre-trained LLM, enabling it commu- nicate interactively with humans. GLAM (Carta et al., 2023) uses LLM as a policy and finetunes it with online RL to improve its abilities to solve text-

---

In-Context Demonstration
Feedback Utilization
Plan
Need Test-Time
Method
Applicability
Refinement
ReAct (Yao et al., 2023)
Yes
Action
N/A
No
Code as Policies (Liang al., 2022)
Yes
Action
N/A
No
Reflexion (Shinn et al., 2023)
Yes
Action
N/A
Yes
Inner Monologue (Huang al., 2023)
Yes
Action
N/A
Yes
RCI (Kim et al., 2023)
Yes
Action & Plan Opt
Single task instance
Yes
DEPS (Wang et al., 2023)
Yes
Action & Plan Opt
Single task instance
Yes
AdaPlanner (Sun et al., 2023)
Yes
Action & Plan Opt
Single task instance
Yes
AutoPlan
No
Action & Plan Opt
All task instances
No
Table I: Comparison of various prompt- methods that employ a LLM agent to solve decision-making tasks.
AutoPlan is the only method that optimizes plan applicable all task instances without any demonstration and requires no test-time refinement of the decision-making process.
based decision-make making tasks. Experiments demon- strate that LLM policy significantly boosts sample efficiency. Other than RI., Xiang et al. also fine- tunes the LLM a supervised manner with experi- ences collected through Monte Carlo Tree Search (MCTS). However, RL and supervised learning re- quire calculating the gradients and updating the model's parameters, which is especially costly for LLMs.
instance. However, Reflexion conducts test-time reflection and the reflection for one environment does not transfer to others. RCI (Kim et al., 2023), DEPS (Wang et al., 2023) and AdaPlanner (Sun et al., 2023) start with an initial plan of the task and refine the plan and the decision-making pro- cess for each specific task instance. Our AutoPlan instead optimizes a task-level plan and directly ap- plies it to all task instances without further test-time refinement, which could be more efficient during inference.
LLM with In-Context Learning As the size of the model and corpus scales, LLM demonstrates In-Context Learning (ICL) abilities, i.e., LLM di- rectly learns from a few demonstrations of a task given in the context. Brown et al. shows that pre-trained LLM performs strongly on traditional NLP tasks, including question answering and cloze tasks, with ICL. More recent works focus the design demonstrations (Sorensen al., 2022; Lu et al., 2022). Sorensen et al. proposes to re- trieve demonstrations with higher mutual infor- mation between model input and output. Glob- alE&LocalE (Lu et al., 2022) uses entropy statis- ties to find the most performant permutation of demonstrations. Nonetheless, the ICL LLM agent is still sensitive to the provided demonstrations and requires additional human effort.
3 AutoPlan
In this section, we describe AutoPlan in detail. We first describe the general procedure of using LLM to solve an interactive decision-making task. Then we present AutoPlan that solves the task by text- based plan, obtained by iterative three-stage pro- cess: AutoPlan 1) collects interaction experiences using the task plan the current step. 2) reflects on the collected experiences, and 3) updates the plan.
3.1 Problem Formulation
We aim to design LLM-based agent accom- plish an interactive task described in natural lan- guage. The agent is provided with a natural lan- guage description of the task, possible actions, and environmental observations. The task description P includes a generic abstract description and a con- crete task instance with an objective. Let M be the LLM agent, A be the set of possible actions, and O be the set of possible observations from the environ- ment. One could augment the input with custom prompt X. At each step t, the agent M generates text action ag E A and receives a text observation Og E O from the environment. op denotes the ini- tial observation, which could be empty. We define
Prompt-based LLM Agent Techniques have re- cently been developed to adapt LLMs to solve decision-making tasks through prompts. Table I illustrates the main difference between works along this line. ReAct (Yao et al., 2023) explicitly reasons over past interactions and determines the follow- ing action based on previous thoughts, actions, and observations. Reflexion (Shinn al., 2023) built on top of ReAct and refines the interaction by iter- atively reflecting on its past failed trials of task

---

Prompt Name
Prompt Content
Thought-prompt
Identify which step of plan you are at. Show your thought about the one next action. Your thought should be faithful the plan step.
Summary-prompt
Summarize the interaction history in steps.
Flaw-prompt
Identify all flawed parts of the plan/action. Remember in this game, things are not like real world. The system message in observation is always correct and the plan plan/action may have flaws.
Rev-prompt
Suggest revision to the current flawed part of the plan. Only the flawed part.
Upd-prompt
Based on the above experiences of the game, rewrite the current game plan. Pay attention summary of successful jobs, and flawed actions and suggested revision of jobs. The plan should be generalizable all job objectives. The actions in the plan should also be in the form as in game description.
Table Prompts that AutoPlan uses in ALFWorld environment.
a reward function ((oor) = 1 if the objective is achieved based on the observations. The problem of AutoPlan is to design optimal prompt X to maximize the expected rewards over all possible task instances,
3.2 AutoPlan
AutoPlan treats custom prompt X' in the form of task-solving plan that includes sequence of abstract actions to exccute in different scenar- ios. Such a plan described in natural language resembles the policy network in deep reinforce- ment learning, but it is more explainable due to its textual form. It is also more -ofen-efficient than in-context demonstrations. Furthermore, state- of-the-art instruction-tunee LLMs demonstrate a strong ability to follow given plan.
X²" arg max Ep (R(o))))
(1)
where is the maximum number of interaction steps allowed.
As shown in Figure 2, we design a three-stage process to optimize plan X' iteratively: 1) experi-
Ideally, the optimal X should be adequate for all task instances of the same problem. Since the space of a custom prompt is vast, we frame such a prompt as a plan, which describes a sequence of actions in natural languages. Figure 2 shows a heat- ing task in ALFWorld (Shridhar ct al., 2021) and how the LLM agent solves this. Task description includes the available actions and an instance- wise objective (e.g., put hot bowl in the sidetable). We aim to find an optimal plan the custom prompt. After the task starts, the agent's current and visible locations constitute the first observation og. Then, the agent acts and observes the environment iter- atively until reaches the maximum number of interaction steps T.
ence collection with the current plan, 2) reflection
the collected experiences, and 3) plan update based reflection.
3.2.1 Experience Collection
AutoPlan starts with an empty plan Xg- At each iteration i, a batch of task instances randomly selected, denoted as P1, P2.... ,PB For each task instance P.. the LLM agent generates a sequence of thoughts and actions in response to observations from the environment.
Let Hi_1 P3 @ X O (op, @op,ão, an, ...... .0--1 be the past interactions before step l. Since we augment the action space with thoughts that do not affect on the environment, at each step t, AutoPlan first obtains the thought,
Following prior work ReAct (Yao et al., 2023), extend the original action space A to include L, the space of thoughts expressed in language. As shown in Figure 2, "thought" action (in the form of "Think(....'") does not elicit any environ- mental feedback and solely manifests the reasoning process of the LLM.
ãt ~ M(HL) + Thought-promph. (2)
where Thought- is provided in Table to make LLM agent act faithfully to the plan X, Then

---

we sample the next action given the thought ot,
In the example of Figure 2, the plan updater aggregates the task instances with their reflections and rewrites the new plan to use the microwave to heat the target objects instead.
d4 ~ M(H ① ã ① "Action:")
(3)
0: - F(a))
(4)
Hi- = H ① āz ① 0e or.
After obtaining a revised plan X1+1, we continue the iterative process until we reach maximum opti- mization iterations I. During inference, we follow the same procedure as experience collection except that now we use the final optimized plan X'.
where is the observation after action and F is a formalizer used to reformat the action to be acceptable by the environment. Details of the for- malizer can be found in Appendix A.I.
To summarize, AutoPlan uses LLM to solve a text-based interactive decision decision-making making task through a task plan described in natural language. The plan is optimized iteratively through a three- stage process. The final plan is then used during inference time.
As shown in Figure 2, at, at and or correspond to "'.ink "Action[.-.]"" and "Observation(..."" in the experience of a task instance, where the LLM agent successfully found the bowl on the sidetable but failed to heat it with the toaster.
3.2.2 SIR Reflection
Experiment
Given the experience H1 and the corresponding reward R(our) (denoted R'), we instruct the
We aim to answer the following questions:
1) Does AutoPlan improve upon baselines?
LLM agent reflect on the interaction history
2) Is AutoPlan efficient during inference?
through SIR reflection procedure: 1) Summa-
rize the interaction history, 2) Identify the flawed
3) Does batching stabilize the optimization?
steps of the plan, 3) Revise the flawed steps,
4) Does trio reflection ensure steady progression?
5j M(H + R + Summary-prompt) (6)
4.1 Data
f3 M(H O R2 Flaw-prompt)
(7)
ALFWorld is a text-based game enabling agents to navigate and interact with simulated household to accomplish six types of tasks. Each task instance comes with a high-level objective (e.g., put a hot tomato on the desk), and the agent can achieve the objective through low-level actions described in text (e.g., heat tomato with microwave 2, go to desk 1). Since the environment feedback of invalid actions provided in the original ALFWorld is too primitive, manually augment the feedback (Table 6) to include possible causes of the invalidity. Further details of ALFWorld can be found in the Appendix B.I.
r5 MMHH + RdC ① Flaw-prompp
(1 Rev-prompt)
(8)
where SummaryyFlaw/Re--rroomg are shown in Table 2. The summarization, flaw, and revision provide necessary and clear information for the plan updater to modify the current plan.
As shown in Figure 2, the reflection summarizes the key actions, successfully identifies the flaw part of the plan, where X₂ treats the toaster the appro- priate heating appliance, and suggests a revision to use the microwave instead.
We randomly sample 24 task instances for each type of task from the training games to opti- mize the task-specific plan and, following prior works (Shridhar et al., 2021; Yao et al., 2023), use 134 unseen validation games' to evaluate our method. ALFWorld evaluates the success/failure of a task instance by checking if the agent in the goal state (e.g. if the hot mug is already on the desk).
3.2.3 Plan Update
With the task descriptions PI, P2....... ,PB. the current task plan X, and the summarizations $1. 8B. identified flaws 11....,Jp and revi- sions r1, TB, we utilize the LLM to revise X, and obtain an improved plan X4+1.
1 MMX (1,,51,11,1)) SB. TB) + Upd-prompt)
(9)
HotpotQA is multi-hop question answering benchmark requiring reasoning over two or more
where Upd-prompt (as shown in Table 2) asks the LLM to generate an updated plan given the task instances and reflections.
'Unseen games have the same task types but different objects, roceptacles and houschold layout.

---

Success Rate
Method
Pick
Light
Clean
Heat
Cool
Pick Two
Supervised method
Method
Acc
BUTLER
46
22
39
74
100
24
Prompt methods w/ ground-truth demonstrations
Supervised Method
AdaPlanner Shot)
100
100
97
96
100
47
Chain-of- Muuin-of-Skille Skills
90
ReAct (2 Shot)
100
100
100
91
96
76
Prompt Methods
Reflexion (2 Shot)
100
100
100
91
100
94
ReAct (0 Shot)
70
Prompt methods w/o ground-truth demonstrations
ReAct Shot)
75
AdaPlanner (0 Shot)
0
0
0
0
0
0
AutoPlan
83
ReAct (0 Shot)
92
94
87
35
71
59
(h) HoqorQQ
Reflexion (0 Shot)
96
100
97
52
81
88
AutoPlan
100
100
97
96
90
82
(a) ALFWorld
Table 3: Accuracy and Success rate (%) of AutoPlan and baselines on HotpotQA and ALFWorld, respectively. AutoPlan consistently outperforms the O-shot baseline, achieves on-par success rates with baselines leveraging ground-truth demonstrations ALFWorld, and even beats the 2-shot ICL. baseline HotpotQA by 8%. Results of AdaPlanner are from the original paper since the author does not provide enough details for reproduction.
Wikipedia pages. As in (Yao et al., 2023), the LLM agent required to answer questions by interact- ing with Wikipedia API. The API supports three types of actions: (1) search[entity]: returns the first five sentences from the Wikipedia page of the en- tity if it exists or suggests top-5 similar e (2) lookup(string): returns the following sentence containing string. (3) finishfanswer):: finishes the task with answer.
4.2 Method Configurations
We use GPT-4-0314 (OpenAI, 2023) the LLM across all experiments. The maximum number of actions is 10 for HotpotQA and 35 for ALFWorld. The default batch size of task instances 4 for both HotpotQA and ALFWorld. We use nucleus sam- pling with = 0.9 during optimization and greedy decoding during evaluation. The full prompt tem- plates of both environments can be found in the Appendix A.2.
We randomly sample 50 hard (question, answer, supporting facts) triples from the official training set to optimize the plan and sample 200 questions from the official development set as the test set?. The final answer is evaluated by three external hu- man annotators rather than exact-match (EM) since
4.3 Baselines
We compare with the following baselines.
ReAct (K Shot): The custom prompt X' consists of demonstrations manually written by human. We reuse the demonstrations provided in (Yao et al., 2023). We have K = 6 for HotpotQA and K - 2 for ALFWorld.
the answer provided by the agent and the gold an- swer can differ drastically in form but share the same meaning. We include the complete annota- tion instruction in the Appendix B.2 and take the majority vote of 3 annotators the final evaluation result. The agreement rate (all three annotators agree with each other) is above 90% for all consid- ered models.
Reflexion (K Shot): Built on top of ReAct, Re- flexion conducts iterative test-time reflection for each environment, using interaction history to revise the actions in the following iterations. We set the number of iterations to be five and use the same custom prompt as in ReAct.
"We notice that this API retrieves the latest information instead of the Wikipedia dump (2017-10-01) used to build HotpoiQA dataset, so we modify to return historical page of entities before 2017-10-01.
AdaPlanner (Sun al., 2023) (K Shot): Ada- Planner also proposes to optimize the plan with LLM but using a code-style custom prompt, which is more rigorous but also more restric-
2200 is trade-off between our budget and evaluation uncertainty.

---

tive than AutoPlan. Note that AdaPlanner still requires human-written demonstrations initial- ize the plan.
baich Lod
0.7
baich sic
batch s
Supervised Baseline: For HotpotQA, we select the best available supervised method Chain-of- Skills (Ma et al., 2023) from the leaderboard of fullwiki setting. For ALFWorld, we choose BUT- LER (Shridhar et al., 2021), an imitation learning agent trained with 10 human demonstrations for each task type.
Success Rate
0.-
0.4
Number of Training Iterations
4.4 Main Results
(a) Batch size
Success Rates The success rate and accuracy of AutoPlan and baselines in ALFWorld and Hot- potQA are shown in Table 3 respectively. In ALF- World, AutoPlan achieves on-par success rates with ReAct (2 Shot), AdaPlanner (1 Shot), and Reflex- ion (2 Shot) all six types of tasks and outper- forms zero-shot baselines by at most 44% Heat task. Notably, AutoPlan accomplishes the first four tasks nearly perfectly with success rates approach- ing 100% and success rates above 90% and 80% for the latter two. In HotpotQA, AutoPlan answers questions even 8% more accurately than ReAct (6 Shot) with human-written demonstrations of how to use the search tool, thanks to the optimized plan.
WiS reflection -4h- Vi rellction
Success Rabe
0.4
02-
0.
Number of Training iterations
(b) Reflection
Figure 3: The success rate of AutoPlan on task Heat of ALFWorld optimized (a) with different batch sizes and (b) with/without complete reflection process. We plot the mean (marked line) and standard deviation (band) of five independent runs. larger batch size significantly improves the success rate average and reduces the variance. The reflection process in AutoPlan ensures the steady improvement over iterations.
Error Analysis Of 137 ALFWorld test instances, AutoPlan fails seven due to the inability to locate the target object. One failure stems from a lexical misunderstanding where the LLM confuses "cup" with "mug". Another results from an atypical ob- ject location, with the apple to be heated found in a garbage can. The remaining five failures occur due to the LLM's erroneous prior assumptions about po- tential object locations, even though the plan points the model towards the most probable ones. Once the agent locates the task instance's 's target ob)ect(s), it performs all subsequent actions correctly. We observe similar failure patterns in cases of ReAct (2 Shot). With neither the optimized plan nor in- context demonstrations, ReAct (0 Shot) struggles to find the correct action sequence to clcan/ccol/heat the object even if it finds the target object(s).
again shows that the ICL method is sensitive to the quality of demonstrations.
Training and Inference Cost We measure the training and inference cost of AutoPlan and base- lines per instance in Table 4. The cost is calculated based on the official documentation'. AutoPlan requires only marginal additional cost compared ReAct (0 Shot) while achieving the best result on ALFWorld and HotpotQA.
4.5 Ablation Study
In HotpotQA, AutoPlan achieves better logical consistency than ReAct (0/6 Shot) thanks to the step-by-step plan. ReAct (6 Shot) performs well when only a few actions are needed but can diverge unreasonable thought and action processes when the number of actions is considerable. One primary reason is that the demonstrations used in ReAct (6 Shot) involve no more than five actions, which
The plan optimization process of AutoPlan can be precarious due to sampling-based decoding. To tackle this, AutoPlan batches multiple task in- stances together in one iteration to stabilize the op- timization and applies explicit 3-step reflection to elicit helpful information from the interaction
ptttps://openai.comm//

---

Method
Training Inference
Method
Training
Inference
ReAct (2 Shot)
N/A
ReAct (0 Shot)
N/A
0.15
Reflexion (2 Shot)
N/A
17
ReAct (6 Shot)
N/A
0.46
AdaPlanner (1 Shot)
N/A
2.1
AutoPlan
0.26
0.23
AutoPlan
1.8
1.6
(b) HotpotQA
(a) ALFWorld
Table 4: Average cost (unit: US Dollar) per question used by methods in ALFWorld and HotpotQA environments. Cost is calculated based on the OpenAl pricing document.
Original Plan
Eaperencces 11s
OrigralFion
Expertacen Inc
hew
beciptucla tuclk meeded the
Searter
Teltorr
Testierr
Hocptacla leaied
nere, Lite
Tallae
5.
the obsect reets he warted,
Lf the oiject siek hastes, Esonsimman
ab rests to be heated.
bos aigect with the toiting
1. Trsec 12 best aned bus
ontact benting
Telewr Et strmare
the tarast renetacle,
Or to the tirrtt resooracie. targer recectatle.
sicroute ssccmssialty
Flace Fasted abject Lr
Snive
Figure 4; An illustration of the impact of batch size on the plan update. The agent with batch size two only tried the toaster heat the object, with batch size four, the agent also tried the microwave, the only allowed heating appliance in this task-the larger the batch size, the more chance the agent can find the correct action sequence.
history. Here, we demonstrate the effectiveness of batching and reflection on task Heat of ALF- World as this is the task that AutoPlan achieves the largest improvement against the baseline ReAct (0 Shot) with no plan. We first run AutoPlan five times with both batch sizes 2, 4, and 8, and then run five times with and without the last two steps of reflection (flaw identification and revision) Then, we measure the mean and standard deviation of test success rates of plans produced in the first three iterations.
Reflection ensures the optimization goes in the right direction. As shown in Figure 3b, Auto- Plan with the complete reflection obtains steady im- provements after cach iteration, while the success rate of AutoPlan with only the interaction summary bounces back and forth between 0% and 30%, even below the success rate of ReAct (0 Shot). Again can visualize such difference in Figure The agent went the microwave and tried to heat the object but failed because of the wrong action se- quence (the correct action sequence can be found in Table 8). AutoPlan with complete reflection ex- plicitly identifies such flawed behavior from the observation and proposes revision, which is later integrated into the new plan. However, AutoPlan without flaw identification and revision does not realize the valid reason for failure and leads to un- desired plan updates.
Larger batch size significantly stabilizes the op- timization process. As shown in Figure 3a, a larger batch size improves the average success rate and reduces the standard deviation during optimiza- tion. We also conducted t-test comparing batch size 2 and 8 results, and the p-value is more than 10 for all iterations (see Table 5). Carefully examining the interaction histories, we find that with larger batch size, the agent is more likely to hit the right action during the experience collection stage. As illustrated in Figure 4, the agent with batch size 2 only tried the toaster to heat the object, but with batch size 4, the agent also tried the mi- crowave, the only correct heating appliance for this task.
5 Conclusion
We propose AutoPlan, a prompt-based method, to enable LLM to solve interactive decision-making tasks without gradient computation or in-context demonstrations. AutoPlan conditions LLM on an additional task plan described in natural language, which is obtained through an iterative three-stage stage process. Experiments show that AutoPlan achieves better results than baselines and is also efficient dur- ing inference. The ablation study further confirms the effectiveness of batching and explicit reflection
*We keep the summary step of reflection since the plan update is meaningless without the interaction summary.

---

Orignal Pann
Sunnayy
New Pan
Fdll Ielacionn
New
Roject
deceaaiee randed 240
racattiaa mecid the Jib,
necettele Hew,, Lite
exject feard take
Samery: Faud bredd the actrbant Tound an e9g
deject foume
Han:: was cary braud attraane
the stcrmat
2o to socroums
heating
coner the unvee falled
ather cijaca,, cake
smite the steretee
FASCIRE
het
Hast
Fiao ti brated 4. Haran
Babe, to cit
oen withed plack the
Leget inceptaa
extect Invve
lorght werppta
Figure 5: An illustration of the impact of reflection on the plan update. With only summary of interactions, the plan updater needs to guess the cause of failure. Eventually leads to the wrong thought that the objects inside the microwave need to move before heating. With flaw identification and suggested revision, the plan updater understands the flawed part of the plan and rewrites the plan to heat the object directly.
in stabilizing the plan optimization process.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Limitation
The improvements of AutoPlan mainly come from two sources: 1) the correct action sequence sam- pled during exploration; 2) the environment feed- back when incorrect actions are sampled by the LLM agent. As shown in Table 6, the feedback directly tells the agent which aspect of the action is invalid. Without such fine fine-granned feedback, the agent needs to collect more experience, i.e., larger batch size, to make sure the correct action sequence is sampled with high probability.
Another limitation is that in order to make Au- toPlan works without any demonstration, we rely on GPT-4-0314 14 to generate action sequences, re- flect on the interactions, and update the plan. We tried to use GPT-335-turbo--03. fails to follow the plan faithfully even explicitly prompted do so; 2) generates too many halluci- nated contents about the environment, which could (possibly) be handled by better prompt design, but that requires excessive human effort, contradicting the goal of AutoPlan to reduce human effort as much as possible It is worth trying other otate-of- the-art LLMs such Claude to see which one also works.
Sébastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Fric Horvizz, Fiee Kamar, Pe- ter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg. Harsha Nori, Hamid Palangi. Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general in- telligence: Early experiments with gpt-4.
but find out 1) it
Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Y Oudeyer. 2023. Grounding large language models in interac- tive environments with online reinforcement learn- ing.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim. Barret Zoph, Alexander Spiridonov, Ryan Scpassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai. Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Frica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy
Ethics Statement
While AutoPlan is capable of functioning solely with task descriptions and observations, it imper- ative to excrcise caution while using it in high- stakes circumstances, given the inherent unpre- dictability of LLMs. Furthermore, we earnestly recommend that users carefully assess if the ob- jectives could inadvertently cause harm to others before putting AutoPlan into action.

---

Hellsr-Hlllttem,, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
Taylor Sorensen, Joshua Robinson, Christopher Ryt- ting, Alexander Shaw. Kyle Rogers, Alexia Delorey. Mahmoud Khalil, Nancy Fulda, and David Wingate. 2022. An coormation-theorete approach to prompt engineering without ground truth labels. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers), pages $19-862, Dublin, Ireland. Association for Computational Linguistics.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and brian ichter. 2023. In- ner monologue: Embodied reasoning through plan- ning with language models. In P Proceedings of The 6th Conference on Robot Learning. volume 205 of Proceedings of Machine Learning Research, pages PMLR.
Haotian Sun, Yuchen Zhuang. Lingkai Kong. Bo Dai, and Chao Zhang. 2023. Adaplanner: Adaptive plan- ning from feedback with language models.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière. Naman Goyal, Eric Hambro. Faisal Azhar, Aurelien Rodriguez. Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023. Language models can solve computer tasks.
Jacky Liang. Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, brian ichter, Pete Florence, and Andy Zeng. 2022. Code as policies: Language model programs for embodied control. In Workshop Language and Robotics at CoRI. 2022.
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023. Describe, explain, plan and se- lect: Interactive planning with large language models enables open-world multi-task agents.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- Proceedings of the
Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang. Zichao Yang, and Zhiting Hu. 2023. Language models meet world models: Embodied experiences enhance language models.
shot prompt order sensitivity. 60th Annual Meeting of the Association for Compu- national Linguistics (Volume 1: Long Papers), pages Dublin, Ireland. Association for Compu- tational Linguistics.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpoiQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369 2380, Brussels, Belgium. Association for Com- putational Linguistics.
Kaixin Ma, Hao Cheng. Yu Zhang, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. 2023. Chain-of-skills: configurable model for open-domain question an- swering.
OpenAL. 2023. Gpt-4 technical report.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations.
OpenAl. 2023. Introducing ChatGPT.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre- trained transformer language models.
A Detailed Implementation of AutoPlan
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal rein- forcement learning.
A.1 Formalizer
The formalizer is again a LLM call with specially designed prompt as shown in Figure 6.
Mohit Shridhar, Xingdi Yuan, Mlee-Alex Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht, 2021. (ALFW)ord Aligning text and embodied environments for interactive learning. In International Conference on Learning Representa- tions.
A.2 Full Prompt of AutoPlan
Full prompts of ALFWorld and HotpotQA are shown in Figure 7 (experience collection and re- flection) and Figure 8 (plan update).

---

Iter 1
Iter 2
Iter 3
p-value (2 & 4)
0.44
0.35
0.013
p-value (2 & 8)
0.007
0.110
0.005
Table 5: P-values of t-test between results batch size 2 & 4 and 2 & 8. Batch size 8 delivers significantly higher success rates than batch size 2.
A.3 Feedback
The examples of augmented feedback of ALF- World are shown in Table 6. We do not add ad- ditional feedback for HotpotQA upon the original one designed in ReAct (Yao et al., 2023).
B Additional Details in Experiments
B.1 Environments
The task types and templates of task objectives of ALFWorld are listed in Table 7. The allowed actions can be found in Figure 7. The correct action sequences for each task can be found in Table 8.
B.2 Human Evaluation
We invite three external human annotators con- duct human evaluation on HotpotQA. Instructions for human annotators are shown in Figure 9. We take the majority votes from human annotators as accuracy and also compute the agreement among three annotators.
B.3 Significant Test
We conduct t-test between success rates of plans generated by batch size 2, and at each iteration. The p-values are shown in Table 5.

---

Volid action formats follows: go to "recep"
take "object" fron "recep" "recep"
put "object" apen "recep"
Valid action fornats are as follous: search(entity]] (ookup[keyword shnsse(
"recep"
use "recep"
clean "object" with "resep" heat "object" with "recep" cool "object" with "racap"
Fornalize the following action strictly into the above valid action formats. there are multiple actions, formalize the First one.
The "object" and "recep" should be replaced with real "desk 1",
and indices, e-g., "apple
Action to formalize: (row (raw_action action) Fornalized action: (fornalized action)
Formalize the following action strictly into the valid action fornats. actions, fornalize the first one.
multiple
Action to formalize: (raw action) Formalized action: (formalized acction)
(b) HotporQA
(a) ALFWorld
Figure 6: The prompts of formalizer for ALFWorld and (b) HotpotQA, With input raw action, the L.M generates the formalized action.
Error Type
Example Action
Augmented Feedback
Missing Index
lake tomato from countertop
You miss index of tomato, e.g., tomato 1.
Wrong Location
take tomato from countertop
You are not at countertop 1.
Invalid Receptacle
take tomato from countertop
countertop is not a valid action in this household.
Closed Receptacle
take tomato from cabinet 1
cabinet is closed.
Inventory Limit
lake tomato from cabinet 1
You cannot hold more than one object.
Not In Inventory
put tomato in/on cabinet
You are not carrying tomato 1.
Not In Inventory
put tomato in/on cabinet
You are not carrying tomato 1.
Invalid Heating Appliance
heat tomato with toaster 1
toaster cannot be used for heating.
Table Examples of fine- fine-grained feedback with respect various erroneous actions.
Task Type
Templates
put obj in recep. put some obj on recep.
Pick
look at obj under the desklamp.
Light
examine the obj with the desklamp.
put a clean obj in recep.
Clean
clean some obj and put it in recep.
put a hot obj in recep.
Heat
heat some obj and put it in recep.
put a cool obj in recep.
Cool
cool some obj and put it in recep.
put two obj in recep. find two obj and put them in recep.
Pick Two
Table 7: Six task types of ALFWorld and their objective templates.

---

NUFWorls: Experience Collection and Reflection
Description:
You need to interact with simalated household to salve
Idantify which stap of plan you are at. Show your thought about the ane next action. Your thought should faithful plan step.
gob. The sted hause has many objects, and receptar les.
Actions on the objects and receptacies are as follmes:
(3) recep
an currently at step 3. should go to the toaster.
(2) object
recep: take object from some recestacle
Action:
and carry 18 with you
go to toaster
(3) put objject in/on recep: put the object you are carrying
diservationn on the toaster 2, you see nothing.
In/on some receptacle
(4) open recep: open some closed receptarle
Identify fy which step of plan you are at. Show your thrught
(5) cllose recep: close soew open receptacle
about the ane next action. your thought should faithful
(8) use necep: use som receptacle
plan step.
(?) cllean abject with recep
an currently at step 4. need to hest the boa) with the toaster.
(R) heat object with recep
(9) cool object with recep
Action:
You job 1s to heat sone object and put it in/on some receptarile. le.
heat bowl with toaster
tiservat ton: Action fai led. Toaster cannot heat houl.
Game Objective: You need to put a hot potato in fridge.
Identify which step of plan you are at, Show your thought about the one next action, Your thought should be faithhful plan step.
Game Plan:
1. Go to the most likely location of the object
It seens toaster is not working, I need to check the eicroave instead,
2. Open the receptacle necessary
3. If the object is not found, continue searching other
Action:
locations
2. go to nicrowive 1 Observation: The nicrowave is open, In it, you see nothing,
4. If the object is found, take the object from the
receptarle
5. Go to the toaster
Identify which stop of plan you are at, Show your thought about the one next action. Your thought should be faithful to the plan step.
6. Heat the object with the toaster
7. Go to the target receptacle
8. Put the object in/on the receptacle
I'1l try using nicrowave to heat the object.
Action:
Gane Starts, You are in the nidole of roos, Looking around you, you see courtertop i, sicrovave 1, toaster 1, sidetable 1, and a sinetable 2.
hest bowl with vicrouae
deservation: You heat boul with nécrowave 1.
Maxdrue number of actions reached. Task roils. Summarize the interaction history in steps.
Identify which step of plan you are at. Show your thought about the one next action. Your thought should be faithful to > Found the bond 1 on sidetable 1. tiited to hest it with the plan step.
the toaster but fodled. I finolly hest it with ricrowvv but failed the task by sxcouding the eaxisun allowe number of actions.
I an currantly at stop 1. I should check sidetable for the boxt.
Action:
go to sidetable 1
Lyentify the flawed part of the plan/action. Renweber in this gare things are not like real world. The systes ressage is always correct and the gane plan/action nay have flans. As the observation said, I need to heat the boal with
Doservation: on the countertop 1, you see a bowl 1, Identify which step of plan you are at. Show your thought
about the one next action. Your thought should be faiehful to the plan step.
eicrouave instead of coaster in this task.
I an currantly as stop 2. I need to take the bowl froe the
sidetable.
suggest revision to the current flaved part of the plan. only the flased part.
Action:
take bowl: Fros sicetable 1, iiservation: You pick up bow) 1.
change "coaster" in stop 5-6 into "hicrowave"
Hotpot(24: Experience Coluution and Doflection
Task Description:
currently at step I need to look for the keyword "tap dancee"" shown in the neturn of search query.
Solve question answering task with interlenwing Thought, about the
Action, doservation steps. Thought
Action:
types: searchfentityl which searches the entity Mikipedia
curnent situation, Action
lockupltap dance]
dbservation: (Result
Hines born 1943 in
paragraph it search.
city, the
Alew (Lauless)
will
nobert
rusician, actor-[2] [2] began career the age of five, studying tap
(2) [ookup[keysars]
containing keysard the returns the sentence passage.
Henry Letang
[1] ...
(3) fanish[andwer), which returns the answer rinishes the task. Call finish]]
his Brosdhay debut in The in Pink Tights (4).
answer not Found.
Identify which step of plan you are at, Show your thought about the are next action. Your thought should be falthful to plan step.
question: Maurice Hines and brother were Farous for what?
as now at step since the anseer "tap dancing" is found.
Tesk Plan:
1. Identify the rain keywords of entities.
Action:
2. Search for the main entity of keyword on Mikipedia using
fintsh[Paurice Hánes and his brother were fasous for tap dancing.]
(entity.
3. Look for the next sentence containing the keyward in the
current Wikipedia page.
Task finished, the ground truth ansuee is "dancer Gregury nines" and the correct entities to search are "Mot heer" and "Maurice Rines", Sumariize the interaction history concisely.
4. sepeat stop 2 and 3 as necessary until the answer is found.
5. Finish the task with finish answen].
searched for "Nourice Hines" on MIkipedia and found Informatio about his and his brother Gregory Hines. then looked the keyaard "tap dance" and found that both Naurice and dregory Hines sere farnus for dancing. The answer provided that Hourice Hines and his brother Gregory ltinss fincus for tap dancing-
Identify which stop of plan are at. Show your thought about: the one next actioon Your thought shouls be faithful to the plan step. currently
1. entity "Maurice Rines". need to search for the entity using saarc]entityy]. Action:
according to
Identify all flaved parts the plan (not flaved action).
Hourch Hinas] American actor, director, singer, and charsographer. [1]. He Gregory Hines. careen the age rive, studying top dance at Henry LeTang Dance Studio Marhatton, o. Shortly after, the brothers began touring as opening act such headliners as Lionel Haspton and Gypsy Roze Lec.
Stop eight already present in such
for the possiollity that first paragraph
Paurice (been December 1963)
retureed search]] necessary, sight can directly proceed to stop to provide the answer.
of
began
lookup[]
Suggest revision to current flaved part the plan. Only flawed part.
Step 3. the answer is not found in the first paragraph returned by sesrch(entity), use [ookup[[rysarr]] to look the next sentince containing the keyword
Identify which step of plan you are at. Show your thought shout the one nest action. Your thought should be faithful
to the plan step.
current Wikipedia page opened by search(entsty)..
Figure 7: Full prompts of experience collection and reflection of ALFWorld and HotpotQA. LLM outputs are preceded with '>'

---

4L%Wo For
For
Gans bescriptta::
Tast Desertatin:
joh, recestarles, voldd arriors abjeets and receptacles
thught ahout current diratiee, touupe, three
1. rellows:
(3) search ey wetch searches the seact entity or iskipedia and returna the
go
First propaast exists sot, retuen
objlct frem recept take object fren neceptacle and wtth
toppp/eerarsl, which
sest contairing the
(3) Eat dbject Lazon noup ext the deject you are carrying Invon sans recoptacle
currect passage
open
fielea]]
receptacle
found.
(5) rece: use sow recestacle
heet objlct with
1. Esentaty made kapwends ancition..
(X) cool abject with nece)
2. search the satn catity leyaoed as vikipedia using cescilfattit)]
receptacle,
and necessary until tapood 5. Pixish the tast with (inist
4.
feurs.
Carrect bule clan:
iitely
2. the recustecle necessery
Question 1: Harrice tines his brother vere famous for whut!
5. 29 the object 1s not fousd contine searching other locetions
Sunnery 1: scarstee for Vaarice tiies"" or Mikipolla and found infermation sbost that tush Harrics Gragory tores faroua
object recestacle
0o the touater
feecing,
prosisss
weat the atgert with rastter
was then Nautice stree and his beorhee aregory sines were Smous tap doviing. present the first panagrggl returel to the serna]] action sed ooes, the losksg]] actios night oot be recessary and the assistant con disectly procedd to
receptacia
Pat the object lnzen receptocle
Samury I foand the bol sideranne tried teat it wits the toaster
focision 1: Step the asser is not foond in the first paragragh returned by search centity, are Ieoto]stgurre last for nest centwace contateiin the page
sut rasled, fisally hest it with alcrouse but falled the tast by eaceeding the
slowe 1: as the sbeercation sa14, 1 need to hear the howl with alcnowee Instwal of raserer mis rack. change
sxerd abose expelleazes of the rask, rearthe curcent tas plan. Pay arre
5-d
auccesated questions,
revision falled qestionn the plan soaule not be specific to axection gereralizable quertiore. The actione in the plas should also in the form
Iesed abore sopertanzee of the gams, reardie corrent gams plan. Pay mara sobe, of Falled 200. the plan sooudd not soscific to dejectties tut pesraatzable abgectives. The artione in the plan should also be the farm
suganetec
hew Plas:
description.
search for the sain enttity of beywond as vikipedia usting ceanchjast (ry].
Nav Plan:
3. ze the anawer containing
found firat zersgruph returned by searet Losk page.
dpee the le 1F ancaccory
cepeor stoy and as neteccarr seril the ansaen is fixnd.
3. 27 the dbpect noe found, cortinus anarching other Locationn
2. Färash the tast with fantanjerawer).
5- Co to the tosster
5. text the seject with toaster
4. Pat the angect te/on the receptacle
Figure 8: Full prompts of plan update of ALFWorld and HotpotQA. LLM outputs are preceded with '>?
Task Type
Correct Action Sequence
Pick
go to the receptacle with target object; pick it up: go to the target receptucle; put it down.
Light
go to the receptacle with target object: pick it up: go to the receptacle with a desklamp: use the desklamp.
Clean
go the receptacle with target object; pick it up: go to a sinkbasin; clean the object with the sinkbasin; go to the target receptacle; put it down.
Heat
go to the receptacle with target object; pick it up: go to microwave; heat the object with the microwave; go to the target receptacle; put it down.
Cool
go to the receptacle with target object; pick it up: go to fridge; cool the object with the fridge; go to the target receptacle; put it down.
Pick Two
go to the receptacle with the first target object; pick it up; go to the target receptacle; put it down; go to the receptacle with the second target object; pick it up: go to the target receptacle; put it down.
Table 8: Correct action sequences for each type of task in ALFWorld.

---

Annotation Instructions
Objective
The primary oiyective is to eveluste the qusltt of predicted answers genersted sted by an sutomsseed method againet the ground-inuth anovers for sei 200 diris points from the comresponding ground-rruh supporting
HolpotQ4
Workflow
components poin (question, ground answer, supporting predicted answer).
Check Accuracy: Deternin whether the predicted answer correctty adcresses the question, considering the ground truth answer and supportng focts.
Consistency: Verlly prodicted arowwe consistam the suppotting facts.
Tagging: Use the annotatin tool to tag the predicted anower az ether Correct 'inoomect, and add comments for clantication, necessary.
Guidelines
Review Data Point
Theroughly road the components (question, ground truth answer, supporting facts, and predicted anover) before making any evaluctions
Check Accuracy
The precicted answer should directly answer the question posed.
Compare
they syrooyymoss
the predicted answer partall correct but mizsing vital information, mark as 'incorrect' and note what is missing the comments.
Check Consistency
The predicted answer must align with the supporting facts provided. the answer beyond or contadies these facts, mark as tmoomeet.
Inconsistencies can include incomeet names, dales, events, any information that deviates from the supportng facts.
Tagging
Use the prosided legging system the annoration tool to categor oe the predicted arewwer as 'Correc or Incoreett
predesed incomplete," etc]
rillly cluilyy sperilically
(e.g.. weong.
Examples
Data Point Example
Question Who
Dreamn speach?
Ground Truth Answer: Martin Luther King
Supporting 1963, Vantin Luther King dellivered Have speach Wisshingron
Predicted Answer:
Correct Annotation
Tooping: 'Corrett'
Figure 9: Instruction for human annotators to conduct human evaluation on model predictions on HotpotQA.


Reviews:
[{"review_id": "QW0n5Dudng", "reviewer": "Reviewer_hFiy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Human Evaluation**\n\nA1: Thank you for the suggestion! We conducted an additional human evaluation of HotpotQA with three external human annotators. The full annotation instructions will be included in the paper's appendix. The annotation process involved four phases: reviewing the data points, checking the accuracy of predictions, verifying the consistency with supporting facts, and logging the results. The results are as follows:\n\n|  | AutoPlan | ReAct (2-Shot) | ReAct (0-Shot) | Supervised |\n| --- | --- | --- | --- | --- |\n| Majority Votes (≥2 votes) | 80% | 73% | 71% | 91% |\n| Agreements (all 3 annotators agree) | 90% | 93% | 92% | 95% |\n\nWe used the majority votes from the three annotators as the evaluation results. AutoPlan still outperforms ReAct, whether or not in-context demonstrations are used. Additionally, we computed the level of agreement among the annotators and found that in over 90% of cases, all annotators were in agreement.\n\n**Q2: Extra Tuning for AutoPlan versus AdaPlanner**\n\nA2: AutoPlan does require a training phase to obtain the task plan. However, during inference, AutoPlan directly generates the action sequence based on the optimized plan and observations. In contrast, baselines like AdaPlanner and Reflexion involve a test-time self-reflection process to iteratively refine the action sequence. This introduces an additional inference cost, as demonstrated below in the ALFWorld (unit: US dollar):\n\n|  | AutoPlan | ReAct | AdaPlanner | Reflexion (5 trials) |\n| --- | --- | --- | --- | --- |\n| Inference Cost per Case | 1.6 | 3 | 2.1 | 17 |\n| Training Cost (24 cases) | 43 | N/A | N/A | N/A |\n\n**Q3: Disadvantages of Using In-Context Demonstrations**\n\nA3: On one hand, using in-context demonstrations increases the inference cost, as shown above and in Table 4 of the paper. On the other hand, generating these demonstrations requires additional human effort to annotate correct action sequences based on given environments and reward functions. AutoPlan, in contrast, only needs the latter two (environment and reward function) to operate effectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The prompting method could transfer an existing langauge model into interactive agent tasks.\n2. The author demonstrates great performance.\n3. To stablize the tuning, the author propose to use trio reflection and large batch sizes.", "reasons_to_reject": "1. The authors themselves did the human evaluation.\n2. Compared with other baseline methods, AutoPlan needs extra tuning for the general task, while baseline such as AdaPlanner does not requires extra training time. Also, the author did not clarify what is the disadvantage of using in-context demonstration", "questions_to_authors": "", "paper_topic_and_main_contributions": "They propose AutoPlan, a prompting method to align LLMs to decision-making tasks. The method achieves similar performance with baseline with human-written demonstration. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RY3Grxxkhd", "reviewer": "Reviewer_gLwg", "summary": "", "questions": "1. Do you keep the entire reasoning context in the context of the LLMs when generating the next action plan?\n\n2. What are the failure cases?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Online or offline optimization?**\n\nA1: Apologies for the confusion. The optimization process occurs in an online and off-policy manner. Specifically, we use the current plan to collect a batch of experiences (trajectories) and update the plan based on the reflection of these experiences. This will be clarified in the revised version.\n\n**Q2: Comparison setup for AutoPlan.**\n\nA2: To clarify, AutoPlan has distinct training and inference phases. During training, we use the current plan to collect experiences from the training set and update the plan based on reflections on these experiences. In the inference phase on test environments, AutoPlan does not reset and directly generates the action sequence in a single round. This is conditioned on the optimized plan and observations from test environments. The entire reasoning process occurs within the context of the LLMs. The only difference between ReAct and AutoPlan during inference is that ReAct uses in-context demonstrations, while AutoPlan uses the optimized task plan. This will be made clear in the revised version.\n\n**Q3: Extra data for training.**\n\nA3: AutoPlan does require training data, but it needs no more than 50 training environments for each dataset tested in order to generate a performant task plan.\n\n**Q4: Larger batch size.**\n\nA4: We further tested AutoPlan with a batch size of 8. The mean (standard deviation) of success rates are as follows:\n\n|  | Iteration 1 | Iteration 2 | Iteration 3 |\n| --- | --- | --- | --- |\n| Batch size 2 | 0.23 (0.32) | 0.4 (0.35) | 0.3 (0.3) |\n| Batch size 4 | 0.37 (0.32) | 0.53 (0.15) | 0.67 (0.15) |\n| Batch size 8 | 0.7 (0.2) | 0.67 (0.23) | 0.73 (0.15) |\n\nWe were unable to test AutoPlan with a batch size of 16 due to the context size limitations of GPT-4. Larger batch size overall delivers better performance.\n\n**Q5: Significance testing of Figure 3.**\n\nA5: We conducted a t-test for each training iteration to compare batch sizes of 2 vs. 4 and 2 vs. 8. Batch size of 8 is significantly better than batch size of 2.\n\n|  | Iteration 1 | Iteration 2 | Iteration 3 |\n| --- | --- | --- | --- |\n| p-value (2 vs 4) | 0.44 | 0.35 | 0.013 |\n| p-value (2 vs 8) | 0.007 | 0.110 | 0.005 |\n\n**Q6: What are the failure cases and does the system have the capability to recover from them?**\n\nA6: The types of failure cases can vary, and a detailed error analysis is provided between lines 401-427. We will include the trajectories of typical failure cases in the appendix. For instance, in ALFWorld, the task might be to place a clean mug in the coffee machine. However, the LLM agent may mistakenly choose a cup, clean it, and place it in the coffee machine. Subsequently, it may engage in repetitive and meaningless actions.\n\nRegarding the system's ability to recover from failures, it largely depends on the nature of the failure. For example, if the task is to place an apple on a table and the apple is dropped, the LLM agent can instruct the robot to clean the apple and continue with the task, making adjustments for similar future scenarios. However, for irreversible failures, such as a broken apple, the system likely cannot recover without human intervention. This aspect will be further discussed in the revised version of the document.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The writing of the paper is decent but could be improved.  For instance, it is unclear to me whether the optimization process is done online or offline based on the writing in Section 3.\n\n2. The performance of the model seems good, and ablation studies of the paper help to understand the method.\n\n3. This method seems to be a nice combination of ReAct and Reflection.", "reasons_to_reject": "1. I have one major concern about the method. It seems that the method runs as follows: given the target instruction, the paper runs AutoPlan on the simulator, and then finally deploys the final actions (reset the entire reasoning trace). I think this is \"cheating\" in the sense that you gather information from the environment, and use the feedback to try again, which means that this is NOT a zero-shot setting, but a trial-and-error approach until you succeed. For ReAct, it does not have a reset process, and it just keeps generating the next actions based on the observation. As a result, I DO NOT think this is a fair comparison. I think what the author should do is keep the entire reasoning trace in the context of the LLMs and see if it can solve the task.\n\n2. Following the above discussion, this method needs a batch of data to run, which means that you need to provide a couple of data points to use.\n\n3. Figure 3 seems to not have statistical significance. And what will happen if the batch size increases to 10 or 20?\n\n4. Another issue is that the method is based on the assumption each state is \"reversible\", in the sense that the agent can go from one state to another state freely. If I want to deploy this method in a real robot, what will happen when the agent fails? Will the plan is able to recover the robot from failure cases?", "questions_to_authors": "1. Do you keep the entire reasoning context in the context of the LLMs when generating the next action plan?\n\n2. What are the failure cases?", "paper_topic_and_main_contributions": "This paper studies the problem of using the LLMs model to come up with a sequence of high-level actions to solve the instruction following tasks. The paper proposes a new method, called AutoPlan, that consists of a \"self-reflection\" mechanism to modify the existing failure plans. The idea is that first LLMs come up with an initial plan, and then execute the actions in the environment, receiving the observations. Once the agent fails, it then does a \"reflection\" step that uses LLMs to summarize the failure and provide what to do to revise the previous plan. Based on the revision, the plan is updated. And the new plan is deployed in the environment again. Based on this repeated process, the plan is optimized, and the final plan is deployed. \n\nSeveral results are provided in the paper. First, Table 3 shows that AutoPlan has better performance than ReAct under the zero-shot setting. for both ALFWorld and Hotpot QA.  Second, table 4 shows that the cost of AutoPlan is lower than the cost of React. Third, figure 3 shows ablate the components of batch size and the reflection step. The observation is that with larger examples, and training iterations, the model has better performance. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6dYqzmFqX4", "reviewer": "Reviewer_kvA8", "summary": "", "questions": "What are the key differences between reflection in AutoPlan and in Reflexion?\n\nWhy did you not directly compare to a version of Reflexion without in-context examples and test time refinement? This seems like a critical comparison that is needed.\n\nHow does the overall performance of AutoPlan compare to the original Reflexion results? What is the actual improvement (if any) is achieved by using in-context examples and/or test time refinement?\n\nResponse to author rebuttal:\n\nThank you for taking the time to answer the questions and perform new experiments to provide further insight and validate your method. Given the high performance of all evaluated methods in ALFWorld, the paper could benefit from validation in more domains to further distinguish AutoPlan and justify its high training cost. \n\nIn light of your responses which address some of the posed weaknesses, I increased the soundness score to a 3.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Ungrammatical English in prompts.**\n\nA1:  We tune the prompt to guide the Language Learning Model (LLM) in generating the desired output. While the resulting prompt may be ungrammatical, it functions as expected.\n\n**Q2: Lack of strong baselines, e.g., zero-shot AdaPlanner and Reflexion.** \n\nA2: Thank you for the suggestion! We have conducted additional experiments with 0-shot AdaPlanner and 0/2-shot Reflexion on ALFWorld. The success rates over all 134 test environments are as follows:\n\n|  | AutoPlan | 0-Shot ReAct | 2-Shot ReAct | 0-Shot AdaPlanner | 0-Shot Reflexion | 2-Shot Reflexion |\n| --- | --- | --- | --- | --- | --- | --- |\n| Success Rate | 95% | 74% | 95% | 0% | 82% | 97% |\n\nAdaPlanner fails when no in-context examples are provided. 0-shot Reflexion improves upon 0-shot ReAct by 8%, and 2-shot Reflexion surpasses both 2-shot ReAct and AutoPlan by 2%.\n\n**Q3: Clarify the difference between AutoPlan and existing methods, e.g., Reflexion.**\n\nA3: Thank you for the suggestion! The key differences between AutoPlan and Reflexion are as follows:\n\n- Reflexion involves test-time reflection, requiring iterative reflection for each test environment. This makes it inefficient, and the reflections do not transfer to other test environments. In contrast, AutoPlan optimizes the plan using training data and generates action sequences for test environments in a single iteration.\n- While Reflexion uses in-context learning inside reflection, AutoPlan employs a three-stage reflection process that captures key information without the need for in-context examples.\n\nWe will include these clarifications and comparisons with other methods in our paper.\n\n**Q4: AutoPlan performance versus few-Shot ReAct.**\n\nA4: AutoPlan achieves comparable results to few-shot ReAct in ALFWorld, but demonstrates a non-negligible advantage over few-shot ReAct in HotpotQA (see human evaluation results in A5). We have also included both training and inference costs (unit: US dollar) for ALFWorld in the table below:\n\n|  | AutoPlan | 2-Shot ReAct | AdaPlanner | 2-Shot Reflexion (5 trials) |\n| --- | --- | --- | --- | --- |\n| Inference cost per case | 1.6 | 3 | 2.1 | 17 |\n| Training cost (24 cases) | 43 | null | null | null |\n\nAutoPlan is the most efficient in terms of inference cost, with reasonable training overhead.\n\n**Q5: Human evaluation.**\n\nA5: We conducted additional human evaluations on HotpotQA with three external human annotators. The full annotation instructions will be included in the paper’s appendix. These instructions involve four phases: reviewing data points, checking the accuracy of predictions, ensuring consistency with supporting facts, and logging the results.\n\n|  | AutoPlan | ReAct (2 Shot) | ReAct (0 Shot) | Supervised |\n| --- | --- | --- | --- | --- |\n| Majority Votes (≥2 votes) | 80% | 73% | 71% | 91% |\n| Aggreements (all 3 annotators agree) | 90% | 93% | 92% | 95% |\n\nWe used the majority vote from the three annotators for evaluation, and AutoPlan outperformed ReAct both with and without in-context examples. Furthermore, the annotators agreed with each other in over 90% of the cases.\n\n**Q6: Grammar errors distract from the paper’s findings.**\n\nA6: Thank you for your suggestion! We will invite a fluent English speaker to carefully review and edit the paper to eliminate any distracting grammatical errors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Provides a detailed qualitative analysis of failures in the ALFWorld domain to understand method performance characteristics.\n\nPerforms ablations to understand the effect of batch size on AutoPlan.\n\nThe paper evaluates the proposed method using GPT-4 thereby demonstrating results using a state-of-the-art LLM.", "reasons_to_reject": "Prompts contain ungrammatical English that may or may not affect experimental results.\n\nThe paper does not evaluate against strong baselines like InnerMonologue, Reflexion, and other LLM planning methods which evaluate on ALFWorld and HotpotQA. These evaluations could be done fairly by modifying these methods to match the inputs to AutoPlan, for example by not providing in-context examples to these methods. AdaPlanner could also be evaluated in the zero-shot setting. The paper should compare to other methods which also allow for plan optimization before testing. A comparison to just React seems incomplete to establish the benefits of the approach compared to methods that use the same resources.\n\nThe paper would be improved by more clearly differentiating its contribution from existing methods in LLM planning. For example it’s not clear whether there’s a significant difference between its “reflection” component and similar mechanisms in existing works like Reflexion.\n\nThe proposed method does not perform significantly better than a 2-shot ReAct baseline in ALFWorld. This calls into question the potential impact of AutoPlan, as AutoPlan requires a complex plan optimization process versus ReAct which does not. Therefore the comparison of method costs only during inference is incomplete. The paper could be improved by comparing total costs including both plan optimization and inference.\n\nThe human evaluation of HotpotQA results is done by the authors of the paper, and the evaluation methodology and criteria are not detailed.", "questions_to_authors": "What are the key differences between reflection in AutoPlan and in Reflexion?\n\nWhy did you not directly compare to a version of Reflexion without in-context examples and test time refinement? This seems like a critical comparison that is needed.\n\nHow does the overall performance of AutoPlan compare to the original Reflexion results? What is the actual improvement (if any) is achieved by using in-context examples and/or test time refinement?\n\nResponse to author rebuttal:\n\nThank you for taking the time to answer the questions and perform new experiments to provide further insight and validate your method. Given the high performance of all evaluated methods in ALFWorld, the paper could benefit from validation in more domains to further distinguish AutoPlan and justify its high training cost. \n\nIn light of your responses which address some of the posed weaknesses, I increased the soundness score to a 3.", "paper_topic_and_main_contributions": "The paper develops a demonstration-free method for generating and iteratively improving a plan for decision making tasks using an LLM. The method is evaluated in both the ALFWorld and HotpotQA domains and outperforms a zero-shot ReAct baseline. In addition to downstream task performance, the authors investigate the inference costs of their method and perform ablations to understand the effect of batch size on the method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Grammatical errors distract from the paper’s findings. The paper should be proofread and edited by a fluent English speaker"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reveiwers have reached consensus in their reviews. While the result are reasonable, they are not conclusive in setting out the advatages of AutoPlan, the presenation can be improved to highlight the value of the approach, i.e. exanding on Table 4 in the paper. The paper should be updated with the results in the rebutal.
Decision: Accept-Findings