Open Grounded Planning: Challenges and Benchmark Construction (2024)

Abstract: The emergence of large language models (LLMs) has increasingly drawn attention to the use of LLMs for human-like planning. Existing work on LLM-based planning either focuses on leveraging the inherent language generation capabilities of LLMs to produce free-style plans or employs reinforcement learning approaches to learn decision-making for a limited set of actions within restricted environments. However, both approaches exhibit significant discrepancies between the open and executable requirements in real-world planning. In this paper, we propose a new planning task—open grounded planning. The primary objective of open grounded planning is to ask the model to generate an executable plan based on a variable action set, thereby ensuring the executability of the produced plan. To this end, we establish a benchmark for open grounded planning spanning a wide range of domains. Then we test current state-of-the-art LLMs along with five planning approaches, revealing that existing LLMs and methods still struggle to address the challenges posed by grounded planning in open domains. The outcomes of this paper define and establish a foundational dataset for open grounded planning, and shed light on the potential challenges and future directions of LLM-based planning. Our code and datasets are at https://github.com/Shiguang-Guo/ Open-Grounded-Planning

heading: 1 Introduction
text: Human life is filled with tasks of varying complexities, from simple activities like brewing coffee to more substantive pursuits such as learning new skills. By utilizing our understanding of the world, we can formulate plans for tasks and execute these
*Equal contribution †Corresponding author
steps in sequence. Although we can employ innumerable strategies and plans to achieve our objectives, the scenario is significantly more complex for artificial intelligence. Grounding plans to open action sets for tasks in open domains poses one of the challenges for AI.
Some prior research has delved into the planning ability of Large Language Models (LLMs) and found that LLMs can engage in planning to some extent using their internal knowledge through
1 4982
common-sense reasoning (Zhao et al., 2023; Brown et al., 2020). However, these plans are often heuristic, coherent, and rational in natural language, yet possess a high degree of freedom and cannot serve as executable instructions for AI agents (Yao et al., 2023; Huang et al., 2022). In other words, these plans are not grounded in an actionable space. In addressing the issue of grounded planning, various approaches have been explored in fields like robot controlling (Ahn et al., 2022; Wang et al., 2023b) and tool use (Qin et al., 2023; Li et al., 2023; Tang et al., 2023). Typically, model fine-tuning is applied for performance improvement in certain restricted scenarios (Song et al., 2023; Shen et al., 2023; Yuan et al., 2023). However, these methods can only enable models to perform planning on a limited set of actions for specific domain tasks (Lin et al., 2023; Wu et al., 2023; Hao et al., 2023). As the task domain becomes broad, the action space becomes vast and open, these grounded planning methods appear too restricted to handle the steeply increased complexity (Wang et al., 2023b).
The capability to perform a wide range of actions and to devise viable, comprehensive plans by selecting suitable actions from an extensive action library for tasks in various domains epitomizes both a vision and a future trend in LLMs. Consequently, in this work, we introduce the concept of Open Grounded Planning to advance the research on LLM planning across broad fields and a rich array of potential actions. We delineate the concept in two distinct dimensions:
• Grounded Planning: LLM is required to compose plans utilizing only the actions available within the executable action sets.
• Open Planning: We aspire for the model to conduct planning within an extensive set of actions in an open domain that contains various task fields.
Moreover, we collect datasets from three major areas, including daily life, tool use, and robot sandbox scenarios. All collected datasets have been transformed into a uniform format, including task objectives, constraint conditions, golden steps, and candidate action sets. Building upon this foundation, we have developed a benchmark to assess the performance of various models and methods in the Open Grounded Planning task.
Besides, to address the open grounded planning challenges, we proposed a novel Retrieve and
Rewrite framework. The method utilizes the LLMs to generate an initial plan and iteratively rewrites this plan using actions retrieved based on the current planning situation.
We conducted comprehensive experiments on four commonly used methods and our Retrieve and Rewrite method for current planning tasks using GPT-3.5, Vicuna-7B, and LLaMA-2-7B fine-tuned with a small amount of domain knowledge. The explored methods include retrieval-based methods and inference-based methods. We observed that fine-tuning contributes much to bridge the gap between smaller models and extremely large-scale language models by raising the instruction following and task understanding abilities. Various methods exhibited trade-offs regarding the executability and quality of generated plans. Generalizing ability from in-domain to out-of-domain planning tasks exists to a certain extent.
Generally speaking, our contributions are:
• We proposed the concept of Open Grounded Planning. We envision future artificial intelligence systems being able to plan tasks within open domains, and having the ability to ground plans onto open executable action sets.
• We constructed a benchmark consisting of datasets from diverse domains for Open Grounded Planning and an automated evaluating procedure to assess the performance of different models and methods.
• We introduced the Retrieve and Rewrite framework to address challenges in Open Grounded Planning tasks, and conducted comprehensive experiments on state-of-the-art models with various methods, and found that current models and methods still struggle with Open Grounded Planning tasks.

heading: 2 Related Work
text: Large language models are trained on data containing extensive common knowledge and exhibit certain planning and common-sense reasoning abilities (Zhao et al., 2023; Brown et al., 2020). Prompting can be employed to guide large language models in generating plans for given tasks, and these plans often possess a high degree of freedom, making them challenging to execute in specific environments (Huang et al., 2022). To facilitate the
2
grounding of generated plans to an AI agent’s executable action space, prior research has extensively explored grounded planning tasks (Lin et al., 2023; Wu et al., 2023). Some approaches opt for a global planning strategy based on the task, aiming to directly generate plans that can be grounded to the execution environment in a single step (Song et al., 2023; Shen et al., 2023; Yuan et al., 2023; Wang et al., 2023a). Conversely, other methodologies employ iterative interactive approaches as the primary means of plan generation to adapt to changes in the environment and conditions (Ahn et al., 2022; Wang et al., 2023b). However, these approaches often demonstrate limited effectiveness, completing constrained tasks with a finite set of actions within a singular domain.
In open-domain environments, the enormity of tasks and action sets poses significant challenges, making it increasingly difficult to bridge the gap between plans generated by large language models and the execution of real-world tasks. Therefore, in our work, we raise the challenge of open grounded planning and compile benchmark data from multiple domains ranging from everyday life to tool use and robot control scenarios, which consist of tens of thousands of tasks and actions. We also utilized our benchmark to assess the performance of mainstream proprietary models and open-source models with various planning methods on open grounded planning tasks.

heading: 3 Open Grounded Planning Benchmark
text: In this section, we initially present the task definition of Open Grounded Planning and its associated challenges. Subsequently, we introduce the Open Grounded Planning Benchmark, encompassing the construction of the dataset, evaluation metrics, and the methodology for automated assessment.

heading: 3.1 Definition of Open Grounded Planning
text: Specifically, for a given task objective G stemming from any domain, along with conditional constraints C (which may be absent for task without additional constraints), we aim to find a plan P composed of a series of actions {si}, where each action si is from an open action set S. In other words, the generated plan P must be grounded onto the action set S which is vast and extendable:
P = (s1, s2, · · · , sn|G,C), si ∈ S, 0 ≤ i ≤ n where n is the length of P . Table 1 shows the grounded planning process.
As discussed in Section 2, many explorations into LLM planning are focusing on heuristic planning in which the generated plans cannot be directly used as instructions for downstream control mechanisms, in other words, they are not "grounded". Some previous studies have demonstrated that LLMs can undertake grounded planning tasks in certain fields. However, these applications have often been limited to constrained scenarios and task domains. As the richness of the task domains and actions increases, the model’s planning proficiency tends to diminish. LLMs still face challenges in executing grounded planning across open domains, which encompass a wide array of tasks and actions from diverse fields.

heading: 3.2 Dataset Construction
text: LLM’s planning capabilities have a variety of application scenarios. We refer to many other works and summarize the three main application areas including daily life, tool usage, and robots. To balance the proportions of data across different categories, we retain a maximum of 500 tasks for each category, forming our evaluation set. All actions related to the original tasks are preserved in the action library as candidate actions.
We split the dataset into two parts. We employ the daily life dataset wikiHow to evaluate the indomain grounded planning capabilities because this dataset covers a very wide range and the action set for selection is more complex. Additionally, we utilize datasets related to tool use and robots
3
to evaluate the generalization of various models and methods for out-of-domain grounded planning. The statistical information of the evaluation set can be found in Table 2. We also provide the more detailed data processing procedure in Appendix A.

heading: 3.2.1 In-Domain Datasets
text: Wikihow Wikihow is an extensive collection of guides and tutorials, encompassing topics ranging from everyday life skills to more complex subjects1. Each guide on WikiHow is presented in a step-bystep manner, making it easy to understand and follow. We gathered the original corpus of WikiHow by referencing Zhang et al. (2020). For each article, we retained only the tasks, methods (if any), and headlines. We eliminated sections containing multiple "parts" as they introduced additional hierarchy. By directly utilizing the original categorization within the WikiHow corpus, we ultimately identified 19 categories, with a total of more than 76,000 tasks. The action libraries for each category are derived from the collective actions of all tasks within the same category, with an average size exceeding 20,000.

heading: 3.2.2 Out-of-Domain Datasets
text: Tools Previous studies have demonstrated the capability of LLMs to utilize tools to accomplish tasks. Effective planning is crucial for tool use, especially when the candidate toolset is extensive. We have collected open-source data relevant to tool usage by LLMs, including contributions from ToolAlpaca (Tang et al., 2023), API-Bank (Li et al.,
1https://www.wikihow.com/
2023), and GPT4Tools (Yang et al., 2023). These datasets encompass various types of tools and provide standard tool invocation sequences to complete the tasks as well. To maintain consistency with other datasets, we only retain the API names and their corresponding description, while ignoring the parameters.
Robot There exists some research related to grounded planning in robotics(Yoshida et al., 2023; Brohan et al., 2023; Ahn et al., 2022), but there is still a lot of room for development. We have converted datasets proposed in VirtualHome (Huang et al., 2022) and SayCan (Ahn et al., 2022) and merged all executable actions as a candidate action set2. It is important to note that complete robot processing involves multiple stages, including visual information processing and action execution. Our dataset, however, only focuses on the planning generation.

heading: 3.3 Evaluation

heading: 3.3.1 Plan Quality Assessment
text: In all the datasets we collected, every task has a corresponding golden plan which is provided by the original datasets and presents one of the possible ways to handle the task. Since the solutions to the tasks in our benchmark could be quite diverse, especially when it involves thousands of candidate operations to form a planned execution path, it is unfair to directly judge whether the generated plans match exactly with the golden plans. Instead, we
2There seems to be some mismatch in the dataset provided by saycan, we fix it manually. A more detailed procedure is provided in Appendix A.
4
make the golden plan a reference and compare the plan generated by the model with it from multiple perspectives to judge which plan is better. The specific evaluation criteria are as follows:
Completeness: Examine whether the plan is comprehensive, with a focus on the coherence and logic between steps, and the avoidance of arbitrarily introduced conditions and missing steps.
Feasibility: Assess the practicality of the plan, considering whether each step can be implemented, whether the plan aligns with common sense, adheres to human ethical standards, and avoids excessive redundant steps.
Relevance to the Task: Evaluate the relevance of the plan to the given task, considering the utilization of the provided task conditions and whether it achieves the goal.
We use ChatGPT to evaluate, which is a widely used evaluation method in previous similar work. During one evaluation, the target task to be solved is delivered to the evaluator along with the model generated plan and the golden plan. Then we prompt ChatGPT to list how good the two plans are regarding the evaluation criteria, and ask it to elect a better plan based on its analyses.
Due to many reported issues with ChatGPT as an evaluator, such as position bias, length preference, and style partiality (Koo et al., 2023; Wu and Aji, 2023; Zheng et al., 2023a), we employ various methods to mitigate those biases. We swap the order of the two plans and average the scores to eliminate positional bias. Additionally, we prompt ChatGPT to penalize the score for redundant steps to reduce length preference. We sampled a small dataset for manual evaluation and verified the plausibility of ChatGPT’s automatic evaluation. For detailed analysis, please refer to appendix C.

heading: 3.3.2 Metrics
text: To more intuitively compare the performance of various models and methods on open domain planning datasets, we define the following metrics to quantify their performance.
Executability is the proportion of executable cases. Executable cases are actions in the plan that all exist within the given action library.
Quality of the executable plans is evaluated from the dimensions in section 3.3.1. We define win rate as the average of the outcomes of two comparisons involving position swaps. Intuitively, quality assesses how complete the generated plan is.
Overall Pass Rate is the proportion of all gener-
ated plans that can be executed while also completing the task. Considering both executability and quality, we choose pass rate as the final evaluation metric to evaluate the overall performance of the model in the entire process. The pass rate is the product of executability and quality.
Executability = #executable cases
#all cases
Quality = #win cases
#executable cases Pass Rate = Executability × Quality

heading: 4 Methods
text: In order to assess the performance of current mainstream models on the Open Grounded Planning task, we endeavored to employ five distinct methods including Retrieve and Rewrite, a new framework we proposed, to address this challenge.
Task-Retrieve: We first adopt a simple and intuitive approach by using the task name as the query for action retrieval. Given a task T and an action set A, we retrieve the relevant action list A′ = retrieve(T,A). LLM selects and orders appropriate actions from this list to generate plan P = select&sort(T,A′).
Plan-Retrieve: Simply searching by task name makes it difficult to recall important steps that are not directly related to the task name. We try to specify the query to improve. We first force LLM to generate an initial plan P0 = {s1, s2, · · · , sn} for the task, then retrieve related actions based on the generated plan to get action list A′ = retrieve(P0, A). Finally, we perform similar selection and rearrangement as in task-retrieve. We also provide the initial plan for LLM to refer to generate the final plan P = select&sort′(T,A′, P0).
Step-wise Select: In addition to retrieval and rearrangement, we also try the step-wise selection method. We adopt a method similar to ReAct (Yao et al., 2023) to our tasks. Each time, LLM generates a possible next step sp for the given task T , Then we obtain a candidate action list A′ = {a1, a2, ...} ⊂ A by retrieving actions based on the generated steps. LLM selects one of the retrieved results as the next step, which means the target plan P is generated step by step, i.e., P = {s1, s2, ...} ∪ {aj} where aj ∈ A′. Selection iteration repeats until 1) LLM outputs None when generating the possible next step, 2) LLM
5
refuses to select from the candidate list, and 3) the maximum number of iterations is reached.
DFS: The Step-wise Selecting method suffers from small searching space. ToolLLM (Qin et al., 2023) proposed a DFS-based method to improve. We implement this by simply extending the Stepwise Selecting method. During the procedure of model selecting the next step, we allow LLM to abandon the selection if it thinks there exists no suitable choice in the retrieved candidate actions to be the next step of the current plan. In this case, we perform a backtracking.
Retrieve and Rewrite: We realize that methods based on retrieval and rearrangement can consider the overall plan, but may not obtain the optimal choice for each step. The step-wise selection approach enables adjustments during the generation process but does not take into account the plan as a whole. We combine the advantages of both methods and propose a new method named Retrieve and Rewrite.Figure 2 illustrates its framework and different rewriting operations.
LLM is first asked to generate an initial plan P0 = {s1, s2, ..., sn0} based on the relevant steps A0 with the given task T . Different from the TaskRetrieve method, P0 does not have to be composed of the exact steps in A0. We mark steps not in action set with underline. We perform several iterations to use steps from the action set to rewrite P0. For iteration i ≥ 1, we choose some of the steps not in action set for retrieval to retrieve relevant actions Ai = {ai1, ai2, · · · , aimi}, where mi is the length of candidate action list of iteration
i. LLM is allowed to perform various operations when rewriting Pi−1, including adding, deleting, and modifying arbitrarily. We only need to ensure using actions in the action set to replace as much as possible those not in the action set. The plan after rewriting might be Pi = { s1, s2, s3, · · · , sni } , where ni is the new length of current plan Pi. Similarly, iteration stops until all actions are in the action set or iteration reaches the maximum number.

heading: 5 Experiment
text: We systematically assess the capabilities of various LLMs and methods in the Open Grounded Planning task by selecting and comparing mainstream proprietary models, open-source models, and opensource models fine-tuned with a small amount of domain-specific data. For each model, we examine the performance of different methods in Section 4. We test the models’ abilities in Open Grounded Planning on both in-domain, the wikiHow dataset, and out-of-domain, the tools and robot datasets.

heading: 5.1 Experiment Settings
text: We chose the proprietary model GPT-3.53 and the open-source model Vicuna-7B-v1.5-16k (Zheng et al., 2023b) for experiments. In addition to this, we fine-tuned Llama-2-7B (Touvron et al., 2023) to check the performance of the SFT model. We believe these three models can represent the capabilities of current mainstream models.
We select 200 tasks from each subcategory below wikiHow as the training set. For each setting,
3We use gpt-3.5-turbo-1106 for our experiments.
6
we perform the inference process of GPT-3.5 on it and select those with high quality for training. We mixed it with the Alpaca dataset (Taori et al., 2023) and fine-tuned the model with 3 epochs to improve the generalization ability. We use OpenAI’s text-embedding-ada-002 to generate embedding for each step in all settings. Additional implementation details are in Appendix B and all prompts are in Appendix F.

heading: 5.2 In-Domain Results
text: We measure the performance of each model and method using metrics in Section 3.3.2. We report the average performance on all wikiHow datasets. Results on in-domain datasets are presented in the left part of Table 3, from which we can derive:
SFT model achieves the best performance Compared to Vicuna and GPT-3.5, the SFT model surpasses them in all methods. The trained model can improve the executability of all methods to close to 100% and maintain high quality at the same time.
Different methods have different focuses Although the initial plan of the Plan-retrieve method may cause interference, it can generate a better plan than Task-retrieve. Compared with the restricted search space in the step-wise selection, DFS usually achieves a higher executability and has better quality. Besides, we find pre-planning grants the final plans higher quality. The Retrieve and Rewrite method we proposed surpasses Step-wise select
and DFS in terms of rationality and completeness of the final plans due to our pre-planning approach and the subsequent rewriting.

heading: 5.3 Out-of-Domain Results
text: The experiment results on the out-of-domain datasets are presented in the right part of Table 3. Due to space constraints, we provide more detailed OOD dataset results in Appendix E. Apart from the DFS method, LLaMA-2-SFT outperforms GPT-3.5 on most datasets using other methods. This suggests that fine-tuning is beneficial for the model to generalize from in-domain grounded planning to out-of-domain scenarios, primarily by enhancing the model’s ability to follow instructions, thereby improving task completion. In the tools dataset, despite our extensive prompt engineering, GPT-3.5 prefers to generate complete steps rather than the provided API names, which results in poor performance.
We observe the performance degradation in SFT model with DFS and find that the average length of the plans generated with DFS on tools dataset is less than 1, while with other methods ranges from 1 to 2, which is also closer to the true length of datasets. We observe extensive backtracking in the generated results,which could be attributed to the higher proportion of rejection in the training data, leading the SFT model to exhibit reduced confidence in its generation. Consequently, this makes LLM more inclined not to select the retrieved re-
7
sults as the next step when facing OOD data. Among various methods, Rewrite method consistently maintains strong performance. However, although it performs well in-domain, the generalization performance is sometimes not as good as simple retrieval and rearrangement.

heading: 5.4 Error Analysis
text: Task How to Build a Small Chick Brooder Method Setting up the Brooder
Plan-Retrieve Set up a brooder. Use absorbent bedding. Provide a warm place. Place any heating apparatus needed for the species that is going to be kept. Monitor the temperature. Set up an attractive feeder. Choose an appropriate hiding spot. Ensure that there is proper ventilation in the stable. DFS Set up a brooder. Point a heat lamp on one side of your brooder to keep the chicks warm. Move your chicks to their brooder. Move the chicks to a brooder. Provide food and water for your chicks. Move the chicks to a brooder. Set up a brooder. Move the chicks to a brooder. Golden plan Fill the bottom of your brooder with paper towels and pine shavings. Point a heat lamp on one side of your brooder to keep the chicks warm. Provide food and water for your chicks. Put a branch in the brooder to create a roost. Rewrite Position the cage away from drafts and direct sunlight. Set up a brooder. Fill the bottom of your brooder with paper towels and pine shavings. Put a branch in the brooder to create a roost. Point a heat lamp on one side of your brooder to keep the chicks warm. Move your chicks to their brooder.
Table 4: Red text indicates steps that are irrelevant to the task and redundant steps, while green text denotes supplementary additional information.
We mainly perform analysis on GPT-3.5 on wikiHow to fairly compare various methods. We analyze from the perspectives of executability and generation quality. As executability shown in table 3, since the output format of the first four methods is usually a sentence or a list, inexecutable plans all come from the hallucination of LLM, which means LLM generate content beyond the given set. However, the output format of the Rewrite method is more complex. We observe that the non-executable plans contains 11.84% format parsing errors. The small format error ratio proves that LLMs have strong instruction following ability, but they still face serious hallucination problem.
To compare the generation quality of different methods, we present cases from the Plan-retrieve, DFS, Rewrite and golden plan from wikiHow in table 4. The plans generated by Plan-retrieve might
generate steps that are correct in meaning but irrelevant in detail to the task. This is because the retrieved actions are not always relevant to the task, and LLM, given a one-time, limited selection, may be forced to choose these steps to make the plan complete. Besides being incomplete or irrelevant to the task, DFS also suffers from duplicate steps, despite being provided with previously selected steps. We find that 19.32% of plans generated by DFS contain repetitions of two or more times. Meanwhile, this step by step generation is also less complete, for example, missing information on the soft bedding material. In contrast, the Rewrite method, through iteration and global consideration, can generate more complete plans. Additionally, with a large pool of candidate steps, LLM can even find supplementary information for the task.

heading: 5.5 Retrieval Amount Influence
text: We find that the number of retrieved actions directly affects the execution rate and quality of the generated plan. We check the result on wikiHowComputers and Electronics for a setup using GPT3.5 and Vicuna for the task-retrieve method4. Figure 3 shows how "Executability", "Quality" and "Pass Rate" change when different numbers of steps are recalled for plan generation. The two solid lines representing "Pass Rate" demonstrate a trend of initial increase followed by a decrease. For GPT-3.5, as the number of recalled actions increases, the proportion of generated plans that
4Different from the main experiment, here we only check the output format.
8
conform to the rules decreases, while the quality of plans lifted. Pass Rate achieves the best performance when recalling 20 actions. Interestingly, we observed that when recalling 5 actions there are still parts of the plan that are not executable. We found that this is because the recalls are too little for LLMs to complete the task using the recall steps. But GPT-3.5 tends to generate complete plans, so steps for supplementing and connecting context are generated. We continue to discuss the results in other settings in Appendix D.

heading: 6 Conclusion
text: In this study, we introduced Open Grounded Planning and developed a benchmark comprising datasets from various domains with vast action sets. Extensive experiments revealed significant limitations in the performance of current models and methods in generating grounded plans for these sets. Furthermore, we observed a pronounced challenge in enabling these models and methods to generalize from in-domain scenarios to out-of-domain datasets. Compared to four other methods, our "Retrieve and Rewrite" approach demonstrates a partial resolution to the challenges inherent in open grounded planning. Our work highlights the need for enhancing the capability of models and methods for expansive planning domains and improving the executability and quality of grounded plans, laying a foundation for future research.
Limitation
Our current implementation relies on a two-stage approach of retrieval and generation. We anticipate that an optimized retriever tailored for this task will achieve better performance. Additionally, our current dataset only explores how to ground tasks into a given set of actions. While this is sufficient for many applications, a more challenging extension would be to introduce parameters for each action, meaning the use of a collection of action functions instead of a collection of actions.
Our current method for evaluating the content of the plan involves using ChatGPT as the evaluator, which is to guide ChatGPT to compare and judge the plans through prompting. This method inevitably brings bias and hallucination into the evaluation results. Although we have employed various methods to alleviate their impact and have sampled some cases for manual evaluation to prove the effectiveness of our evaluation method, bias
and hallucination persist. In future work, we may introduce more diverse objective evaluation metrics and ways to reduce bias and hallucination.

heading: Acknowledgements
text: We sincerely thank all anonymous reviewers for their insightful comments and valuable suggestions. This research work is supported by the National Natural Science Foundation of China under Grants no. 62106251, no. 62122077 and no. 62306303, and the Basic Research Program of ISCAS, Grant No. ISCAS-JCZD-202303.

heading: A Dataset Details
text: For the datasets of WikiHow, Tools, and Robot, although they have different original formats, they can all be transformed into a uniform format with three parts: task name, method, and steps. For WikiHow, each life guide (task) contains a title and several steps, with each step including a headline and a detailed explanation of the step. We retain only the headline part as the step to complete the task. Some guides may have different methods to achieve this. For example, one can create a poster by either hand drawing or using paper cutting. We randomly select one of these methods as the method to accomplish the task. The final example for WikiHow is shown in Table 1.
For Tools scenarios, we extract all the API calls under each task as steps. To maintain consistency with other tasks, we only retain the API name and description without including API parameters, as this would require additional training, and many works have already explored this kind of capability(Qin et al., 2023). We will also set the default method to some tasks to make the results they generate more consistent with requirements. Here is an example from GPT4Tools. Given a task of Generate a real image of xxx from the sketch image and a training instance for API call, we will transform it into a process of Sketch Detection On Image -> Generate Image Condition On Sketch Image. {
"title": "Generate a real image of a cat sitting on a table next to
a bowl from the sketch image", "method ": "One or two steps are
usually enough to complete the task , and there are only a few cases where more may be required .",
"steps": [ "Sketch Detection On Image
DESCRIPTION: useful when you want to generate a scribble of the image. like: generate a scribble of this image , or generate a sketch from this image , detect the sketch from this image. ",
"Generate Image Condition On Sketch Image DESCRIPTION: useful when you want to generate a new real image from both the user description and a scribble image or a sketch image. "
] }
In the Robot scenario, we can also obtain tasks, methods, and corresponding steps in the same way.
11
As described in the footnotes, the SayCan repository provides two files: one providing tasks and environment states, and the other providing tasks and corresponding plans, but they do not completely match. Since the tasks in this dataset have similar processes, such as instructing the robot to reach a certain place or pick up an item, we filled in the inconsistent parts based on this style. We also provide an example here. {
"title": "I'd like a clear soda.", "method ": "As a robot with only one
gripper , you are surrounded by a far counter , a near counter , a table , and a trash can. You are located near the table. You can only perform one action at a time , such as moving or picking up and putting down. Environmental status:water on table , 7up on table",
"steps": [ "find a 7up", "pick up the 7up", "bring it to you", "put down the 7up"
] }
B Implementation Details
We use FastChat (Zheng et al., 2023b) for training, and the training parameters are consistent with vicuna-7B. For all generation steps, we perform generation with temperature = 1.0. We perform up to five retries per generation to avoid formatting errors. We also performed a rule review on the output of LLM to obtain the best performance of LLM in the open grounding planning task. For example, if LLM is required to choose one of several options as the next step, and it does not output a sentence that meets the requirements, we will regard this output as a failure and regenerate it. This retry will count towards the five retries above. For tools datasets, our output combines API name and API description, and the input format is "{api name} DESCRIPTION: {API description}". Since we only care about choosing the correct step, we accept both API name and "{API name} DESCRIPTION: {anything}" as input.
We simply use OpenAI’s text-embedding-ada002 for embedding generation in all settings. We used different recall numbers for different methods. For the plan-retrieve method, each generation step recalls the two most relevant choices. For the taskretrieve method, we retrieve the 20 most relevant candidate steps from the task name. Stepwise Se-
lecting and DFS methods are similar in that we both perform recalls of size 5. In the Rewrite method, we will select at most the first three steps that have not been replaced in each round, and dynamically control the recall number of each step to around 10. In all settings, we will first perform deduplication on the recall steps and then hand it over to LLM for other operations.
In addition, since the Stepwise Selecting, DFS, and Rewrite methods will iterate multiple times, we set an upper limit of 20, 30, and 20 iterations for them. These upper bounds are usually sufficient to complete the task, but if the LLM reaches the upper limit of the number of iterations, it means that the generated steps may be incomplete. If the plan complies with the rules, we still think the plan is executable, but incomplete plans will have an impact on the quality of the plan.

heading: C Evaluation Details
text: In the evaluation set, we randomly selected a total of 200 cases and conducted human evaluations on the three models and five methods we used in our experiments. The Spearman rank correlation coefficient between the results of human and automated is 80.76%, which indicates that the automated evaluation results using ChatGPT present a significant consistency with those of human evaluation. Therefore, this automated assessment approach is deemed both reasonable and feasible.

heading: D Different Retrieved Numbers
text: The experimental results concerning the impact of the retrieved item number are illustrated in Figure 4 for both the Plan-Retrieve and Step-wise Select methods. For the plan-retrieve method, as the number of retrieved items increases, the available actions for the model to select also increase, leading decrease in plan executability, possibly due to excessive choices causing interference for the model, leading to the generation of illusory steps and consequently preventing the generated plan from being fully grounded in actions from the candidate sets. However, the generated quality shows a trend of decline after improvement.
For the select method, vicuna and GPT-3.5 show different properties. As the number of options available for vicuna increases, its executability rate will also increase. This has caused its Pass Rate to also show a slightly upward trend. The results of GPT-3.5 show a downward trend.
12

heading: E More Result of OOD Datasets
text: In this section, we provide more detailed performance results on the five OOD datasets. Tables 5, 6, and 7 show the results on the Tools dataset, while Tables 8 and 9 present the performance of LLMs on the Robot dataset. We can draw some similar conclusions to those found in the In-Domain dataset. For example, the Task-Retrieve method has a higher execution rate, but Plan-Retrieve achieves better plans. Additionally, the performance of methods such as Select and DFS is limited by the model’s capabilities, as the model needs to follow the single-step selection instruction and choose the next step reasonably. Existing models often stop too early or too late and lack attention to previous
13
steps, leading to poorer performance.

heading: F Prompts for All Methods
text: Our prompt contains two parts: instruction and input. In GPT-3.5, we use instruction for system messages, while in Vicuna and SFT models, since they are not trained on different system messages, we place it at the beginning of user input.
We use 0-shot in most settings and only use 2- shot in a few such as generating initial plans and rewriting. Prompts set to 0-shot often only have formatting requirements. The rest of the operations need to match the characteristics of the dataset or are more complex, so we think they require additional information.
14

heading: 1. step1

heading: 2. step2

heading: 3. step3...

heading: 1. Open the Disney+ app.

heading: 2. {other steps}…

heading: 1. Improve your core muscles with deep abdominal stretching.

heading: 2. {other steps}…
text: 15

heading: 1. You need to generate a plan that satisfies the given tasks and methods.

heading: 2. The initial plan is a reference only, which means you should not output the steps in the initial plan directly.

heading: 3. You can only use the steps in Actions in Library to complete a given task, even though the provided steps may not complete the task.

heading: 4. You must use the actions in the library exactly. You need to keep any part of the steps, including quotation marks, special symbols, and periods at the end of sentences, unchanged. Send your answer in the following format and do nothing else: 1. step1

heading: 2. step2

heading: 3. step3...

heading: 1. Spend time bonding with your rabbit every day.

heading: 2. {other steps}…

heading: 1. You need to generate the next step in the plan that needs to meet the given tasks and
text: 16

heading: 1. You need to select the next step in the plan from the candidate actions that satisfies the

heading: 1. Based on your plan for a given task and method, you need to generate three candidates for the first step of completing the task with the specified method. These three steps must be different. I would choose one of the three candidates you generated as an initial step in
text: 17

heading: 1. You need to select the next step in the plan from the candidate actions that satisfies the
text: 18

heading: 1. You need to generate three candidates for the next step of the current plan based on the existing plan to satisfy the plan for the given task and method. These three steps must be

heading: 1. You need to refer to the content in <References> to generate a plan that can complete the task in the specified method.

heading: 2. step2

heading: 3. step3...
text: 19

heading: 1. {<TO BE REPLACED> Avoid eating out.}

heading: 2. {<TO BE REPLACED> Cancel unused subscriptions and memberships.}

heading: 3. {<TO BE REPLACED> Bike or walk to work, rather than drive.}

heading: 4. {<TO BE REPLACED> Find free or low-cost entertainment options instead of expensive outings.}

heading: 5. {<TO BE REPLACED> Reduce your rent.}
text: 20

heading: 1. {<TO BE REPLACED> Mix buttermilk, tarragon, salt and pepper in a large bowl.}

heading: 2. {<TO BE REPLACED> Add chicken pieces to mixture to marinate.}

heading: 3. {<IN LIB> Refrigerate marinated chicken for 4 to 6 hours.}

heading: 4. {<IN LIB> Remove chicken pieces from the buttermilk mixture and dredge in flour.}

heading: 5. {<IN LIB> Heat oil in skillet to 350 °F (177 °C).}

heading: 6. {<IN LIB> Fry the chicken in a frying pan until crispy and golden in color.}

heading: 7. {<TO BE REPLACED> Once cooked, place the fried chicken on a wire rack to drain excess oil.}

heading: 8. {<TO BE REPLACED> Serve and enjoy!}
text: 21
22

heading: Future Work
text: Our work highlights the need for enhancing the capability of models and methods for expansive planning domains and improving the executability and quality of grounded plans, laying a foundation for future research. Limitation
Our current implementation relies on a two-stage approach of retrieval and generation. We anticipate that an optimized retriever tailored for this task will achieve better performance. Additionally, our current dataset only explores how to ground tasks into a given set of actions. While this is sufficient for many applications, a more challenging extension would be to introduce parameters for each action, meaning the use of a collection of action functions instead of a collection of actions. Our current method for evaluating the content of the plan involves using ChatGPT as the evaluator, which is to guide ChatGPT to compare and judge the plans through prompting. This method inevitably brings bias and hallucination into the evaluation results. Although we have employed various methods to alleviate their impact and have sampled some cases for manual evaluation to prove the effectiveness of our evaluation method, bias
and hallucination persist. In future work, we may introduce more diverse objective evaluation metrics and ways to reduce bias and hallucination