SlowGuess/NeurIPS_2023_OCR huggingface dataset:
On the Planning Abilities of Large Language Models : A Critical Investigation
Karthik Valmeekam
School of Computing & AI  
Arizona State University Tempe.  
kvalmeek@asu.edu
Sarath Sreedharan*
Department of Computer Science, Colorado State University, Fort Collins. sarath.sreedharan@colostate.edu
Matthew Marquez
School of Computing & AI  
Arizona State University, Tempe.  
mmarqu22@asu.edu
School of Computing & AI  
Arizona State University, Tempe.  
rao@asu.edu
Intrigued by the claims of emergent reasoning capabilities in LLMs trained on general web corpora, in this paper, we set out to investigate their planning capabilities. We aim to evaluate (1) the effectiveness of LLMs in generating plans autonomously in commonsense planning tasks and (2) the potential of LLMs as a source of heuristic guidance for other agents (AI planners) in their planning tasks. We conduct a systematic study by generating a suite of instances on domains similar to the ones employed in the International Planning Competition and evaluate LLMs in two distinct modes: autonomous and heuristic. Our findings reveal that LLMs' ability to generate executable plans autonomously is rather limited, with the best model (GPT-4) having an average success rate of  $\sim 12\%$  across the domains. However, the results in the heuristic mode show more promise. In the heuristic mode, we demonstrate that LLM-generated plans can improve the search process for underlying sound planners and additionally show that external verifiers can help provide feedback on the generated plans and back-prompt the LLM for better plan generation.
It would be no exaggeration to say that transformer-based large language models (LLMs) have revolutionized the field of natural language processing (NLP). Kicked off by the advances presented by the GPT- $\mathbf{x}$  models developed by OpenAI [27], these types of language models currently provide state-of-the-art performance in many of the standard NLP tasks. Although LLMs were originally developed mostly to do word sequence completion tasks, with no guarantees about the completion beyond its coherence, there have been increasing claims and anecdotal evidence that they have other emergent capabilities that are not normally associated with sequence completion. Indeed, the hints of such emergent capabilities has started a veritable land rush, with researchers probing (prompting) and studying LLM behavior almost as if they were artificial organisms (c.f. [16]). Of particular interest to us in this paper is the thread of efforts that aim to investigate (and showcase) reasoning abilities of LLMs—including commonsense reasoning [35, 29, 7], logical reasoning [33], and even ethical reasoning [15]. The macro-tenor of the drumbeat of these works has been suggesting that LLM's are indeed capable of doing such kinds of reasoning [19, 37, 4].
*Author was at Arizona State University during part of this work
One type of reasoning task that has been well studied in the AI community is planning and sequential decision making. At its simplest, planning involves developing a course of actions (policy) which when executed takes the agent to a desired state of the world. Planning has generally been studied primarily as an inference on world and reward models—whether specified by humans or learned by the agent by interacting with its world. In this paper, we are interested in seeing what planning abilities, if any, LLMs may already have, given their high capacity functions (with billions of tunable parameters) trained on web-scale corpora. Specifically, we are interested in answering two broad questions:
1. How effective are LLMs by themselves in generating simple plans in commonsense planning tasks (of the type that humans are generally quite good at)?
2. How good are LLMs in being a source of heuristic guidance for other agents in their planning tasks?
Notice that in theory, it is possible for LLMs to be very effective as idea generators for external sound planners or humans in the loop in computer-supported cooperative work scenarios, while themselves being very bad at generating plans that are guaranteed to be correct. This is especially likely because the chief power of LLMs comes from their pattern-finding abilities than from first-principles simulations over world models. Compared to a planner that is guaranteed to be correct in a narrow set of domains, LLMs may likely be good at generating plausible (but not guaranteed to be correct) plan heuristics/suggestions in many more domains.
To investigate these questions in a systematic rather than anecdotal manner, we generate a suite of planning problem instances  ${}^{2}$  based on the kinds of domains employed in the International Planning Competition [14]. To eliminate the subjective aspect of analysis that forms the core part of many earlier efforts on evaluating the reasoning capabilities of LLMs, we automate the evaluation by leveraging models and tools from the automated planning community. The evaluation itself is done
Figure 1: The diagrammatic overview of the two modes of LLMs for planning.
in two modes (shown in Figure 1). In the first "autonomous" mode, LLMs are used standalone, and we directly assess the quality and correctness of plans they generate. As we shall see, the results in the autonomous mode are pretty bleak. On an average, only about  $12\%$  of the plans that the best LLM (GPT-4) generates are actually executable without errors and reach their goals. We will show that the choice of the specific LLM (we have tested the family of GPT LLMs including GPT-4 [25], GPT-3.5 [24], InstructGPT-3.5, InstructGPT-3 [26] and GPT-3 [3]), as well as fine tuning does not seem to have a major effect on this dismal performance. We also show that the performance deteriorates further if the names of the actions and objects in the domain are obfuscated-a change that doesn't in anyway affect the performance of the standard AI planners. To shed further light on the performance of GPT4, we present an evaluation of the plans it generates under a series of more relaxed (more forgiving) executability conditions. Further, we provide a human baseline for the simplest domain in our set of domains, by presenting the planning instances to human subjects (through IRB-approved studies) and evaluating the quality and correctness of their plans. These results are substantially better than those of LLMs—confirming that LLMs can't plan even in a simple common sense domain in the autonomous mode. In the second "heuristic" mode, the plans produced by LLMs are given as input to an automated planner working off of a correct domain model to check whether the LLM's plans help with the search process of the underlying planner to come up with correct plans. Specifically we show that a well known automated planner called LPG [6], that uses
2Link to the github repo: https://github.com/karthikv792/LLMs-Planning
local search to locate and remove flaws in a candidate plan to make it correct, is able to repair the LLM plans with relative ease. We compare the LLM+LPG combination with two baselines, one where an empty plan is used as the seed plan for the LPG and two, where a random plan is provided as the seed plan to the LPG. We show that the average search steps by the LLM+LPG combination is much lesser than both the baselines, thereby revealing that LLMs' plans are indeed helping with the search process of the underlying planner. Further, instead of having LPG correct the plans, we use an external verifier, VAL [11], to point out the errors in the LLM-generated plans and back-prompt the LLM for a new plan with this feedback. We show that this repeated interaction indeed improves the plan correctness in common-sense domains. Overall, our findings demonstrate that, with respect to planning, LLMs' perform poorly in the autonomous mode but the generated plans can help AI planners in the search process or can be given to external verifiers and back-prompt the LLM for better plans. In this paper, we first present an overview of the related work. Following that, we describe the necessary background and the prompt generation pipeline. Finally, we provide the results and analysis of various experiments undertaken in both autonomous and heuristic evaluation modes.
In this work, we look at LLMs' planning capabilities when the domain is given as part of the prompt (as is the standard practice in automated planning [8]). Our evaluation focuses on zero-shot (just domain and problem specification), and few-shot (example problems with plans) modes. There have been a few works that looked at the planning capabilities of LLMs. Most of them, such as [12, 2] focus on commonsense domains/tasks (e.g. moving things in kitchens, wedding/menu planning etc.) and thus evaluate LLMs in a mode wherein the prompt doesn't include any information about the specific domain. Plans generated in that way are hard to evaluate as they are not directed at any plan executor and the humans often wind up giving the benefit of doubt for a plausible-but not actually executable-plan. This is why in SayCan [2], where executability is critical, they try to filter out/interpret the LLM plans in terms of the skills/actions that are actually available to the executor. While SayCan does this in a rather convoluted way that requires access to the internal log probabilities of the LLM, our approach simplifies this by specifying the domain as part of the prompt. In all our experiments, we found that LLMs only use the actions listed as part of the domain specification.
One other mode of evaluation of planning capabilities in the literature involves the user incrementally interacting with the LLM, and re-prompting it to point out flaws in its plans, with the hope that the LLM eventually reaches an executable plan [13, 39, 28]. Such evaluations are notorious for their Clever Hans effect [1] with the actual planning being done by the humans in the loop rather than the LLMs themselves. We thus separate our evaluation into two modes—autonomous and as assistants to external planners/reasoners. There have also been efforts which mostly depended on LLMs as "translators" of natural language problem/goal specification into formal specifications, which are then thrown over to sound external planners [38, 21]. Such efforts don't shed any light on the internal planning capabilities of the LLMs themselves, as our evaluations in autonomous and assistive modes do. Finally, after our initial study and benchmark were made public, other groups did parallel studies that largely corroborate our results on the ineffectiveness of LLMs in finding executable plans [32, 21].
Taking a broader perspective, making plans in the world involves (1) discovering actions (and their precondition/effect causal dependencies), and (2) sequencing an appropriate subset of available/discovered actions to achieve the agent's goals. The former requires broad knowledge about actions available in the world and their individual effects, while the latter requires deep drilling-down over a given set of actions to ensure that all goals are supported (causal chaining) without any undesirable interactions. LLMs have an edge on the former—they do indeed have web-scale broad knowledge! As we shall see however, they are very bad at the second phase of developing valid interaction-free plans (in part, because LLMs don't have the ability to do combinatorial search). Most cases in literature (as outlined in [18]) where LLMs are claimed to have "planned" turn out, upon close examination, to be instances of phase 1—your wedding plans, recipe plans etc.—where you are either using a very forgiving plan correctness criterion, or the phase 2 is vacuous. Standard AI planners-on the other hand-assume that the discovery part is done and handed down as a compact domain model, and focus mostly on the second part: selecting among known actions to establish causal chains and sequencing them to make them interaction free. In this sense, LLMs and AI planners can be complementary, as we have shown in this paper—with the former helping with phase 1—either with a candidate/approximate plan or domain model—and the latter with phase 2.
3 Prompt Generation for Classical Planning Problems
Given that we are interested in investigating the basic reasoning about actions and change problem, we want to look at the most fundamental planning formalism first, namely the goal-directed deterministic planning problem. Colloquially referred to as classical planning problem, these problem classes consist of a problem domain, an initial state and a goal state. The problem domain consists of a set of fluents which correspond to predicates with some arity and a set of actions. The state-space for the planning problem is defined by the possible truth assignment over the predicates. Each action consists of preconditions and effects where preconditions is a set of predicates that describe when an action can be executed and effects are set of predicates that describe what happens when an action is executed. The effects can further consist of add effects, which is the set of predicates that will be set true by the action, and delete effects, which is the set of predicates that will be set false. The solution for a planning problem is a sequence of actions, or a plan, that when applied in the initial state will result in a state where the goal conditions are satisfied. A standard representation to specify such kind of planning problems is the Planning Definition and Domain Language (PDDL) [22]. Below is a snippet of an action from a popular benchmark problem called Blocksworld, in PDDL. The action corresponds to picking up a block in that domain.
(:action pickup parameters(?ob) :precondition (and (clear ?ob) (on-table ?ob) (arm-empty)) effect (and (holding ?ob) (not (clear ?ob)) (not (on-table ?ob)) not (arm-empty))))
A more detailed description on classical planning problems is provided in Appendix A.1. We now will describe how we generate the prompts that are given to the LLMs.
3.2 Prompt Generation
Figure 2: The diagrammatic overview of the prompt generation pipeline. The prompt configurations for the different experiments are generated from PDDL domain files and are modified with an example generator and natural language translator as needed depending on the experiment requirements.
Prompt Configurations: We have developed a suite of unique planning problems to test LLMs' abilities to generate plans. We have multiple prompt configurations based on this suite of problems, varying in both the method of presentation as well as number of examples given to the LLM. In particular, we use two methods of presentation, natural language and PDDL, as well as two different methods of providing examples, zero shot (with no examples provided) and one shot (with an example provided), giving us four different configuration combinations for our experiments.
Within a prompt, LLMs are first provided with a lifted domain description. For one shot configurations, the prompt additionally contains an example instance of a planning problem (consisting of a description of the initial state and the goal) and the corresponding plan (which ends with a tag, referred to as the plan-end tag, that denotes the end of the plan). All prompts end with a planning problem description. The text generated by the LLM until the plan-end tag is used as the candidate for extracting the plan. If the extractor cannot reasonably extract an instance, it is marked as incorrect.
Table 1: Results of GPT-4, GPT-3.5 (popularity known as ChatGPT), Instruct-GPT3.5, Instruct-GPT3 (text-davinci-002) and GPT3 (davinci) for the Plan Generation task with prompts in natural language.
Domain	Method	Instances correct
GPT-4	GPT-3.5	I-GPT3.5	I-GPT3	GPT-3
Blocksworld (BW)	One-shot	206/600 (34.3%)	37/600 (6.1%)	54/600 (9%)	41/600 (6.8%)	6/600 (1%)
Zero-shot	210/600 (34.6%)	8/600 (1.3%)	-	-	-
COT	214/600 (35.6%)	-	-	-	-
Logistics Domain	One-shot	28/200 (14%)	1/200 (0.5%)	6/200 (3%)	3/200 (1.5%)	-
Zero-shot	15/200 (7.5%)	1/200 (0.5%)	-	-	-
Mystery BW (Deceptive)	One-shot	26/600 (4.3%)	0/600 (0%)	4/600 (0.6%)	14/600 (2.3%)	0/600 (0%)
Zero-shot	1/600 (0.16%)	0/600 (0%)	-	-	-
COT	54/600 (9%)	-	-	-	-
Mystery BW (Randomized)	One-shot	12/600 (2%)	0/600 (0%)	5/600 (0.8%)	5/600 (0.8%)	1/600 (0.1%)
Zero-shot	0/600 (0%)	0/600 (0%)	-	-	-
The prompt is either formatted in natural language or PDDL. Natural language prompts utilize complete natural language sentences to describe feasible actions in the domain. Initial conditions are also reported as complete sentences. Plans in the natural language setting take the form of a series of commands such as "stack the orange block on top of the blue block". As implied by the name, PDDL prompts format all elements (domain description, initial state, goal state, and plans) using PDDL. We point the reader to the supplementary material for examples on each of these prompt configurations.
Chain of Thought Prompting: In addition to the four experiments above, we look at a fifth experiment using a state tracking chain of thought prompting technique in a natural language one shot setting. Within this configuration, we provide an annotated example where each action is annotated with the state prior to the action, the reason for why the action is applicable in the prior state, and the resulting state after applying the action. After the example, a meta-explanation about plan correctness is provided. The LLM is then asked to return a response making the same state tracking and justification annotations that were included in the example.
Prompt Generation Pipeline: We've developed a prompt generation pipeline (visualized in Figure 2) that accepts PDDL domain files as input and outputs prompts that follow the experiments described above. The prompt generation component takes care of creating the set of PDDL problems to be solved for all experiments. Following that, examples are added to the prompt in one shot experiments. While our setup utilizes a planner during example generation, any example generation technique could be used here so long as the examples generated are valid plans. In the state tracking experiment, we also have developed a component to add justification annotations for examples so that the examples reflect what we expect of the LLM. The last step before finishing is translation: since problems at this point are currently in PDDL, prompts for all natural language experiments (whether an example was added or not) need to be translated into natural language. We utilize a domain-specific translator to do so.
4 Evaluating Planning Capabilities of LLMs in Autonomous Mode
In the autonomous mode, we treat the LLM as an automated planner and perform a single run of the dataset on the LLMs for each domain and prompt configuration. In this mode, the plan generated by the LLM is back-translated from natural language to forms that can be used by external plan translators. For each domain, we perform template-based translation to translate between PDDL and natural language for the natural language prompt configurations. We use VAL [11] to evaluate the translated plan with the corresponding domain and problem file. Our evaluation here primarily
Table 2: Results of GPT-4 and GPT-3.5 (popularity known as ChatGPT) for the Plan Generation task with one or zero examples in the prompt by directly providing the domain and problem in PDDL.
Domain	Method	Instances correct
GPT-4	GPT-3.5
Blocksworld (BW)	One-shot	75/600 (12.5%)	12/600 (2%)
Zero-shot	106/600 (17.6%)	12/600 (2%)
Logistics Domain	One-shot	28/200 (14%)	1/200 (0.5%)
Zero-shot	11/200 (5.5%)	0/200 (0%)
Mystery BW (Deceptive)	One-shot	17/600 (2.8%)	1/600 (0.1%)
Zero-shot	3/600 (0.5%)	0/600 (0%)
focuses on the GPT family of LLMs. We tested GPT-4 [25] and GPT-3.5 (commonly known as Chat-GPT) [24] on all the prompt configurations while we tested the older versions of GPT (namely, Instruct-GPT3 and GPT3) on one-shot natural language prompts across the domains. We set the temperature for all models to be 0, thereby making them deterministic. In this section, we detail the evaluation of LLMs on these domains and prompt configurations. We would like to point the reader to the Appendix for example prompts.
Evaluation of LLMs on the Blocksworld domain: Blocksworld problems capture common sense block manipulations and consist of a set of blocks. Blocks are identified with unique colors and are placed either on a table or on top of other blocks. The goal is to arrange some of these blocks in a stack in a particular order. The general expectation here would be that one can pick up a block if it is clear, i.e., there are no other blocks on top of that block and you can only stack a block on top of another block if it is clear. The choice of this particular domain is motivated by both the fact that this is a simple common sense domain and is a very popular domain in planning literature, that has a long history of being used in various planning challenges. The instances were generated using a PDDL generator employed in the IPC competitions. We permitted the generation of problems that varied in terms of the number of blocks (3-5), optimal plan length, and goal properties (positive, negative, or no interactions between subgoals).
As shown in Table 1 and Table 2, GPT-4 improves upon previous versions of GPT models in the Blockworld domain across all four prompt configurations. However, the overall performance is still approximately  $34\%$  in the Blockworld dataset. Even the chain of thought style prompting (indicated by COT in the tables) had little effect on improving the performance. GPT-4 performs better with natural language prompts (206 and 210 instances for one-shot and zero-shot prompts, respectively) as opposed to PDDL prompts (75 and 106 instances). The performance drops significantly with other GPT models. We also discovered that for instances where Instruct-GPT3 generated the correct plans, replacing the example plan in the prompt with another example plan led to an even greater drop in accuracy. This suggests that the LLM seems to rely primarily on pattern matching, rather than inducing some internal model from the prompts. Overall, even in a seemingly simple common-sense domain like Blockworld, which humans typically find easy to navigate, LLMs prove to be quite ineffective in planning autonomously.
Finetuning GPT-3 on Blocksworld: Along with directly testing the LLMs from the GPT family, we have also looked at the utility of fine-tuning the LLMs. Specifically, we fine-tuned GPT-3 (Davinci) in the Blocksworld domain. For this, we prepared a dataset comprising the initial state, goal state, and the respective plan for 1,000 distinct Blocksworld instances. It's important to note that these instances were separate from our test set of 600 instances. By using the default hyperparameters provided by OpenAI and an 80-20 train-validation data split, we carried out the fine-tuning process. Our results revealed that the fine-tuned GPT-3 solved only 122 instances out of the 600 in our set, representing approximately  $20\%$  of the total. This suggests that fine-tuning has a limited impact on improving the performance of LLMs in Blocksworld planning. This outcome aligns with the observations of [40], who argue that language models trained for reasoning tend to concentrate on the inherent statistical features instead of the causal structure, which in turn affects their performance on such tasks.
Open-source models: In addition to the GPT family of LLMs, we have also conducted preliminary experiments with an open-source LLM, BLOOM [31], and found that BLOOM too is ineffective in plan generation. We assessed BLOOM's performance in the blockworld and mystery blockworld (deceptive) domains using a one-shot natural language prompt configuration. In the blockworld domain, BLOOM correctly handled only 4 out of 250 instances, representing a  $1.6\%$  success rate. In the mystery domain, it failed to produce a single correct response in all 50 instances.