Title: On the Planning Abilities of Large Language Models - A Critical Investigation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Large Language Models

Planning

LLMs for autonomous and heuristic planning guidance
Abstract: Intrigued by the claims of emergent reasoning capabilities in LLMs trained on general web corpora, in this paper, we set out to investigate their planning capabilities. We aim to evaluate (1) the effectiveness of LLMs in generating plans autonomously in commonsense planning tasks and (2) the potential of LLMs as a source of heuristic guidance for other agents (AI planners) in their planning tasks. We conduct a systematic study by generating a suite of instances on domains similar to the ones employed in the International Planning Competition and evaluate LLMs in two distinct modes: autonomous and heuristic. Our findings reveal that LLMs’ ability to generate executable plans autonomously is rather limited, with the best model (GPT-4) having an average success rate of ~12% across the domains. However, the results in the heuristic mode show more promise. In the heuristic mode, we demonstrate that LLM-generated plans can improve the search process for underlying sound planners and additionally show that external verifiers can help provide feedback on the generated plans and back-prompt the LLM for better plan generation.

On the Planning Abilities of Large Language Models : A Critical Investigation

Abstract

Intrigued by the claims of emergent reasoning capabilities in LLMs trained on general web corpora, in this paper, we set out to investigate their planning capabilities. We aim to evaluate (1) the effectiveness of LLMs in generating plans autonomously in commonsense planning tasks and (2) the potential of LLMs as a source of heuristic guidance for other agents (AI planners) in their planning tasks. We conduct a systematic study by generating a suite of instances on domains similar to the ones employed in the International Planning Competition and evaluate LLMs in two distinct modes: autonomous and heuristic . Our findings reveal that LLMs' ability to generate executable plans autonomously is rather limited, with the best model (GPT-4) having an average success rate of ∼ 12% across the domains. However, the results in the heuristic mode show more promise. In the heuristic mode, we demonstrate that LLM-generated plans can improve the search process for underlying sound planners and additionally show that external verifiers can help provide feedback on the generated plans and back-prompt the LLM for better plan generation.

1 Introduction

It would be no exaggeration to say that transformer-based large language models (LLMs) have revolutionized the field of natural language processing (NLP). Kicked off by the advances presented by the GPT-x models developed by OpenAI [27], these types of language models currently provide state-of-the-art performance in many of the standard NLP tasks. Although LLMs were originally developed mostly to do word sequence completion tasks, with no guarantees about the completion beyond its coherence, there have been increasing claims and anecdotal evidence that they have other emergent capabilities that are not normally associated with sequence completion. Indeed, the hints of such emergent capabilities has started a veritable land rush, with researchers probing (prompting) and studying LLM behavior almost as if they were artificial organisms (c.f. [16]). Of particular interest to us in this paper is the thread of efforts that aim to investigate (and showcase) reasoning abilities of LLMs-including commonsense reasoning [35, 29, 7], logical reasoning [33], and even ethical reasoning [15]. The macro-tenor of the drumbeat of these works has been suggesting that LLM's are indeed capable of doing such kinds of reasoning [19, 37, 4].

Matthew Marquez

Subbarao Kambhampati

One type of reasoning task that has been well studied in the AI community is planning and sequential decision making. At its simplest, planning involves developing a course of actions (policy) which when executed takes the agent to a desired state of the world. Planning has generally been studied primarily as an inference on world and reward models-whether specified by humans or learned by the agent by interacting with its world. In this paper, we are interested in seeing what planning abilities, if any, LLMs may already have, given their high capacity functions (with billions of tunable parameters) trained on web-scale corpora. Specifically, we are interested in answering two broad questions:

How effective are LLMs by themselves in generating simple plans in commonsense planning tasks (of the type that humans are generally quite good at)?

How good are LLMs in being a source of heuristic guidance for other agents in their planning tasks?

Notice that in theory, it is possible for LLMs to be very effective as idea generators for external sound planners or humans in the loop in computer-supported cooperative work scenarios, while themselves being very bad at generating plans that are guaranteed to be correct. This is especially likely because the chief power of LLMs comes from their pattern-finding abilities than from firstprinciples simulations over world models. Compared to a planner that is guaranteed to be correct in a narrow set of domains, LLMs may likely be good at generating plausible (but not guaranteed to be correct) plan heuristics/suggestions in many more domains.

To investigate these questions in a systematic rather than anecdotal manner, we generate a suite of planning problem instances 2 based on the kinds of domains employed in the International Planning Competition [14]. To eliminate the subjective aspect of analysis that forms the core part of many earlier efforts on evaluating the reasoning capabilities of LLMs, we automate the evaluation by leveraging models and tools from the automated planning community. The evaluation itself is done

Back-prompting

Language

Model

plan

Seed

Paninar

LPG

Candidate

Planning System

plan

VAL

Language

Model

LLM asheuristics

tosoundplannerslikeLPGorverifierslikeVAL

PlanningSystem

LLMasanautonomousplanner

in two modes (shown in Figure 1). In the first 'autonomous" mode, LLMs are used standalone, and we directly assess the quality and correctness of plans they generate. As we shall see, the results in the autonomous mode are pretty bleak. On an average, only about 12% of the plans that the best LLM (GPT-4) generates are actually executable without errors and reach their goals. We will show that the choice of the specific LLM (we have tested the family of GPT LLMs including GPT-4 [25], GPT-3.5 [24], InstructGPT-3.5, InstructGPT-3 [26] and GPT-3 [3]), as well as fine tuning does not seem to have a major effect on this dismal performance. We also show that the performance deteriorates further if the names of the actions and objects in the domain are obfuscated-a change that doesn't in anyway affect the performance of the standard AI planners. To shed further light on the performance of GPT4, we present an evaluation of the plans it generates under a series of more relaxed (more forgiving) executability conditions. Further, we provide a human baseline for the simplest domain in our set of domains, by presenting the planning instances to human subjects (through IRB-approved studies) and evaluating the quality and correctness of their plans. These results are substantially better than those of LLMs-confirming that LLMs can't plan even in a simple common sense domain in the autonomous mode. In the second 'heuristic" mode, the plans produced by LLMs are given as input to an automated planner working off of a correct domain model to check whether the LLM's plans help with the search process of the underlying planner to come up with correct plans. Specifically we show that a well known automated planner called LPG [6], that uses

local search to locate and remove flaws in a candidate plan to make it correct, is able to repair the LLM plans with relative ease. We compare the LLM+LPG combination with two baselines, one where an empty plan is used as the seed plan for the LPG and two, where a random plan is provided as the seed plan to the LPG. We show that the average search steps by the LLM+LPG combination is much lesser than both the baselines, thereby revealing that LLMs' plans are indeed helping with the search process of the underlying planner. Further, instead of having LPG correct the plans, we use an external verifier, V AL [11], to point out the errors in the LLM-generated plans and back-prompt the LLM for a new plan with this feedback. We show that this repeated interaction indeed improves the plan correctness in common-sense domains. Overall, our findings demonstrate that, with respect to planning, LLMs' perform poorly in the autonomous mode but the generated plans can help AI planners in the search process or can be given to external verifiers and back-prompt the LLM for better plans. In this paper, we first present an overview of the related work. Following that, we describe the necessary background and the prompt generation pipeline. Finally, we provide the results and analysis of various experiments undertaken in both autonomous and heuristic evaluation modes.

2 Related Work

In this work, we look at LLMs' planning capabilities when the domain is given as part of the prompt (as is the standard practice in automated planning [8]). Our evaluation focuses on zero-shot (just domain and problem specification), and few-shot (example problems with plans) modes. There have been a few works that looked at the planning capabilities of LLMs. Most of them, such as [12, 2] focus on commonsense domains/tasks (e.g. moving things in kitchens, wedding/menu planning etc.) and thus evaluate LLMs in a mode wherein the prompt doesn't include any information about the specific domain. Plans generated in that way are hard to evaluate as they are not directed at any plan executor and the humans often wind up giving the benefit of doubt for a plausible-but not actually executable-plan. This is why in SayCan [2], where executability is critical, they try to filter out/interpret the LLM plans in terms of the skills/actions that are actually available to the executor. While SayCan does this in a rather convoluted way that requires access to the internal log probabilities of the LLM, our approach simplifies this by specifying the domain as part of the prompt. In all our experiments, we found that LLMs only use the actions listed as part of the domain specification.

One other mode of evaluation of planning capabilities in the literature involves the user incrementally interacting with the LLM, and re-prompting it to point out flaws in its plans, with the hope that the LLM eventually reaches an executable plan [13, 39, 28]. Such evaluations are notorious for their Clever Hans effect [1] with the actual planning being done by the humans in the loop rather than the LLMs themselves. We thus separate our evaluation into two modes-autonomous and as assistants to external planners/reasoners. There have also been efforts which mostly depended on LLMs as 'translators" of natural language problem/goal specification into formal specifications, which are then thrown over to sound external planners [38, 21]. Such efforts don't shed any light on the internal planning capabilities of the LLMs themselves, as our evaluations in autonomous and assistive modes do. Finally, after our initial study and benchmark were made public, other groups did parallel studies that largely corroborate our results on the ineffectiveness of LLMs in finding executable plans [32, 21].

Taking a broader perspective, making plans in the world involves (1) discovering actions (and their precondition/effect causal dependencies), and (2) sequencing an appropriate subset of available/discovered actions to achieve the agent's goals. The former requires broad knowledge about actions available in the world and their individual effects, while the latter requires deep drilling-down over a given set of actions to ensure that all goals are supported (causal chaining) without any undesirable interactions. LLMs have an edge on the former-they do indeed have web-scale broad knowledge! As we shall see however, they are very bad at the second phase of developing valid interaction-free plans (in part, because LLMs don't have the ability to do combinatorial search). Most cases in literature (as outlined in [18]) where LLMs are claimed to have "planned" turn out, upon close examination, to be instances of phase 1-your wedding plans, recipe plans etc.-where you are either using a very forgiving plan correctness criterion, or the phase 2 is vacuous. Standard AI planners-on the other hand-assume that the discovery part is done and handed down as a compact domain model, and focus mostly on the second part: selecting among known actions to establish causal chains and sequencing them to make them interaction free. In this sense, LLMs and AI planners can be complementary, as we have shown in this paper-with the former helping with phase 1-either with a candidate/approximate plan or domain model-and the latter with phase 2.

3 Prompt Generation for Classical Planning Problems

3.1 Background

Given that we are interested in investigating the basic reasoning about actions and change problem, we want to look at the most fundamental planning formalism first, namely the goal-directed deterministic planning problem. Colloquially referred to as classical planning problem , these problem classes consist of a problem domain, an initial state and a goal state. The problem domain consists of a set of fluents which correspond to predicates with some arity and a set of actions. The state-space for the planning problem is defined by the possible truth assignment over the predicates. Each action consists of preconditions and effects where preconditions is a set of predicates that describe when an action can be executed and effects are set of predicates that describe what happens when an action is executed. The effects can further consist of add effects, which is the set of predicates that will be set true by the action, and delete effects, which is the set of predicates that will be set false. The solution for a planning problem is a sequence of actions, or a plan, that when applied in the initial state will result in a state where the goal conditions are satisfied. A standard representation to specify such kind of planning problems is the Planning Definition and Domain Language (PDDL) [22]. Below is a snippet of an action from a popular benchmark problem called Blocksworld, in PDDL. The action corresponds to picking up a block in that domain.

A more detailed description on classical planning problems is provided in Appendix A.1. We now will describe how we generate the prompts that are given to the LLMs.

3.2 Prompt Generation

OneShotNL

ExampleGeneration

(withStateTracking)

Example

StateTracking

Generator

Annotator

OneShotNL

PDDL

Natural

PDDL

Problem

Language

ZeroShot NL

Domain

Files

Generator

Translator

One ShotPDDL

Zero ShotPDDL

Prompt Configurations: We have developed a suite of unique planning problems to test LLMs' abilities to generate plans. We have multiple prompt configurations based on this suite of problems, varying in both the method of presentation as well as number of examples given to the LLM. In particular, we use two methods of presentation, natural language and PDDL, as well as two different methods of providing examples, zero shot (with no examples provided) and one shot (with an example provided), giving us four different configuration combinations for our experiments.

Within a prompt, LLMs are first provided with a lifted domain description. For one shot configurations, the prompt additionally contains an example instance of a planning problem (consisting of a description of the initial state and the goal) and the corresponding plan (which ends with a tag, referred to as the plan-end tag, that denotes the end of the plan). All prompts end with a planning problem description. The text generated by the LLM until the plan-end tag is used as the candidate for extracting the plan. If the extractor cannot reasonably extract an instance, it is marked as incorrect.

The prompt is either formatted in natural langauge or PDDL. Natural language prompts utilize complete natural language sentences to describe feasible actions in the domain. Initial conditions are also reported as complete sentences. Plans in the natural language setting take the form of a series of commands such as "stack the orange block on top of the blue block". As implied by the name, PDDL prompts format all elements (domain description, initial state, goal state, and plans) using PDDL. We point the reader to the supplementary material for examples on each of these prompt configurations.

Chain of Thought Prompting: In addition to the four experiments above, we look at a fifth experiment using a state tracking chain of thought prompting technique in a natural language one shot setting. Within this configuration, we provide an annotated example where each action is annotated with the state prior to the action, the reason for why the action is applicable in the prior state, and the resulting state after applying the action. After the example, a meta-explanation about plan correctness is provided. The LLM is then asked to return a response making the same state tracking and justification annotations that were included in the example.

Prompt Generation Pipeline: We've developed a prompt generation pipeline (visualized in Figure 2) that accepts PDDL domain files as input and outputs prompts that follow the experiments described above. The prompt generation component takes care of creating the set of PDDL problems to be solved for all experiments. Following that, examples are added to the prompt in one shot experiments. While our setup utilizes a planner during example generation, any example generation technique could be used here so long as the examples generated are valid plans. In the state tracking experiment, we also have developed a component to add justification annotations for examples so that the examples reflect what we expect of the LLM. The last step before finishing is translation: since problems at this point are currently in PDDL, prompts for all natural language experiments (whether an example was added or not) need to be translated into natural language. We utilize a domain-specific translator to do so.

4 Evaluating Planning Capabilities of LLMs in Autonomous Mode

In the autonomous mode, we treat the LLM as an automated planner and perform a single run of the dataset on the LLMs for each domain and prompt configuration. In this mode, the plan generated by the LLM is back-translated from natural language to forms that can be used by external plan validators. For each domain, we perform template-based translation to translate between PDDL and natural language for the natural language prompt configurations. We use VAL [11] to evaluate the translated plan with the corresponding domain and problem file. Our evaluation here primarily

focuses on the GPT family of LLMs. We tested GPT-4 [25] and GPT-3.5 (commonly known as Chat-GPT) [24] on all the prompt configurations while we tested the older versions of GPT (namely, Instruct-GPT3 and GPT3) on one-shot natural language prompts across the domains. We set the temperature for all models to be 0, thereby making them deterministic. In this section, we detail the evaluation of LLMs on these domains and prompt configurations. We would like to point the reader to the Appendix for example prompts.

Evaluation of LLMs on the Blocksworld domain: Blocksworld problems capture common sense block manipulations and consist of a set of blocks. Blocks are identified with unique colors and are placed either on a table or on top of other blocks. The goal is to arrange some of these blocks in a stack in a particular order. The general expectation here would be that one can pick up a block if it is clear, i.e., there are no other blocks on top of that block and you can only stack a block on top of another block if it is clear. The choice of this particular domain is motivated by both the fact that this is a simple common sense domain and is a very popular domain in planning literature, that has a long history of being used in various planning challenges. The instances were generated using a PDDL generator employed in the IPC competitions. We permitted the generation of problems that varied in terms of the number of blocks (3-5), optimal plan length, and goal properties (positive, negative, or no interactions between subgoals).

As shown in Table 1 and Table 2, GPT-4 improves upon previous versions of GPT models in the Blocksworld domain across all four prompt configurations. However, the overall performance is still approximately 34% in the Blocksworld dataset. Even the chain of thought style prompting (indicated by COT in the tables) had little effect on improving the performance. GPT-4 performs better with natural language prompts (206 and 210 instances for one-shot and zero-shot prompts, respectively) as opposed to PDDL prompts (75 and 106 instances). The performance drops significantly with other GPT models. We also discovered that for instances where Instruct-GPT3 generated the correct plans, replacing the example plan in the prompt with another example plan led to an even greater drop in accuracy. This suggests that the LLM seems to rely primarily on pattern matching, rather than inducing some internal model from the prompts. Overall, even in a seemingly simple common-sense domain like Blocksworld, which humans typically find easy to navigate, LLMs prove to be quite ineffective in planning autonomously.

Finetuning GPT-3 on Blocksworld: Along with directly testing the LLMs from the GPT family, we have also looked at the utility of fine-tuning the LLMs. Specifically, we fine-tuned GPT-3 (Davinci) in the Blocksworld domain. For this, we prepared a dataset comprising the initial state, goal state, and the respective plan for 1,000 distinct Blocksworld instances. It's important to note that these instances were separate from our test set of 600 instances. By using the default hyperparameters provided by OpenAI and an 80-20 train-validation data split, we carried out the fine-tuning process. Our results revealed that the fine-tuned GPT-3 solved only 122 instances out of the 600 in our set, representing approximately 20% of the total. This suggests that fine-tuning has a limited impact on improving the performance of LLMs in Blocksworld planning. This outcome aligns with the observations of [40], who argue that language models trained for reasoning tend to concentrate on the inherent statistical features instead of the causal structure, which in turn affects their performance on such tasks.

Open-source models: In addition to the GPT family of LLMs, we have also conducted preliminary experiments with an open-source LLM, BLOOM [31], and found that BLOOM too is ineffective in plan generation. We assessed BLOOM's performance in the blocksworld and mystery blocksworld (deceptive) domains using a one-shot natural language prompt configuration. In the blocksworld domain, BLOOM correctly handled only 4 out of 250 instances, representing a 1.6% success rate. In the mystery domain, it failed to produce a single correct response in all 50 instances.

Human Baseline for the Blocksworld: We have previously mentioned that planning tasks on the blocksworld domain are anecdotally simple enough for humans to perform. To establish this and come up with a preliminary baseline to compare LLMs performance, we conducted an IRB-approved user study where we asked 50 participants to come up with a plan for a blocksworld instance picked at random, from the set of 600 instances that we used for the evaluation of LLMs. We presented the same domain description as we did for the LLMs and then primed them with an example instance. We point the reader to the supplementary material for further details on the study.

Out of the 50 participants, 39 of them (78%) came up with a valid plan. Along with validity, we also tested the optimality of their plans even though they were not required to come up with an optimal plan. Out of the 39 participants, 35 (89.7%) participants came up with an optimal plan. These initial results show that the blocksworld domain is a simple enough domain where most humans are able to come up with plans (which are also optimal) while LLMs, on the other hand, showcase subpar performance.

Evaluation of LLMs on the Logistics domain: Logistics is also a widely recognized domain in the planning literature. In this domain, the objective is to transport packages within cities via trucks, and between cities via airplanes. Within a city, the locations are directly linked, allowing trucks to travel between any two of these locations. Similarly, cities are directly connected to each other allowing airplanes to travel between any two cities. Each city is equipped with one truck and has a designated location that functions as an airport. We generated 200 instances on this domain.

From Tables 1 and 2, we see that in the one-shot setting with natural language input, GPT-4 only solved 14% of the instances (28/200), and this rate dropped to 7.5% (15/200) when using zero-shot prompting. When provided with the domain and problem in PDDL format, GPT-4's performance remained the same in the one-shot setting (14% or 28/200) but decreased to 5.5% (11/200) in the zero-shot setting. GPT-3.5 did even worse.

Obfuscating names to test the brittleness of LLM Planning: Although the domain specification is part of our prompts, the names of the objects (e.g. blocks, trucks), predicates (e.g. on-table, in-city) and actions (e.g. pickup, drive) still do provide connections to the commonsense knowledge that the pretrained LLMs possess. One intriguing question is whether the planning performance is based really only on the domain model or these other background connections. To test this, we experimented with a variation of the Blocksworld domain, where we obfuscate the action names (for example pickup becomes attack, and unstack becomes feast) and predicate names (for example ontable becomes planet, and handempty becomes harmony). Note that from the perspective of standard planners, these domains are essentially identical. 3 In addition to such deceptive obfuscation, we also considered a variation where random alphanumeric names were substituted for the action and object names.

Tables 1 and 2, we see that this simple obfuscation leads to a catastrophic drop in performance. Specifically, with zero-shot prompting and natural language input, GPT-4 is able to solve 210 instances out of 600 in the Blocksworld domain, but it could only solve 1 instance in the deceptive Mystery Blocksworld domain and 0 instances in the randomized mystery domain. A similar result is observed with the PDDL-style prompts: GPT-4 could solve 106 instances in Blocksworld, but only 3 instances in the deceptive Mystery Blocksworld. Notably, chain of thought prompting does not significantly improve performance over one-shot natural language prompts. GPT-3.5 does not solve even a single instance in the entire set of natural language instances. For most of the instances, GPT-3.5 outputs that the instance can't be solved. These results strongly suggest that whatever accidental planning performance LLMs show is likely connected to pattern matching rather than reasoning (which should be robust to name change).

Planning Failure Modes by Relaxation Type for Blocksworld

Planning Failure Modes by Relaxation Type for Blocksworld

600

onNaturalLanguagePrompts

on PDDL Prompts

600

Non Goal Reaching

500

500

Inexecutable

One shot

400

400

'///,Zero shot

Instances

Instances

300

300

200

200

100

100

DeleteandPrecondition

Precondition Only

Delete Only

Non-Relaxed

DeleteandPrecondition

Precondition Only

Delete Only

Non-Relaxed

Relaxation Type

Relaxation Type

Analyzing GPT-4 failures: To get a better sense of the type of failures LLM generated plans encounter, we wondered whether they will fare much better with a more forgiving test of the validity of the generated plans. In automated planning community, the notion of relaxations of the domain model are used to simplify the problem-chiefly to derive heuristics for planning problems [8]. Taking a leaf from them, we considered two types of relaxations: (i) delete relaxation involves ignoring all the delete conditions of the domain actions (thus making sure that there can be no negative interactions between subgoals) and (ii) precondition relaxation involves ignoring all the preconditions of the domain actions-thus assuming that the the actions are executable from any state giving their effects. Our idea is to evaluate the plans produced by GPT4 with respect to domain models that are delete relaxed, precondition relaxed or both. It should be clear that a plan that is correct with respect to the normal (unrelaxed) model will also be correct with respect to all the relaxed models. Figure 3 shows the results for blocksworld. We see that while the correctness of LLM generated plans increased under more forgiving (relaxed) assessments (area in green), even in the most lenient assessment mode (Delete+Precondition Relaxed), there still are plans ( ∼ 39%) that are incorrect (because they still don't reach the goals) across all the prompt configurations. The plots further classify the failure cases in terms of whether they were inexecutable , shown in maroon, or could be executed but didn't reach the goals (shown in red). Note that when preconditions are relaxed, all plans are executable. We provide additional details on the relaxed assessments in Appendix A.2.

5 Evaluating LLMs as Idea Generators

While the preceding discussion establishes that LLMs are not capable of generating correct plans in autonomous mode, there is still the possibility that they can be useful idea generators for other sound external planners, verifiers or even humans-in-the-loop. In this section, we investigate this possibility and demonstrate that LLMs show promise on this front (especially with external planners and verfiers).

5.1 LLMPlans as Heuristics to Sound Planners

To see if the LLM generated plans can provide heuristic guidance to sound external planners, we use a local-search planner LPG [6] which generates plans by starting with a seed plan and iteratively repairing flaws until a correct plan is found. We feed the LLM-generated plan as the initial seed plan

for LPG's iterative search. Our hypothesis is that this might put LPG on the right path and reduce the time for it to generate a correct plan. It is interesting to note the similarities between this LLM+LPG approach, and the approaches used in case-based planning in the past [9, 17]. Here the LLM can be loosely viewed as 'retrieving a potentially useful plan case/sketch' out of thin air, which the LPG adapts/corrects.

We utilized the plans that were generated by LLMs in the one-shot natural language prompt configuration on all three of our previous domains - Blocksworld, Mystery Blocksworld, and Logistics - as the "seed plans" from which LPG would begin its local search for a valid plan. For the Blocksworld domain, both GPT-4 and Instruct-GPT3 were evaluated, whereas for the Logistics and Mystery domains only GPT-4 was evaluated. We confirmed that all the plans that were generated by this LLM+LPG combination for both the domains were valid (which is as expected given that the underlying planner, LPG, is sound). To get an idea of how far the initial LLM generated plans were from the final correct solutions generated by LPG, we measured the Levenshtein edit distance between them. While the default LPG local search doesn't aim to minimize the changes to the suggested plan (there do exist versions of LPG that do this; see [23]) , the edit distances also give an idea of how partially or approximately correct the original LLM plan is. Along with the edit distance, we also measured the number of search steps that were taken by the LPG to come up with a correct plan.

As shown in Table 3, the edit distances across domains are approximately half the length of the seed plans generated by the LLMs, indicating that 50% of the final plan retains the elements of the initial LLM plan. For each problem, we performed two additional plan initializations to serve as baselines: initializing with an empty plan and initializing with a random plan of the same length as the plan generated by the LLM for that problem. In the Blocksworld and Logistics 4 domains, we see a significant improvement in search steps over the empty seed plan when GPT-4 is used and an even larger one over the random seed plan. Consistent with our findings in the autonomous mode, the usefulness of this assistance wanes in domains where the relationships between predicates can no longer be inferred from common sense understandings of their names: in the Mystery Blocksworld domain, the LLM only has meager reduction in step size over the random plan and actually uses more steps than the empty plan.

5.2 Verifier-assisted repeated backprompting of LLMs

The interaction between LLM and LPG was unidirectional-with LLM sending a seed plan that LPG aims to repair. One supposed advantage of LLMs is that they can be prompted to improve their solutions. Suppose we have access to a sound automated verifier that not only checks the plan correctness but also pinpoints faults (in terms of unsatisfied preconditions or delete interactions). Such feedback can be easily converted into a "backprompt" to the LLM, with the hope that LLM comes up with a better plan. This is what we do with the help of V AL[11]-an AI planning tool that uses the domain model to validate the correctness of the plans (and point out errors).

Table 4: GPT4 Performance with Backprompting by VAL [11]. Mystery BW had deceptive disguising. I.C - Instances correct (within 15 feedbacks); A.F.R - Avg. feedback rounds for correct instances.

While, as we mentioned earlier, there can be thorny 'clever hans' issues about humans prompting LLMs, an automated verifier mechanically backprompting the LLM doesn't suffer from these.

We tested this setup on a subset of the failed instances in the one-shot natural language prompt configuration using GPT-4, given its larger context window. We set a threshold of 15 backprompting rounds . We tested on three domains-Blocksworld, Logistics and Mystery BW-with 50 failed instances from each domain. Table 4 shows the results. We provide the prompt+feedback examples in Appendix A.9. We found that GPT4 is able to come up with correct plans 82% of the Blocksworld instances and 70% of the Logistics one. The average number of backprompting rounds for these

successful cases was 3.68 for BW and 3.31 for Logistics. The performance on the Mystery BW however remained quite poor-suggesting that even with back prompting, GPT4 cannot do well unless it can tease out commonsense patterns for the domain.

In this backprompting configuration, LLM serves as the candidate plan generator while VAL serves as the external sound verifier. While it is tempting to have a self-critiquing architecture with LLM also serving as the verifier, our recent work shows that approach to be of questionable utility as LLMs are no better at verifying plans than they are at generating them [36, 34].

5.3 LLMs as idea generators for humans-in-the-loop

Along with external planners and verifiers, LLMs may also offer their insights as plan suggestions directly to the human-in-the-loop which might potentially guide the user to the correct plan. After all, this sort of computer supported cooperative work (CSCW) use case has been the staple of LLM applications. We explored the efficacy of LLMs in assisting human planners through a betweensubjects user study, structured similarly to the study outlined in Section 4, but with two primary distinctions: (1) The study involved two separate participant groups. The first group received no assistance in devising plans, paralleling the approach in Section 4, while the second group had access to LLM-generated suggestions. (2) both participant sets were asked to offer subjective feedback via the NASA-TLX assessment tool [10], gauging their cognitive load. Additionally, participants from the second group evaluated the correctness of the LLM suggestions presented to them. We utilized the plans generated by GPT-4 to provide plan suggestions.

The study included 49 participants in the unassisted group and 48 in the LLM-assisted group. We evaluated the statistical significance regarding accuracy, time taken, and cognitive load between the groups. Our findings revealed no statistical significance between the groups across all three aspects. 5 . Notably, 3 out of 48 participants mistakenly accepted incorrect LLM suggestions, with two submitting these erroneous suggestions as their plans. This shows the potential for automation bias in such methodologies [5]. We have provided the details of the user-study in Appendix A.12.

6 Conclusion and Future Work

In this paper, we presented a critical investigation of the planning abilities of large language models (LLMs). To this end, we evaluated the plan generation abilities of LLMs in two different modes. In the autonomous mode, our results show that even in simple common-sense planning domains where humans could easily come up with plans, LLMs like GPT-3 exhibit a dismal performance. Even though there is an uptick in the performance by the newer GPT-4 in the blocksworld domain, it still fails miserably on the mystery blocksworld domain, indicating their inability to reason in an abstract manner. In the heuristic mode, we have seen that plans generated by LLMs can help improve the search of sound planners like LPG. Further, we showed that using external verifiers, we can point out the errors and back-prompt LLMs for a better plan. We showed that this indeed helps in common-sense domains. In the supplementary material, we show the prompt examples for all the configurations and the details of the user-studies (Appendix A.11). From our studies, we see that LLMs as autonomous planners fail miserably, but we also see that the generated plans improve the search when used by an underlying sound planner and that better plans can be obtained by back-prompting the LLM with feedback from an external verifier.

References

Clever Hans. https://en.wikipedia.org/wiki/Clever_Hans.

Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 , 2022.

Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Mary L Cummings. Automation bias in intelligent time critical decision support systems. In Decision making in aviation , pages 289-294. Routledge, 2017.

Alfonso Gerevini and Ivan Serina. Lpg: A planner based on local search for planning graphs with action costs. In AIPS , volume 2, pages 281-290, 2002.

Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics , 9:346-361, 2021.

Malik Ghallab, Dana S. Nau, and Paolo Traverso. Automated Planning and Acting . Cambridge University Press, 2016.

Kristian J. Hammond. CHEF: A model of case-based planning. In Tom Kehler, editor, Proceedings of the 5th National Conference on Artificial Intelligence. Philadelphia, PA, USA, August 11-15, 1986. Volume 1: Science , pages 267-271. Morgan Kaufmann, 1986.

Sandra G Hart and Lowell E Staveland. Development of nasa-tlx (task load index): Results of empirical and theoretical research. Advances in psychology , 52:139-183, 1988.

Richard Howey, Derek Long, and Maria Fox. VAL: Automatic plan validation, continuous effects and mixed initiative planning using PDDL. In 16th IEEE International Conference on Tools with Artificial Intelligence , pages 294-301. IEEE, 2004.

Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning , pages 9118-9147. PMLR, 2022.

Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608 , 2022.

IPC. International planning competition, 1998.

Liwei Jiang, Jena D. Hwang, Chandrasekhar Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. Delphi: Towards Machine Ethics and Norms. ArXiv , abs/2110.07574, 2021.

Subbarao Kambhampati. AI as (an Ersatz) Natural Science? https://cacm.acm.org/blogs/blogcacm/261732-ai-as-an-ersatz-natural-science/fulltext, Jun 2022.

Subbarao Kambhampati and James A. Hendler. A validation-structure-based theory of plan modification and reuse. Artif. Intell. , 55(2):193-258, 1992.

Subbarao Kambhampati, Karthik Valmeekam, Matthew Marquez, and Lin Guan. On the role of large language models in planning, July 2023. Tutorial presented at the International Conference on Automated Planning and Scheduling (ICAPS), Prague. https://yochan-lab.github. io/tutorial/ICAPS-2023/ .

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners. arXiv preprint arXiv:2205.11916 , 2022.

Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning , pages 6565-6576. PMLR, 2021.

Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477 , 2023.

Drew McDermott, Malik Ghallab, Adele E. Howe, Craig A. Knoblock, Ashwin Ram, Manuela M. Veloso, Daniel S. Weld, and David E. Wilkins. Pddl-the planning domain definition language. 1998.

Tuan Anh Nguyen, Minh Do, Alfonso Emilio Gerevini, Ivan Serina, Biplav Srivastava, and Subbarao Kambhampati. Generating diverse plans to handle unknown and partially known user preferences. Artificial Intelligence , 190:1-31, 2012.

OpenAI. Introducing chatgpt by openai, 2022.

OpenAI. Gpt-4 technical report, 2023.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022.

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.

Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. Planning with large language models via corrective re-prompting. arXiv preprint arXiv:2211.09935 , 2022.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 8732-8740, 2020.

Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A Rothkopf, and Kristian Kersting. Large pre-trained language models contain human-like biases of what is right and wrong to do. Nature Machine Intelligence , 4(3):258-268, 2022.

Big Science. BigScience Large Open-science Open-access Multilingual Language Model. https://huggingface.co/bigscience/bloom , 2022.

Tom Silver, Varun Hariprasad, Reece S Shuttleworth, Nishanth Kumar, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. PDDL planning with pretrained large language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop , 2022.

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 , 2022.

Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati. Gpt-4 doesn't know it's wrong: An analysis of iterative prompting for reasoning problems. arXiv preprint arXiv:2310.12397 , 2023.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937 , 2018.

Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. Can large language models really improve by self-critiquing their own plans? arXiv preprint arXiv:2310.08118 , 2023.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022.

Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05128 , 2023.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations , 2023.

Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van den Broeck. On the paradox of learning to reason from data. arXiv preprint arXiv:2205.11502 , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper conducts a systematic study by generating a suite of instances on domains similar to the ones employed in the International Planning Competition and evaluate LLMs in two distinct modes: autonomous and heuristic. The experiments show that LLMs' ability to generate executable plans autonomously is rather limited, while the results in the heuristic mode show more promise.\n\nSTRENGTHS:\n1. The paper is well-written and easy to understand.\n2. The paper provides a detailed investigation of GPT's planning abilities in different domains and presents some interesting findings.\n\nWEAKNESSES:\n1. Some of the conclusions presented in the paper, such as re-planning makes better performance, have already been widely applied in robotics task and motion planning applications, which is not considered novel. The community has developed a range of interesting algorithms to enhance the planning capabilities of LLM, including re-plan and generating feasible plans [1,2,3,4,5]. While the authors ignores these efforts.\n2. The author's investigation of GPT's planning abilities in certain domains overlooks the fact that GPT's greatest strength lies in its zero-shot or few-shot capabilities across different domains, without the need for pre-defined action spaces. Additionally, the metrics used to evaluate GPT are not fair to the model itself. At least human evaluation should be introduced for a more comprehensive assessment. More detailed experiments and metrics can be found in [zero-shot].\n3. For a survey paper, it is not sufficient to only consider OpenAI's GPT-level models. Open-source models like LLaMA and Vicuna should also be included in the analysis to better understand if there are fundamental differences in planning capabilities across different levels of language models.\n4. More demonstrations in the prompts seem to effectively improve planning performance. The author only conducted zero-shot and one-shot experiments, which is insufficient. A ablation experiment to explore the importance of the \"how much-shot\" factor would be valuable to the community.\n5. Some relevant papers are not cited, which are listed in the references.\nReferences:   \n[1] Inner Monologue: Embodied reasoning through planning with language models    \n[2] Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents   \n[3] ReAct: Synergizing Reasoning and Acting in Language Models    \n[4] Reflexion: Language Agents with Verbal Reinforcement Learning     \n[5] Text2Motion: From Natural Language Instructions to Feasible Plans\n\nQUESTIONS:\nAnswer questions in weakness section.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a systematic evaluation of the Planning abilities of a class of LLM (GPT series until the latest GPT-4), using standardized planning problems such as those provided in symbolic planning competitions. It analyzes LLM as autonomous planners, but also as heuristic planners providing suggestions to sound planners (heuristic mode). The experiments use a sophisticated prompting mechanism allowing both NL prompting and PDDL prompting. In the heuristic mode, LLM plans can be repaired via a local search planner (LPG), or use a verifier to send feedback to the LLM ('backprompting'). While LLMs score poorly as autonomous planners, they can achieve high scores (70-80%) in heuristic mode on standard benchmarks. \n\nPost-rebuttal comments: the authors have fully answered my questions, in particular providing additional results for another LLM (BLOOM, which appears a very reasonable choice, due to how its ecosystem differs from that of the GPT family). Their commitment to release some of the evaluation resources can only enhance the contribution of this work, which is reflected in my increasing the 'contribution' score in this review. Having also considered other reviewers' comments and the author responses, I remain very positive about this paper and maintain my original score.\n\nSTRENGTHS:\nThe paper introduces a comprehensive evaluation method for the Planning abilities of LLM, harnessing the full methodology of traditional Planning in terms of benchmarks and reasoners. The choice of the latter is particularly appropriate to the experiments at hand, since it includes both a local planner, well-suited to Plan repair, and a validator that can send feedback by identifying gaps or flaws in the proposed plan under the 'heuristic mode'. It is fairly impressive to have automated a process previously taking place in an interactive form with a human in the loop, which was vulnerable to the Clever Hans effect, also observed in other forms of LLM reasoning [1]. \nThe prompting mechanism is particularly sophisticated without being over-engineered, as their is a clear rationale for supporting each option and a rather elegant design starting with PDDL domains and branching out to generate NL or PDDL prompts. \nIn line with some claims that LLM reasoning tend to reproduce human reasoning to some extent, the choice of planning domains known to be solvable by humans is of high interest, although some of the user/human experiments have been moved back to supplementary material. \nThe paper is highly readable and quite systematic, and has all the elements to become a reference paper on the topic, not least for the results produced and the contrast between autonomous and heuristic modes, the latter avoiding pitfalls of interactivity or CoT limitations. \n\n[1] Shapira, N., Levy, M., Alavi, S.H., Zhou, X., Choi, Y., Goldberg, Y., Sap, M. and Shwartz, V., 2023. Clever hans or neural theory of mind? stress testing social reasoning in large language models. arXiv preprint arXiv:2305.14763.\n\nWEAKNESSES:\nThe paper is technically sound with very few weaknesses. Perhaps one issue is the limited number of Planning test domains used in the experiments, especially compared to [Silver et al., 2022] (ref [24] in the paper). This might be in relation to the need to explore human solutions to Planning problems as described in the supplementary material, still it could be worth justifying explicitly. \nAnother potential issue would be concentrating on the GPT family, as LLM may vary in their real-world knowledge depending on their training base.\n\nThe paper has similarities with the following preprint: https://arxiv.org/abs/2302.06706\nThis is not a major issue, either in terms of novelty or in terms of anonymity, since the submission has substantial new material and there is no direct link to the authors of the preprint. Regardless of whether the preprint is from the same authors (or a subset), it would still be appropriate to reference it in the final version of the paper.\n\nQUESTIONS:\nCan any of the observations be related to the autoregressive nature of the LLM explored? \nWhat variability in performance would you expect across LLMs (other than GPT versions)?\n\nLIMITATIONS:\nThe conclusion section leaves little space to discuss limitations of the approach. Since CoT prompting has also been explored, it could have been interesting to discuss the proposed coupling of LLM to reasoners via PDDL exchange proposed as part of \"Faithful CoT\" [1].\n\nThe paper rightly identifies the potential role of the natural language semantics of predicates or operators' names in LLM's planning abilities, for which it designs various methods of obfuscation. However, further discussions would be interesting for this phenomenon reported in [2] (\"semantics of the English terms used in the PDDL problems\"), such semantics of PDDL contents having also been proposed as a mechanism for planning model extension and planning repair [3].\n\n[1] Lyu, Q., Havaldar, S., Stein, A., Zhang, L., Rao, D., Wong, E., Apidianaki, M. and Callison-Burch, C., 2023. Faithful chain-of-thought reasoning. arXiv preprint arXiv:2301.13379.\n[2] Silver, T., Hariprasad, V., Shuttleworth, R.S., Kumar, N., Lozano-Pérez, T. and Kaelbling, L.P., 2022, November. PDDL planning with pretrained large language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop. - ref [24] of the paper\n[3] Porteous, J., Ferreira, J.F., Lindsay, A. and Cavazza, M., 2021. Automated narrative planning model extension. Autonomous Agents and Multi-Agent Systems, 35(2), p.19.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work evaluates the planning abilities of LLMs in two distinct settings:  (1) As generators of final plans, with or without feedback from a validator, and (2) as generators of seed plans which are then corrected by a standard planner. The evaluations are performed on two commonsense domains for which humans tend to produce high-quality plans:  Blocksworld and Logistics. Four LLMs are tested, including GPT-4, which generates more correct plans than the other models. Still, GPT-4 is found to fail on most of the problems, even with the benefit of one-shot prompting and CoT reasoning. But feedback from a validator (VAL) dramatically boosts the solution rate to 82% in BW and 70% in logistics, after just 3-4 feedback loops on average. In the other setting, the standard planner (LPG) produces correct plans in significantly fewer steps when starting with seed plans generated by GPT-4.\n\nSTRENGTHS:\nThis is excellent work, carefully detailed, and clearly presented. It avoids the Clever Hans effect that often arises when humans are involved in evaluations.\n\nThe different evaluation settings are very well chosen, and the results provide valuable guidance as industries work to understand how these LLMs can best be leveraged.\n\nGPT-4 is thoroughly evaluated on all benchmarks. Inclusion of three other LLMs on many of the evaluations provides additional insight.\n\nWEAKNESSES:\nCovering more domains beyond these two would be a useful contribution. But two are sufficient to support the conclusions drawn, and they light the way for others to run similar evaluations on additional planning domains.\n\nQUESTIONS:\nLine 97 says that “our approach of specifying the domain as part of the prompt ensures that the generated plans only use the actions in the domain specification.” How does domain specification provide this guarantee? Can't the LLM still hallucinate nonsense?\n\nLine 191 says:  “We set the temperature for all models to be 1, thereby making them deterministic.” Is this a typo? The appendix correctly identifies zero as the temperature that produces deterministic behavior.\n\nLIMITATIONS:\nNo concerns.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the (lack of) capabilities of pretrained LLM for solving classical well-known planning benchmarks. They study both the case of fully autonomous LLM without any external feedback and the case of using external tools, for validation feedback or as a seed for improving an external planner. As no fine-tuning is done, the work involves variations of prompts for the task considered. Fig 2 is an excellent summary of the studied tasks including zero and one-shot tasks with the problem in NL or in the original PDDL. GPT-4 is reported as the most powerful LLM, but it is still not satisfactory. Moreover, it's shown to be sensitive to the name description, performing better when using the standard names for well-known benchmarks.\n\nSTRENGTHS:\n- Some LLMs such as GPT-4 are being used to obtain plans, so it's important to investigate their capabilities. \n- Classical planning benchmarks are well-understood so offer a solid ground for evaluation.\n- Consider explicitly the case of autonomous mode vs external but automatic feedback.\n- Chain of thought is investigated, answering a question that people familiar with LLMs might have.\n\nWEAKNESSES:\n- LLMs are trained in language and human-written code. The evaluation with Randomized Disguising might be less meaningful.\n\t- However, Randomized Disguising is a small part of the work.\n\t- An alternative not explored in the paper is to add human-readable description to the domains. Even though that requires human intervention, it's reasonable to assume the ones providing the domain can also provide a description.\n- The domain-specific translator is not discussed.\n\nQUESTIONS:\n- Did you try COT with PDDL prompts?\n- Are there any indications of how LLM would perform in problems with shallow plans? If the number of objects or actions is big enough, that might be challenging for classical planners.\n- For the interactive scenario, does it make sense to keep the temperature at 0? Perhaps randomization might help the LLM to recover to deviate from earlier commitments.\n- Did you attempt relaxations with other planning problems? Perhaps it's not \"natural\" in blocks world, but there are other problems where the plans are equivalent to their relaxed versions.\n\nLIMITATIONS:\nA potential limitation of this work is that it might inform about the relevant problems that required planning from LLMs. That's not the scope of the work, so that argument should be left aside. Instead, this work is a systematic investigation of well-known planning benchmarks. Those problems might be close to the distribution of the LLMs.\n\nI miss a discussion on the complexity of the planning tasks per se. There are some simple algorithms for solving block world problems. While logistics can be a complex problem, in the PDDL benchmark solving a single problem is bounded by the complexity of moving one package, a simple problem that as cities are fully connected and it takes one airplane trip for a package to the right city. The relaxed plans discussed in the paper might be related to my question about shallow plans.\n\nOther comments:\n- Fig 2 summarizes well the approaches studied, but that's sometimes not mentioned in the table captions. For instance, Table 1 and 2 should mention that they consider the automated approach.\n- What's I-GPT-3 in Table 3?", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}