SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Human Baseline for the Blocksworld: We have previously mentioned that planning tasks on the blocksworld domain are anecdotally simple enough for humans to perform. To establish this and come up with a preliminary baseline to compare LLMs performance, we conducted an IRB-approved user study where we asked 50 participants to come up with a plan for a blocksworld instance picked at random, from the set of 600 instances that we used for the evaluation of LLMs. We presented the same domain description as we did for the LLMs and then primed them with an example instance. We point the reader to the supplementary material for further details on the study.
Out of the 50 participants, 39 of them  $(78\%)$  came up with a valid plan. Along with validity, we also tested the optimality of their plans even though they were not required to come up with an optimal plan. Out of the 39 participants, 35  $(89.7\%)$  participants came up with an optimal plan. These initial results show that the blocksworld domain is a simple enough domain where most humans are able to come up with plans (which are also optimal) while LLMs, on the other hand, showcase subpar performance.
Evaluation of LLMs on the Logistics domain: Logistics is also a widely recognized domain in the planning literature. In this domain, the objective is to transport packages within cities via trucks, and between cities via airplanes. Within a city, the locations are directly linked, allowing trucks to travel between any two of these locations. Similarly, cities are directly connected to each other allowing airplanes to travel between any two cities. Each city is equipped with one truck and has a designated location that functions as an airport. We generated 200 instances on this domain.
From Tables 1 and 2, we see that in the one-shot setting with natural language input, GPT-4 only solved  $14\%$  of the instances (28/200), and this rate dropped to  $7.5\%$  (15/200) when using zero-shot prompting. When provided with the domain and problem in PDDL format, GPT-4's performance remained the same in the one-shot setting (14% or 28/200) but decreased to  $5.5\%$  (11/200) in the zero-shot setting. GPT-3.5 did even worse.
Obfuscating names to test the brittleness of LLM Planning: Although the domain specification is part of our prompts, the names of the objects (e.g. blocks, trucks), predicates (e.g. on-table, in-city) and actions (e.g. pickup, drive) still do provide connections to the commonsense knowledge that the pretrained LLMs possess. One intriguing question is whether the planning performance is based really only on the domain model or these other background connections. To test this, we experimented with a variation of the Blocksworld domain, where we obfuscate the action names (for example pickup becomes attack, and unstack becomes feast) and predicate names (for example ontable becomes planet, and handempty becomes harmony). Note that from the perspective of standard planners, these domains are essentially identical. In addition to such deceptive obfuscation, we also considered a variation where random alphanumeric names were substituted for the action and object names.
Tables 1 and 2, we see that this simple obfuscation leads to a catastrophic drop in performance. Specifically, with zero-shot prompting and natural language input, GPT-4 is able to solve 210 instances out of 600 in the Blocksworld domain, but it could only solve 1 instance in the deceptive Mystery Blocksworld domain and 0 instances in the randomized mystery domain. A similar result is observed with the PDDL-style prompts: GPT-4 could solve 106 instances in Blocksworld, but only 3 instances in the deceptive Mystery Blocksworld. Notably, chain of thought prompting does not significantly improve performance over one-shot natural language prompts. GPT-3.5 does not solve even a single instance in the entire set of natural language instances. For most of the instances, GPT-3.5 outputs that the instance can't be solved. These results strongly suggest that whatever accidental planning performance LLMs show is likely connected to pattern matching rather than reasoning (which should be robust to name change).
<sup>3</sup>Such obfuscated domains were originally introduced into the Intl. Planning Competition by Drew McDermott in 1998—specifically to check if the competition planners were truly domain independent.
Figure 3: Assessment of GPT-4 plans with relaxations in Blocksworld domain
Table 3: Evaluation of GPT-4 and Instruct-GPT3 (I-GPT-3) plans as heuristics for a local search planner LPG, on blocksworld (BW), logistics and mystery blocksworld domains.
Domain	LLM	Avg. Search Steps	Avg. Plan Length	Avg. Lev. Distance
Empty Seed Plan	Random Seed Plan	LLM Seed Plan	Empty Seed Plan	Random Seed Plan	LLM Seed Plan
BW	I-GPT-3	15.8	20.07	14.5	8.45	9.62	11.7	7.22
GPT-4	15.8	20.07	8.9	8.45	9.62	10.76	4.15
Logistics	GPT-4	77.5	144.39	51.3	23.7	32.72	32.24	15.04
Mystery BW	GPT-4	15.8	20.45	16.09	8.45	9.78	11.53	7.77
Analyzing GPT-4 failures: To get a better sense of the type of failures LLM generated plans encounter, we wondered whether they will fare much better with a more forgiving test of the validity of the generated plans. In automated planning community, the notion of relaxations of the domain model are used to simplify the problem—chiefly to derive heuristics for planning problems [8]. Taking a leaf from them, we considered two types of relaxations: (i) delete relaxation involves ignoring all the delete conditions of the domain actions (thus making sure that there can be no negative interactions between subgoals) and (ii) precondition relaxation involves ignoring all the preconditions of the domain actions—thus assuming that the actions are executable from any state giving their effects. Our idea is to evaluate the plans produced by GPT4 with respect to domain models that are delete relaxed, precondition relaxed or both. It should be clear that a plan that is correct with respect to the normal (unrelaxed) model will also be correct with respect to all the relaxed models. Figure 3 shows the results for blocksworld. We see that while the correctness of LLM generated plans increased under more forgiving (relaxed) assessments (area in green), even in the most lenient assessment mode (Delete+Precondition Relaxed), there still are plans ( $\sim 39\%$ ) that are incorrect (because they still don't reach the goals) across all the prompt configurations. The plots further classify the failure cases in terms of whether they were inexecutable, shown in maroon, or could be executed but didn't reach the goals (shown in red). Note that when preconditions are relaxed, all plans are executable. We provide additional details on the relaxed assessments in Appendix A.2.
5 Evaluating LLMs as Idea Generators
While the preceding discussion establishes that LLMs are not capable of generating correct plans in autonomous mode, there is still the possibility that they can be useful idea generators for other sound external planners, verifiers or even humans-in-the-loop. In this section, we investigate this possibility and demonstrate that LLMs show promise on this front (especially with external planners and verifiers).
5.1 LLM Plans as Heuristics to Sound Planners
To see if the LLM generated plans can provide heuristic guidance to sound external planners, we use a local-search planner LPG [6] which generates plans by starting with a seed plan and iteratively repairing flaws until a correct plan is found. We feed the LLM-generated plan as the initial seed plan
for LPG's iterative search. Our hypothesis is that this might put LPG on the right path and reduce the time for it to generate a correct plan. It is interesting to note the similarities between this LLM+LPG approach, and the approaches used in case-based planning in the past [9, 17]. Here the LLM can be loosely viewed as "retrieving a potentially useful plan case/sketch" out of thin air, which the LPG adapts/corrects.
We utilized the plans that were generated by LLMs in the one-shot natural language prompt configuration on all three of our previous domains - Blockworld, Mystery Blockworld, and Logistics - as the "seed plans" from which LPG would begin its local search for a valid plan. For the Blockworld domain, both GPT-4 and Instruct-GPT3 were evaluated, whereas for the Logistics and Mystery domains only GPT-4 was evaluated. We confirmed that all the plans that were generated by this LLM+LPG combination for both the domains were valid (which is as expected given that the underlying planner, LPG, is sound). To get an idea of how far the initial LLM generated plans were from the final correct solutions generated by LPG, we measured the Levenshtein edit distance between them. While the default LPG local search doesn't aim to minimize the changes to the suggested plan (there do exist versions of LPG that do this; see [23]), the edit distances also give an idea of how partially or approximately correct the original LLM plan is. Along with the edit distance, we also measured the number of search steps that were taken by the LPG to come up with a correct plan.
As shown in Table 3, the edit distances across domains are approximately half the length of the seed plans generated by the LLMs, indicating that  $50\%$  of the final plan retains the elements of the initial LLM plan. For each problem, we performed two additional plan initializations to serve as baselines: initializing with an empty plan and initializing with a random plan of the same length as the plan generated by the LLM for that problem. In the Blocksworld and Logistics<sup>4</sup> domains, we see a significant improvement in search steps over the empty seed plan when GPT-4 is used and an even larger one over the random seed plan. Consistent with our findings in the autonomous mode, the usefulness of this assistance wanes in domains where the relationships between predicates can no longer be inferred from common sense understandings of their names: in the Mystery Blocksworld domain, the LLM only has meager reduction in step size over the random plan and actually uses more steps than the empty plan.
5.2 Verifier-assisted repeated backprompting of LLMs
The interaction between LLM and LPG was unidirectional—with LLM sending a seed plan that LPG aims to repair. One supposed advantage of LLMs is that they can be prompted to improve their solutions. Suppose we have access to a sound automated verifier that not only checks the plan correctness but also pinpoints faults (in terms of unsatisfied preconditions or delete interactions). Such feedback can be easily converted into a "backprompt" to the LLM, with the hope that LLM comes up with a better plan. This is what we do with the help of VAL[11]—an AI planning tool that uses the domain model to validate the correctness of the plans (and point out errors).
Table 4: GPT4 Performance with Backprompting by VAL [11]. Mystery BW had deceptive disguising. I.C - Instances correct (within 15 feedbacks); A.F.R - Avg. feedback rounds for correct instances.
Domain	I.C	A.F.R
GPT-4	GPT-4
Blocksworld (BW)	41/50 (82%)	3.68
Logistics	35/50 (70%)	3.31
Mystery BW	5/50 (10%)	7.0
While, as we mentioned earlier, there can be thorny "clever hans" issues about humans prompting LLMs, an automated verifier mechanically backprompting the LLM doesn't suffer from these.
We tested this setup on a subset of the failed instances in the one-shot natural language prompt configuration using GPT-4, given its larger context window. We set a threshold of 15 backprompting rounds. We tested on three domains—Blocksworld, Logistics and Mystery BW—with 50 failed instances from each domain. Table 4 shows the results. We provide the prompt+feedback examples in Appendix A.9. We found that GPT4 is able to come up with correct plans  $82\%$  of the Blocksworld instances and  $70\%$  of the Logistics one. The average number of backprompting rounds for these
For the logistics domain, we considered only those instances for which a plan could be generated in less than 550 search steps (179 instances).
successful cases was 3.68 for BW and 3.31 for Logistics. The performance on the Mystery BW however remained quite poor-suggesting that even with back prompting, GPT4 cannot do well unless it can tease out commonsense patterns for the domain.
In this backprompting configuration, LLM serves as the candidate plan generator while VAL serves as the external sound verifier. While it is tempting to have a self-critiquing architecture with LLM also serving as the verifier, our recent work shows that approach to be of questionable utility as LLMs are no better at verifying plans than they are at generating them [36, 34].
5.3 LLMs as idea generators for humans-in-the-loop
Along with external planners and verifiers, LLMs may also offer their insights as plan suggestions directly to the human-in-the-loop which might potentially guide the user to the correct plan. After all, this sort of computer supported cooperative work (CSCW) use case has been the staple of LLM applications. We explored the efficacy of LLMs in assisting human planners through a between-subjects user study, structured similarly to the study outlined in Section 4, but with two primary distinctions: (1) The study involved two separate participant groups. The first group received no assistance in devising plans, paralleling the approach in Section 4, while the second group had access to LLM-generated suggestions. (2) both participant sets were asked to offer subjective feedback via the NASA-TLX assessment tool [10], gauging their cognitive load. Additionally, participants from the second group evaluated the correctness of the LLM suggestions presented to them. We utilized the plans generated by GPT-4 to provide plan suggestions.
The study included 49 participants in the unassisted group and 48 in the LLM-assisted group. We evaluated the statistical significance regarding accuracy, time taken, and cognitive load between the groups. Our findings revealed no statistical significance between the groups across all three aspects. Notably, 3 out of 48 participants mistakenly accepted incorrect LLM suggestions, with two submitting these erroneous suggestions as their plans. This shows the potential for automation bias in such methodologies [5]. We have provided the details of the user-study in Appendix A.12.
In this paper, we presented a critical investigation of the planning abilities of large language models (LLMs). To this end, we evaluated the plan generation abilities of LLMs in two different modes. In the autonomous mode, our results show that even in simple common-sense planning domains where humans could easily come up with plans, LLMs like GPT-3 exhibit a dismal performance. Even though there is an uptick in the performance by the newer GPT-4 in the blockworld domain, it still fails miserably on the mystery blockworld domain, indicating their inability to reason in an abstract manner. In the heuristic mode, we have seen that plans generated by LLMs can help improve the search of sound planners like LPG. Further, we showed that using external verifiers, we can point out the errors and back-prompt LLMs for a better plan. We showed that this indeed helps in common-sense domains. In the supplementary material, we show the prompt examples for all the configurations and the details of the user-studies (Appendix A.11). From our studies, we see that LLMs as autonomous planners fail miserably, but we also see that the generated plans improve the search when used by an underlying sound planner and that better plans can be obtained by back-prompting the LLM with feedback from an external verifier.
<sup>5</sup>We conducted independent-samples t-tests on plan accuracy, formulation time, and task-related cognitive load between the two groups, with a significance threshold set at  $\alpha = 0.05$ . The t-tests yielded statistic values of -1.21, 0.09, and -0.63, with p-values of 0.22, 0.92, and 0.52 for accuracy, time, and cognitive load, respectively. These results imply that the null hypothesis cannot be rejected for any of the tests, indicating no statistically significant differences in accuracy, time taken, or cognitive load between the two groups.
[37] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
[38] Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05128, 2023.
[39] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In *The Eleventh International Conference on Learning Representations*, 2023.
[40] Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van den Broeck. On the paradox of learning to reason from data. arXiv preprint arXiv:2205.11502, 2022.