Title: Thought of Search: Planning with Language Models Through The Lens of Efficiency
Venue: NeurIPS 2024
Accepted: yes
Keywords: planning

large language models

search
Abstract: Among the most important properties of algorithms investigated in computer science are soundness, completeness, and complexity. These properties, however, are rarely analyzed for the vast collection of recently proposed methods for planning with large language models. In this work, we alleviate this gap. We analyse these properties of using LLMs for planning and highlight that recent trends abandon both soundness and completeness for the sake of inefficiency. We propose a significantly more efficient approach that can, at the same time, maintain both soundness and completeness. We exemplify on four representative search problems, comparing to the LLM-based solutions from the literature that attempt to solve these problems. We show that by using LLMs to produce the code for the search components we can solve the entire datasets with 100% accuracy with only a few calls to the LLM. In contrast, the compared approaches require hundreds of thousands of calls and achieve significantly lower accuracy. We argue for a responsible use of compute resources; urging research community to investigate sound and complete LLM-based approaches that uphold efficiency.

Thought of Search: Planning with Language Models Through The Lens of Efficiency

Abstract

Among the most important properties of algorithms investigated in computer science are soundness, completeness, and complexity. These properties, however, are rarely analyzed for the vast collection of recently proposed methods for planning with large language models. In this work, we alleviate this gap. We analyse these properties of using LLMs for planning and highlight that recent trends abandon both soundness and completeness for the sake of inefficiency. We propose a significantly more efficient approach that can, at the same time, maintain both soundness and completeness. We exemplify on four representative search problems, comparing to the LLM-based solutions from the literature that attempt to solve these problems. We show that by using LLMs to produce the code for the search components we can solve the entire datasets with 100% accuracy with only a few calls to the LLM. In contrast, the compared approaches require hundreds of thousands of calls and achieve significantly lower accuracy. We argue for a responsible use of compute resources; urging research community to investigate sound and complete LLM-based approaches that uphold efficiency.

1 Introduction

Recent work has addressed the issue of planning in Large Language Models (LLMs), spurred by their abilities in natural language tasks. The approaches vary widely from giving a planning problem to an LLM and asking it to output an entire plan to asking an LLM to plan step by step, including backtracking. Roughly, these approaches can be partitioned into two sets. The first exemplifies yet another capability of language models, while the second aims at presenting practical tools for solving planning problems. In the latter case, however, since planning problems are often computationally hard, it is crucial to understand the properties and the complexity of the algorithms proposed.

The purpose of our work is precisely that. Going over a collection of recent methods for planning with large language models, we analyse the most important properties of the proposed methods such as soundness and completeness. We find all these methods to be neither sound nor complete. We further investigate the computational efficiency of these methods in terms of the number of invocations of their most expensive routine - LLM evaluation. From a pragmatic perspective, the cost of LLM evaluations is significant, either in terms of GPU resources, or expensive API calls to hosted LLMs. We find that abandoning the soundness and completeness does not provide any benefit

in computational efficiency, as the proposed methods are prohibitively inefficient, expensive, and most importantly harming the environment.

We propose an alternative named Thought of Search: thinking before searching, an approach that exploits the strengths of LLM, while mitigates the weaknesses of the existing approaches, doing so in an efficient manner. We propose using the language models for deriving the symbolically represented search components that allow for performing the search itself without calling LLMs. That way, the search space correctness can be checked before the search is performed, allowing for soundness and completeness (and sometimes optimality) of the search algorithms imply these properties of the overall solution. Specifically, we focus on the two essential components of any search, successor generator and goal test, using the large language models to obtain their implementation in Python. We exemplify our proposed approach on four representative search problems, comparing to the LLMbased solutions from the literature that attempt to solve these problems and show these approaches to be prohibitively expensive. We show that by using LLMs, possibly with human feedback, to produce the code for the search components we can solve the entire datasets with 100% accuracy with only a few calls to the LLM. We argue for a responsible use of compute resources; urging research community to investigate sound and complete LLM-based approaches that uphold efficiency.

2 Properties and Complexity of Planning using LLMs

In this section, we review the existing methods for planning with language models from the point of view of the soundness and completeness properties of the proposed algorithms, as well as their LM Evaluation Complexity . We quantify the number of LLM requests required by each algorithm. An algorithm is sound if it produces only valid solutions and it is complete if it is guaranteed to produce a solution before terminating successfully. For consistency, we unify the notation here. In what follows, we denote the bound on the number of successors generated per state by b , the number of rollouts by T , and the length of a rollout/path by L .

IO The Input-Output (IO) prompting is the straightforward use of LLMs to generate an output for a given input. Some examples that employ IO for planning include querying a pre-trained model [21] or fine-tune a model [17, 5]. Here, the model is usually evaluated once, generating an output that may include a sequence of steps, making the complexity of this approach O (1) . The method is neither sound nor complete for planning, as it can generate incorrect solutions and not guaranteed to provide a solution if one exists. Further, the methods that fine-tune a model can have a computationally intensive step of data generation.

CoT The Chain-of-Thought approach [22] prompts the model with a predefined example of a chain of steps (thoughts) to resolve the question, in an attempt to make the model generate similar chains in its output. Here, still the model is evaluated once, so the complexity is O (1) and the approach is still neither sound nor complete for planning, for the same reasons as before.

ReAct The ReAct approach [25] aims at interleaving CoT with acting steps. Each acting step may result in a separate evaluation of the language model. Therefore, the number of evaluations is worst case linear in the number of steps L in the expected output, O ( L ) . This approach is akin to re-planning at each step; or treating LLM as a policy, referenced for each state in the trajectory. Still, no guarantees of soundness or completeness can be obtained for this approach.

ReWOO ReWOO [23] aims at tackling the inefficiency in the number of LLM evaluations, requesting the first model evaluation to plan all future steps, then each step is executed without model evaluation if possible (and with, if not), finalizing by a model evaluation with the concatenated output of previous states as the input. The best case complexity is therefore O (1) , with only two evaluations performed and when external tools are not LLM. The worst case complexity is however still O ( L ) , where L is the plan's length, since each of the L external tool calls can be to an LLM. Same as before, no guarantees of soundness or completeness can be obtained.

RAP Reasoning via Planning (RAP) approach performs a search (e.g., MCTS) using the LLM for expansion (generate successors) and for heuristic or reward prediction of a state [7]. Here, the complexity is the worst among the approaches explored so far, being linear in the size of the search

space. MCTS is an incomplete algorithm, where the search tree size can be controlled by bounding the number of successors generated per state b , number of rollouts T , and their depth L . The overall complexity is O ( T × b × L ) . Since the LLM is used for generating successors, it can generate incorrect successors, making the approach also not sound.

ToT Tree of Thoughts [24] approach is similar to that of RAP, where the 'thought generator' is expanding a state by calling an LLM and the state evaluator is calling an LLM to evaluate the generated states. The evaluation can be done per state (one call per state), or across states (one call across all the current generated states). Then, a search is performed on the search tree. The worse case overall complexity is O ( N ) , where N is the number of states generated by the search algorithm. The authors use bounded incomplete versions of the well-known Breadth-First Search (BFS) and Depth-First Search (DFS) algorithms, with the depth bound L , branching bound b , and beam size m , restricting the complexity to O ( b × m × L ) . Here as well, the successor generation by LLM eliminates soundness of the otherwise sound algorithms.

GoT Graph of Thoughts (GoT) [1] is similar to the tree of thought approach, except that it can connect the thought nodes in the tree above into a graph for more flexible representations through (a) aggregation of thoughts a , (b) improvement of a thought to refine it further i , (c) generation of a thought based on existing ones g . In GoT, in addition, repeat operation k defines how often a particular operation is repeated. The complexity is similar to that of the ToT, O ( N ) , where N is the number of nodes in the graph generated, in practice bounded by the branching bound b and the depth bound L , resulting in the overall complexity of O ( b × L ) . Here as well, there are no guarantees of soundness or completeness, as the graph is constructed with the language model.

Reflexion Reflexion [19] is a feedback incorporated approach where in addition to a so-called actor model, which can be either a CoT or a ReAct approach, it has an internal and external feedback component, named an evaluator model and a self-reflection model. The feedback is given within a bounded number of trials and a bounded memory that provides additional context to the agent. The number of evaluations is dependent on which actor model is used plus a constant (two evaluations, one for the evaluator model and one for the self-reflection model) times the number of trials, T . So its either O ( T ) if CoT is used, or O ( L × T ) if ReAct is used. While the authors report 100% accuracy given a large enough T in some domains, the approach is not a complete approach. Similar to the other cases, the approach is also not sound.

LATS Similarly to [7], Language Agent Tree Search (LATS) [28] performs a Monte-Carlo Tree Search (MCTS), using language models as algorithm components. Using our notation of b for the number of successors generated per state, T for the number of rollouts, and L for their depth, the algorithm complexity here as well is O ( T × b × L ) . The authors note that in practice, however, their method produce more calls to the LLM than RAP, exploring larger portions of the search space and producing somewhat higher accuracy solutions. As mentioned before, MCTS is already an incomplete algorithm by itself, and LATS limits it further. Since the successors here as well are generated by a language model, the approach is not sound.

AoT Algorithm of Thoughts (AoT) [18] combines the CoT and the ToT approach in a following sense: rather than a standard CoT prompting, it incorporates the search strategy (DFS or BFS) within the prompt through the in-context examples. That is they use a standard DFS or BFS to produce the in-context examples. The approach is neither sound nor complete, as it relies completely on the language model. Further, it assumes an existence of a search component that can solve the in-context examples. If such component already exists, the only possible reason to use this approach would be if the search component is not efficient, which would imply that the problems are too large to be handled by BFS/DFS (e.g., above 10 12 states). But then, these search spaces would certainly be beyond any imaginable future capabilities of LLMs.

While technically AoT makes one call to the LLM per problem instance, the input and output number of tokens grows exponentially with the instance size. Therefore it would be misleading to say that the complexity of this approach is O (1) . From the cost perspective, when charged based on the number of input and output tokens, it might well be more expensive than the other approaches. Since we do not have an adequate uniform way of transforming this single call to multiple calls of similar size to other approaches, we skip this approach in our analysis.

LLM-Modulo Framework The framework suggests to perform planning by a so-called GenerateTest-Critique loop [10]. The LLM is generating candidate plans, which are validated/critiqued by external means, providing feedback to the LLM. These means are assumed to be sound, thus the overall framework is claimed to inherit soundness guarantee. It is not mentioned, however, how to obtain such means of sound validation or critique for cases where no symbolic model already exists. Once critiques deemed a plan valid, the algorithm stops. Otherwise, next candidate is generated. There is no guarantee of convergence towards a valid plan and therefore the algorithm is not guaranteed to terminate. This can be fixed by employing a bound on the number of iterations. This, however, does not provide completeness guarantees.

Policy generation with LLMs Focusing on problems where no search is required (e.g., polynomial cases) and a general policy (or a generalized plan) exists, the authors propose querying LLMs to provide a policy implementation in python directly [20]. The policy is then checked on a few held out problem instances from the domain, providing a feedback on problems that are not solved by the policy. The number of calls to the LLM is therefore O (1) per domain , and the result can be amortised among all the problem instances of that domain. This approach, while tackling the simpler case of problems where no search is needed, has served as an inspiration for our proposed approach.

3 Proposed Approach

In this work, we argue for an efficient and responsible use of compute resources. First and foremost, this means using the computationally expensive large language models efficiently and appropriately. In the case of solving search problems, as in the focus of this work, we argue against using an LLM at each expansion and evaluation. Such an implementation of search algorithms is inefficient, but equally importantly, it also sacrifices important properties that search algorithms possess, such as soundness and completeness. While completeness is often sacrificed knowingly, in order to limit the number of possible calls to the language model, soundness can be lost unintentionally. If the state successors are generated by a large language model, there is no guarantee that the produced successors are valid . An additional mechanism that validates the produced output would be required to render the algorithm sound. Such a mechanism would be symbolic by nature, since it must guarantee correctness. However, if such mechanism exists, it may be possible to use that mechanism to produce successors, without the need for performing the calls to a large language model at each evaluation. The large language models would be a prime candidate for producing such a mechanism. The mechanism can be a symbolic planning model, e.g., Planning Domain Definition Language (PDDL), from which all the search components can be computed, as is done by classical planners that perform heuristic search [9]. Alternatively, large language models can also directly produce the search components code: a successor function, a goal test, and even a heuristic function [8] or reward. While the former approach has been explored in the literature [6, 16], here we focus on the latter. We propose to use large language models for obtaining a Python implementation of two critical search components, successor functions and goal test . We query the language model for each component separately, using a textual description of the problem at hand. We assume the process to be iterative and user-guided, by a person capable of validating the code obtained, leaving automation of the process for future work. It is worth noting that if the expensive evaluation of large language models is not performed during search, there is no need to artificially restrict the algorithms to their incomplete variants. Still, the code must be validated for soundness, ensuring that all produced successors are correct, as well as completeness, ensuring that all possible immediate successors are produced.

4 Experiments

We exemplify the proposed approach with a variety of popular search problems, demonstrating the use of large language models to obtain the search components code for a representative set of the same search problems that the vast recent work on planning using LLMs used. Our intention is to exemplify the efficiency and accuracy gap created by the unsupervised use of large language models at every search step. For each search problem, we count the number of interactions with the language model to obtain valid 1 successor function and goal test implementations. We then run a standard implementation of either a BFS or a DFS algorithm with these two functions on a set of instances

User : The 24 Game is a mathematical card game in which the objective is to find a way to manipulate four integers so that the end result is 24. The game is played with a list of four numbers, and the player must use all four numbers exactly once, using any combination of addition, subtraction, multiplication, or division, to arrive at the number 24. If a state is defined by a list of numbers (4 or less), how are successor states defined? Provide only the python code that returns a list of successor states for a state.

ChatGPT : Below is the Python code that defines how to generate the successor states for a given state in the 24 Game. A state in the 24 Game is a list of numbers. The successor states are generated by taking two numbers from the state, applying an operation (addition, subtraction, multiplication, or division if the divisor isn't zero), and then forming a new list with the result and the remaining unused numbers.

import itertools

This function takes a state (a list of numbers), iterates over all pairs of numbers, applies the operations to these pairs, and then forms new lists with the result of the operation and the other numbers that were not used in the operation. This ensures that each successor state is a valid step towards finding a solution to reach 24, using all four numbers exactly once along with any of the allowed operations.

User : Provide only the python code that tests whether a state is a goal state.

ChatGPT : Here's the Python code that tests whether a given state is a goal state in the 24 Game. A goal state is achieved if the list contains exactly one number, and that number is 24.

This function checks if the length of the state list is 1 and whether the sole element in the list is the number 24.

and report the accuracy and the total time to solve the instances. We repeat the experiment 5 times, obtaining 5 implementation variants and compare the efficiency of these variants in terms of total time. Note that if the implementation of successor function and goal test obtained from the large language model is correct, the accuracy is guaranteed to be 100% (if sufficient resources are given to the search). The opposite is not necessarily true, as the set of tested instances might not include some corner cases that make the code perform incorrectly. However, we see that as one of the strengths of our approach, as such cases can be used for testing and debugging the obtained code. Under the assumption of correctness of the successor function and goal test implementations, our approach inherits the soundness and completeness guarantees of the search algorithms used [4]. In all our experiments we use the GPT-4 model [14], in a chat mode. The code obtained from GPT-4 in all 5 correspondences is provided in the appendix. The search with the obtained code was run locally, on a machine with 2.3 GHz 8-Core Intel Core i9 CPU, no GPUs were used.

24 game The 24 game [24] takes 4 integers as an input that can be manipulated through the four most common arithmetic operations: addition, subtraction, multiplication, and division. The goal of the game is to produce a formula (also representable by a sequence of operations) that evaluates to 24, if one exists. As mentioned before, [24] achieve that with an incomplete algorithm that performs ≈ 100 calls to the LLM, according to their report. The authors report a success rate of 75% on a subset of 100 medium difficulty problems out of a collection of 1362 problems [24].

As mentioned before, we propose instead to use the LLM to produce the code for two functions: successor function and goal test . We leave the heuristic function to future work, noting that even without it, the search space of the 24 game is rather small, up to 1 + 36 + 648 + 3888 = 4573 states. Listing 1 describes an example correspondence with the LLM. We observe that the model required 1.2 interactions on average for the successor function and 1 interaction for the goal test. We see

that, with a minor help from the user, the model is able to produce correct successor and goal test functions for this problem. We further tested all 5 successor functions and goal test within a simple BFS implementation on the entire suite of 1362 instances from ToT [24]. For each of the 5 tested successor functions, BFS found a solution in the 1361 cases where a solution exists and report that no solution exists in the only one unsolvable case, a 100% success rate. The total time to solve all 1362 problems varies over these 5 cases from 1.92s to 6.83s in our naive BFS implementation, hinting that some successor functions can be more efficient than other. This is comparable to a single LLM evaluation time; which is ≈ 7 s for GPT-4 Chat [15]. Note that the generated successor functions are generic enough to be able to solve the generalized version of the 24game, Countdown [5], with only minimal adaptation to the goal test.

Mini crosswords The mini crosswords [24] is a 5x5 crosswords dataset that includes 20 games, where the input describes the 5 horizontal and 5 vertical clues and the output is the full 25 letters board. We used GPT-4 to produce the two functions: successor function and the goal test and repeated the experiment 5 times. The correspondence with the model can be found in the appendix. We observe that the model required 2.4 interactions on average to produce a valid successor function, and 1.4 interactions on average to produce the goal test, with errors primarily related to not handling the corner cases of incorrect input. In all cases, after providing the exact error to the model, it added safeguards which fixed the issues. We tested the obtained functions within a standard implementation of a DFS with a closed list on the entire suite of 20 games [24]. As our focus in this work is on the search aspects, we assume that each clue in each crossword has 10 variants of possible answers, including the correct one. All 20 games were solved by all five generated pairs of implementation of successor function and goal test (100% accuracy), with a total time for all 20 games varying from 5.5s to 346s, yet again signifying the importance of efficient implementation of the successor function.

To compare to the ToT approach, with the bound of 100 on state expansions, it calls the LLM ≈ 200 times in the worst case for each of the 20 games. To be fair to the ToT approach, it does not assume the availability of possible answers, deriving the candidates, at each step, with the help of the language model. This, however is a mixed blessing, since the language model does not provide a guarantee of including the correct answer among produced variants.

BlocksWorld Probably the most famous planning domain is BlocksWorld, where the blocks can be picked up from the table, put down on the table, unstacked from other blocks or stacked on other blocks to transform the initial configuration to a given goal configuration. The domain has a known planning model, described in PDDL and it is one of the tasks considered by the reasoning-as-planning approach [7]. As in the previous cases, we use the GPT-4 model to obtain the successor function and the goal test implementation in Python. We use the same textual description of the domain as [7]. A correct successor function and goal test are obtained after 2.8 and 1 iterations on average, respectively.

The mistakes GPT-4 makes when producing the code repeat from one experiment to another, and are often easy to fix inline, without additional iterations (e.g., using shallow copy when deep copy is needed). In our experiments, however, we did count these as additional iterations. In order to evaluate the obtained functions, we used them within a standard implementation of a BFS and experimented with the collection of 502 instances from [7]. All 502 tasks were solved by all five generated pairs of successor function and goal test (100% accuracy), with the total time for all 502 tasks varying from 0.56s to 9.7s. The more time efficient approaches represented a state as a set of strings representing boolean variables (or logical predicates), while the less efficient representation used dictionaries of lists of objects or pairs of objects with predicates as keys. The simpler state representation also resulted in a more efficient duplicate detection in our rather naive implementation.

Note that the accumulated number of expanded 2 and generated states in the most efficient case (with duplicate detection) was 50143 and 129408, respectively. If we needed to call the GPT-4 model on each expansion and generation, by the most conservative estimation, it would take approximately 14 days and cost over $1000 , while not guaranteeing correctness of the outcome. Note that the instances considered are quite small - they have only 4 to 5 blocks. Larger BlocksWorld instances would require significantly more resources. This is true for the proposed approach as well, and larger instances mean larger state spaces, becoming too large for an uninformed search such as BFS. In such cases, a heuristic function may be of help, either, similarly, implemented in Python or automatically derived

User : Logical deduction is performed by applying rules of the form if X then Y to our current knowledge about a world, extending our knowledge with each rule application. A rule if X then Y is applicable if we know X. If a state is defined as our current knowledge of the world, how are successor states defined, given a collection of rules? Provide only the Python implementation of successor states function.

ChatGPT

:

from a PDDL representation, which in turn may be obtained with the help of a large language model [16]. It is also worth noting that BFS not only guarantees that the obtained solution is correct, but also guarantees that it is optimal, while LLMs can never provide such a guarantee - checking whether a given solution is optimal is as hard as finding an optimal solution.

PrOntoQA Logical reasoning can be viewed as a search problem of finding a sequence of logical rules that when applied to the known facts, derive or disprove the target hypothesis. Previous work applies Monte-Carlo Tree Search (MCTS) with successor function and rewards obtained by calling an LLM, to examples from the PrOntoQA dataset to derive the answer but also the proof, a sequence of reasoning steps [7]. The authors report performing 20 iterations for MCTS and 20 samples for self-consistency, resulting in 94.2% correct answer rate and 78.8% proof accuracy.

Similarly to the previous cases, we have generated the successor function and the goal test with the help of GPT-4 model and obtained the answer and the proof by running BFS at most twice per question: once trying to prove the positive hypothesis and, if not successful, once more trying to prove the negation of the hypothesis. The search, if successful, returns a path from the initial state to a found goal state, which corresponds precisely to the sequence of applications of the reasoning rules - the proof. We performed the experiment 5 times, with the language model being able to produce a correct successor function and goal test after 1.6 and 1 iterations on average, respectively. We tested the 5 obtained pairs of functions on the entire collection of 4000 questions generated by [7]. All 4000 questions were answered correctly, with all generated proofs guaranteed to be valid, resulting in

100% accuracy for both, with the total time for all 4000 questions varying between 2.16s and 2.53s. Listing 2 shows an example correspondence with the model, with non-code text removed for brevity.

5 Discussion

It is hard to overstate the importance of the ability to solve search problems and it is natural to solve these problems by exploring some portion of their state space. All the methods we discuss here do that in one way or another. The difference between those approaches is in how big is the portion of the explored state space and what is the cost of exploring that portion. As all the discussed approaches are greedy, unsound, and incomplete, the accuracy of the solutions they provide can only be evaluated experimentally for particular problem and a dataset, and the results do not reflect on other problems or even datasets within the same problem.

In this section we perform a thought experiment. We ignore the accuracy of the approaches and only compare their estimated cost and the portion of the search space they have the potential to explore. For each approach we estimate two values. First, we estimate the number of calls to a language model performed if the approach was run on the four datasets from the previous section, as a proxy for the cost of using the approach. Second, we estimate the number of states explored during search, as a proxy for a portion of the state space size. The actual state space size for these datasets can be calculated precisely, and we provide these values. Note that the number of states explored is an over-estimation for the portion of the state space explored, since the same state can be explored several times. This is true for sound and complete search algorithms as well. These algorithms often employ a duplicate detection mechanism to prevent re-evaluating and re-expanding such states, when such re-expansions are not needed to guarantee algorithm properties (e.g., optimality when admissible but inconsistent heuristics are used). We also provide the actual number of calls to the language model and the number of states explored using our proposed approach. The actual number of calls for our proposed approach is the number of calls that was sufficient for obtaining soundness and completeness (and sometimes optimality). The number of states explored indicated an upper bound on the portion of the states space necessary for exploration.

Datasets size We use the same four representative search problem datasets from previous section. The size of each dataset is denoted by D . For the 24 game, D is 1362 , for Crossword it is 20 , for BlocksWorld it is 502 , while for PrOntoQA it is 4000 . We used the datasets provided by the approaches we compare to, but these numbers could also have been significantly larger. For these datasets, we have also computed the size of the search problems in terms of the summed number of states over the instances in the dataset. For the 24 game, the number of states per instance is 4 , 573 and therefore the summed number of states is 6 , 228 , 426 . For Crossword, the number of states is ∑ 10 i =0 ( n i ) 10 i = (10 + 1) 10 per puzzle, and therefore the summed number of states is 518 , 748 , 492 , 020 . For BlocksWorld, our dataset includes 447 instances with 4 blocks and 55 instances with 5 blocks. The instances with 4 blocks have 125 states each and the instances with 5 blocks have 866 states each, giving us the total of 103 , 505 states. Finally, for PrOntoQA, the number of states per task varies between 12 and 54 , with the total summed number being 97 , 608 .

Uniform restriction of the search space Each approach sets limitations to restrict the number calls to the language model, which makes it difficult to compare. In this thought experiment, we choose to use only the bounds on the breadth/depth of the search and the number of trials, ignoring the absolute constant bounds on the number of explored states used by some approaches. For a fair comparison across the approaches, we will use the same parameter value for all, even if the original work used a different value. We aimed at smaller values among used across the different approaches for each parameter. This results in under-approximation of the number of calls in most cases, and possibly would have resulted in an accuracy that is lower than the one reported by the approaches. In this experiment, however, we ignore the accuracy of the approaches. We use the branching bound b = 5 , the number of trials T = 10 , and the beam size m = 5 . The only parameter we set according to the dataset is L , the bound on the sequence length. This is due to the fact that if L is not sufficiently large, the algorithms will not be able to find a solution. In the 24 game L = 3 is sufficient, while in Mini crosswords it must be at least 10 . In both BlocksWorld and PrOntoQA the length of a plan/proof varies, with the maximal length over the instances in the datasets in BlocksWorld being 16 and in PrOntoQA being 6 .

Analysis The complexity analysis performed in the previous section does not tell the whole story. It hides the constant multiplier, which in some cases could be significant. We also need an additional information about the number of states traversed. Let us take another look at the explored approaches.

Both IO and CoT only call a language model once per instance. Assuming that the output is a plan, it traverses L states. ReAct and ReWOO (worst case) make exactly L calls to the LLM, exploring L states. Both RAP and LATS perform MCTS, but the actual number of calls to the LLM varies. RAP performs LT + bLT calls (see Algorithm 1 in [7]), while LATS performs 2 LT + bLT calls (see Algorithm 1 in [28]). The number of states explored in both cases is bLT . Reflection with ReAct performs (2 + L ) T calls, exploring LT states. ToT introduced their own variants of well-known BFS and DFS algorithms with bounds on the branching factor, search depth, but also a limit on the open list size (beam size). While their performance can vary greatly on individual tasks, with DFS typically being greedier and therefore faster, in the worst case they both call the LLM bmL times and explore bmL states. The GoT approach calls the LLM bL times, exploring bL states.

Table 1 shows the estimated numbers according to the parameters values and the analysis above. For comparison, the last row of the table depicts our proposed approach. %States columns denotes the search effort in terms of the number of states explored, possibly with repetitions, described as a percentage of the overall number of states. For our approach, this is the actual value from our experiments. Recall, in the case of PrOntoQA the BFS search is performed once or twice, until the hypothesis or its opposite is proven. For the other approaches, this is an estimate, under the assumptions that the search does not explore the same states multiple times. Clearly, this assumption does not hold in practice, with the methods exploring the same state many times. This is also true for a sound and complete search, albeit to a lower extent, it can generate the same state multiple times. An important property of a sound and complete search is that it generates states systematically , and expands them at most once. The number of times the same state is generated tends to be higher in MCTS than in BFS/DFS. Non-systematic successor generation is also a major contributor to that inefficiency. Having in mind that these numbers are very crude over-approximations, we observe that the investigated approaches explore only a very small portion of the search space. This is one of the major sources of their low accuracy - if the solution is not completely included in that explored portion, then there is no way for that solution to be found. In 24 game and crossword, the portion explored tends to be very low, one or two orders of magnitude smaller compared to our method. In BlocksWorld it is comparable to our method, and in PrOntoQA it is often much higher, order of magnitude larger than the size of the state space, indicating that the same states are explored over and over again.

Looking at the overall number of calls, some methods are more expensive than other. Assuming an average of 500 tokens per input 3 and 50 tokens per output, according to the current pricing of

GPT-4-turbo (the cheaper of GPT-4 models), the overall cost varies from $40 for IO/CoT and $200 for ReAct/ReWOO, to $14 , 000 for RAP and $16 , 000 for LATS.

6 Related work

As the related work on the problem of planning with language models is detailed in Section 2, here we review the work related to our proposed method.

Generation of planning representation Our work proposes to use LLM to generate python code for successor and goal functions. By using these two functions, along with a search algorithm, one can thoroughly explore the entire state space of a problem and compute a plan. Recently, LLMs have also been used to generate PDDL [6, 16, 11, 30], which can be used with existing planners to search for a plan. Of these work, Guan et al. [6] is quite closely connected to our work as they also use corrective human feedback to generate the PDDL descriptions. While related, our work differs from theirs in three major ways. First, they assume the goal state is provided in the PDDL form. Our work does not make this assumption; rather asks LLM to generate a goal function. Second, in their approach action descriptions are obtained one by one (separate calls to LLMs) along with predicates to use. We provide entire description of the problem and asks for a single successor function. Third, their feedback on the generated PDDL is explicit and requires nuance understanding of modeling in PDDL (e.g., 'unnecessary precondition "no other object stacked on object ?x"'). While ours is mostly generic python code feedback (e.g., issues with shallow copy of a dictionary with list values, error trace) and generic logic (e.g., 'two of the operations are not symmetric, division and subtraction').

Code generation Our work proposes to use LLMs to generate code that helps solve the search problem, instead of using LLMs to solve the search problem directly. Similar idea has been explored by Zhou et al [29] for mathematical reasoning problems. In that, they illustrate that LLMs can be used to generate, execute (verify), and self-refine the python code that helps solve the math problem. Their study corroborate our findings that LLMs can be used for generating verifiable code with some feedback. Additional evidence supporting the feasibility of our approach of using LLMs to generate code comes from the recent advancements in LLM-based program synthesis literature [13, 27, 3, 26, 12].

7 Conclusions and Future Work

The current trends in planning with large language models focus on performing a search when the search components are realized through the large language models. We analyze the existing approaches and show them to be unsound, incomplete, and quite expensive. We propose to use the large language models to instead generate a code for these search components, to be reused throughout the entire dataset. The significant differences in the use of computational resources and the performance measures clearly demonstrate that LLMs can be effectively employed for planning purposes without compromising on soundness, completeness, efficiency or accuracy.

For future work, we would like to explore the use of large language models to obtain the code for search guidance or search pruning techniques. But more importantly, we would like to relax the need for human feedback in coming up with valid implementations of the search components. This could be done in ways similar to the way a policy is obtained in generalized planning [20]. The first attempt at automating the process was recently made with AutoToS [2], which replaced the human experts in the loop of feedbacking the language model with an automated feedback, based instead on human-generated unit tests. These unit tests attempt to find evidence of non-soundness and incompleteness and may in turn be automated in the future.

References

Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models. In Jennifer Dy and Sriraam Natarajan, editors, Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI 2024) , pages 17682-17690. AAAI Press, 2024.

Daniel Cao, Michael Katz, Harsha Kokel, Kavitha Srinivas, and Shirin Sohrabi. Automating Thought of Search: A journey towards soundness and completeness. In NeurIPS 2024 Workshop on Open-World Agents , 2024.

Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In ICLR . OpenReview.net, 2024.

Thomas H. Cormen, Charles E. Leiserson, and Ronald L. Rivest. Introduction to Algorithms . The MIT Press, 1990.

Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D. Goodman. Stream of Search (SoS): Learning to search in language. arXiv:2404.03683 [cs.LG], 2024.

Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. In Proceedings of the Thirty-Seventh Annual Conference on Neural Information Processing Systems (NeurIPS 2023) , 2023.

Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023) , 2023.

Peter E. Hart et al. A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics , 4(2):100-107, 1968.

Malte Helmert. The Fast Downward planning system. Journal of Artificial Intelligence Research , 26:191246, 2006.

Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Kaya Stechly, Mudit Verma, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. LLMs can't plan, but can help planning in LLM-Modulo frameworks. CoRR , abs/2402.01817, 2024.

Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. LLM+P: empowering large language models with optimal planning proficiency. CoRR , abs/2304.11477, 2023.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In NeurIPS , 2023.

Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. In ICLR . OpenReview.net, 2024.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, et al. GPT-4 technical report, 2024.

OpenAI Dev. Forum. Performance analysis of Assistants versus Chat completion. https://community. openai.com/t/performance-analysis-of-assistants-versus-chat-completion-chat-co mpletion-seems-somewhat-faster-for-complete-message-generation-streaming-taken -into-account/628368 , Feb 2024.

James Oswald, Kavitha Srinivas, Harsha Kokel, Junkyu Lee, Michael Katz, and Shirin Sohrabi. Large language models as planning domain generators. In Sara Bernardini and Christian Muise, editors, Proceedings of the Thirty-Fourth International Conference on Automated Planning and Scheduling (ICAPS 2024) . AAAI Press, 2024.

Vishal Pallagani, Bharath Muppasani, Keerthiram Murugesan, Francesca Rossi, Lior Horesh, Biplav Srivastava, Francesco Fabiano, and Andrea Loreggia. Plansformer: Generating symbolic plans using transformers. arXiv:2212.08681 [cs.AI], 2022.

Bilgehan Sel, Ahmad Al-Tawaha, Vanshaj Khattar, Lu Wang, Ruoxi Jia, and Ming Jin. Algorithm of thoughts: Enhancing exploration of ideas in large language models. CoRR , abs/2308.10379, 2023.

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Proceedings of the Thirty-Seventh Annual Conference on Neural Information Processing Systems (NeurIPS 2023) , 2023.

Tom Silver, Soham Dan, Kavitha Srinivas, Josh Tenenbaum, Leslie Pack Kaelbling, and Michael Katz. Generalized planning in PDDL domains with pretrained large language models. In Jennifer Dy and Sriraam Natarajan, editors, Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI 2024) . AAAI Press, 2024.

Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models - A critical investigation. In Proceedings of the Thirty-Seventh Annual Conference on Neural Information Processing Systems (NeurIPS 2023) , 2023.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the Thirty-Sixth Annual Conference on Neural Information Processing Systems (NeurIPS 2022) , pages 24824-24837, 2022.

Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, and Dongkuan Xu. ReWOO: Decoupling reasoning from observations for efficient augmented language models. arXiv:2305.18323 [cs.CL], 2023.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Proceedings of the Thirty-Seventh Annual Conference on Neural Information Processing Systems (NeurIPS 2023) , 2023.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR 2023) . OpenReview.net, 2023.

Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. Self-edit: Fault-aware code editor for code generation. In ACL (1) , pages 769-787. Association for Computational Linguistics, 2023.

Li Zhong, Zilong Wang, and Jingbo Shang. Debug like a human: A large language model debugger via verifying runtime execution step by step. In ACL (Findings) , pages 851-870. Association for Computational Linguistics, 2024.

Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. CoRR , abs/2310.04406, 2023.

Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and Hongsheng Li. Solving challenging math word problems using GPT-4 code interpreter with code-based self-verification. In ICLR . OpenReview.net, 2024.

Max Zuo, Francisco Piedrahita Velez, Xiaochen Li, Michael L. Littman, and Stephen H. Bach. Planetarium: A rigorous benchmark for translating text to structured planning languages. CoRR , abs/2407.03321, 2024.

A Search implementation

The search components successor_states and goal_test are plugged into the search algorithms. In this work we used BFS and DFS algorithms, implemented as follows.

B Model correspondences

B.1 24 game

The prompt for the successor function is as follows.

The 24 Game is a mathematical card game in which the objective is to find a way to manipulate four integers so that the end result is 24. The game is played with a list of four numbers, and the player must use all four numbers exactly once, using any combination of addition, subtraction, multiplication, or division, to arrive at the number 24. If a state is defined by a list of numbers (4 or less), how are successor states defined? Provide only the python code that returns a list of successor states for a state.

The prompt for the goal test is as follows. Provide only the python code that tests whether a state is a goal state.

B.1.1 Experiment Run 1.

User : The 24 Game is a mathematical card game in which the objective is to find a way to manipulate four integers so that the end result is 24. The game is played with a list of four numbers, and the player must use all four numbers exactly once, using any combination of addition, subtraction, multiplication, or division, to arrive at the number 24. If a state is defined by a list of numbers (4 or less), how are successor states defined? Provide only the python code that returns a list of successor states for a state.

ChatGPT :

User : Provide only the python code that tests whether a state is a goal state.

ChatGPT :

B.1.2 Experiment Run 2.

User : The 24 Game is a mathematical card game in which the objective is to find a way to manipulate four integers so that the end result is 24. The game is played with a list of four numbers, and the player must use all four numbers exactly once, using any combination of addition, subtraction, multiplication, or division, to arrive

at the number 24. If a state is defined by a list of numbers (4 or less), how are successor states defined? Provide only the python code that returns a list of successor states for a state.

ChatGPT :

User : Provide only the python code that tests whether a state is a goal state.

ChatGPT :

User : In the generate_successor_states function, two of the operations are not symmetric, division and subtraction. Please fix the code

ChatGPT : Here's the updated Python code for generating successor states that considers the non-symmetry of subtraction and division operations:

This code correctly handles the non-commutativity of subtraction and division by explicitly computing the result for both a ◦ b and b ◦ a whenever the operator is '-' or '/'.

B.1.3 Experiment Run 3.

User : The 24 Game is a mathematical card game in which the objective is to find a way to manipulate four integers so that the end result is 24. The game is played with a list of four numbers, and the player must use all four numbers exactly once, using any combination of addition, subtraction, multiplication, or division, to arrive at the number 24. If a state is defined by a list of numbers (4 or less), how are successor states defined? Provide only the python code that returns a list of successor states for a state.

ChatGPT : Below is the Python code that defines how to generate the successor states for a given state in the 24 Game. A state in the 24 Game is a list of numbers. The successor states are generated by taking two numbers from the state, applying an operation (addition, subtraction, multiplication, or division if the divisor isn't zero), and then forming a new list with the result and the remaining unused numbers.

This function takes a state (a list of numbers), iterates over all pairs of numbers, applies the operations to these pairs, and then forms new lists with the result of the operation and the other numbers that were not used in the operation. This ensures that each successor state is a valid step towards finding a solution to reach 24, using all four numbers exactly once along with any of the allowed operations.

User : Provide only the python code that tests whether a state is a goal state.

ChatGPT : Here's the Python code that tests whether a given state is a goal state in the 24 Game. A goal state is achieved if the list contains exactly one number, and that number is 24.

This function checks if the length of the state list is 1 and whether the sole element in the list is the number 24.

B.1.4 Experiment Run 4.

User : The 24 Game is a mathematical card game in which the objective is to find a way to manipulate four integers so that the end result is 24. The game is played with a list of four numbers, and the player must use all four numbers exactly once, using any combination of addition, subtraction, multiplication, or division, to arrive at the number 24. If a state is defined by a list of numbers (4 or less), how are successor states defined? Provide only the python code that returns a list of successor states for a state.

ChatGPT : In the 24 Game, the concept of a "state" can be represented by a list of numbers, with the game beginning with four numbers. The "successor states" are generated by applying a binary operation (addition, subtraction, multiplication, division) to any pair of numbers in the current state, then replacing these two numbers with the result of the operation, which gives a new state with one fewer number. Here's Python code that defines a function to generate all possible successor states from a given state:

This code leverages combinations from the itertools module to select each pair of numbers from the current state, and then applies every possible operation to these numbers. The apply_operation function computes possible outcomes for the operations between two numbers, including division only when the divisor is not zero. Each result is used to create a new state, which is added to the list of successor states.

User : Provide only the python code that tests whether a state is a goal state.

ChatGPT : Here's Python code that defines a function to check whether a given state is a goal state in the 24 Game. The goal state is when the list contains only the number 24.

This function returns True if the list state contains exactly one element and that element is 24, which would indicate that the goal of achieving 24 has been successfully reached.

B.1.5 Experiment Run 5.

User : The 24 Game is a mathematical card game in which the objective is to find a way to manipulate four integers so that the end result is 24. The game is played with a list of four numbers, and the player must use all four numbers exactly once, using any combination of addition, subtraction, multiplication, or division, to arrive at the number 24. If a state is defined by a list of numbers (4 or less), how are successor states defined? Provide only the python code that returns a list of successor states for a state.

ChatGPT : To generate successor states for a given state in the 24 Game, we need to consider pairs of numbers from the current state and all possible operations between them. After selecting a pair of numbers and an operation, we replace the pair in the current state with the result of the operation to create a new state. This process is repeated for every possible pair and operation.

Here's a Python function that returns all successor states for a given state:

This function works as follows:

It iterates over all combinations of two indices from the state.

For each pair, it applies all possible arithmetic operations.

It forms new states by replacing the chosen pair with the result of the operation and includes all other numbers from the original state.

It collects and returns all these new states as potential successors.

User : Provide only the python code that tests whether a state is a goal state.

ChatGPT : Here's a Python function to check if a given state is a goal state in the 24 Game. The goal is achieved if the state contains exactly one number which is 24:

This function checks whether the list state has only one element and that element is 24, indicating that the operations on the initial numbers have successfully resulted in 24.

B.2 Mini crossword

The prompt for the successor function is as follows.

The task is a 5x5 mini crosswords. Given an input of possible answers to horizontal clues and vertical clues, come up with a successor state function written in python, where a state is a 2D array representing the current puzzle state (initially all empty).

The prompt for the goal test is as follows.

Can you come up with a goal test function ensuring that all cells are not None and that all words horizontally and vertically match at least one of the answers?

B.2.1 Experiment Run 1.

User : The task is a 5x5 mini crosswords. Given an input of possible answers to horizontal clues and vertical clues, come up with a successor state function written in python, where a state is a 2D array representing the current puzzle state (initially all None). The possible clues for each row and each column are given separately. Here is an example:

ChatGPT : To create a successor state function for a 5x5 mini crossword puzzle in Python, we'll consider the following:

State Representation: A state will be represented by a 5x5 grid, where each cell can either contain a letter or be empty (represented by None or an empty string).

Input: The inputs to the function will be a list of possible answers for horizontal and vertical clues. Each answer is exactly 5 characters long since it's a 5x5 grid.

Successor State Function: This function generates all possible successor states from the current state. For each empty row or column, the function will try to place every suitable word from the clues into the grid and generate a new state for each valid placement.

Here is a Python implementation of the successor state function for the 5x5 mini crossword puzzle:

is_valid_placement: Checks if the word can be placed on the grid without conflicting with existing characters.

place_word: Places the word on the grid and returns the new grid state.

successor_states: Iterates over each row and column to find empty spaces and attempts to place each clue word, generating new successor states.

This function will give you all the potential successor states by attempting to fill each empty row and column with every possible word from the clues. Note that this example assumes the initial state has no conflicting

pre-filled characters and does not handle partial word placements or overlaps, which could be extensions to this basic successor function.

User : Can you come up with a goal test function Come up with a goal test function ensuring that all cells are not None and that all words horizontally and vertically match at least one of the answers?

ChatGPT : Certainly! To create a goal test function for the 5x5 mini crossword puzzle, we need to ensure that:

No Empty Cells: There should be no cell in the grid that contains None.

Valid Words Horizontally: Each row must contain a word that is present in the list of possible horizontal answers for that specific row.

Valid Words Vertically: Each column must contain a word that is present in the list of possible vertical answers for that specific column.

Here is how you could implement such a function in Python:

Function Explanation:

The goal_test function iterates over all rows and columns. For rows, it concatenates the elements to form a word and checks against the list of possible answers for that row. It repeats a similar process for columns. If any word doesn't match the possible answers or if there's any None value in the grid, the function returns False.

If all checks pass, it returns True, indicating that the grid is in a goal state.

User :

For the successor state generation I am getting the following error:

ChatGPT : The error message "IndexError: string index out of range" typically indicates an attempt to access an index that does not exist in a string. This issue often arises when working with grid-related operations and is usually due to one of two reasons in your crossword context:

Mismatch in Word Length: Since your grid cells are designed for 5x5 crosswords, every word used as a clue should be exactly 5 characters long. If any word is shorter or longer than this, you'll encounter this error when you try to place it in the grid.

Out-of-Bounds Access: This happens if the code attempts to place a word in a position where it doesn't fit entirely within the grid's boundaries, although based on your function setup (starting placements at row 0 and column 0 for horizontal and vertical placements respectively) and the checks you seem to have intended, this shouldn't normally occur unless the word length is incorrect.

How to Fix:

Verify Word Lengths: Ensure that all words provided in the horizontal_answers and vertical_answers lists are exactly 5 characters long. You can add a validation step or filter out any incorrect entries.

Debugging Additions: Include checks in your is_valid_placement function to ensure you don't run into out-of-bounds errors. This is more of a safeguard than a solution to the current issue but is good practice.

Here's how you might modify your code to include these fixes:

Verify the Data: Before running your grid operations, it's beneficial to confirm that your input data is as expected:

This code snippet ensures that every word in both the horizontal and vertical clues lists is exactly 5 characters long. If this assertion fails, you will need to revise your input data to meet this requirement. This helps prevent runtime errors related to string length mismatches in a fixed-size grid context like this.

B.2.2 Experiment Run 2.

User : The task is a 5x5 mini crosswords. Given an input of possible answers to horizontal clues and vertical clues, come up with a successor state function written in python, where a state is a 2D array representing the current puzzle state (initially all None). The possible clues for each row and each column are given separately. Here is an example:

Note that some of the possible answers are not exactly 5 character long.

ChatGPT : To approach the creation of a successor state function for this 5x5 mini crossword problem in Python, let's consider a few key aspects:

State Representation: We'll represent the state of the puzzle as a 2D list of characters. Each cell in the list can either be a specific character or empty (denoted by a space or some placeholder).

Initial State: The initial state will have all cells empty.

Successor State Function: This function will generate all possible states that can be reached from the current state by making a single move. A "move" in this context would involve placing a word in a row or column that fits with the already placed characters.

Constraints:

Words must match the allowed words in horizontal_answers for rows and vertical_answers for columns.

Any placed word must not conflict with characters already placed in intersecting rows or columns.

Given the constraints and specifications, we'll proceed as follows:

Successor State Function: For a given state, iterate over each row and column, trying to place each possible word from the respective lists. Generate a new state for each valid placement.

Here's a simplified Python implementation of this successor state function:

This code assumes the answers are all 5 characters long to match the 5x5 grid exactly. For words that are not 5 characters, additional preprocessing would be necessary to either filter them out or adjust the grid size accordingly. Also, this code only illustrates the initial placement of words without handling intersections; further logic is needed to ensure words do not contradict each other at intersecting cells.

User : Can you write a goal test as well?

ChatGPT : Certainly! A goal test function for the 5x5 mini crossword puzzle would determine if a complete and valid crossword has been formed. Here are the criteria for a complete and valid crossword:

Full Grid: All cells in the grid must be filled with valid letters (no empty spaces allowed).

Valid Words: All words formed in rows and columns must match one of the possible answers given for that specific row or column.

Let's write a Python function that checks if a given state meets these criteria. This goal test function will:

Verify that all grid cells are filled.

Check that every word in each row is in the respective list from horizontal_answers.

Check that every word in each column is in the respective list from vertical_answers.

Here is the implementation:

This goal_test function relies on final_state being fully filled and accurately matching the corresponding answer lists. This function can be used in a puzzle-solving algorithm to check if the current state of the board is the solution.

User : I am getting an error

ChatGPT : The error you're experiencing suggests that there's a mismatch between the indices being accessed and the size of the grid or word. This usually happens if the grid is initialized incorrectly, or if the word doesn't fit exactly into the intended row or column. Since the code assumes each word perfectly fits a 5x5 grid (5 letters per word), any deviation from this can cause an index out of range error.

Let's address this by ensuring that:

The grid and the words are correctly aligned in size.

The placement functions are handling indexing safely.

I'll revise the place_word function to ensure it correctly handles the dimensions of both rows and columns:

This version of place_word adds checks to ensure that words are only placed if they match the expected length, preventing indexing errors.

Additionally, ensure that the initial grid is correctly populated with None values in each cell:

If you are still encountering issues, please double-check that:

All words used in horizontal_answers and vertical_answers are exactly 5 characters long.

The grid dimensions are strictly 5x5.

If you need to handle words that aren't 5 characters long, you'd need additional logic to either skip those words or adjust the grid accordingly. If you're working with dynamic sizes or irregular word lengths, the implementation would require more complex adjustments to manage these variations.

User : I am now getting this error:

ChatGPT : The error "IndexError: string index out of range" in the is_valid_word_placement function suggests that there's an attempt to access an index in the word that is beyond its length. This can occur if:

The length of the word being checked does not match the expected column height (which should be 5 for your 5x5 grid).

The column index i exceeds the length of the word.

To resolve this issue, we should verify that:

All words used for placement in rows and columns are exactly 5 characters long (as your grid is a 5x5 grid).

The checks within is_valid_word_placement are safely accessing characters in words.

Here's an updated version of the is_valid_word_placement function with added safety checks:

This modified function includes a check to ensure the word length matches the grid dimension (5 for your 5x5 grid) before proceeding with any further logic. This should help prevent the "IndexError" you encountered by immediately returning False if the word length is incorrect.

Please ensure that:

All words in horizontal_answers and vertical_answers are exactly 5 characters before using them in your code.

The place_word function is also safely handling word placements, particularly in the columns.

These changes should help eliminate the index errors by ensuring all accessed indices are within the valid range for both the grid and the words. If you have any additional needs or run into further issues, feel free to ask for more assistance.

B.2.3 Experiment Run 3.

User : The task is a 5x5 mini crosswords. Given an input of possible answers to horizontal clues and vertical clues, come up with a successor state function written in python, where a state is a 2D array representing the current puzzle state (initially all None). The possible clues for each row and each column are given separately. Here is an example:

Note that some of the possible answers are not exactly 5 character long.

ChatGPT : To address the mini crossword puzzle problem as described, we need to define a successor function for generating possible states from a given state. Each state in this problem is represented by a 5x5 grid, where each cell can contain a letter or be empty.

Here's a high-level plan for the successor function:

Initialize State: Start with an empty 5x5 grid.

Identify Next Cell to Fill: Decide on a strategy to fill the grid. A simple approach is row-wise from top to bottom, left to right.

Generate Successors: For the next empty cell, consider all possible words that could fit in the row and column intersecting at that cell, ensuring that the words do not conflict with already filled cells.

Check Validity: Each attempted word must not only fit the current cell but also not contradict any existing letters in intersecting rows and columns.

Create New State: For each valid placement, create a new grid state.

Below is a Python function to implement this idea. It uses recursion to try placing words in the grid row by row and backtracks when no valid placement is possible. It assumes that the lists of words contain only valid 5-character words fitting exactly in the grid. We'll have to pre-process the input to ensure this:

This function assumes that all entries in horizontal_answers and vertical_answers are valid 5-character words. The function is_valid checks if a word can be placed at a given position without conflicts, and place_word places the word on the grid. The successor function generates all possible successor states by trying to place words in the first available empty spot.

User : Can you write a goal test as well?

ChatGPT : A goal test function for the mini crossword puzzle checks if the crossword puzzle is completely filled with valid words according to both the horizontal and vertical constraints. Here's how you can implement such a function:

Check Completion: Verify that all cells in the grid are filled (i.e., none are None).

Validate Words: Ensure that every row and column contains a word from the respective lists of possible words (horizontal_answers and vertical_answers).

Here's the Python code that implements the goal test:

Explanation:

is_complete: This function checks if there are any None cells left in the grid. If there are, the grid is not yet complete.

extract_word: This function extracts a word from a specified starting point either horizontally or vertically.

validate_words: This function checks each row or column to ensure the extracted word is in the provided list of possible answers for that row or column.

goal_test: This function combines all the checks to determine if the grid meets the goal conditions.

This set of functions together forms the complete goal test for the mini crossword problem.

User :

ChatGPT : The "IndexError: string index out of range" error you're encountering suggests that either word[i] or new_grid[row + i][col] is trying to access an index that doesn't exist. This could be happening because of several reasons in the context of your crossword puzzle:

Word Length Mismatch: If word is shorter than 5 characters (since you are working with a 5x5 grid), trying to access word[i] where i ranges from 0 to 4 will result in an index error if word is shorter than this range.

Grid Boundaries: The same error could occur if you try to place a word such that it would extend beyond the boundaries of the grid, but this usually would throw a different kind of error unless the grid's structure itself is misdefined.

Steps to Resolve: First, ensure that all words used in the successors function are exactly 5 characters long. This needs to be enforced before words are passed into your place_word and is_valid functions. You can preprocess your horizontal_answers and vertical_answers lists to filter out any words that are not exactly 5 characters long:

Second, double-check that you are not attempting to place a word vertically or horizontally in a position where it doesn't fit within the grid boundaries. This check should be integrated within your is_valid function to ensure that you do not start placing a word too close to the edge of the grid:

Implementing these checks should help prevent the IndexError by ensuring that only suitable words are considered and they are always placed within the valid boundaries of the grid. If these solutions do not resolve the issue, I recommend printing out the values of row, col, and word just before the error occurs to better understand the problematic scenario.

I am still getting this error

ChatGPT : The error message "IndexError: string index out of range" at new_grid[row + i][col] = word[i] suggests an issue with accessing an index that doesn't exist, either because the word length is incorrect or because you're trying to access a grid position that's out of range. Since we've already checked the word lengths to ensure they're exactly 5 characters, let's make sure of a few additional things:

Debugging Steps:

Check Grid Initialization: Verify that your grid is indeed a 5x5 grid and initialized correctly. This error might occur if the grid was not initialized as expected.

Confirm Word Lengths: Double-check that the word lengths are exactly 5 characters when they are passed into the place_word function. Even though you've filtered them before, it's good to check right at the point of use.

Boundary Conditions in Placement: The error might be occurring when trying to place a word vertically near the bottom of the grid or horizontally near the right edge, even though you have checks to prevent this. Let's add explicit checks right in the place_word function to avoid going out of bounds.

Here's how you can add debugging statements and ensure the grid dimensions and word length:

Updated Code with Debugging:

Explanation:

Debug Prints: Added debug print statements that will show exactly what word and position are being attempted if an error occurs or if a boundary check fails.

Boundary Checks: Explicit checks are added right in the place_word function to avoid placing words that would extend beyond the grid's boundaries.

Error Handling: If an IndexError is caught, the code will print out detailed debug information before re-raising the exception. This can help identify exactly what went wrong.

Using these additions, you should be able to pinpoint why the error occurs. Ensure all words are properly filtered to 5 characters before they are used in the place_word function, and always check your grid initialization and the indices you're accessing.

User : Instead of throwing error, just skip over the words that are not exactly 5 character long and revise the definition.

ChatGPT : To avoid errors related to word length mismatches and to make your code more robust, I'll provide you with a revised version of the crossword puzzle functions where any words that are not exactly 5 characters long are skipped during the placement process. This will ensure that only suitable words are considered when generating successor states and checking grid validity.

Here's the updated Python code:

Revised Code to Skip Invalid Length Words and Placement

Key Changes:

Word Length Check: Before attempting to place a word or validate its placement, the functions now check if the word is exactly 5 characters long. If not, they skip processing that word.

Code Efficiency: Filters are applied to ensure that all words considered are of appropriate length before any operations are performed with them. This helps avoid unnecessary runtime errors and simplifies the logic within the successor functions.

This approach ensures your crossword solver doesn't crash due to incorrect word lengths and that it handles words dynamically based on actual content suitability.

B.2.4 Experiment Run 4.

User : The task is a 5x5 mini crosswords. Given an input of possible answers to horizontal clues and vertical clues, come up with a successor state function written in python, where a state is a 2D array representing the current puzzle state (initially all None). The possible clues for each row and each column are given separately. Here is an example:

Note that some of the possible answers are not exactly 5 character long.

ChatGPT : To solve this problem, we'll need to define a successor function for a 5x5 crossword puzzle using the given possible answers for horizontal and vertical clues. The successor function will produce all possible next states from the current state by placing valid words into the crossword, ensuring that the new word fits both the horizontal and vertical constraints.

Here's a Python function that outlines this process. The function assumes that the state is represented by a 5x5 grid where each cell can contain a letter or be empty (denoted by a space). The function will attempt to place a word from the list of possible answers into the next available row or column that is currently empty, ensuring it does not violate any constraints imposed by already placed letters.

This function generates successor states by checking each row and column to see if they are completely empty and, if so, attempts to place every possible word from the respective answers list into that row or column. Each time a word is successfully placed (meaning it does not conflict with any existing letters), a new successor state is created. This approach does not yet handle more complex situations where the grid is partially filled and might require backtracking or more sophisticated constraint checking, which would be typical in a full crossword puzzle solver.

User : Can you write a goal test as well?

ChatGPT : Certainly! The goal test function for a crossword puzzle needs to determine if the crossword is fully filled with valid words according to the given horizontal and vertical clues. This involves verifying that every row and column contains a word from the respective clues list and that these words also agree at every intersection point.

Here's a Python function for the goal test that you can use in conjunction with the successor function:

print("Is this a goal state?", is_goal_state(test_state, horizontal_answers, vertical_answers)) This function checks each row and column to ensure that they contain valid words from the provided lists. If all rows and columns contain valid words and the entire grid is filled (no None or unfilled spaces), the crossword is considered complete, and the goal state is achieved. This example assumes that the state passed to is_goal_state is a completed state. For a fully functional solver, you would typically integrate this check into your search or solving algorithm, to be performed after each new state generation. User : Getting this error: File "./gpt4_goal4.py", line 15, in is_goal_state row_word = ''.join(state[i]) TypeError: sequence item 0: expected str instance, NoneType found

ChatGPT : The error you're encountering occurs because the function tries to join elements of the list that contains None values, which join() cannot handle because it expects strings. This can happen if the test data provided to the function still contains None values, or if the state isn't fully filled yet.

To address this, you should ensure that the state being tested is fully filled with strings (i.e., no None values). If you want to test partial states or states that may still contain None, the function needs to be adjusted to skip or properly handle these cases.

Here's a revised version of the is_goal_state function that checks for None values before attempting to create a string from the row or column:

Revised is_goal_state Function: This function now ignores words that are not exactly 5 characters long and ensures all cells are filled with valid strings.

These modifications should handle scenarios where words are not exactly 5 characters long and avoid errors due to incomplete data or mismatches in expected data types.

B.2.5 Experiment Run 5.

User : The task is a 5x5 mini crosswords. Given an input of possible answers to horizontal clues and vertical clues, come up with a successor state function written in python, where a state is a 2D array representing the current puzzle state (initially all None). The possible clues for each row and each column are given separately. Here is an example:

Note that some of the possible answers are not exactly 5 character long.

ChatGPT : To solve the given 5x5 mini crossword puzzle task, we'll write a Python function that defines the successor state function. This function will generate possible next states based on a given state by trying to fill the crossword grid progressively with valid words from the provided clues, both horizontally and vertically.

Each state in our context is a 2D list representing the crossword grid, where each element can be a letter or a placeholder (like a dot .) indicating an empty space. The successor state function will take the current state and return a list of all valid states that can be reached by making a single valid word placement.

Here's a simple approach using Python:

Start with an empty grid.

For each cell in the grid, if it's empty, try placing each of the potential words that can fit in the row or column.

For each word placement, check if the word is compatible with any existing letters in the intersecting rows or columns.

If a word placement is valid, create a new state with this word placed and return this as a potential successor state.

Let's define the Python function:

This function starts with an empty grid and tries to fill it by placing each possible word in every row and column, respecting the existing letters. The is_valid_placement function checks whether a word can be legally placed in a given position, and place_word updates the grid with the new word. The example usage initializes the grid and finds the next possible states from an empty board.

Keep in mind that this approach is simplistic and might not efficiently solve more complex puzzles due to combinatorial explosion in larger grids or with more word options. For more efficient solving, one might consider using constraint satisfaction techniques or more advanced search algorithms.

User : Can you write a goal test as well? ChatGPT :

Certainly! A goal test function in the context of a crossword puzzle checks whether the entire grid has been filled with valid words from the given clues without any empty cells remaining. Here's how you could write a goal test function for our crossword puzzle setup, assuming the grid uses None to represent empty cells:

Here's what this goal_test function does:

Check Rows and Columns: The function loops through each row and column of the state.

Validation of Content: For each row and column, it forms the word by joining the characters. If any cell in the row or column is None, it means the crossword is incomplete, so it returns False.

Word Verification: It checks if the formed word is in the list of possible answers for that row or column. If not, the function returns False.

Completion Check: If all rows and columns have valid words and no None values, it returns True.

This setup ensures that your goal test is strictly validating both the completion of the grid and the correctness according to the provided answers. Make sure that the example state you test with corresponds to possible valid completions as per your horizontal_answers and vertical_answers.

The error you're encountering arises because the function get_column_word joins elements into a string, and if any of those elements are None, the process of joining fails to include them directly. However, the immediate issue seems to be the misunderstanding in the error message. When converting None types to a string during the joining process, it doesn't handle None explicitly, thus trying to check None in the resulting string is not directly possible as None would not exist in a string.

To correct the function and the error, you need to ensure that None checks are performed before attempting to join the elements. Let's adjust the get_column_word and its usage in the goal_test function to properly handle columns with None:

Changes Made:

Both get_row_word and get_column_word functions now check for None before attempting to join elements into a word. If None is present, these functions return None.

This way, when you check row_word and col_word in the goal_test function, you simply test if they are None or not valid according to the clues lists. This avoids trying to work with invalid types and addresses the error you were encountering.

This should resolve the error and improve the robustness of the goal_test function.

B.3 BlocksWorld

The prompt for the successor function is as follows.

I am playing with a set of blocks where I need to arrange the blocks into stacks. Here are the actions I can do

Pick up a block Unstack a block from on top of another block Put down a block Stack a block on top of another block

I have the following restrictions on my actions: I can only pick up or unstack one block at a time. I can only pick up or unstack a block if my hand is empty. I can only pick up a block if the block is on the table and the block is clear. A block is clear if the block has no other blocks on top of it and if the block is not picked up. I can only unstack a block from on top of another block if the block I am unstacking was really on top of the other block. I can only unstack a block from on top of another block if the block I am unstacking is clear. Once I pick up or unstack a block, I am holding the block. I can only put down a block that I am holding. I can only stack a block on top of another block if I am holding the block being stacked. I can only stack a block on top of another block if the block onto which I am stacking the block is clear. Once I put down or stack a block, my hand becomes empty.

If a state is defined by a list of following predicates "clear block", "on-table block", "arm-empty", "holding block", and "on block1 block2", how are successor states defined? Provide only the python code that returns a list of successor states for a state.

The prompt for the goal test is as follows.

Provide only the python code that tests whether a state is a goal state. Assume that the goal configuration is provided in the same format.

B.3.1 Experiment Run 1.

User : I am playing with a set of blocks where I need to arrange the blocks into stacks. Here are the actions I can do

Pick up a block Unstack a block from on top of another block Put down a block Stack a block on top of another block

I have the following restrictions on my actions: I can only pick up or unstack one block at a time. I can only pick up or unstack a block if my hand is empty. I can only pick up a block if the block is on the table and the block is clear. A block is clear if the block has no other blocks on top of it and if the block is not picked up. I can only unstack a block from on top of another block if the block I am unstacking was really on top of the other block. I can only unstack a block from on top of another block if the block I am unstacking is clear. Once I pick up or unstack a block, I am holding the block. I can only put down a block that I am holding. I can only stack a block on top of another block if I am holding the block being stacked. I can only stack a block on top of another block if the block onto which I am stacking the block is clear. Once I put down or stack a block, my hand becomes empty.

If a state is defined by a list of following predicates "clear block", "on-table block", "arm-empty", "holding block", and "on block1 block2", how are successor states defined? Provide only the python code that returns a list of successor states for a state.

ChatGPT :

To solve this problem, we can write Python code that represents each state of the blocks world and generates all valid successor states based on the actions that can be taken. The state can be represented by a dictionary where each predicate (like "clear", "on-table", "holding", "on") is a key with a list of items (blocks or tuples of blocks) that satisfy the predicate. Here's a Python function to generate successor states:

return successors

This function performs the following:

When the arm is empty, it attempts to pick up clear blocks from the table or unstack blocks. When a block is being held, it considers putting the block down on the table or stacking it onto another clear block. Each action generates a new state which is added to the list of successors. The code assumes that the initial state is provided with the correct format and content, and does not handle any kind of errors or inconsistencies in the input state. You can test this function with different states to explore how it generates successors based on the actions allowed.

User : Provide only the python code that tests whether a state is a goal state. Assume that the goal configuration is provided in the same dictionary format.

ChatGPT :

To determine if a given state matches a goal configuration in the blocks world, you need to compare relevant predicates that define the configuration of blocks. The goal might involve checking specific blocks being "on" other blocks, blocks being on the table, and the state of being "clear". Here's a Python function to check if a state is a goal state:

This function works as follows:

It checks that all on relationships defined in the goal state are present in the current state. It checks that all blocks that should be on the table according to the goal state are indeed on the table in the current state. It checks that all blocks that should be clear according to the goal state are clear in the current state. The function returns True if all conditions are met, meaning the current state matches the goal state, and False otherwise. This function assumes that the state and goal_state dictionaries are properly formatted and contain the necessary keys. Adjustments might be needed based on the exact structure of your state definitions.

User : The successor_states function uses a shallow copy, please rewrite with deep copy

ChatGPT : To ensure that the changes to nested lists or dictionaries in the state don't affect other states, it's necessary to use a deep copy of the state in the successor_states function. Here is the revised code using Python's copy module for deep copying:

In this code, copy.deepcopy(state) is used to create a new independent copy of the state dictionary. This prevents modifications to the new_state from affecting the original state object or other generated successor states. This deep copy is essential when dealing with mutable objects like lists or dictionaries within the state that may change from action to action.

B.3.2 Experiment Run 2.

User : I am playing with a set of blocks where I need to arrange the blocks into stacks. Here are the actions I can do

Pick up a block Unstack a block from on top of another block Put down a block Stack a block on top of another block

I have the following restrictions on my actions: I can only pick up or unstack one block at a time. I can only pick up or unstack a block if my hand is empty. I can only pick up a block if the block is on the table and the block is clear. A block is clear if the block has no other blocks on top of it and if the block is not picked up. I can only unstack a block from on top of another block if the block I am unstacking was really on top of the other block. I can only unstack a block from on top of another block if the block I am unstacking is clear. Once I pick up or unstack a block, I am holding the block. I can only put down a block that I am holding. I can only stack a block on top of another block if I am holding the block being stacked. I can only stack a block on top of another block if the block onto which I am stacking the block is clear. Once I put down or stack a block, my hand becomes empty.

If a state is defined by a list of following predicates "clear block", "on-table block", "arm-empty", "holding block", and "on block1 block2", how are successor states defined? Provide only the python code that returns a list of successor states for a state.

ChatGPT :

To define a function that calculates the successor states based on the rules you provided, we'll write a Python function that takes the current state as input. The state is represented by a list of predicates such as "clear blockA", "on-table blockB", "arm-empty", "holding blockC", and "on blockX blockY".

Here's a breakdown of how each action affects the state:

Pick up a block - This action can be taken if:

The block is on the table. The block is clear. The arm is empty. It results in the block being held by the arm and the arm no longer being empty. Unstack a block - This action can be taken if:

The block is on top of another block. The block is clear. The arm is empty. It results in the block being held, the block below it becoming clear, and the arm no longer being empty. Put down a block - This action can be taken if:

The block is being held. It results in the block being on the table, clear, and the arm becoming empty. Stack a block - This action can be taken if:

A block is being held. The target block is clear. It results in the block being on top of the target block, the target block no longer clear, and the arm becoming empty. Let's write a Python function that generates successor states for a given state:

This function computes the successor states by iterating through possible actions and checking the constraints of each action. It then returns a list of new states based on the applicable actions.

User : Provide only the python code that tests whether a state is a goal state. Assume that the goal configuration is provided in the same format.

ChatGPT : To check whether a given state matches a goal configuration, you can write a Python function that compares the two states. Here's a simple Python function that does this by ensuring every predicate in the goal state is present in the current state:

This function works by converting both the current state and the goal state into sets and then checking if all elements of the goal state set are contained within the current state set. This approach handles cases where the current state might have extra predicates not relevant to the goal, as long as all predicates describing the goal configuration are satisfied.

User : The generate_successor_states function in the case of stack a block does not add clear for the block that it was holding. Please rewrite.

ChatGPT : You're correct about the issue in the "Stack a block" section of the generate_successor_states function where it does not correctly handle the clearing status of the block being held after it is stacked. Here's the corrected version of the function that ensures the state is correctly updated:

{block}") return successors Now, when stacking a block, the function correctly adds the clear predicate for the block that was previously held, reflecting the fact that after being placed on another block, it is accessible again (unless it gets covered later). This is important for maintaining an accurate description of the state for future actions. User : Also, when getting the on_top blocks from the state, we must differentiate the predicate "on x y" from "on-table x". They both start with "on". Please fix. ChatGPT : You're right; the current way the function checks for blocks on top of each other can mistakenly include "on-table" predicates due to the substring match. To differentiate properly between "on x y" and "on-table x", we can use a more precise filtering method. Here's the updated version of the generate_successor_states function with this correction:

This update ensures that the function specifically checks for predicates that exactly match the format "on x y" by splitting the string into exactly three parts, which helps accurately identify the "on" relationships between blocks, without confusing them with "on-table" predicates.

B.3.3 Experiment Run 3.

User : I am playing with a set of blocks where I need to arrange the blocks into stacks. Here are the actions I can do

Pick up a block Unstack a block from on top of another block Put down a block Stack a block on top of another block

I have the following restrictions on my actions: I can only pick up or unstack one block at a time. I can only pick up or unstack a block if my hand is empty. I can only pick up a block if the block is on the table and the block is clear. A block is clear if the block has no other blocks on top of it and if the block is not picked up. I can only unstack a block from on top of another block if the block I am unstacking was really on top of the other block. I can only unstack a block from on top of another block if the block I am unstacking is clear. Once I pick up or unstack a block, I am holding the block. I can only put down a block that I am holding. I can only stack a block on top of another block if I am holding the block being stacked. I can only stack a block on top of another block if the block onto which I am stacking the block is clear. Once I put down or stack a block, my hand becomes empty.

If a state is defined by a list of following predicates "clear block", "on-table block", "arm-empty", "holding block", and "on block1 block2", how are successor states defined? Provide only the python code that returns a list of successor states for a state.

ChatGPT : To define the successor states for a block stacking game based on the rules and actions you've described, we can create a Python function that checks the current state against possible actions (Pick up, Put down, Stack, Unstack) and returns all valid successor states. Each state is a set of predicates that describe the current situation, such as which blocks are clear, which are on the table, whether the arm is empty, which block is being held, and the stacking relations between blocks.

Here's a Python function to calculate successor states:

stack_relations[top] = bottom

This function performs the following steps:

Parse the current state to identify clear blocks, blocks on the table, the block being held, and relationships between stacked blocks. Determine possible actions based on whether the arm is empty or holding a block. Generate successor states for each valid action, taking care to update predicates accurately to reflect the new state of the world after the action. Return all successor states. You can use this function by providing it with a set of predicates that describe a current state, and it will return a list of all possible successor states.

User : The generate_successor_states function in the case of stack a block does not add clear for the block that it was holding. Please rewrite.

ChatGPT : You're right; when stacking a block, the previously held block should still be clear after the action, since nothing is on top of it in the new state. I'll update the code to handle this correctly. Here's the revised version of the function:

In this corrected version, when the held block is stacked on another block, the clear predicate for the held block is now added to the state to indicate that nothing is on top of it. This maintains the integrity of the game's logic and ensures the state reflects the actual conditions after each move.

User :

Provide only the python code that tests whether a state is a goal state. Assume that the goal configuration is provided in the same format.

ChatGPT : Here's a Python function that checks whether a given state matches a specified goal state. This function assumes both the current state and the goal state are provided as sets of predicates in the same format. The function will return True if all predicates in the goal state are present in the current state, indicating that the current state meets all the conditions of the goal state.

This function uses the issubset method of Python sets, which returns True if all elements of the goal_state are present in the current_state. This approach ensures that the current state contains at least all the conditions defined by the goal, but it may also contain additional predicates not specified in the goal, reflecting the possibility of multiple configurations that satisfy the goal requirements.

B.3.4 Experiment Run 4.

User : I am playing with a set of blocks where I need to arrange the blocks into stacks. Here are the actions I can do

Pick up a block Unstack a block from on top of another block Put down a block Stack a block on top of another block

I have the following restrictions on my actions: I can only pick up or unstack one block at a time. I can only pick up or unstack a block if my hand is empty. I can only pick up a block if the block is on the table and the block is clear. A block is clear if the block has no other blocks on top of it and if the block is not picked up. I can only unstack a block from on top of another block if the block I am unstacking was really on top of the other block. I can only unstack a block from on top of another block if the block I am unstacking is clear. Once I pick up or unstack a block, I am holding the block. I can only put down a block that I am holding. I can only stack a block on top of another block if I am holding the block being stacked. I can only stack a block on top of another block if the block onto which I am stacking the block is clear. Once I put down or stack a block, my hand becomes empty.

If a state is defined by a list of following predicates "clear block", "on-table block", "arm-empty", "holding block", and "on block1 block2", how are successor states defined? Provide only the python code that returns a list of successor states for a state.

ChatGPT : To define the successor states for a given state in this blocks world, we can implement a function in Python. This function will take the current state as input and return a list of all possible successor states based on the actions described (pick up, put down, stack, unstack). Each state can be represented as a set of predicates that describe the conditions in the environment.

Here is the Python code that defines the function to generate successor states:

This code defines four actions (pick up, put down, unstack, stack) and checks for the necessary conditions before applying an action. Each action, when applicable, modifies the state accordingly and adds the new state to the list of successors. The function assumes that each state is uniquely represented by a set of predicates and each block involved in the predicates is uniquely identifiable.

User : Provide only the python code that tests whether a state is a goal state. Assume that the goal configuration is provided in the same format.

ChatGPT : To test whether a given state is a goal state in the blocks world, you can implement a Python function that compares the current state against a specified goal configuration. The function will check if all the predicates in the goal configuration are present in the current state. Here's the Python code for this function:

This function uses a set containment check (<=), which checks if all elements of the goal state are present in the current state, thus ensuring that all conditions required by the goal are satisfied in the current state.

User : The successo_states function in the case of stack a block does not add clear for the block that it was holding. Please rewrite.

ChatGPT : You're correct; in the case of stacking a block on top of another, the code should ensure that the block being stacked becomes "clear" after the action, as there would be nothing on top of it. Let's revise the stack function within the successor_states method to correctly reflect this:

This updated version of the stack function now correctly adds the clear block predicate for the block being stacked, ensuring it's clear because there's nothing on top of it after the action. This update reflects the logic needed to maintain a consistent state representation after stacking operations.

User : The blocks are not correctly identified from this state representation. Please fix

ChatGPT : You're right; the approach for identifying blocks from the state representation in the previous code is flawed because it inaccurately captures block names when parsing predicates. I'll correct this by implementing a more reliable method to extract block names directly from the predicates involved in the state.

Here's the revised version of the successor_states function that includes an improved way to determine the set of blocks based on the current state:

In this revised version, I have improved the block identification by iterating over all predicates and extracting block names from them directly, ensuring that only valid block names are considered for actions. The revised code assumes that block names can be identified as they don't contain the word "table", thereby excluding predicates like "on-table". This approach provides a more robust way to handle different naming conventions or additional predicates that might be added to the state.

User : There is no need for checking if the string "block" is part of the block name, it might not be. Please update the code.

ChatGPT : You're right; assuming that the term "block" will be a part of the block's name in predicates is restrictive and may not apply to all scenarios. Instead, we'll directly gather the block names by processing the appropriate parts of each predicate more accurately. Here's an updated version of the successor_states function that extracts block names based on the structure of predicates without assuming specific naming conventions:

This version extracts block names from predicates based on their position in the string, which is typically the last part for predicates like "clear", "on-table", and "holding". For predicates starting with "on", both blocks involved are considered. This method ensures that all blocks are appropriately identified and handled without presuming the presence of specific substrings like "block" in the block names.

B.3.5 Experiment Run 5.

User : I am playing with a set of blocks where I need to arrange the blocks into stacks. Here are the actions I can do

Pick up a block Unstack a block from on top of another block Put down a block Stack a block on top of another block

I have the following restrictions on my actions: I can only pick up or unstack one block at a time. I can only pick up or unstack a block if my hand is empty. I can only pick up a block if the block is on the table and the block is clear. A block is clear if the block has no other blocks on top of it and if the block is not picked up. I can only unstack a block from on top of another block if the block I am unstacking was really on top of the other block. I can only unstack a block from on top of another block if the block I am unstacking is clear. Once I pick up or unstack a block, I am holding the block. I can only put down a block that I am holding. I can only stack a block on top of another block if I am holding the block being stacked. I can only stack a block on top of another block if the block onto which I am stacking the block is clear. Once I put down or stack a block, my hand becomes empty.

If a state is defined by a list of following predicates "clear block", "on-table block", "arm-empty", "holding block", and "on block1 block2", how are successor states defined? Provide only the python code that returns a list of successor states for a state.

ChatGPT : To generate successor states based on the actions and restrictions you've described, you'll want a function that checks the current state (represented by a set of predicates), applies possible actions based on the rules, and then produces the new states. Here is a Python function that accomplishes this:

This function assumes that the state parameter is a set of strings representing the predicates. The actions available depend on whether the robot's arm is empty or holding a block, and whether certain blocks are clear and available on the table or can be unstacked from another block. This function then modifies the state according to the rules of each action and adds each resulting new state to the list of successor states.

User : Provide only the python code that tests whether a state is a goal state. Assume that the goal configuration is provided in the same format.

ChatGPT : To test whether a given state matches a goal configuration, you can use a Python function that simply checks if all predicates specified in the goal are present in the current state. Here's a concise function to perform this check:

This function receives two sets: state and goal, where each is a set of strings representing the predicates. The function returns True if all predicates in the goal are contained within the current state, signifying that the state meets the goal configuration, and False otherwise. This approach is straightforward and leverages Python's set operations for efficiency.

User : The successor_states function does not correctly parse the state into predicates. Please rewrite.

ChatGPT : Certainly! It seems there was a misunderstanding in how predicates like "on" were being parsed. I'll adjust the parsing logic in the successor_states function to correctly handle the predicates and relations between blocks. Here's the corrected version of the function:

In this revised function:

I ensure proper parsing of each predicate to correctly identify which blocks are clear, which are on the table, which one is being held, and the relationships between blocks (on top of). Operations like picking up a block from the table and unstacking are allowed only when the arm is empty, and the corresponding block is clear and either on the table or on top of another block. The stacking and putting down operations are processed when holding a block, respecting the restrictions on clear targets and table placements. This code is more structured and should correctly handle the transition between states according to the provided rules and predicates.

User : When getting the on_top blocks from the state, we must differentiate the predicate "on x y" from "on-table x". They both start with "on". Please fix.

ChatGPT : You're right; the initial parsing of the predicates didn't differentiate between "on-table" and "on" predicates correctly. I'll adjust the function to parse these predicates appropriately. Here's the updated successor_states function:

In this revised function:

The on_top_of set now ensures that only predicates starting with "on" and having exactly three components (which corresponds to "on X Y") are included. This ensures accurate recognition and handling of the predicates representing blocks stacked on each other while avoiding confusion with "on-table" predicates. The operations within the function continue to account for blocks being picked up, placed, or stacked according to the rules.

B.4 PrOntoQA

The prompt for the successor function is as follows.

Logical deduction is performed by applying rules of the form if X then Y to our current knowledge about a world, extending our knowledge with each rule application. A rule if X then Y is applicable if we know X. If a state is defined as our current knowledge of the world, how are successor states defined, given a collection of rules? Provide only the Python implementation of successor states function

The prompt for the goal test is as follows.

Provide only the python code that tests, given a target fact, whether a state is a goal state.

B.4.1 Experiment Run 1.

User : Logical deduction is performed by applying rules of the form if X then Y to our current knowledge about a world, extending our knowledge with each rule application. A rule if X then Y is applicable if we know X. If a state is defined as our current knowledge of the world, how are successor states defined, given a collection of rules? Provide only the Python implementation.

ChatGPT : In the context of logical deduction using a set of rules, a state can be considered as a set representing our current knowledge about the world. Each element in this set is a fact or a truth that we know about the world. Successor states are defined by applying applicable rules to the current state to extend our knowledge.

Here is a Python implementation to model this behavior. This implementation defines a function that takes the current state and a collection of rules, and produces the set of all successor states by applying these rules:

In this code:

apply_rule checks if the antecedent of a rule (first element of the tuple) is in the current state. If it is, it adds the consequent (second element) to the state. generate_successor_states iterates over all rules and applies them to the current state, collecting new states that result from applying each rule. Each new state is added only if it is different from the current state, avoiding duplicates. This model assumes that the rules are simple implication relationships and that the state can be represented as a set of facts. The generated successor states are those that include at least one new fact derived from applying the rules.

User : Provide only the python code that tests, given a target fact, whether a state is a goal state.

ChatGPT : To test whether a given state is a goal state based on the presence of a target fact, you can define a simple function in Python. This function will check if the target fact is in the current state. Here is how you can implement this:

This function is_goal_state takes two arguments: the state (a set of facts) and the target_fact (a fact that you want to check for). It returns True if the target fact is present in the state, indicating that this state is a goal state with respect to the target fact.

B.4.2 Experiment Run 2.

User : Logical deduction is performed by applying rules of the form if X then Y to our current knowledge about a world, extending our knowledge with each rule application. A rule if X then Y is applicable if we know X. If a state is defined as our current knowledge of the world, how are successor states defined, given a collection of rules? Provide only the Python implementation of successor states function.

ChatGPT : To implement the function for generating successor states using a logical deduction approach, you'll need a way to represent the current state of knowledge and the rules. Here's how you might do it in Python:

State Representation: Represent the state as a set of facts. Each fact can be a simple string or a more complex structure, depending on your application. Rule Representation: Represent rules as tuples (X, Y), where X is a condition (or set of conditions) and Y is the conclusion. X can be a single fact or a set of facts. Successor Function: The function checks each rule to see if its conditions are met in the current state. If so, it adds the rule's conclusion to the state, forming a new successor state. Below is the Python implementation for generating successor states:

This function checks each rule to see if the conditions are met within the current state. If they are, it creates a new state that includes the conclusion of the rule. This implementation also supports both single conditions and sets of conditions. Adjustments might be needed depending on the specific requirements and the complexity of the conditions and conclusions.

User : Provide only the python code that tests, given a target fact, whether a state is a goal state.

ChatGPT : Here's a Python function that tests whether a given state contains a target fact, effectively checking if it is a goal state:

This function simply checks if the target_fact is a member of the state set, returning True if the state meets the goal criteria (i.e., contains the target fact) and False otherwise.

B.4.3 Experiment Run 3.

User : Logical deduction is performed by applying rules of the form if X then Y to our current knowledge about a world, extending our knowledge with each rule application. A rule if X then Y is applicable if we know X. If a state is defined as our current knowledge of the world, how are successor states defined, given a collection of rules? Provide only the Python implementation of successor states function.

ChatGPT : To provide a Python implementation for defining successor states based on a given collection of rules and a current state of knowledge, we need a function that takes the current state (a set of known facts) and the rules (formulated as "if X then Y") and produces all possible successor states by applying these rules.

Here's a straightforward Python function to achieve this:

This function works by iterating through each rule and checking if the antecedent (condition) of the rule is already a known fact in the current_state. If it is, it creates a new state by adding the consequent (result) of the rule to a copy of the current state. Each new state is added to a set of successor states, ensuring that only unique states are stored. Each state is stored as a frozenset to ensure immutability and to allow sets of sets.

User : Provide only the python code that tests, given a target fact, whether a state is a goal state.

ChatGPT : Here's a Python function to check whether a given state includes a specific target fact, thus determining if it's a goal state:

User : In successor_states, I get AttributeError: 'frozenset' object has no attribute 'add'

ChatGPT : The error arises because I incorrectly used frozenset for the new_state, which is immutable and doesn't support the add method. We should first modify the mutable set before converting it to a frozenset when adding it to new_states. Let's correct the implementation:

This revised function ensures that the current_state is copied into a mutable set before adding new elements. After the modifications are done, it is then converted into a frozenset for storage in the new_states set. This approach maintains the immutability of states within new_states and avoids errors related to modifying a frozenset.

B.4.4 Experiment Run 4.

User : Logical deduction is performed by applying rules of the form if X then Y to our current knowledge about a world, extending our knowledge with each rule application. A rule if X then Y is applicable if we know X. If a state is defined as our current knowledge of the world, how are successor states defined, given a collection of rules? Provide only the Python implementation of successor states function

ChatGPT : To implement a Python function for generating successor states from a given state using a set of logical deduction rules, we'll assume that the state is represented as a set of facts (truths known about the world), and the rules are represented as tuples (X, Y), meaning "if X then Y". A rule is applicable if all elements of X

are present in the current state. When a rule is applied, Y is added to the state to create a successor state. Here's how this can be implemented:

This function simply checks if the target_fact is a part of the current state. indicating that the state is a goal state; otherwise, it returns False.

User : In successor_states function, assume that the rules are just pairs of facts, so a single fact condition. Also, you cannot add to a frozenset

ChatGPT : Given the revised assumptions where each rule is just a pair of facts and each condition is represented by a single fact, and considering that we cannot add to a frozenset, we can refactor the successor_states function. The function will now operate on a copy of the current state converted to a mutable set, apply the changes, and then convert back to a frozenset for immutability and use in a set of states. Here's the updated implementation:

This code:

Converts the immutable frozenset of the current state to a mutable set to allow modifications. Iterates over each rule, checking if the condition is met in the current state. If the condition is met, it creates a new state by copying the current state, adding the result of the rule, and then converting it back to a frozenset for use in a set (to ensure immutability and unique state management). Returns a set of all new states generated this way.

B.4.5 Experiment Run 5.

User : Logical deduction is performed by applying rules of the form if X then Y to our current knowledge about a world, extending our knowledge with each rule application. A rule if X then Y is applicable if we know X. If a state is defined as our current knowledge of the world, how are successor states defined, given a collection of rules? Provide only the Python implementation of successor states function

ChatGPT : In a logical deduction system, we can model successor states using a function that takes the current state of knowledge (a set of facts) and a collection of rules. Each rule can be represented as a tuple (X, Y) where if X (a precondition) is satisfied by the current state, then Y (a consequence) can be added to the state.

Here's a Python function that computes the successor states by applying all applicable rules to the current state:

This function works as follows:

It initializes the new state with the current state. It iterates over the rules and checks if the precondition of any rule is met by the current state. If a precondition is met and the consequence is not already in the state, the consequence is added. This process repeats until no new facts can be added, ensuring all applicable rules have been exhausted. The function ensures that all direct and indirect consequences of the initial facts are considered, making it useful for computing successor states in logical deduction systems.

User : Provide only the python code that tests, given a target fact, whether a state is a goal state.

ChatGPT : Certainly! Below is a Python function that tests whether a given state contains a target fact, which would indicate it's a goal state:

This function takes a set state containing all known facts and checks if the target_fact is among them, thus determining if the state is a goal state.

The implementation of apply_rules applies all possible rules iteratively. That is not what I asked for, I asked for a successor states function, returning all immediate successor states

ChatGPT : Thank you for the clarification. To implement a function that generates all immediate successor states, we need a different approach. Instead of applying all rules iteratively until no more can be applied, we'll generate a new state for each single rule application, thereby obtaining all the states that are one rule application away from the current state.

Here's the Python implementation for the function that generates all immediate successor states:

This function iterates through each rule and checks if the precondition is met and the consequence is not already in the state. If so, it creates a new state with the consequence added and adds this new state to the list of successors. Each state in the list represents a distinct state reached by applying one rule from the initial state.

Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: abstract and introduction does reflect the main claim of the paper, assumptions and limitations. Further the claims are supported in the rest of the paper (Section 2, 3, 4 and discussed further in Section 5).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please see the future work section which discusses the limitation of the work. "For future work, we would like to explore the use of large language models to obtain the code for search guidance or search pruning techniques. But more importantly, we would like to relax the need for human feedback in coming up with valid implementations of the search components. This could be done in ways similar to the way a policy is obtained in generalized planning. "

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The proofs are either inline or known results and are referenced.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please see Section 4 and Appendix A and B for all the details.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Please see Section 4 and Appendix A and B for all the details.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/gui des/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public /guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please see section 4 for details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Appropriate values are reported in Section 4 and further discussed in the Discussion section.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have reported information on compute resources in Section 4 (Experiments).

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read NeurIPS Code of Ethics and comply with it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We do mention negative impact of prior work with respect to their inefficient use of LLMs in the introduction. In particular we mention "harming the environment". Our proposed work mitigates this negative impact and instead it argues for a responsible use of compute resources.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use GPT-4 and make appropriate references.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Please see Appendix A and B for details. We provide the DFS and BFS implementation we used as well as all the interactions with chat GPT.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer:[NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyst the use of LLM in planning, and propose to write the successor function and goal test in code instead of directly solving the problem. The paper showed experiments that using the code can get a higher accuracy and lower number of calls to the LLM compared to LLM-based solutions.\n\nSTRENGTHS:\nThe paper is very well-written and easy to follow. The discussion is very detailed and provide extra insights on the topic.\n\nWEAKNESSES:\n- First of all, I think the conclusion is not surprising or bring any insights at all. The idea of letting LLM to write code instead of directly solve problems has extensively appear in LLM for reasoning [1], and more specifically on LLM for planning [2][3]. \n- Because the conclusion does not provide any big insights, I am expecting to see a strong experiment, which is not the case in this paper. The benchmark selected, are quite classic from different manners, and similar code has appeared on GitHub for at least a year. It is hard to justify whether the success of generating code is coming from LLMs remembering similar context from Github or they do have the ability to correctly generate code. And even in this case, the authors mentioned “The mistakes GPT-4 makes when producing the code repeat from one experiment to another”, which is not a good sign if one wants to deploy similar methods to more general applications. \n- One minor drawback is that I feel the authors failed to cover some related works that are quite close, for example the ones I mentioned in the first weakness. A more thorough literature review are recommended for a more convincing paper.\n\n[1]. Zhou, Aojun, et al. \"Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification.\" arXiv preprint arXiv:2308.07921 (2023).\n\n[2]. Liu, Bo, et al. \"Llm+ p: Empowering large language models with optimal planning proficiency.\" arXiv preprint arXiv:2304.11477 (2023).\n\n[3]. Guan, Lin, et al. \"Leveraging pre-trained large language models to construct and utilize world models for model-based task planning.\" Advances in Neural Information Processing Systems 36 (2023): 79081-79094.\n\nQUESTIONS:\nI am concerned about the current need of human feedback to successfully generate the code. While I can see from the appendix that a large portion of them might need such feedback, can you provide some statistics about this?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a position paper that argues current works for LLMs for planning waste significant compute, on top of having poor algorithmic and empirical performance. The authors also propose ideas on how to use LLMs more efficiently and effectively by using them to preprocess search algorithms instead of using them directly during search. More specifically, the proposed method consists of using the LLM to generate the successor generator and goal state checker, alongside user feedback.\n\nSTRENGTHS:\nThe paper has several strengths as a position paper. It provides a simple yet original idea, namely that researchers should strive for responsible usages of LLMs for planning in terms of computing efficiency and also provides arguments that this would actually improve the performance of LLMs for planning as well.\n\nThe idea is complemented by an extensive survey of LLMs for planning methods with a summary of worst case complexities, and whether the algorithms are sound or complete. Furthermore, it is complemented by a novel methodology for using LLMs for planning which adheres to the authors' proposition: more efficient and effective LLMs for planning research. \n\nThe experiments are extensive with a wide variety of planning benchmarks, implementation details, and results. The results are quite positive as summarised in Table 1, with minimal calls to LLMs in comparison to existing approaches. The authors are also transparent about the limitations of their approach in its current state, being that it requires user interaction. Nevertheless, this does not undermine the proposition of the position paper.\n\nWEAKNESSES:\nWith regards to the idea of the paper, there are no major weaknesses. Nevertheless, the paper could benefit from some additional details regarding experiments, and improved clarity in certain areas.\n\n- It may not be clear to some readers how %States could go over 100%. By its definition in line 344-345, it is not clear whether visiting the same state twice double dips into the percentage or not, but from the next 2 sentences, it does seem to be the case.\n- Minor formatting issue: the benchmark domains are not consistently capitalised, e.g. 24 game and crossword in line 357 but elsewhere they are capitalised such as in Table 1 and page 8.\n- Although the focus of the paper is in LLMs for planning, the paper misses more general related work regarding learning for planning/generalised planning. Such methods are magnitudes of orders more efficient than LLMs in evaluating learned heuristics (ToT or GoT in LLM terminology) or policies and solve problems magnitude of orders larger than problems solved by LLM research. Example works include learned heuristics [1] or generalised policies [2], as well as foundation models for planning [3].\n\n[1] Simon Ståhlberg, Blai Bonet, Hector Geffner: Learning General Optimal Policies with Graph Neural Networks: Expressive Power, Transparency, and Limits. ICAPS 2022: 629-637\n\n[2] Dominik Drexler, Jendrik Seipp, Hector Geffner: Expressing and Exploiting Subgoal Structure in Classical Planning Using Sketches. J. Artif. Intell. Res. 80 (2024)\n\n[3] Dillon Ze Chen, Sylvie Thiébaux, Felipe W. Trevizan: Learning Domain-Independent Heuristics for Grounded and Lifted Planning. AAAI 2024: 20078-20086\n\nQUESTIONS:\nNo questions or clarifications that could change my opinion.\n\nLIMITATIONS:\nThe authors adequately addresses limitations of their work, and also the checklist with appropriate justification.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nExisting LLM-based planning approaches usually involve searching and multiple passes of the model, which leads to significant inefficiency and cost while failing to guarantee the correctness of the generated plans. Motivated by this issue, this paper first analyzes the soundness, completeness, and complexity of a series of existing planning methods, arguing that they do not meet the standard of these properties. A new algorithm, Thought of Search (ToS), is proposed to alleviate the heavy computing demand of the searching operation for planning. It queries LLMs for generating the code implementation of successor functions and goal tests. The proposed method achieves 100% success rate for four representative search problems, while the total time spent (on CPU) is shorter than or comparable with a single LLM evaluation time. Further discussion describes that, compared with other approaches, ToS is not only sound and complete but also more cost-effective and able to explore a much larger portion with only O(1) complexity.\n\nSTRENGTHS:\n1. Impactful motivation. Recently, there have been increasing efforts to improve the planning ability of LLMs. However, since this line of research is still at its early stage, the efficiency of the proposed methods is easy to overlook, especially when the system contains multiple agents. Therefore, I strongly agree with the motivation of this paper (i.e.,  the need for sound and complete LLM-based approaches that uphold efficiency) and believe in its importance for future research.\n2. Comprehensive analysis. The paper systematically and comprehensively studies the properties of twelve related works that are commonly used or recently proposed, providing convincing support for the authors’ claim and, more importantly, valuable information to the community.\n3. Solid results. Across all four evaluated tasks, ToS consistently generates valid final solutions only and reaches 100% accuracy in a relatively short time. Notably, the searching operation is run on the CPU. These results are sufficient to demonstrate the effectiveness of the proposed method.\n\nWEAKNESSES:\n1. Current works on code generation using LLMs show that the correctness of the generated code is not guaranteed. In this paper, the obtained implementations in the experiments are also not always valid at the first trial and require human feedback. Thus, I am slightly concerned about whether ToS can generalize to more difficult tasks while maintaining its nice properties. Nonetheless, this is likely to be alleviated by combining with some automated optimization techniques as discussed by the authors. Therefore, I am still relatively optimistic about this approach at this point. Further experiments (if any) to address this concern are welcome.\n\n2. While the paper's contents are generally well organized, which I appreciate, there are quite a few typos and wrong words/phrases. For exemplification, a non-exhaustive list is written below. I highly recommend a careful and thorough check of the whole paper to fix all the mistakes.\n  - Line 22: ‘The purpose of our work is precisely that.’\n  - Line 99: ‘Reflection’\n  - Line 323: ‘a lower than reported by the approaches accuracy’\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nThe limitations have been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a Thought of Search: thinking before searching strategy to solve Automated Planning problems using LLMs. They use the GPT-4 LLM to generate a Python code for generating successor states and goal test functions which are the crucial parts of any search. They argue that this method is sound and complete and requires the least calls to the LLM before successfully solving the problem when compared to the relevant literature.\n\nSTRENGTHS:\n1. Analysis of Complexities for the existing methodologies of using LLM for Planning.\n2. Innovative use of LLM to reduce the number of calls made to solve the problems.\n\nWEAKNESSES:\nHowever, there are several concerns regarding the proposed work:\n1. If the Large Language Model (LLM) is being used to generate successor and goal test functions, which are already intrinsic components of automated planners, it is unclear what improvement is being made. The inherent memory and time complexities associated with solving these planning problems are not addressed or mitigated by the proposed approach. It appears that the LLM is simply re-writing (a few components of) a more basic planner, which does not seem sufficiently innovative for a NeurIPS-level conference.\n2. The process is not fully automated, as it requires human intervention to re-prompt the LLM until the correct code is generated. \n3. The methodology would have been more compelling if it had included an exploration of generating useful heuristic functions.\n\nQUESTIONS:\n1. Why did the authors use only GPT-4 for their experimentation? No reasons are presented in the paper. Presenting a comparison of the performance with different LLMs would have been interesting to see.\n2. What is the need to use LLMs to generate successor and goal test functions and solve the planning problems with a naive blind search? Is the objective to evaluate GPT-4's capability to generate Python code, or to assess its reasoning ability in solving planning problems?\n3. Comparing the number of calls to the LLM with other approaches may not be appropriate. Your methodology necessitates generating the search components once for each domain, whereas other approaches involve the LLM in solving every problem within the domain. A more relevant comparison would be the average time required to solve these problems and the success rate across different methodologies.\n\nLIMITATIONS:\n1. Showcasing results with only GPT-4 model.\n2. Incomplete comparisons with the other existing approaches as mentioned above.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}