Leveraging Environment Interaction for Automated PDDL Translation and Planning with Large Language Models
Authors: Sadegh Mahdavi
Raquel Aoki
Keyi Tang
Yanshuai Cao
Venue: NeurIPS 2024 poster (2024)
Keywords: Planning
PDDL
Large Language Models
Primary area: natural_language_processing
TL;DR: We automate PDDL file generation for planning problems using LLMs and environment interaction

Abstract: Large Language Models (LLMs) have shown remarkable performance in various natural language tasks, but they often struggle with planning problems that require structured reasoning. To address this limitation, the conversion of planning problems into the Planning Domain Definition Language (PDDL) has been proposed as a potential solution, enabling the use of automated planners. However, generating accurate PDDL files typically demands human inputs or correction, which can be time-consuming and costly. In this paper, we propose a novel approach that leverages LLMs and environment feedback to automatically generate PDDL domain and problem description files without the need for human intervention. Our method introduces an iterative refinement process that generates multiple problem PDDL candidates and progressively refines the domain PDDL based on feedback obtained from interacting with the environment. To guide the refinement process, we develop an Exploration Walk (EW) metric, which provides rich feedback signals for LLMs to update the PDDL file. We evaluate our approach on $10$ PDDL environments. We achieve an average task solve rate of 66\% compared to a 29\% solve rate by GPT-4's intrinsic planning with chain-of-thought prompting. Our work enables the automated modeling of planning environments using LLMs and environment feedback, eliminating the need for human intervention in the PDDL translation process and paving the way for more reliable LLM agents in challenging problems. Our code is available at https://github.com/BorealisAI/llm-pddl-planning

Leveraging Environment Interaction for Automated PDDL Translation and Planning with Large Language Models
Sadegh Mahdavi* University of British Columbia smahdavi@ess @ece. wbc.uuc..o
Raquel Aoki Borealis AI
raquel. aoki1borrealisai.coom
Keyi Tang
Yanshuai Cao Borealis AI
Borealis AI
keyi. tang@borealliaaaa. com
yanshual.cao@boeallise caw@borealisaa com
Abstract
Large Language Models (LLMs) have shown remarkable performance in various natural language tasks, but they often struggle with planning problems that require structured reasoning. To address this limitation, the conversion of planning prob- lems into the Planning Domain Definition Language (PDDL) has been proposed as potential solution, enabling the use of automated planners. However, generating accurate PDDL files typically demands human inputs or correction, which can be time-consuming and costly. In this paper, we propose novel approach that lever- ages LLMs and environment feedback to automatically generate PDDL domain and problem description files without the need for human intervention. Our method introduces an iterative refinement process that generates multiple problem PDDL candidates and progressively refines the domain PDDL based feedback obtained from interacting with the environment. To guide the refinement process, we develop Exploration Walk (EW) metric, which provides rich feedback signals for LLMs to update the PDDL file. We evaluate our approach on 10 PDDL environments. We achieve an average task solve rate of 66% compared to 29% solve rate by GPT-4's intrinsic planning with chain-of-thought prompting. Our work enables the automated modeling planning environments using L.MSs and environment feedback, eliminating the need for human intervention in the PDDL translation process and paving the way for more reliable LLM agents in challenging problems. Our code is available at https//giihub.coo//orraliii palpssslAmmmm--llllllpiii
1
Introduction
Large language models (LLMs) have demonstrated remarkable success across various domains, including mathematics, coding, and even the bar exam [ These models excel at understanding and generating natural language, offering flexibility and adaptability wide range of tasks. However, when comes to planning and monghooizon reasoning, LLMs have shown limited performance despite some promising results (].
28
Planning is crucial aspect of intelligence that involves reasoning to find a sequence of actions to achieve desired goal state from an initial state. The Planning Domain Definition Language (PDDL) 18 a widely used formalism for describing planning problems. PDDL provides a structured way
Work performed while interning at Borealis AL
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

to define the domain, which includes the types of objects, predicates, and actions, as well as the problem instance, which specifies the initial state and goal conditions. PDDL enables the application of search-based algorithms, such as breadth-fitg first search (BFS) or A search, which can guarantee to find a valid solution if one exists. However, the downside of PDDLL is that it requires a well-defined and structured domain and problem definition, which can be challenging to create, especially for complex scenarios. Figure [ showcases snippets of some PDDL problems and domain files along with an action plan produced by classical planner.
Recent studies explored combining the strengths of LLMs and PDDL-based planning 13
9 The
idea is to leverage LLM for translation from natural language (NL) problem descriptions into PDDL formal descriptions, and then use classical planner to solve the translated PDDL problem (9]. This hybrid approach could theoretically take ad udvantage of the flexibility of NL input and the correctness guarantees provided by the classical planner. If the translation from NI. to PDDL is accurate, the resulting plan is guaranteed to be valid.
Unfortunately, existing approaches have not been able to generate both PDDL problem and domain descriptions with reasonable success rates without humans in the loop, as we shall elaborate in Sec. 2 While translating PDDL problems is feasible given the domain PDDL description [5), generating domain PDDL from NL correctly is more nuanced and challenging problem. To do so requires identifying causally relevant objects to design predicates, as well as their inter-eelaioonshiss, in a way that accurately reflects the possible states and transitions of the environment. A small error, for example in predicate design, could lead to entirely incorrect domain description and failed planning (see Appendix A.2 for real example). Guan et al. [9] take step toward this goal relying on human-in-then toop to detect and correct mistakes made by LLMSs.
In this work, we develop a fully automated method for generating PDDL domain and problem definitions using LLMs and environment feedback without relying on human intervention. Intuitively, our method lets an LLM build hypothetical "mental models" of the environment, in the form of proposed PDDL domain descriptions. The LLM then verifies and updates the "mental model" by observing discrepancies between the feasibility of actions under its "mental model" and the real environment. This method enables LLMs to use classical planners to solve complex planning problems whose solutions may require hundreds or thousands of steps that all need to be correct.
We first highlight the challenges of this task and then propose solution. In particular, our contributions are as follows:
We demonstrate that even small modifications to PDDL domains can render plan search infeasible, limiting the feedback information for LLMs to perform in context update.
To address this, we introduce a new Exploration Walk (EW) metric, which is a smooth similarity measure between two domains by comparing the executability of random action sequences sampled from one domain on the other. Crucially, EW only requires access to the action interface and executability of the environments, not directly the ground-truth PDDL.
We propose an guided tree search approach that leverages LLMs to generate and refine the PDDL domain and problem files iteratively and automatically.
We evaluate our method 10 challenging PDDL domains, where a number of them from the International Planning Competition, and show that outperforms a baseline that generates PDDL files in a single attempt without refinement. Our method solves out of 10 environments, achieving an average task solve rate of 66% and average EW score of 0.84, compared 34% task solve rate and 0.53 EW score for the baseline, and 29% solve rate by GPT-4 (gpt-4 1-06--16---00 preview)'s 's intrinsic planning with chain-of tfain-of--hought prompting.
To the best of our knowledge, this the first work that enables modeling planning environment via PDDL translation using LLMs and environment interaction, without the need for human intervention.
2
Related Work
LLMs and Classical Planning. There has been recent interest in integrating LLMs with PDDL [15]
28
g
20 26). and more generally neural networks with PDDL 24 2 Silver et al. leverage M. to take domain PDDLs and problem PDDL, specifications, and synthesize a
7
30
23
10
25
Python function to generate domain-specific plans, as a replacement for cearch-based planning. Liu
2

---

Table Summary of comparison to most closely related prior studies. Buuddes..Requunn "Require at least one problem instance to be translated by a human into the target domain as an in-context example.
(sthod(s)
Translate Problem
Translate Domain
No Human Intervention
LLM+P
[5.. LLM-DP 7
LLM World Models 9
X
Ours
et al. [15] show that using LLMs to translate problem specification to PDDL, and using classical solvers results into a higher planning accuracy that using LLM directly as a planner. Dagan et al. 77) consider a similar setting, but assume that the list of objects is partially observable, and the LLM needs to interact with the world to observe the list of objects. All of the mentioned works, however, assume that domain PDDL files is already provided. Oswald et al. [0] generate domain PDDL from natural language and propose heuristics for comparing PDDL action domains. However, their approach assumes that predicates are provided, whereas our work makes no such assumption. Additionally, Oswald et al. (20) rely on ground-truth problem instances for domain compatibility evaluation, whereas we directly translate problem PDDL without any such assumptions. Guan et al. ( translate both Domain and Problem from natural language description but rely on human experts to correct mistakes in the domain translation before generating problem PDDLs. In this work, our goal is to lift the and verification. See Table I for summary of related work comparison.
assumption, and instead, use domain interaction for evaluation
Direct Reasoning with LLMs. Recent research has explored eliciting direct reasoning capabilities within Large Language Models (LLMs). This reasoning can be either entirely direct [1 29 or partially direct with the assistance of basic external tools [16]. However, the primary limitation of these approaches lies in the inherent tendency of auto-regressive LLMS produce errors in long- horizon reasoning tasks (28). Even a minor mistake in a single reasoning step can lead to cascading errors, ultimately resulting in incorrect final answer [8]. When applied to classical planning, this approach delegates the entire plan generation process to an LLM instead of leveraging dedicated classical planner. Studies have demonstrated that this strategy is suboptimal compared to generating PDDL code directly [9] [5], highlighting the importance of incorporating classical planning tools for faithful plan generation in classical planning tasks.
This last line of work focuses on generating executable code from natural language instructions such as SQL Python code generation [4] 19 5 32). Here, the LLM often acts as a code translator, and the reasoning logic lies within the generated code. Chen et al. 4 show that LLMs are capable of Python code generation from docstrings to high accuracy. The authors also find that taking multiple code samples from an LLM and picking the best samples results in an accuracy boost. Later works show that iterative refinement of LLM responses improves the accuracy on the downstream task [17 5]. especially given external feedback such as unit tests or human feedback. Our work related to code generation as we produce structured PDDL files. However, our setting presents three challenges: (1) there are two types of PDDL files, in contrast to single Python script, and the two files need to be consistent with each other; (2) more importantly, getting external feedback and the evaluation of a generated PDDL code is not as easy as python unit tests, and as we show in Section 4.3, (domain generation) errors are abundant and hard to trace; (3) LLMMs are trained with a lot more Python code compared to PDDL., as the later is much scarcer.
External Reasoning and Code Generation.
3 Notation and Background
Notation. We denote [[-] as the indicator function. The notation 1 N refers to the sequence of integers ranging from to N. For a set AA, we define A as the set comprising all possible sequences of elements drawn from A, and define 2-4 as the power set of A.
PDDL. Planning Domain Definition Language (PDDL) is a formal language used to describe and specify planning problems for automated planning. Here, we have two types of PDDL files: (1) Domain PDDL, which defines possible predicates (i.e., states), and actions in the environment. Executing each action requires some precondition (i.e., set of predicates to have specific value), and the execution leads to some effect (i.e., a change in the values of some predicates). (2) Problem PDDL, which contains set of initial predicates and a set of goal predicates.
3

---

The problem PDDL instantiates the domain definition PDDL to form a concrete environment. Together, the planning problem fully defined and formalized. A classical planner takes in both files and searches for plan based on the provided specification. A plan is sequence of actions, starting from the initial state, leading to state satisfying the goal conditions, with each action respecting the rules of the environment. Formally, let DD P. AA be the set of all possible domains, problems, and actions, respectively. Then, given a domain E 'D and problem p E P. a classical planner C:: D x P + A U } takes in domain d and plan p, and produces a plan q == C(d, which is either set of actions from A", or a planning error L. A planning error may be due to an infeasible plan search (i.e., plan not found), syntax errors, or incompatible domain and problem. A plan validator verifies whether a plan q is executable and achieves the desired problem goal given a domain PDDL d and problem PDDL p, i.e., whether q solves the planning problem instance. The validator function, denoted as Va.p A + {0, 1), is 1 if the plan is valid, and O otherwise. For convenience, we assume Va.((1)
O. Similarly, we define plan execution checker Ed..p : A {0, 1). which only checks whether an action sequence is executable in a domain or not. Note that the difference between V and E is that the former checks for both plan executability and goal satisfaction, while the latter only checks for plan executability, We also define S as the set of all possible states. Function Ad.p : :S 2- delineates the set of legal actions given the current states (i.e., actions that would not immediately result in Ed.p returning 0). The function Sa,p A x S + S denotes the state transition function ( i.e., Sd.p(a,
(a. s) determines the subsequent state given the current state s and action a). Finally, we denote the initial state induced by d and p to be sd.p, E S. See Table 3 in the Appendix for a summary of notations.
To illustrate the definitions with an example, consider the Grippers 13 environment with several rooms containing robots and boxes. Robots move balls between rooms us- ing their left and right grippers. Given an initial setting of robots and balls in differ- ent rooms, the main goal is move specific balls specific rooms using the robots. Figure | shows an annotated exam- ple domain, problem, and plan for this environment. The domain deter- mines predicates and actions. Pred- icates such as at at-robby keep track of object states (e.g., whether a par- ticular robot is in particular room)
Domain PDDL
Problem PDDL
(define (damain grippar-trips)
(dorine (probiem grippe--)
Eilpes room obj robot grioper) Epredicates
(oobjects robot1 robos2- robat
Object List
Predicates
(al-obbyy robot ruom)
room1 room2- room
(fre Pg gripper)
(inil
Initial State
and defining suitable predicates is a crucial part of domain design. The move action for moving a robot from one room to another has three param- eters: robot r, departure room from, and destination room to. Each action has preconditions and effects, which comprise the main logic of the domain for determining the actionability of an action,
(of-robby robott room2) (tree rebeti rgripper1)
:action mave
pparameters (2r robot 7from "to room) -4 precondtion (and (at-robiyy 7r ?from))-4
saffect (and (st-mbhy 7r ?h0)
Coool
Gool Conditions
(nol (al-robby (from)))
(st ball1 room2)
[astion pick
Effects
[adion drop
Preconditions-
Plan
Action Interface
(move robet2 room3 room1))
(pick robot2 ballz roomt (groppr)2)
action. In the case of the the precondition is that the robot must be in the from room, and the effect is that it will no longer be in that room and will be in the to room. A problem PDDL p specifies the initial state of robots, boxes, rooms, and the final goal. For instance, (at-robby robot2 room3) means that robot2 is initially at room3. The predicate (at ball1 room2) specifies the goal condition that ball1 must eventually be moved to room2. A plan constitutes a sequence of actions to reach the goal. For instance, one action could be (move robot2 room3 room1), moving robot2 from room3 to room1. If robot2 is not already in room3, this action is considered illegal, and the environment will produce an error. For complete example of domain d, problem p. and plan q. see Listings I 2 and 7 respectively in the Appendix.
Actions
Figure 1: Snippets of PDDL domain, problem, and plan.
Large Language Models (LLMs). We assume access to a powerful language model LLM. LLM, (X) denotes sampling n responses from the LLM given prompt X. Following the prior works, we set a temperature of T = 0 for sampling with n = (i.e., greedy sampling), and a tempera- ture of - 0.7 for n > 1 (5]. Whenever possible, use zero-shot or one-shot chain-of-though
prompts 44 29 for the LLM to reason before generating response.

---

4
Method
Given environment e, its domain NI. description and task NI. description, the environment's object list and action interface, our goal is to model the environment by generating a domain PDDL E DD and problem PDDL P E P. such that applying a classical planner C' on the PDDL files produces valid plan for the environment, i.e., C(d, P) is valid plan for e, Le., (C, 1.
4.1 Setup
For evaluation, we assume there exists ground truth domain PDDL DD and a corresponding problem instance p P. However, the ground truth is not directly compared generated d, p. but to validate the plan q == C(d. p) by executing the validator of the ground-trute truuh environment, Va,
Formally, for each environment with domain PDDL d E DD and N tasks with their corresponding ground-truth problem PDDLs pI:N- == (pi, (P1,2,,-
PI:N PN, our goal is to generate domain PDDL d, and a sequence of task PDDLS pI.N = (f1,f2..... .... .r)) such that the average solve rate V is maximized:
El,,Pa,.... ...,..... PNNN
argmax V(d, PENEEN
Σ Van. (c)d. N
1
Generating accurate d and PI.N in one attempt is often impractical [9]. and some form of feedback is required to refine the response. Guan et al. 9 leverage human expert feedback on d to correct the generated domain. However, human feedback may not always be reliable and is not scalable. Before introducing our method that relies on environment feedback instead, we first state our assumptions:
Assumption (Environment access) We assume the list of objects and action interfaces are known. Furthermore, we assume that executability and verifiability of actions can be observed (through the functions E..p and V...).
Assumption (Natural language description) We assume the natural language descriptions of the domain and task are both given.
The action interfaces are equivalent to APIs available to LLM agents. So it is reasonable to assume that the exact API call signatures are known. On the other hand, one may wonder why the object list, which appears in problem PDDLs as illustrated in Figure | needs to be assumed to be given, when the NL problem description should describe the objects involved in the planning tasks. This is because the NL description may not refer to the object instances using exactly the same label as the environment induced by d and p. If p refers language problem description calls it Jarvis, then the environment only recognizes robot1 and not Jarvis, so the L.M would have no way to correct this mistake due to trivial name mismatch. See Appendix I. for detailed example of our assumptions on the Grippers environment.
robot as robot1 but the user specifying the natural
Note that our assumptions do not require the underlying environment to be PDDL environment, but it can be any environment as long as PDDL is expressive enough to capture the working mechanisms of the environment. For digital agents in virtual environments, the list of objects and action interfaces are just different data objects and APIs available. The assumptions could even hold true for physical agents in the real world, provided recognition and control are sufficiently accurate. In this work, we focus on PDDL environments only, although our framework is more general.
4.2 Difficulty of domain PDDL generation
Generating the correct domain PDDL is challenging, as small mistakes could make the plan search fail. To demonstrate this brittleness, we simulate random omission of k terms, where S < 10, from the action precondition and effects of the original domain d. For instance, in the case of the Grippers (Figure D. we may create a new synthetic domain by removing the (at robby ?r ?to) term from the effects of the move action. Namely, we define dę (x))), where (x(d) represents the uniform random removal of k terms. Then, for each generated de, coupled with the ground truth task PDDLs, we compute whether the classical planner is able to find a plan without error and compute the Plan-Not- FounF rate under k omissions, PNF of the environment.
5

---

Plan-Not-Foud Fraction
Average
0.6
0.4
Tury LPferance
1mm 1PPrrros
1.0
0.0
9
10
Remaved Terms (k)
5
(a)
Figure 2: (a) Effect of the number of removed terms on plan search failure. Fach gray line shows the PNF; Pllan-Not-Funnd metric for one environment. The line is the average of all 15 environments. (b) Correlation between average exploration walk (EW) score and average domain difference. The z-axis shows how many terms each pair of domains differs in. The y-axis shows the average EW score over various pairs, All the domains show the average monotonicity of the EW score with respect to term difference.
We empirically measure the value of PNFx using Monte-Carlo estimation on 15 environments. As shown in Figure 2a PNF; has average of 0.14 among different environments. This means that on average 14% the terms in domain PDDLs are so critical that removing them results plan-not-found error. This situation is exacerbated for larger A: at k 3, the average PNF reaches around 0.3. In practice, the problem PDDL P. also needs to be generated, and the generated domain may have extra terms, both of which may further increase the planning-not- rate.
4.3 Domain alignment measure via Exploration Walk metries
Whenever the plan search fails, absolutely no information is available to the LLM about which part of the problem or domain has issues. This is because the underlying search algorithm (such as BFS and A*) fails and as result, it does not produce any output. For example, with BFS, enumerates all paths (possibly several thousand paths more), and finds none satisfy the goal conditions, leaving the plan search without any useful insights. As an alternative, we introduce the Exploration Walk (EW): smooth feedback signal that provides incremental feedback for LLM in-context learning. EW both provides a mechanism to gather richer feedback information that feeds into LLM context for PDDL refinement, well computing a smooth scoring metric that to compare multiple PDDLs and guide the refinement process forward.
Intuitively, the idea is to take legal random action sequences and verify their executability under LLM's "mental model" environment induced by an --gnerrrtte PDDL domain. This is analogous to the retrodiction step in scientific methodology, where existing observations and experimental data need to be explained by the existing model.
And in the other direction, EW takes executable random action sequences from an LLM- -enerrtteeed PDDL domain and verifies whether they are correct in the real environment. This is analogous to hypothesis testing in scientific methodology, where new predictions are verified experimentally,
We now describe the EW and EW metries formally. We define Exploration Walk of length T be any action sequence sampled from a strictly positive distribution Pd.p.r over executable T-step action sequences in A corresponding to domain and task p. We assume the probability of non-executable action sequences to be zero under P. In other words, VqL.T, P...p. (q1.r) > 0 iff (q.(() 1.
For the rest of this paper, use the simplest possible EW, with uniform distribution over valid actions at each step. Note that to sample uniform random EW from the ground truth environment induced by d and p, do not need direct access to the full d and p. We only need the list of objects in and the action interface in d, and executability checker E.,p. consistent with our Assumption At each step, running Ea,p on all possible actions yields the legal actions at that step for EWW
Given an EW distribution, we define an EW metric using the fractions executability of EW walks from one domain under another, averaged over all different lengths.

---

Definition 1 (EW Metries) Let P1:N and PI-N be problems in domain d and respectively, such that the set of objects Pj and Pj are consistent. We define the one-sided measure m a-d and the for the degree of alignment between two domains d and d as:
symmetric one aasa for
1
E
N Tame
21= Eyppapy, Eas, (0)
NT Th d PVN): = 2/ al/ma-a(pnnn where Tmax is the largest EW walk length.
ma-d measures what fraction of EWs sampled from domain d are executable on the domain d. Then, maedd takes the harmonic mean of ma-d and d-d to produce the final EW measure. This metric has two favourable properties: (1) it ensures that dedd meddd thereby providing consistent measure of similarity regardless of the order of domain comparison. (2) the harmonic mean is resistant to trivial domain similarity inflation. By employing the harmonic mean rather than the arithmetic mean, the symmetric EW metric prevents domains that are overly permissive (e.g.. domains where all actions are permissible without any preconditions) from being similar more restrictive domains. For example, in scenario where domain d allows all possible actions without restrictions, ma-d 1. An arithmetic mean in this context would yield maedd 2 0.5, overestimating the similarity. In contrast, the harmonic mean results in m E, where < 1) for most cases.
Note that while the PDDL problems PI:N and PI-N appear in the definition of EW metrics, we only the fact there are aligned object sets in them. We could also use arbitrarily sampled object list to form P and pair P with D and Î for EW metrics. But since for PDDL generation, already generate PI:N, it is more convenient to use them.
Importantly, EW metrics can be computed without direct access to the full ground truth domain d and problems p's o'.. As established before, to sample uniform random EW, we just need access to the object list and action interface, plus the environment executability checker of the source domain. So even for a where the EW action sequences come from d, we do not need more than what is available through Assumption I
To demonstrate the relationship between m aedd and domain disparity, we use the same simulated random omission study setup from Sec. 222 For a pair of modified domains, we count the number of terms that differ, and inspect m d as function of increasing number of differing terms in Figure 2b for six example domains (see Figure in the Appendix for the full set). We observe that, average, a greater discrepancy in the number of terms between two domains correlates with a reduced EW score m d This observation provides additional support to the use of the EW score an effective measure for domain differences.
4.4 Leveraging LLMs to generate PDDL files
We now show our overall LLM-based method for PDDL generation using the EW score to guide and measure the progress of domain generation. To illustrate the process, we first focus on a domain d with a single task p. Recall that we are given NL description of the environment domain dsr. and problem PNL (Assumption 2) as well the object list in p and action interface from d (Assumption| Dp. Then, by using dsL, PNL: and access to environment action feedback, we seek to generate d DD, E P.
Our method starts by initializing templated d()) based on action interfaces and templated ((() using object list. Example template /i0) and (00) are shown in Listings 6 and of Appendix A.1 We then use an LLM to improve the initial di(0) and ((0).
Given that domain PDDL files are typically more complex than problem PDDL files, our strategy prioritizes the generation of a problem PDDL file P first, followed by the domain d. This approach enables us assess the quality of the generated domain immediately. Moreover, prior works on code generation 4 tree-of-thoughe [31]. and self-debug [5] have found that taking multiple samples from the LLM response and taking the best response leads to better performance. However, they often require an evaluation metric on the generated response (such as unit test cases, or execution traces). Here, we use the EW metric introduced in Section 4.3 to serve as an evaluator of the generated domain. These considerations lead to proposed Algorithm [
We emphasize again that the
7

---

Transation
Rafnement
LLM
Py
Xhans
Exp Walks
P&D Chain
P&D Tree
Planner
Figure 3: Overview of our method. Right: The process begins with natural language descriptions translated into problem PDDL by the LLM (red arrows). Then domain is generated and refined through iterative cycles involving exploration walks in the environment, interaction with a classical planner, and feedback from the LLM (blue/black arrows). Left: The iterative refinement process depicted on the right corresponds to single paths in the structures shown on the left. Fach node represents a state in the refinement process, with arrows indicating problem translation (red), domain refinement (blue).
ground-truth domain and problem d, only used to take exploration walks and evaluate plan through the environment in
Algorithm Generating Domain PDDL and Problem PDDL Using Environment Feedback
Require: Natural language descriptions dNL, PNL, environment action interface.
1: g()) p(2) p(.),.... p(mp) +- LLMM,, (PSL) (Problem PDDL candidates)
2: for = 1,2, .......,,,, np do
3: A(t) (0(1, dvu.] (Keep history of conversation).
4: (00) +- dNL. (Initialize with an empty template).
5:
for 1,2.... ..., Cmux do
6:
(c.1)) )(.2) ddtmm)) LLM (h()))
7:
d(c) +- argmax (dr)),,,... M maerd (p. (0)) {Evaluate LLM responses using EW)
8:
f(0) Natural language feedback from EW d,
9:
AO9 < h09 + [d f(r]]
10:
(0)
best + argmax.je (do)), m masdd (p, (p.p(0)
11: end for
end for
13:
p +- argmax (de)) ii, de+ (p. B(0)
return d P (Return the final refined domain and problem PDDLs)
Note that each environment contains N problems, therefore, we need to translate all problem instances into PDDL.. Similar to Liu al. [15]. given one problem Pisa. and its generated translation fr, we translate the rest of the problems P2.Nst in a one-shot manner. That is, we generate p; == LLM, (pisL PI, ) as the final problem translation for problem i for 2 < S N.
5
Experiments
Dataset. We consider PDDL files from real environments, taking nine domains from a combination of domain PDDLs from Liu et al. (15) and Seipp al. [22]. The LLM may have seen the mentioned domains in its pre-training data, which is a common issue for current benchmarks. To mitigate this issue, we also modify the original Grippers domain, and create modified domain called "Grippers- ood" domain, to ensure LLM has seen it previously. We generate natural domain descriptions for

---

all PDDL files by bock-translating them using GPT-4 and manually inspecting and modifying the translations for conrectness. For each environment, we consider one domain PDDL d and N 10 problem PDDLs PI-N- We use one problem for domain translation and HW evaluation, and all problems for evaluating final domain response. We reserve the Blocksworld environment as an in- context example for prompting the LLM. As such, we do not evaluate the Blocksworld environment itself in our evaluations. See Appendices and [ more details on dataset curation.
Feedback Format. The natural language feedback given to LLM is in the following form: [Action sequence] [State description]. That is, we first provide LLM with the sequence of actions taken from one exploration walk, up until one action fails. Then, we provide the environment state description from the last step. We show an example of environment feedback and LLM response for the Termes environment in Listings i in the Appendix. We deliberately choose simple feedback format to maintain the general applicability of our framework.
Baselines and Metrics. We use GPT-4 [ (gpt-4-1 1004--10--000 106-previer)) as the LLM since models with lower capability may struggle with syntax errors [9] We consider the following methods: (1, 2) Intrinsic Planning (CoT):: where the language model generates a complete plan without the help of any external planning library, based the given descriptions, both with and without chain-of-thought prompting. This baseline does not leverage any classical planner or PDDL translation. (3) P&D
Chain: Our proposed method (Algorithm TT with nd = ", 1. (4) P&D Tree: Our proposed method with multiple response generations (nd = 10, ?p = 5). (5) P&D Tree + DomProp: Our proposed method with multiple response generations and domain proposals for each problem (see Appendix B.2). Following prior works [1 5
we set a maximum conversation turns of Cmax 4.
We run each algorithm for four seeds and compute the Best metric, which takes the highest score among the four seeds. We report two metrics: (1) tasks solved measuring the fraction of the N = 10 tasks successfully solved (Eq. (I). and (2) EW score, comparing the final domain through running exploration walks on all N problems (Eq. (2) with Tmax 10). We use the original fast-downward [ library for planning, the modified fast-downward library from text-world 66 for python-compatible state explorations, and the VAL 12] library to validate plans.
Results. Table 2 shows the final results various environments. We consider domain generation to be solved if a method achieves > 0.5 solve rate since we observe the rest of the errors are problem translation errors rather than domain translation errors. Our proposed method solves 7 out of 10 domains, compared to 3 solved by the Intrinsic CoT baseline. We also generally observe the correlation of EW score with task solve rate. Particularly, even when the task solve rate is zero, the EW metric shows signs of progress, e.g., in domains such as Barman and Childsnack where all task solve rates are zero, the EW metric shows a clear distinction between method performances. Moreover, when the EW metric is high, such as 1.0, we observe generated PDDL domain to be very close the ground-truth domain, and differing in very few predicates. For instance, in the case of the "Hiking" environment, the P&D Chain achieves zero solve rate, but perfect EW score, which we observe perfect solution in the case of P&D Tree.
Computational Cost. For the results in Table 2 using the GPT-4 model, we used 12.40 million input tokens and 8.73 million output tokens. Computing the EW is relatively negligible compared to the cost of LLM inference. In our experiments, computing the EW score for single Nomain--roblee pair takes less than two minutes on a 64-core server CPU.
6 Conclusion
In this work, we present novel approach for modeling planning environments via PDDL translation using large language models (LLMs) and environment feedback, without relying on human inter- vention. The key contributions include introducing the Exploration Walk (EW) metric to measure domain similarity and guide domain refinement, and an iterative method that leverages LLMs to generate and refine PDDL domain and problem files. Evaluation on 10 real-world PDDL domains demonstrates the effectiveness of the proposed approach, outperforming a baseline that generates PDDL files in single attempt without refinement. The method solves 7 out of 10 environments, achieving an average task solve rate of 66% and an average EW score of 0.84.
Noote that perfect task solve rate does not guarantee exact domain equivalency of the generated domain to the ground truth domain.

---

Table @ (Tasks solved Exploration Walk) for different domains, For intrinsic planning no domain is generated, therefore the EW score is not defined.
Intrinsic No CoT
Intrinsic CoT
P&D Chain
P&D Tree
P&D Tree + DomProp (ma 10, Mp = 5)
(nd = 1, 1, = 1)
(nd
10, np = 5)
Barman
0.00/
0.00/
0.00 / 0.93
0.00 1.00
0.00 1.00
Childsnack
0,00/
0,00/
0.00 / 0.57
0.00 1.00
0.00 1.00
Driverlog
0
0.00/
0.00 / 0.05
0,00 / 0.05
0.00 0.60
Floortile
0,00/
0.00/
0.00/ 0.07
0,90/ 0.94
0,00 0.07
Grippers
0.40/-
0.60/-
0.10/ 0.39
1.00
1.00 1.00
Grippers-ood
0.30/-
0.30/-
0.30 / 0.35
0.70
1.00 1.00
Hiking
0.00
0.00/-
0.00/ 1.00
1.00
1.00 1.0/1.00 1.00
Miconic
0.90
1.00/-
/.80
1.00 0.85
1.00 1.00
Movie
1.00/
1.00/-
1.00/ 0.07
1.00
1.00 0.86
Termes
0.00
0.00/-
1.00/ 1.00
1.00/ 1.00
1.00 1.00
Average
0.26/-
0.29/-
0.34/
0.66 / 0.84
0.60 0.85
The current limitations include potentially insufficient and efficient exploration caused by random F.W. More sophisticated EW strategies could improve the success rate while lowering the cost in the future. For cxample, strategies from the reinforcement learning litcrature (e.g., [27 21) could be adapted to improve exploration efficiency and success rates. Another limitation is that have only applied the framework to PDDL environments, despite it being applicable to digital or even physical environments. We hope this work will inspire further research at the intersection of language models and planning, enabling the development of more advanced and autonomous planning systems.
References
[1] Achiam, Adler. S. Agarwal, L. Ahmad, Akkaya, F. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint :.Xix:2303.08777 2023.
[2] A. Ahmetoglu, M. Y. Seker, J. Piater, E. Oztop, and E. Ugur. Deepsym: Deep symbol generation and rule learning for planning from unsupervised robot interaction. Journal of Artificial Intelligence Research, 75:709-745, 2022.
[3] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Li, S. Lundberg, et al. Sparks artificial general intelligence: Early experiments with gpt-4. arXiv preprint 233. 2023.
[4] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. O. Pinto. J. Kaplan, H. Edwards, Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint trXiv:21072 033744 2021.
[5] X. Chen, M. Lin, N. Schärli, and D. Zhou. Teaching large language models to self-debug, In The Twelfth International Conference Learning Representations, 2024. URL https / //openrevie net/forum?iiiii wtw.formmmid-uuiiIIii...
[6] M.-A. Côté, A. Kádár, X. Yuan, B. Kybartas, T. Barnes, E. Fine, J. Moore, R. Y. Tao, M. Hausknecht, L. Asri, M. Adada, W. Tay, and A. Trischler. Textworld: learning environment for text-based games. CoRR, abs/1806.1 1532, 2018.
[7] G. Dagan, F. Keller, and A. Lascarides. Dynamic planning with a 11m. arXiv preprint arXiv:2308.06
2023.
[8] N. Dziri, X. Lu, M. Sclar, X. Li, L. Jiang, B. Y. Lin, S. Welleck, P. West, C. Bhagavatula, R. Le Bras, et al. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36, 2024.
[9] L. Guan, K. Valmeekam, Sreedharan, and S. Kambhampati. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL httpss
//openreview het//oorumiddzDDDsSeee
10

---

[10] M. Han, Y. Zhu, S.-C. Zhu, N. Wu, and Y. Zhu. Interpret: Interactive predicate learning from language feedback for generalizable task planning. In Robotics: Science and Systems (RSS), 2024.
[11] M. Helmert. The fast downward planning system. Journal of Artificial Intelligence Research, 26:191-246 2006.
[12] R. Howey, D. Long, and M. Fox. Val: Automatic plan validation, continuous effects and mixed initiative planning using pddl. In Proceedings of the 16th IEEE International Conference on Tools with Artificial Intelligence, ICTAI '04, page 294-301, USA, 2004. IEFE Computer Society. ISBN 076952236X. doi: 10. 109/CTA112000 120. URL
1109/ICTAI 2004. 120
[13] IPC. International planning competition, 1998. URI.
/g/competitions/
[14] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero- shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages
22199-22213 2022.
[15] B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang. J. Biswas, and P. Stone. Llm+p: Empowering large language models with optimal planning proficiency. ArXiv, abs/2304.11477, 2021, URL
weemanticscholl.eo htsp::///pp.semmttiicchhh.. /stps://ap....
Terg/CorpusIlie 258298051
[16] Q. Lyu, S. Havaldar, A. Stein, L. Zhang. D. Rao, E. Wong, M. Apidianaki, and C. Callison- Burch. Faithful chain-of-thought reasoning. In J. C. Park, Y. Arase, B. Hu, W. Lu, D. Wijaya, A. Purwarianti, and A. A. Krismadhi, editors, Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference the Asia-Pacific Chapter of the Association for Computational Linguisties (Volume 1: Long Papers), pages 305-329, Nusa Dua, Bali, Nov. 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1/2033 ijenlp-main.20 20. URL /aattps:/aacaantee www.ollogyooooooooooooooooo wtgg/022..eo. jicnnl-mmlim..222 20
[17] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark. Self-refine: Iterative refinement with self-feedback. Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL BtLpttts:://// http://pppereveeeeeeeeeeeee
[18] D. McDermott. The 1998 ai planning systems competition. Al Magazine, 21(2):35-55, 2000.
[19] A. Ni, S. Iyer, D. Radev, V. Stoyanov, W.-t. Yih, S. I. Wang, and X. V. Lin. Lever: Learning to verify language-to-code generation with execution. In Proceedings of the 40th International Conference Machine Learning (CCML223), 2023.
[20] J. Oswald, K. Srinivas, H. Kokel, J. Lee, M. Katz, and S. Sohrabi. Large language models as planning domain generators. In 34th International Conference on Automated Planning and Scheduling, 2024. URL
[21] S. Sagar, A. Taparia, and R. Senanayake. Failures are fated, but can be faded: Characterizing and mitigating unwanted behaviors in large-scale vision and language models. arXiv preprint arXiv:2406.075 07145, 2024.
[22] Seipp. Á. Torralba, and J. Hoffmann. PDDL generators. zenodo. 6382173, 2022.
[23] T. Silver, V. Hariprasad, R. S. Shuttleworth, N. Kumar, T. Lozano-Pérez and L. P. Kael- bling. PDDL planning with pretrained large language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022. URL htt.:ttts///ppeeeeeeeee wtwp://www..e//ooummimii. 1QMMUB4ZfI
[24] T. Silver, R. Chitnis, N. Kumar, W. McClinton, T. Lozano-Pérez, L. Kaelbling, and J. B. Tenenbaum. Predicate invention for bilevel planning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 12120-12129, 2023.
[25] T. Silver, S. Dan, K. Srinivas, J. Tenenbaum, L. Kaelbling, and M. Katz. Generalized planning in PDDL domains with pretrained large language models. In AAAI Conference on Artificial Intelligence (AAAI), 2024.
[26] K. Stein and A. Koller. Autoplanbench: Automatically generating benchmarks for Im planners from pddl. arXiv preprint arXiv:2311.0988 2023.

---

[27] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
[28) K. Valmeekam, M. Marquez, A. Olmo, S. Sreedharan, and S. Kambbhappti. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023, URL MtLLttts:///pperrrr//
[29] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Lc, D. Zhou, et al. Chain-of- thought prompting clicits reasoning in large language models. Advances in neural information processing systems, 55:24824-24887 2022.
[30] Y. Xie, C. Yu, Zhu, J. Bai, Z. Gong, and Soh. Translating natural language to planning goals with large- languaan models. arXiv preprint 23..:2300..052288 2023.
[31] S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. R. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
id=5Xclecx01n
[32] T. Zhang, T. Yu, T. Hashimoto, M. Lewis, W.-t. Yih, D. Fried, and S. Wang. Coder reviewer reranking for code generation. In International Conference on Machine Learning, pages 41832-41846. PMLR, 2023.
12

---

Table Summary Notation and Definitions
Notation
Description
1 N
Sequence of integers ranging from to N
A"
Set comprising all possible sequences of elements drawn from set AA
24
Power set of
D
Set of all possible domains in PDDL
P
Set of all possible problems in PDDL
A
Set of all possible actions in PDDL
1
Planning error
C:DXP=A'ULL
Classical planner function that takes domain E D and a problem P and produces plan q
Va A - 10,1)
Plan validator function for domain d and problem p. returns 1 if plan is valid, otherwise 0
Eap A² + 10
Plan execution checker for domain d and problem returns 1 if action sequence is executable, otherwise 0
S
Set of all possible states
Ad,p :8 -8-+2 24
Function delineating the set of legal actions given the current state for domain d and problem
Sd.p :AxS +
State transition function, determines the next state given the cur- rent state and action in domain d and problem p
$d.p.0
Initial state induced by domain d and problem
(X)
Sampling responses from the LLM given prompt X
A Dataset
A.1 Dataset Details.
Dataset Examples.
We provide example of each file for the Grippers environment: (1) The ground-truth domain (Listing] 1 of ground truth PDDL domain (2) One ground-truth problem (Listing 2 (3) Domain natural language deseription along with PDDL template for action interfaces dNL (Listings 5 and 6) (4) Problem natural language description along with PDDL template with the list of objects (Listings aand
(define (domain gripper
(: (:requrreennss :strips :typing)
t:types room obj robot gripper)
(:predicates (at-robby ?r robot ?x room)
(at 70 obj ?x room)
(free ?r robot ?g gripper)
(carry ?r robot ?o obj ?g gripper)
(:action move
10
parameters (?r robot ?from ?to room)
precondition (and (at-robbl ?r ?from)) :effect (and (at-robby ?r ?to) (not (at-robby ?r ?from))))
11
12
13
(:action pick
13

---

Storage
Floorble
Hiking
Childsnack
1.0
1.00
1.0
1.0
EW ScOr
9.75
0.5
2.
2.6-
0.50
0.0
2.5
0.4
0.2
10
10m Difeer--0 10
11
0 WTT
7mm Differncce
Logistics
Tormes
Driverlog
Satellite
1.00
EW Score
Mesroya CA Score
0.75
0.50
0.6
0.5
0.00
Depot
BuMRAc EW Soore
8.8-
0.5
0.4
Parking
1.00
0.75
Torn
once
Figure 4: Correlation between average exploration walk score and average domain difference
16
parameters (?r robot ?obj obj ?room room ?g gripper)
17
precondition (and (at ?obj ?room) (at-robb) ?r ?room) free ? ?g))
:fffect (and (carry ?r ?obj ?g) (not (at ?obj ?room)) (not (free ?r ?g))))
20
21
22
(:action drop
23
: :araameeers (?r robot ?obj obj ?room room ?g gripper)
precondition (and (carry ?r ?obj ?g) (at-robby ?r ?room)) efffect (and (at ?obj ?room)
26
(free ?r ?g)
27
(not (carry ?r ?obj ?g)))))
Listing I: Grippers domain PDDL [15
(define (problem gripper- -2-3-4) d:domiin gripper-stripp (:objects robot1 robot2 robot
rgripperi 1gripperi rgripper2 1gripper2 - gripper room1 room2 room3 room
ball1 ball2 ball3 ball4 obj)
(:init
(at (at-robby robot1 room2)
14

---

(free robotl rgripper1)
10
(free robot1 1gripper1)
11
(at-robby robot2 room3)
12
(free robot2 rgripper2)
13
(free robot2 1gripper2)
(at ball1 room3)
15
(at ball2 rooml)
16
(at ball3 room1)
17
(at bal14 room3)
18
19
(: (:goal
20
(and
(at ball1 room2)
(at ball2 room2)
(at ball3 room3)
24
(at ball4 room3)
25
26
21
Listing Grippers problem PDDL.
You control two robots, each equipped with left and right gripper, capable of moving objects (balls) between different rooms.
Initially:
Robot1 is room2 and both its grippers (rgripperi and 1gripper1) are free. Robot2 is in room3 and both its grippers (rgripper2 and 1gripper2) are free. Ball1 and Bal14 are in room3.
Bal12 and Ball3 are in room1.
Your goal is to achieve the following configuration
10
Ball1 must be moved to room2.
Bal12 must be moved to room2.
Ball3 must remain in room3.
13 Bal14 must remain in room3. Listing 3: Grippers problem natural language.
(define (problem gripper-2-3-1
domani gripper -neersttin strips) (:objects 1gripperi 1gripper2 rgripperi rgripper2 gripper ball1 ball2 bal13 ball4 obj robot1 robot2 robot room1 room2 room3 room)
(:init
(:goal (and )
Listing Grippers problem template PDDL..
The gripper domain involves world with multiple rooms, robots, and objects (balls). Each robot has two grippers
15

---

that can be used to pick up and drop objects. The goal is to move objects from their initial locations to the desired goal locations using the robots and their grippers.
The domain includes three actions:
1. move: This action allows robot to move from one room to another. The precondition is that the robot must be in the starting room. The effect is that the robot is no longer in the starting room and is now in the destination room.
2. pick: This action allows robot to pick up an object using one of its grippers. The preconditions are that the object and the robot must be in the same room, and the specified gripper must be free (not holding any object). The effect is that the robot is now carrying the object with the specified gripper, the object is no longer in the room, and the gripper is no longer free.
3. drop: This action allows robot drop an object it is carrying in specific preconditions are that the robot must be carrying the object with the specified gripper and the robot must be in the specified room. The effect is that the object is now in the room, the gripper is free, and the robot is no longer carrying the object with that gripper.
using of its grippers. The
Listing Grippers domain natural language.
(define (domain gripper-strie
(:requirements : :strips :typing)
(:types room obj robot gripper) (:predicates)
(:action move
:parameters (?r robot ?from ?to room)
:precondition ()
:effect ())
10
11
(:action pick
:parameters (?r robot ?o -bj ?room room ?g gripper)
precondition ()
14
:effect ())
15
16
(:action drop
17
:aarameters (?r robot ?o -bjj ?room room ?g gripper) :precondition ()
18
19
:effect ()))
Listing 6: Grippers domain PDDL template.
(move robot2 room3 room1)
(pick robot2
ball2 room1 1gripper2)
(nove robot2 room1 room2)
(drop robot2 bal12 room2 1gripper2)
16

---

Cutoff
Frequency
60
40
20
0
100
10a
10a
10a
10a
10a
Number Lines
Figure 5: Historgram of average number of lines of domains in
(move robot1 room2 room1)
(pick robot1 bal13 room1 1gripperi)
(move robot1 room1 room3)
(pick robot1 ball1 room3 rgripper1)
(drop robot1 ball3 room3 igripperi)
10
(move roboti room3 room2)
11 (drop robot1 ball1 room2 rgripper1)
Listing 7: Grippers problem plan example.
A.2 Criticality of predicate design.
Here, we give an example on the delicacy of predicate design. Consider the Grippers environment, where each robot has two grippers: left gripper and right gripper. In our experiments, one of the main predicates that the LLM incorrectly generates is the free predicate (see Listing [8. This predicate keeps track of whether gripper is free or not. Therefore, at first sight, (free ?g gripper) seems a natural choice to show a particular gripper not occupied and hence capable picking ball. However, when designed this way. in contrast to (free ?r robot ?g gripper) (missing the robot argument), this small detail causes the final domain to be entirely wrong! The reason is that there would no longer be any association between robot and its two grippers. Therefore, on the incorrect domain, one robot will able to pickup an object with the gripper of another robot! In fact, observe that this incorrect design for the free predicate, is the reason behind the failure of the "P&D Chain" " method in Table 2
We provide one more example from the Barman environment, illustrating the criticality of predicate design. The Barman environment involves actions related to manipulating containers (e.g., shot glasses, shakers) to prepare and serve drinks using various ingredients. One of the key predicates used in the domain is (used ?c container beverage has been used in a specific container. This is important for actions like refilling cleaning, where knowing the specific beverage type is essential to ensure conformation to the environment rules (e.g.& container can be refilled only with the beverage that it already had, otherwise, it needs to be cleaned first). However, we have observed that when the LLM generates the domain, it sometimes (used ?c container).
beverage), which keeps track of which
mistakenly omits the beverage argument, simplifying the predicate At first glance, this might seem like a harmless simplification, as the container usage is still tracked. However, this change results in significant problems in the overall domain behavior. Since the beverage is no longer specified, the domain can no longer differentiate between containers used for different types of beverages. This leads situations where container that has already been used for one beverage can be incorrectly treated as if it can hold another beverage without requiring proper cleaning resetting actions. Such a mistake can cause the final domain to generate invalid plans, as the planner will fail to ensure that containers are used properly with respect to their contents, leading cascading errors in tasks like mixing drinks, cleaning containers, or pouring from shakers.
(define (problem gripper -2-3--2-33 -2-3-4) d:domain gripper-stripg
(:objects robot1 robot2 robot rgripperi 1gripperi rgripper2 1gripper2 gripper
17

---

room1 room2 room3 room
ball1 ball2 bal13 bal14 obj )
(:init
(at-robby robot1 room2)
(free rgripper1) ; Correct: (free robot1 rgripper1)
(free 1gripper1) Correct: (free robot1 lgripper1)
(at-robby robot2 room3)
12
(free rgripper2) Correct: (free robot2 rgripper2)
13
(free 1gripper2)
Correct: (free robot2 Igripper2)
14
(at ball1 room3)
15
(at ball2 rooml)
16
(at bal13 room1)
17
(at bal14 room3)
(: gogoll
(and 20
21
(at ball1 room2)
22
(at ball2 room2)
23
(at ball3 room3)
24
(at ball4 room3)
Listing 8: Incorrect generated grippers problem PDDL. The free predicate has only one parameter.
B
Implementation Details
In this section, we explain our implementation details.
B.1 One-shot prompting
To generate PDDL files (problem PDDL and domain PDDL), we always include one-shot example prompt from the BlocksWorld environment. This environment is concise easy enough to fit into context, and explanatory enough to demonstrate example to the LLM for better output stecrability. This includes problem generation, domain proposal, and problem refinement. For instance, when prompting the LLM to generate problem translation from natural language, e.g. LLM(pNL.), we also prompt the LLM with an example from Blocksworld.
B.2 P&D Tree with Domain Proposal
As discussed in A.2 predicate design is challenging. Therefore, in one variant of our method, which we call "P&D Tree DomProp".. we propose for the LLM first draft domain proposal, then generate problem PDDL based on the predicates found in the draft. This way, the LLM first generates domain-avee aware predicates, then generates the problem PDDL. Formally, line one in Algorithm [ will be changed to the following two lines:
(11 (22)
j(np)
LLM,, (dwa.)
p(0) + LLM,( (d .PNL) for all S i iii
where the problem PDDL is generated by first creating domain proposal.
B.3 Domain Refinement Strategy
Refinement Interface. For the domain refinement stage, in our early experiments we observed that prompting the LLM to regenerate the domain results into redundant output generation and
18

---

Problem Translation
Domain Proposal Draft
P&D Chain
P&D Tree
P&D Tree Domain Proposal
Figure 6: Overview of our method with domain proposal. To generate problem PDDL, the LLM first drafts domain proposal to find suitable predicates for the problem PDDL. Then, the draft is discarded, and the domain refinement stage starts.
more importantly, sometimes modifies incorrect parts of the domain. For instance, the LLM had high tendency towards changing the action interface signature, despite the instructions explicitly mentioning not to change the signature. As such, we provide a python interface for the LLM to modify domain. The interface provides the LLM with the following two functions:
add add_ uutioorrupdat_peeddcteee :redicates: List
modify_action aetion_name:: str,
preppoccoodii [ist
new sew_effects: List[str]
The first function adds predicates to the already created predicates, and the second one modifies the preconditions and effects of particular action. Guan et al. [9] use similar approach where they generate the domain PDDL one action at a time, and gradually create predicates. However, our python function interface allows for more flexibility, such as convenient implementation well as enabling the LLM to modify an action several times, or introduce predicates between reasoning steps.
Domain Rating. Our main domain rating originates from the EW metric. When generating domain refinement strategies, the LLM may make mistakes hence failing before even the EW metric could be computed. For instance, the modification may invalid, containing syntax error, or failing to fill parts of the template. To facilitate incorporating these into the EW metric strategy, we create the following rating system for each domain refinement modification:
Description
Rating
Exploration Walk Executable
S EW Score S 1
No initial action possible
Invalid domain modification (e.g., undefined predicates)
Domain sanity check failure (e.g., empty effect list)
Invalid domain modification
No domain modification
5
19

---

when the EW metric is perfect (i.e., equals 1.0), we also run the planning on the environment (i.e., evaluate V.p(.)) and stop early the plan is valid.
User:
Incorrect. The environment returned the following error:
Sampled a set of consecutive random actions from the ground truth environment, but the actions are not executable in the generated environment.
Executing the following actions sequentially the
environment:
(move pos-2-0 pos-3-0 n0)
(move pos-3-0 pos-2-0 n0)
(create-block pos-2-0)
(move pos-2-0
poa-3-0 n0)
10 (move pos-3-0
pos-2-0 n0)
u1
block pos-2-0 pos-1-0 n0 n1)
block pos-2-0 pos-1-0 m1 n0)
(place-block pos-2-0 pos-2-1 n0 n1)
14 blecove-block pos-2-0 pos-2-1 n1 n0)
15 (place-block pos-2-0 pos-3-0 no n1)
16 (create-block pos-2-0)
11 (place-block pos-2-0 pos-2-1 n0 n1)
18 (create-block pos-2-0)
block pos-2-0 pos-1-0 no m1)
(create-block pos-2
(move-up pos-2-0 n0 pos-1-0 n1)
22
23
Result: Error when executing the action (move-up pos 22- no pos-1-0 n1).
24
Current state: The robot is position pos-2-0.
25
This action is not executable the environment.
Please reason about the issue with your generated code. The current domain pddl is as follows:
28
29
'pddl
90
[Current Generated PDDL]
In your response, please generate code to fix the issue.
34
35 Assistant:
36 seems that there is an issue with the 'move-up' action precondition. The precondition for move-upp should check that the destination position "?hto" is one less than the starting position 'Thfrom', not the other way around. This is because the robot is moving to higher position, the destination height should be greater than the starting height.
37
38
Similarly, the precondition for 'dove should check that the destination position '?hto' is one less than the starting position '?hfrom', indicating that the robot is moving to lower position.
20

---

40 Let correct the preconditions for 'move-up up and 'move down
41
42
"' "python
43
[Python Domain Refinement Function Calls]
With these corrected preconditions, the 'move-up" and 'move-down actions should now accurately reflect the natural language description of the Termes domain, and the error should be resolved.
Listing 9: Example of domain feedback from the Termes environment, where the LLM output refinement results into a correct domain.
B.4 Further experiment detaíls
LLM calls per task. For each task in Algorithm 1 the overall complexity of LLM calls is O( Dd x Cmaz + N). The complexity of domain generation O(p X Cmax X n2a). This because at first, Bp problem candidates are generated and for each problem candidate the algorithm goes through refinement procedure (lines I and 2 of Algorithm[] I. The refinement is a tree with depth Cmax (where Omax is the maximum number of refinement turns) (line 5), and each level of the tree, one node is expanded with children (where na is the number of domain refinement candidates) (line 6), which leads to On X Cmax na)) complexity. Once the domain is ready, the complexity of task generation for N tasks is O since for each task we only call the LLM once to get a problem translation.
Number of successful seeds. In Table we report the results analysis, we report the number of seeds a domain was successful in successfully generating a correct domain. The number of seeds that succeed in generating correct domain for the Termes, Movie, Miconic, Grippers, Hiking, Grippers-ood, and Floortile, are 4, 3, 3, 3, 2, 1, respectively.
four seeds. To provide further
C Natural Language Description Generation
To generate natural language description of domains, problems, and environment states, we use the following strategies:
Domain:
We a few-shot translation strategy. We first pick three diverse environments of "Grippers", "Childsnack", and "Termes" to manually (with assistance of GPT-4) curate domain translation. Then, we use these three domains three-shot in-context examples to translate the rest of domains. The example prompt is provided in Listing I0
Problems:
We use a similar few-shot translation strategy for problem translation. We first pick two diverse environments of "Termes" and "Satellite" for problem two-shot problem translation. Once one problem from target domain translated, we use that problem translation as in-context example translate the rest of the problems. This step crucial to ensure all problems from the same domain are translated in a consistent manner. The example prompt is provided in Listing
Natural Language Predicate Description: To generate natural language description of states, we generate python files for each domain, with one function produce natural language description of predicates for state description. The example prompt is provided in Listing 12
Your task is to translate PDDL files into natural language. Ensure that the resulting text covers natural language description of its actions, their preconditions, and effects.
21

---

DO NOT translate the problem PDDL files, only use problem PDDL to understand the domain. ALWAYS wrap your code in the appropriate markdown syntax.
Two examples are provided below.
Q:
Domain PDDL:
'p'ddl
(define (domain gripper-stripg (:requirements :strips :typing)
10
(:types room obj robot gripper)
n
(:predicates (at-robby ?r robot ?x room)
12
(at ?o obj ?x room)
13
(free ?r robot ?g gripper)
(carry ?r robot ?o obj ?g gripper))
15
(: acction move
16
:parameters (?r robot ?from ?to room) preconnitionn (and (at-robby ?r ?from)) reffect (and (at-robby ?r ?to) (not (at-robby ?r ?from))))
17
18
19
20
(:action pick
parameters (?r robot ?obj obj ?room room ?g gripper)
22
precondition (and (at ?obj ?room) (at- ?r ?room) (free ?r ?g))
23
effect (and (carry ?r ?obj ?g) (not (at ?obj ?room))
24
25
(not (free ? ?g))))
(:action drop
parameters (?r robot ?obj obj Proom room ?g gripper)
28
preconnition (and (carry ?r ?obj ?g) (at- ?r ?room))
29
reffect (and (at ?obj ?room)
30
(free ?r ?g)
(not (carry ?r ?obj ?g)))))
Problem PDDL:
34 'pddd
35
(define (problem xripper-2-4-0
36
domaiin gripper-strips)
37
obojects robot1 robot2 robot
38 rgripperi 1gripperi rgripper2 1gripper2 gripper room1 room2 room3 room4 room
40
ball1 ball2 ball3 bal14 ball5 ball6 obj)
41 i:ini
42
(at-robby robot1 room2)
43
(free robot1 rgripper1)
44
(free robot1 1gripper1) rab-robby robot2 room3)
45
(free robot2 rgripper2)
(free robot2 1gripper2)
88
(at ball1 room3)
49
(at ball2 room1)
50 (at bal13 room3)
22

---

(at bal14 room2)
52
(at bal15 room4)
53 (at bal16 room4)
54
55 (:goal
56 (and
(at ball1 room4)
(at ball2 room1)
59
(at ball3 room1)
60
(at ball4 room2)
61
(at bal15 room1)
62
(at bal16 room1)
63
66 <c.
67 A:
68 c markdowwn
69 The gripper domain involves a world with multiple rooms, robots, and objects (balls). Each robot has two grippers that can be used to pick up and drop objects. The goal is to move objects from their initial locations to the desired goal locations using the robots and their grippers. 70 The domain includes three actions:
n 1. move: This action allows a robot to move from one room to another. The precondition starting room. The effect that the robot is no longer in the starting
that the robot must be the
and is in the destination room.
2. pick: This action allows robot to pick up object using one of its grippers. The preconditions are that the object and the robot must be in the same room, and the specified gripper must be free (not holding any object). The effect is that the robot now carrying the object with the specified gripper, the object no longer in the room, and the gripper is longer free.
3. drop: This action allows robot to drop object it is carrying in specific room using one of its grippers. The preconditions are that the robot must be carrying the object with the specified gripper and the robot must be in the specified room. in the room, the gripper is free, and the robot is no longer carrying the object with that gripper.
effect is that the object is now
Q:
76
Domain PDDL:
77
''pddl
78
(define (domain child-snack
79
(:requirements :typing :equality)
80
t:yye child bread-portion content tochent-portion portion sandvich tray place)
(: constants kitchen place)
(: predicates (at_kitchen_bon bread ?b -oradonoorton)
83
(at_ content ?c content
81
wer.kitccennsaadviiee ?s sandwich)
23

---

(no_gluten_bread ?b bread portion)
86
(no_gluten_content ?c content -portion)
87
(ontray ?s sandwich ?t tray)
88
(no_gluten_sandvie ?s sandwich)
89
(allergic_gluten ?c child)
(not_allergic.glun
?c child)
(served ?c child)
(waiting ?c child ?p place)
93
(at ?t tray ?p place)
94
(notexist ?s sandwich)
95
96
(: action moke_sandrich.nogglutee
97
parameters (?s sandwich ?b bread -potioon ?c
content-portion
:precondition (and Sat_kitchen_bread ?b)
99
(at_kittche,connent ?c)
100
(no_gluten_bread ?b)
101
(noggluten_conten, ?c)
102
(notexist ?s))
:effect (and
(not (at_kitccen_bbead ?b))
(not (at_kitccen_conter ?c))
(at_kitcchn_saadvion) ?s)
107
(no_gluten_sandviee ?s)
108
(not (notexist ?s))
109
)
110
acatioon make_sandvich
parameters (?s sandwich ?b Pread-portion ?c
content-porticn
precondition (and brr_ ?b)
113
(at_kitchen_contee) ?c)
114
(notexist ?s)
115
116
:effect (and
(not (at_kitcchn_bbead ?b))
(not (at_kitchen_ content ?c))
kat_ ?s)
(not (notexist ?s))
121
)
122
(: acction put_on_tray
123
parameters (?s sandvich ?t tray)
precondition (and wtw.kkichhenssandvec.. ?s)
(at ?t kitchen))
:effect (and (not (at_kitchhen ?s))
128
(ontray ?s ?t)))
129
(: actioo serve_saadsich..o_guuu
130
parameters (?s sandwich ?c child tray ?p - -place)
precondition (and
(allergic_gluten ?c)
(ontray ?8 ?t)
(waiting ?c ?p)
135
(no_gluten_san sandwich ?s)
136
(at ?t ?p)
24

---

138
efffect (and (not (ontray ?s ?t))
139
(served ?c)))
140
(: actioo serve_sandwich
141
parameters (?s sandwich
child tray ?p place) precondition (and (not_allergic_ ?c)
(waiting ?c ?p)
(ontray ?s ?t)
145
(at ?t ?p))
146
:effect (and (not (ontray ?s ?t))
147
(served ?c)))
action move_tray
parameters (?t tray ?p1 ?p2 place) precondition (and (at ?t ?p1)) :effect (and (not (at ?t ?p1))
152
(at ?t ?p2)))
153
154
155
<C.
Problem PDDL:
'pdddl
child-snack task with 6 children and 0.4 gluten factor
constant factor of 1.3
160
random seed: 234324
161
(define (problem snacks domain child-snack:
162
o:obects
child1 child2 child3 child4 child5 child6 child
bread1 bread2 bread3 bread4 bread5 bread6 bread contenti content2 content3 content4 content5 content6
content-portion tray1 tray2 tray
167
168
table1 table2 table3 place
169
sandw1 sandw2 sandæ3 sandw4 sandw5 sandw6 sandæ7 sandw8 sandwich
(:init (at trayl kitchen) (at tray2 kitchen)
172
173
174
(at_kitchen.breal bread1)
175
(at.kiitchen_bred bread2)
(at_kittchen.brad bread3)
(at_kitcche__bead bread4)
(at_kiitchen_bread bread5)
179
(at_kitcche_bbeed bread6)
180
(at_kiitchen_contentt contenti)
181
(at_kittchen.content content2)
182
(st_kkitccee_coontentt
content3)
fat_Rhitchhen.content
content4)
lat_kittchen.content content5)
lat_kittchen.coneett content6)
(no_glutoe__bead bread2)
187
(no_gluten_bread bread5)
188
(no_gluten,content content3)
25

---

(no_glutto_conttnt content6)
190
(allergic.gluten child1)
191
(allergic.gluten child5)
192
(mot_alleergc..gluten child2)
193
(not_alllergic_gluten child3)
(not_allergic.glutee) child4)
(not_alleeggc.glluten child6)
(waiting child1 table2)
(vaiting child2 table1)
198
(vaiting child3 table3)
199
(waiting child4 table2)
200
(waiting child5 tablel)
201
(waiting child6 table1)
(notexist sandw1)
(notexist sandw2)
204
(notexist sandw3)
205
(notexist sandw4)
206
(notexist sandw5)
207
(notexist sandw6)
(notexist sandw7)
(notexist sandw8)
(:
(.goal (and
212
213
(served child1)
214
(served child2)
215
(served child3)
(served child4)
(served child5)
(served child6)
219
220
221
222
A:
mmmakdoown
The child-snack domain is designed to plan the preparation and serving of sandwiches to children, some of whom have gluten allergies. The domain includes actions for making sandwiches with and without gluten, placing sandwiches on trays, moving trays between places, and serving the sandwiches to the children.
The actions defined in this domain include: 1. make saaaee__andvichno__glle gluten-free sandwich. The preconditions for this action are that there must be gluten-free bread and gluten-free content available in the kitchen, and the sandwich must not already exist. The effect this action is that the gluten-free bread and content the kitchen, and gluten-fres sandwich is now available in the kitchen.
This action is for making a
no longer available in
2. make_sandvich: This action is for making regular sandwich that may contain gluten. The preconditions are that there must be bread and content available in the kitchen, and
26

---

the sandwich must not already exist. The effect of this action is that the bread and content are no longer available in the kitchen, and a sandwich is now available in the kitchen.
229
3. put_on_tray: This action for placing sandwich on a tray. The preconditions the kitchen and the tray must be at the kitchen. The effect is that the sandwich is no longer in the kitchen and is now on the tray.
that the sandwich must be in
230
4. sorve_aandrich.no.gluteee gluten sandwich The preconditions are that the child is allergic to gluten, waiting
This action is for serving a child who is allergic to gluten.
place, and there is a gluten-free sandwich on tray at the same place. The effect is that the sandwich is no longer on the tray and the child is served.
231 5. serve_sandvich: This action is for serving a sandwich to a child who is not allergic to gluten. The preconditions are that the child not allergic to gluten, waiting at place, and there is sandwich on a tray at the same place. The effect is that the sandwich is no longer the tray and the child is served.
6. move_tray: This action allovs tray to be moved from one place to another. The precondition is that the tray is at the starting place. The effect is that the tray is no longer at the starting place and is now at the destination place.
Q:
Domain PDDL:
" 'pddl
236
237
(define (domain termes)
(: requirements :typing :negative- ppeccoddiioonss))s
238
239
types
numb object
position object
243
(: predicates
244
(height ?p - position ?h numb)
245
(at ?p - position)
246
bhas-block)
static predicates (SUCC ?n1 numb ?n2 numb)
250
(NEIGHBOR ?p1 position ?p2 position) (IS-DEPOT ?p position)
351
252
253
(: action move
254
parameters (?from position ?to position ?h numb) precondition (and
(at ?from)
258
(NEIGHBOR ?from ?to)
259
(height ?from ?h)
27

---

(height ?to ?h)
261
262
efffect (and
263
264
(not (at ?from))
(at ?to)
268
(: actioo move-up
269
parameters (?from position Phfrom numb ?to position ?hto numb)
270
precondition
271
(and
(at ?from)
(NEIGHBOR ?from ?to)
274
(height ?from ?hfrom) (height ?oo ?hto)
275
276
(SUCC ?hto ?hfrom)
277
278
:fffect (and
(not (at ?from)) (at ?to)
282
283
294
(: action move -own
285
:parameters (?from position ?hfrom numb ?to position ?hto numb)
precondition (and
288
(at ?from)
289
(NEIGHBOR ?from ?to)
290
(height ?from ?hfrom)
291
(height ?to ?hto)
(SUCC ?hfrom ?hto)
eeffect (and
295
296
(not (at ?from))
297
(at ?to)
208
299
(: action place-bloct parameters (?rpos position ?bpos position ?hbefore numb ?hafter numb) precondition (and
302
303
304
(at ?rpos)
(NEIGHBOR ?rpos ?bpos)
(height ?rpos ?hbefore)
(height ?bpos ?hbefore)
(SUCC ?hafter ?hbefore)
309
(has-block)
310
(not (IS-DEPOT ?bpos))
28

---

312
reffect (and
313
314
(not (height ?bpos ?hbefore))
315
(height ?bpos ?hafter) (not (has-block))
319
(: acction remove-block
320
parameters (?rpos position ?bpos position ?hbefore numb ?hafter numb) precondition (and
321
(at ?rpos)
(NEIGHBOR ?rpos ?bpos)
325
(height ?rpos ?hafter)
326
(height ?bpos ?hbefore)
327
(SUCC Thbefore ?hafter)
328
(not (has-block))
eeffect (and
(not (height ?bpos ?hbefore))
333
(height ?bpos ?hafter) bhackblock)
334
335
336
(: actiion create-block
parameters (?p position)
precondition (and
340
341
(at ?p)
342
(not block))
343
(IS-DEPOT ?p)
efffect
(and
347
(hac-block)
348
349
350
action destroy-bles -bookk
351
parameters (?p position)
precondition (and
354
(at ?p)
355
(has-block)
356
(IS-DEPOT ?p)
357
358
:effect
(and
(not (has -aa-biie block)
)
362
363
29

---

CCC
365
Problem PDDL:
366
'dddl
367
(define (problem prob)
368
(: dommain termes)
Initial state:
0 ROD
0
372
0
373
Goal state:
374
375
376
Maximal height: 1
(: (bojects
379
n0 numb
380
n1 numb
381
pos - - position
382
pos
position
pos - position
pos pos-1- 1-0 position
pos-1-1 position
pos-1-2 position
387
pos-2-0 position
388
pos-2-1 position
389
pos-2-2 position
390
i:ini
(height pos-0-0 n0)
(height pos-0-1 n0)
394
(height pos-0-2 n0)
395
(height pos-1-0 n0)
396
(height pos-1-1 n0)
397
(height pos-1-2 m0)
(height pos-2-0 n0)
(height pos-2-1 n0)
(height p2s n0)
401
(at pos -2-)
402
(SUCC m1 n0)
403
(NEIGHBOR pos-0-0 pos-
404
(NEIGHBOR pos-0-0 10s-0-1)
(NEIGHBOR pos-0-1 p08-1-1)
(NEIGHBOR pos-0-1 pos-0-0)
(NEIGHBOR pos-0-1 pos-0-2)
408
GHEIGHBOR pos-0-2 pos-i-2)
409
(NEIGHBOR pos-0-2 -os-0-1)
410
(NEIGHBOR pos -10 00s-0-0)
411
(NEIGHBOR pos -0s-
(NEIGHBOR pos pos-
(NEIGHBOR pos pos-0-1)
(NEIGHBOR pos-1-1 pos p28-2-1)
(NEIGHBOR pos-1-1 pos pos-1-0)
416
(NEIGHBOR pos-1-1 pos-1-2)
417
(NEIGHBOR 1-8 pos -0-
30

---

(NEIGHBOR pos-1-2 pos-2-2)
419
(NEIGHBOR pos-1-2 pos-1-1)
420
(NEIGHBOR pos-2-0 pos-1-0)
421
(NEIGHBOR poa-2-0
pos-2-1)
422
(NEIGHBOR pos-2-1 pos-l-1)
(NEIGHBOR pos-2-1 pos-2-0)
(NEIGHBOR pos-2-1 pos-2-2)
(NEIGHBOR pos-2-2 pos pos-1-20
(NEIGHBOR pos-2-2 pos 22-1)
427
(IS- (IS-DEPOT pos 2-00
428
429
g:goal
(and
(height pos-0-0 m0)
(height pos-0-1 n0)
433
(height pos-0-2 n0)
434
(height pos-1-0 n0)
435
(height pos-1-1 m1)
436
(height pos-1-2 n0)
(height pos-2-0 n0)
(height pos-2-1 n0)
(height pos-2-2 n0)
(not (hac-block))
441
442
443
444
A:
markkoow
The Termes domain is a planning domain that simulates the behavior of robotic agents (inspired by termites) that can move around, pick up blocks, stack them to build structures, and remove blocks from structures. The domain includes actions for moving the robot, placing and removing blocks, and creating and destroying blocks depot.
The actions defined in this domain include:
1. move: This action allows the robot to move from one position to another at the same height. The preconditions are that the robot starting position positions have the same height. The position, and both effect is that the robot is longer at the starting position and is now at the destination position.
the starting position, the neighbor to the destination
450 2. move-up: This action allows the robot to move from a lower position to a neighboring higher position. The preconditions are that the robot is at the starting position, the starting position is a neighbor to the destination position, the starting position has certain height, and the destination position's height is one less than the starting position's height. The effect is that the robot is no longer at the starting position and is now at the destination position.
31

---

3. move-down: This action allows the robot to move from higher position to a neighboring lower position. The preconditions are that the robot is at the starting position, the starting position is a neighbor to the destination position, the starting position has certain height, and the destination position's height is one less than the starting position's height. The effect is that the robot is no longer at the starting position and is now at the destination position.
452 4.
This action allovs the robot to place a block at a neighboring position, increasing the height of that position by one. at a position next the block position, both positions have the same height, the robot has block, and the block position is not depot. The effect is that the height of the block position is increased by one, and the robot no longer has a block.
preconditions are that the robot
453
5. remove-block: This action allows the robot to remove a block from a neighboring position, decreasing the height of that position by one. The preconditions are that the robot is at position next to the block position, the robot's position is position, and the robot does not have a block. The effect is that the height of the block position is decreased by one, and the robot now has a block.
height unit higher than the block
454
breate-block: This action allows the robot to create a block at a depot. The preconditions are that the robot at the depot and does not have block. The effect is that the robot now has block.
7. destroy bbockk: This action allows the robot to destroy a block at a depot. The preconditions are that the robot is at the depot and has a block. The effect is that the robot no longer has a block.
Q:
Domain PDDL:
'pddl
[Target Domain PDDL Code]
461
462
Problem PDDL:
463
pddl
[Target Problem PDDL Code]
Listing 10: Domain lack-translation prompt template, with domain PDDL and problem PDDL placeholders for each target domain. Some PDDL credit comments are omitted for clarity.
Your task is to translate problem PDDL files into natural language. Ensure that the resulting description covers all initial state and goal conditions.
DO NOT be lazy in your response, be extremely precise in your descriptions such that all conditions are covered your description and there If you do not find any common rule about some conditions, list all of them.
ambiguity in your description.
32

---

For the initial conditions, start with "Initially:", and for the goal conditions, start with "Your goal is to". ALWAYS wrap your code in the appropriate markdown syntax. Two examples are provided below. Q:
Domain Description:
markkoown
10 The Termes domain is a planning domain that simulates the behavior of robotic agents (inspired by termites) that can move around, pick up blocks, stack them to build structures, and remove blocks from structures. The domain includes actions for moving the robot, placing and removing blocks, and creating and destroying blocks depot.
The actions defined in this domain include:
12 1. move: This action allows the robot to move from one position to another at the same height. The preconditions are that the robot is at the starting position, the starting position position, and both positions have the same height. The effect is that the robot is longer at the starting position and is now at the destination position.
neighbor to the destination
13 2. move-up: This action allows the robot to move from a lower position to a neighboring higher position. The preconditions are that the robot is at the starting position, the starting position is a neighbor to the destination position,
starting position has certain height, and the destination position's height is less than the starting position's height. The effect is that the robot is no longer at the starting position and is now at the destination position.
14 3. dove- This action allows the robot to move from a higher position to a neighboring lower position. The preconditions are that the robot is at the starting position, the starting position is neighbor to the destination position, the starting position has certain height, and the destination position's height is one less than the starting position's 's height. The effect is that the robot is no longer at the starting position and is now at the destination position.
is 4. place-block: This action alloss the robot to place a block at a neighboring position, increasing the height of that position by one. The preconditions are that the robot is at a position next to the block position, both positions have the same height, the robot has a block, and the block position is not a depot. The effect is that the height of the block position is increased by one, and the robot no longer has a block.
16 5. block: This action allows the robot remove a block from neighboring position, decreasing the height of that position by one. The preconditions are that the robot is at a position next to the block position, the robot's position is one height unit higher than the block position, and the robot does not have a block. The effect
33

---

is that the height of the block position is decreased by one, and the robot now has a block.
n 6. create-blocke This action allovs the robot to create a block at a depot. The preconditions are that the robot at the depot and does not have block. The effect is that the robot now has block.
7. destroy-block: This action allows the robot to destroy a block at a depot. The preconditions are that the robot is at the depot and has block. The effect is that the robot no longer has a block.
19 ...
20 Problem PDDL:
21
" "pdddl
(define (problem prob)
d:mmmiin termes)
24
Initial state:
25
ROD
26
21
28
Goal state:
0
0
0
32
Maximal height: 1
33
obbects
34
numb
35
numb
36
pos - position
pos pos-0-1 position
38
pos pos-0-2 -022 position
39
pos - position
40
pos-1-1 position
41
pos-1-2 position
42
pos-2-0 position
pos-2-1 position
p-s-2-2 position
46
(: finit
47
(height pos-0-0 n0)
48
(height pos-0-1 m0)
49
(height pos-0-2 n0)
50
(height pos-1-0 n0)
(height pos-1-1 n0)
(height pos-1-2 m0)
53
(height pos-2-0 n0)
54
(height pos-2-1 n0)
55
(height -2s-2-2 n0)
56
(at -2s- (SUCC n1 n0)
57
(NEIGHBOR pos-0-0 pos-1-0)
(NEIGHBOR pos-0-0 pos-0-13
(NEIGHBOR 00s-0-1 pos-1-1)
61
(NEIGHBOR pos-0-1 pos -00-
62
(NEIGHBOR pos 0-1 pos -0-
34

---

(NEIGHBOR pos-0-2 pos-1-2)
64
(NEIGHBOR poa-0-2 pos-0-1)
65
(NEIGHBOR pos-1-0 pos-0-0)
66
(NEIGHBOR pos-1-0 pos-2-0)
67
(NEIGHBOR pos-1 pos-l-1)
(NEICHBOR pos 1-11 pos-0-1)
(NEIGHBOR pos-1 pos pos-2-1)
(NEIGHBOR pos-1-1 pos pos-1-0)
71
(NEIGHBOR pos-1-1 pos-1-2)
72
(NEICHBOR pos-1-2 pos-0-2)
73
(NEIGHBOR pos-1-2 p2s-2-2)
74
(NEIGHBOR pos-1-2 pos-1-1)
75
(NEIGHBOR poa-2-0 pos-1-0)
(NEIGHBOR pos-2-0 pos-2-1)
(NEIGHBOR pos-2-1 pos-1-1)
78
(NEIGHBOR pos-2-1 pos-2-0)
79
(NEIGHBOR pos-2-1 pos-2-2)
80
(NEIGHBOR poa-2-2 pos-1-2)
81
(NEIGHBOR
pos 22
82
(IS-DEPOT pos pos-2-0)
(: gogaal
(and
86
(height pos-0-0 n0)
87
(height pos-0-1 n0)
88
(height pos-0-2 m0)
89
(height pos-1-0 m0)
90
(height pos-1-1 m1)
(height pos-1-2 n0)
(height pos-2-0 n0)
93
(height pos-2-1 n0)
94
(height pos-2-2 n0)
95
(not block))ock))
96
100 A:
101
<< markdown
102
You control a robot capable of building structures by moving and manipulating blocks.
The environment consists of grid of positions and two heights, numbered and 1. The robot can move at the same height, move up one height, can also place or renove block at a neighboring position, or create or destroy a block at the depot. A block's height increases by one when placed and decreases by one when removed. Initially:
move down one height. It
the grid have height of 0. The robot is at position pos-2-0, which is the depot The robot does not have a block.
All positions
108
The positions have the folloving neighboring relationships:
109
pos-0-0 neighbors pos-1-0 and pos -0-1
35

---

pos pos-0-1
neighbors pos-1-1, pos-0-0, and pos -0-2
111
pos-0-2 neighbors pos-1-2 and pos
112
pos-1-0 neighbors pos-0-0. pos-2-0, and pos-1-4 -
113
pos-1-1 neighbors pos-0-1, pos-2-1, pos-1-0, and pos -1-2
114
pos-1-2 neighbors pos-0-2, pos-2-2, and pos 10s-1-1
pos-2-0 neighbors pos-1-0 and pos pos-2-1, and is the depot
pos-2-1 neighbors pos-1-1, pos-2-0, and pos-2-2
p-s-2-2 neighbors pos-1-2 and pos -2-1
There is a successor relationship between the numbers n1 and no
119
Your goal is to achieve the following configuration: The height at pos-1-1 needs be
All other positions must remain height 0.
The robot should not have block at the end of the task.
Q:
125
Domain Description:
126
c markkownn
127
The satellite domain is designed model the operation of satellites that can take images of various targets in different modes. Each satellite is equipped with instruments that can be turned used to take images. The domain includes actions for turning the satellite to point at different directions, switching instruments on and off, calibrating instruments, and taking images.
and off, calibrated, and
128
The actions defined in this domain include:
1. turn_to: This action changes the direction the satellite is pointing. The preconditions are that the satellite must be pointing at a previous direction, and both the new and previous directions are valid. The effect is that the satellite is now pointing at the new direction and no longer pointing at the previous direction.
2. switch_on: This action turns on an instrument on board the that the instrument must be board the satellite and there must be power available on the satellite. The effect is that the instrument is powered on, it is no longer calibrated, and the satellite no longer has power available.
satellite. The preconditions
131 3. switch_off: This action turns off an instrument on board the satellite. The preconditions are that the instrument must be on board the satellite and it must be powered on. The effect is that the satellite has power available and the instrument is no longer powered on.
132 4. calibrate: This action calibrates an instrument on board the satellite. The preconditions are that the satellite must be pointing at a calibration target for the instrument, the instrument must be on board the satellite and powered on. The effect is that the instrument is calibrated.
5. take imag_: This action uses instrument board the satellite to take an image in specific mode of a direction the satellite is pointing at. The preconditions are that the satellite must be pointing at the direction,
36

---

the instrument must be calibrated, on board the satellite, support the mode, and be powered on. The effect is that an image of the direction in the specific mode is now available.
Problem PDDL:
'ppddl
(define (problem strips-sat, -x-1
138
(: domain satellite)
139
(: objectt
140
satellite0
instrument0
satellitel
instrument1
instrument2
145
instrument3
146
satellite2
147
instrument4
148
instrument5
instrument6
satellite3
instrument7
satellite4
153
instrument8
154
thermograph2
155
image3
156
infrared1
spectrograph4
infrared0
Star1
160
Star4
161
Star0
162
GroundStation3
Star2
Star5
Planet6
Phenomenon7
167
Star8
168
Phenomenong
169
Star10
170
Star11
Star12
Planet13
Planet14
174
Phenomenon15
175
Planet16
176
Star17
177
Star18
Planet19
inin
(satellite satellite0)
182
(instrument instrumentO)
183
(supports instrumento spectrograph4)
37

---

(calibration_target instrumento Star0)
185
(on_board instrumento satellite0)
186
(power_avail satellite0)
187
(pointing satellite0 Star8)
188
(satellite satellitel)
(instrument instrumenti)
(supports instrumenti infrared0)
(supports instrumenti infraredi)
collibattiontarger instrumenti GroundStation3)
193
(instrument instrument2)
194
(supports instrument2 infrared1)
(supports instrument2 infrared0)
(calibration,target instrument2 Star2)
(instrument instrument3)
(supports instrument3 spectrograph4)
(supports instrument3 infraredi)
199
(supports instrument3 thermograph2)
200
201
(calibration_targee instrument3 Star0)
202
(on_board instrumenti satellitel)
(on_board instrument2 satellitel)
(on_board instrument3 satellitel)
(power_avail satellitel)
(pointing satellitel GroundStation3)
207
(satellite satellite2)
208
(instrument instrument4)
209
(supports instrument4 infraredi)
210
(supports instrument4 image3) (supports instrument4 infrared0)
(calibration_target instrument4 Star2)
(instrument instrument5)
(supports instrument5 thermograph2)
214
(supports instrument5 spectrograph4)
215
316
fealibration_targer instrument5 Star0)
217
(instrument instrument6)
(supports instrument6 infrared0)
(calibration_targee instrument6 GroundStation3)
(on_board instrument4 satellite2)
221
(on_board instrument5 satellite2)
222
(on_board instrument6 satellite2)
223
(pover_avail satellite2)
234
(pointing satellite2 Star4)
225
(satellite satellite3)
(instrument instrument7)
(supports instrument7 image3)
228
(calibration_target instrument7 Star2)
229
(on_board instrument7 satellite3)
230
(power_avail satellite3)
231
(pointing satellite3 Phenomenon9)
232
(satellite satellite4)
(instrument instrument8)
(supports instrument8 infrared0)
(supports instrument8 spectrograph4)
(supports instrument8 infraredi)
236
237
(calibration.targer instrument8 Star2)
38

---

(on_board instrument8 satellite4)
239
(power_avail satellite4)
240
(pointing satellite4 Phenomenon9)
241
(mode thermograph2)
242
(mode image3)
(mode infrared1)
(mode spectrograph4)
(mode infrared0)
246
(direction Star1)
247
(direction Star4)
248
(direction Star0)
249
(direction GroundStation3)
250
(direction Star2)
(direction Star5)
(direction Planet6)
253
(direction Phenomenon7)
354
(direction Star8)
255
(direction Phenomenon9)
256
(direction
Star10)
257
(direction Stari1)
(direction Star12)
(direction Planet13)
(direction Planet14)
261
(direction Phenomenoni5)
262
(direction Planet16)
263
(direction Star17)
264
(direction Star18)
(direction Planet19)
(: goal (and
268
(pointing satellite0 Phenomenon9)
269
(pointing satellitel Star4)
370
(pointing satellite4 Starl1)
271
(have_image Star5 image3)
(have_image Planet6 infraredi)
(have_image Phenomenon7 infraredi)
(have_imal
Star8 image3)
275
(have_image Star10 thermograph2)
276
(have_image Starl1 infrared1)
277
(have_image Planet13 spectrograph4)
278
(have_image
Planet14 thermograph2)
279
(have_image
Phenomenon15 infrared0)
(have_image
Planet16 image3)
Thave_image Star17 infrared0)
282
)
283
281 <.*
285
286
< markdown
You operating constellation of satellites tasked with taking images of various targets in different modes. Initially:
280
There are five satellites (satellited to satellite4) with corresponding instruments (instrument0 to instrument8).
39

---

Each instrument supports specific modes and has a calibration target:
291
InstrumentO supports spectrograph4 and targets Star0.
292
Instrumenti supports infrared0 and infrared1, targeting GroundStation3.
Instrument2 supports infrared1 and infraredo, targeting Star2.
Instrument3 supports spectrograph4, infraredi, and thermograph2, targeting Star0.
295
Instrument4 supports infraredi, image3, and infraredo, targeting Star2.
296
Instrument5 supports thermograph2 spectrograph4,
targeting Star0.
Instrument6 supports infraredo, targeting GroundStation3, Instrument7 supports image3, targeting Star2. Instrument8 supports infrared0, spectrograph4, and infrared1, targeting Star2.
299
300
Instruments are on board their respective satellites, and all satellites have power available.
Satellites are pointing at various directions:
Satellite0 is pointing at Star8.
Satellitel is pointing at GroundStation3.
Satellite2 is pointing at Star4.
305
Satellite3 is pointing at Phenomenon9.
306
Satellite4 is pointing at Phenomenon9.
307
There are various modes (thermograph2, image3, infrared1, spectrograph4, infrared0) and directions (Star1 to Star18, GroundStation3, Planet6, Phenomenon7, Phenomenon9, Planet13, Planet14, Phenomenoni5, Planet16, Planet19). Your goal is to:
309
Point satellited at Phenomenon9.
310
Point satellitel at Star4.
311
Point satellite4 at Starli.
312
Have images of the following targets the specified modes: Star5 in image3 mode.
Planet6 in infraredi mode.
Phenomenon7 in infrared1 mode.
316
Star8 in image3 mode.
317
Star10 in thermograph2 mode.
318
Starl1 in infraredi mode.
319
Planet13 in spectrograph4 mode.
Planet14 in thermograph2 mode.
Phenomenon15 in infrared0 mode.
Planet16 in image3 mode.
323
Star17 in infrared0 mode.
324
To achieve these goals, you will need to turn the satellites to point at the correct directions, switch on and calibrate the necessary instruments, and take images using the calibrated instruments in the supported modes.
Q:
Domain Description:
328
marrkoow
329
[Target Domain Natural Language Description]
40

---

CCC
331
Problem PDDL:
332
" "pddl
333
[Target Problem PDDL Code]
334
A:
Listing 11: Domain trak-translttion prompt template, with domain natural language description and problem PDDL placeholders for each target domain.
Your task is to generate python predicate descriptor for each environment. You are given the natural language description of the domain along with the PDDL code.
Q:
Domain Description:
< markddww
The robot has four actions: pickup, putdown, stack, and unstack. The domain assumes world where there are set of blocks that can be stacked top of each other, an arm that can hold one block at time, and a table where blocks can be placed.
The actions defined in this domain include:
pickup: allows the arm to pick up block from the table if it is clear and the arm empty. After the pickup action, the arm will be holding the block, and the block will longer be on the table or clear.
putdown: allovs the arm to put down block on the table if it is holding a block. After the putdown action, the arm will be empty, and the block will be on the table and clear. stack: allows the arm to stack a block on top another block if the arm is holding the top block and the bottom block is clear. After the stack action, the the top block will be top of the bottom block, and the bottom block will no longer be clear.
will be empty,
10 unstack: allovs the arm to unstack a block from on top of another block if the arm is empty and the top block is clear. After the unstack action, the arm will be holding the top block, the top block will no longer be on top of the bottom block, and the bottom block will be clear.
Domain PDDL: ''ddl
13
14
(define (domain blocksworld-4ope
15
(:requirements :strips)
16
(:predicates (clear ?x)
17
(on-table ?x)
(arm (arm-empty)
(holding ?x)
20
(on ?x ?y))
21
acctiion pickup
22
:parameters (?ob)
:precondition (and (clear ?ob) (on-table ?ob) (arm (arm-empty))
23
24
:effect (and (holding ?ob) (not (clear ?ob)) (not (on table ?ob))
(not (arm-))pty))))
41

---

:: action putdown
27
parameters (?ob)
28
precondition (holding ?ob)
29
:effect (and (clear ?ob) (arm-empty) (on-table ?ob)
30
(not (holding ?ob))))
(: actiion stack
parameters (?ob ?underob)
precondition (and (clear ?underob) (holding ?ob))
34
efffcc (and (arm-empty) (clear ?ob) (on ?ob ?underob) (not (clear ?underob)) (not (holding ?ob))))
35
36
(: actioon unstack
37
parameters (?ob ?underob)
38
precondition (and (on ?ob ?underob) (clear ?ob) (arm empty))
:effect (and (holding ?ob) (clear Punderob) (not (on ?ob ?underob)) (not (clear ?ob)) (not
(arm-enpty))))
41
.c.
42 A:
43
python def desdribe.predicate (epeeidiiaaaeeeeiciitttteeeeeeee (predicate_name, predicate_args)
<
Predicates:
(clear ?x)
48
(on-table ?x)
49
(arm- (arm-empty)
50
(holding ?x)
51
(on ?x ?y)
52
pparam predicate_name: str
param predicate_arge: [urre
:return: Tupre[str, str] (positive, negative)
55
56
(clear ?x)
57
predicate_name "clear": (x,) predicate_args
58
return fBBlock {x} is clear.", fBBlock {x} is not clear." (on-table ?x)
61
elif predicate_name == "on-table":
62
(x,) arag
63
return fBBlock {x} is on the table.", f"Block {x} is not on the table." (arm empty)
64
elif predicate_name == "arm "mrm-empty";
return " "Arm is empty.", "Arm is not empty."
67
(holding ?x)
68
elif predicate_nam == "holding":
69
(x,) predicate_args
70
return f"Arm is holding block (x}.", f"Arm is not holding block {x}."
(on ?x ?y)
elif predicate_name == "on":
73
(x, y) = predicate_args
74
return fBBlock {x} is on block fy).", f"Block (x} is not on block {y}."
42

---

else:
76
raise ValueError (nukno predicate:
{predicate_name]")
77
18
Q:
Domain Description:
'markdown
[Target Domain Natural Language Description]
82
ccc
83
Domain PDDL:
84
'pddl
85
[Target Domain PDDL Code]
A:
Listing 12: Predicate translation python code generation prompt.
43

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims are supported by experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In the conclusion section, we discuss limitations of our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
44

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe our method Algorithm T and Appendix The code will be made publicly available upon publication.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
45

---

Answer: [No]
Justification: The code will be made publicly available upon publication. We explain our method Algorithm For the data, the PDDL files are publicly available, and we provide examples on how to obtain natural language descriptions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
public//uides//odeeeeeee https//ooooouuummsssoooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nips.c//pppp
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our method uses pre-trained large language models, and we provide examples in Algorithm Tn how we prompt the LLMs.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Figure 2b and Figure 4 show standard error. Table 2 does not contain any confidence intervals as the experiments become computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
46

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the experiments section, we reported the number of tokens used create Table
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The setup used in our work does not have direct societal impact as it is centered around generating PDDL code.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
47

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work is a fundamental work and poses no such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the sources from which we use library, data, and code.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
48

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The code will be made publicly available upon publication
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
49


Reviews:
[{"review_id": "XRtwd0czGF", "reviewer": "Reviewer_iKF8", "summary": "This work presents an approach for modeling planning environments via PDDL generation using LLMs and environment feedback, without relying on human intervention. This is achieved by an Exploration Walk (EW) metric to measure domain similarity and guide domain refinement, and an iterative rectifying method that leverages LLMs to generate and refine PDDL domain and problem files. The evaluation of this method is performed alongside baselines on ten different standard planning domains from IPC.", "questions": "1. This work on generating programs for planning problems [https://arxiv.org/abs/2305.11014] may also need to be cited in related work.\n2. The authors may benefit from more sophisticated exploration approaches in this paper [https://arxiv.org/abs/2406.07145] that solve a different problem.", "limitations": "Limitations have been addressed by the authors.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The presentation of the work is quite clear and concise. \n2. The exploration walk method included in the PDDL file correction loop is a rather simple and unsophisticated way to obtain approximate scores for the domain generation process.\n3. A decent amount of experimentation and analysis have been performed and stated in the work.", "weaknesses": "1. Section 4.2 - demonstration of the brittleness of PDDL generation can be made more realistic such as additionally including hallucinated object identifiers or actions or symbols, which are highly probable with LLM-based code format generators.\n2. The PDDL generation with LLMs approach is not as novel and the exploration walk/ environment feedback approach may not be as useful in generating completely new domains from descriptions or making custom modifications to existing domain files. The method does not really differ for rectifying problem files.\n3. There are intrinsic problems with generating domain files from descriptions for domains such as Barman - where the levels of shaker and actions such as clean shot, empty shot do not translate well for LLM-based generation. More analysis and description in this line of argument are necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments. Below, we respond to the weaknesses and questions raised by the reviewer.\n\n**W1: Section 4.2 - demonstration of the brittleness of PDDL generation can be made more realistic such as additionally including hallucinated object identifiers or actions or symbols, which are highly probable with LLM-based code format generators.**\n\nWe already have a qualitative demonstration of the brittleness of PDDL generation in Appendix A.2, where we show a real (common) example where the LLM makes a subtle error in predicate design, and as such, the whole domain produces invalid plans. This is in addition to the quantitive metrics shown in Section 4.2. We will update the writing to highlight this result more. \n\n**W2: The PDDL generation with LLMs approach is not as novel and the exploration walk/environment feedback approach may not be as useful in generating completely new domains from descriptions or making custom modifications to existing domain files. The method does not really differ for rectifying problem files.**\n\nPDDL generation is not novel, and we do not claim it to be our contribution. We should emphasize that the main contribution of our work is to provide a framework to eliminate the need for human feedback to generate PDDL domains and problems and put the first steps towards such automation. We are certainly excited to see extensions of our method to “custom domain modification”, or an exploration-walk-like metric for PDDL problem files, but we believe these are out of the current scope and are interesting directions for future research.\n\n**W3: There are intrinsic problems with generating domain files from descriptions for domains such as Barman - where the levels of shaker and actions such as clean shot, empty shot do not translate well for LLM-based generation. More analysis and description in this line of argument are necessary.**\n\nIndeed predicate design is a crucial part of LLM-based generation, and we already have qualitative analysis for the Grippers environment in Appendix A.2. We observe similar challenges happening in the Barman environment, and kept the Grippers environment example since the environment is simple enough to be illustrative throughout the paper. As per the suggestion of the reviewer, we will add the Barman example to Appendix A.2 for more comprehensiveness. \n\n**Q1: This work on generating programs for planning problems [https://arxiv.org/abs/2305.11014] may also need to be cited in related work.**\n\nWe have already cited and discussed this work in our related work section.\n\n**Q2: The authors may benefit from more sophisticated exploration approaches in this paper [https://arxiv.org/abs/2406.07145] that solve a different problem.**\n\nWe thank the reviewer for pointing out this work. There is a literature on exploration strategies and we will add citations to some papers in that literature including the one mentioned by the reviewer, but the method itself in this suggested reference is for a very different problem. Adaptation to our problem would be highly non-trivial and could warrant a new research effort.\n\nAdditionally, we should stress two points: 1) as we have well-established through empirical results on 10 domains, our current simple EW already works. 2) our approach is a general framework, and components in the current implementation can be improved in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eL375Q9Yhj", "reviewer": "Reviewer_zxRN", "summary": "This work talks about generating PDDL domain and problem files with LLMs. Specifically, it improves existing frameworks, particularly Guan et al. [8], in terms of increasing the degree of automation & eliminating the need for human corrective feedback. The core contribution of this work is the EW score. To compute the score, it only requires access to a set of executable plans and an executability checker (which can be either a simulator or the actual environment). The EW score is used to select sampled domain models given by the LLMs.", "questions": "See the weakness section.", "limitations": "See the weakness section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written, with precise and rigorous wording and formalism.\n- The attempt to reduce the need for human feedback in domain generation is a meaningful and useful step forward. \n- The introduction of the EW score not only forms the foundation of this work but also holds potential for future applications/research (e.g., for evaluation or as a heuristic)", "weaknesses": "1. The domain model sampling/generation is done in a relatively simple way. The feedback message could be more informative than just indicating the inexecutable step or action in a plan.\n\n2. Regarding the structure of the related work section, the distinction between \"intrinsic reasoning\" and  \"external reasoning\" seems unclear to me, especially given that \"with the assistance of basic external tools [14]\" is mentioned under the \"intrinsic reasoning\" subsection. Also, even for the task of PDDL generation,  certain degree of \"intrinsic reasoning\" is needed. Rather than saying \"intrinsic reasoning\", I guess the authors likely meant \"direct plan generation.\"\n\n3. It doesn’t make it clear how much knowledge is provided in the domain NL description. Clarifying on this helps readers understand whether this work leverages LLMs as a knowledge source or a \"translator.\" -- from the examples in appendix, it seems that LLMs are used as the latter in this work (note that I am not saying translation is trivial)\n\n4. While the EM score may be effective for selecting candidate domain models and guiding their generation, its suitability as an evaluation metric is questionable. We know there exists a \"ground-truth\" PDDL and our goal is to fully recover its functionality. This is a binary 0/1 problem. This is not like a generated model with 0.8 avg solve rate is more usable than a model with 0.2 avg solve rate.\n\n5. Also, I think it's important to mention that the avg solve rate can only serve as an approximate measure of the equivalency between two domain models. A 100% avg. solve rate doesn't guarantee model equivalency (but this seems to be an easier-to-compute measure)\n\n6. Assumption 2 is stated in a loose way. An NL description of a domain can be given at different degrees of detail (which correspond to different levels of difficulty in domain PDDL generation).\n\n7. Line 211: I think it's better to say \"as long as PDDL is expressive enough to capture the dynamic / working mechanisms of the environment\" rather than \"the env supports PDDL action interface.\"\n\n8. It's unclear what the takeaway of Sec. 4.2 should be. Firstly, \"plan-not-found\" only accounts for a certain fraction of consequences caused by removing a term or predicate. Other consequences, such as producing invalid plans, can also occur. Secondly, it is well known that obtaining a valid domain model is challenging, even for humans. The authors should better explain the connection between Sec. 4.2 and the other parts of the paper.\n\n9. The authors should give more information on the computational complexity/cost (e.g., time consumption) associated with the calculation of EW score per candidate model.\n\n------------------------\nOverall, I find this manuscript well-written, and the idea can be valuable to the community. Therefore, I am leaning towards recommending acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments. Below, we address the main questions raised by the reviewer:\n\n**W1: The domain model sampling/generation is done in a relatively simple way. The feedback message could be more informative than just indicating the inexecutable step or action in a plan.**\n\nRespectfully, our perspective here differs. The simplicity of our feedback messages is a deliberate choice to showcase the general applicability of our framework. As we improved the solve rate from 29% to 66% with a simple variant of our framework, it shows the impact of the core idea of the framework and a path forward for future innovation. More detailed feedback messages can be useful for future direction among other possibilities enabled by our approach.  \n\n**W2: Regarding the structure of the related work section, the distinction between \"intrinsic reasoning\" and \"external reasoning\" seems unclear to me, especially given that \"with the assistance of basic external tools [14]\" is mentioned under the \"intrinsic reasoning\" subsection. Also, even for the task of PDDL generation, certain degree of \"intrinsic reasoning\" is needed. Rather than saying \"intrinsic reasoning\", I guess the authors likely meant \"direct plan generation.\"**\n\nWe appreciate the reviewer's feedback on the structure of the related work section. To clarify, we will replace the term \"intrinsic reasoning\" with \"direct reasoning.\" This should better capture the distinction we intended to make between different types of reasoning approaches in our study.\n\n**W3: Are LLMs used as knowledge sources or translators?**\n\nThe LLMs are mainly used as translators, however, some elementary general knowledge and reasoning are required for the LLM to be able to come up with correct predicates and preconditions/effects.\n\n**W4: While the EM score may be effective for selecting candidate domain models and guiding their generation, its suitability as an evaluation metric is questionable. We know there exists a \"ground-truth\" PDDL and our goal is to fully recover its functionality. This is a binary 0/1 problem. This is not like a generated model with 0.8 avg solve rate is more usable than a model with 0.2 avg solve rate.**\n\nWhile achieving exact functionality with ground-truth PDDL is ideal, this is often unattainable in hard domain PDDLs, resulting in incompatible output plans (and consequently functionally inequivalent PDDL domains). Therefore, a proxy metric is essential for meaningful comparison. Such proxy metrics are crucial in comparing different models and understanding trends that would have been unpredictable through the primary metric [5]. In Section 4.2, we demonstrate that the EW metric is a suitable proxy for our setting.\n\n**W5: It's important to mention that the avg solve rate can only serve as an approximate measure of the equivalency between two domain models. A 100% avg. solve rate doesn't guarantee model equivalency (but this seems to be an easier-to-compute measure)**\n\nWe will add a sentence to clarify complete task solve rate does not mean exact domain equivalency. However, we should note that it is common practice to test a generated PDDL code using task solve rate (e.g., [1], [2]). More generally, the majority of code generation literature (e.g., Codex [2], AlphaCode [3]) tests the generated Python, or C++ code using unit test cases and decides the final accuracy based on performance on test cases.\n\n**W6: Assumption 2 is stated in a loose way. An NL description of a domain can be given at different degrees of detail (which correspond to different levels of difficulty in domain PDDL generation).**\n\nThis assumption is a part of the input to our framework., The amount of details provided only changes the accuracy, and not the framework itself. Our natural language descriptions maintain a degree of detail comparable to previous works (e.g., [1]). Studying the trade-off between the amount of detail in natural language and accuracy is an interesting area for future work.\n\n**W7: Line 211: I think it's better to say \"as long as PDDL is expressive enough to capture the dynamic / working mechanisms of the environment\" rather than \"the env supports PDDL action interface.\"**\n\nThank you for your constructive feedback. We will change this line as suggested by the reviewer.\n\n**W8: It's unclear what the takeaway of Sec. 4.2 should be. Firstly, \"plan-not-found\" only accounts for a certain fraction of consequences caused by removing a term or predicate. Other consequences, such as producing invalid plans, can also occur. Secondly, it is well known that obtaining a valid domain model is challenging, even for humans. The authors should better explain the connection between Sec. 4.2 and the other parts of the paper.**\n\nSection 4.2 aims to convey the message that even the smallest divergence from the actual domain PDDL leads to no plan being found, let alone the plan being valid. This sensitivity is a significant motivation for our shift from traditional plan search approaches to the introduction of the exploration walk mechanism\n\n**W9: The authors should give more information on the computational complexity/cost (e.g., time consumption) associated with the calculation of EW score per candidate model.**\n\nComputing the EW is relatively negligible compared to the cost of LLM inference. In our experiments, computing the EW score for a single domain-problem pair takes less than two minutes on a 64-core server CPU. We will add this information to our paper.\n\n---\n[1] Liu et al., LLM+P: Empowering Large Language Models with Optimal Planning Proficiency, 2023\n\n[2] Guan et al.,  Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning, NeurIPS 2023.\n\n[3] Chen et al., Evaluating Large Language Models Trained on Code, 2021\n\n[4] Li et al., Competition-Level Code Generation with AlphaCode, 2022\n\n[5] Schaeffer et al.,  Are Emergent Abilities of Large Language Models a Mirage?, Neurips 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zLQgGCPNZ0", "reviewer": "Reviewer_wCSx", "summary": "The paper presents an approach that leverages LLMs to generate PDDL domain and problem files from natural language descriptions, and refine them iteratively based on environment interactions. In particular, it proposes an Exploration Walk (EW) metric that provides feedback signals to guide the iterative refinement process. In experiments, the proposed method successfully recovers PDDL files in 7 of 10 domains, outperforming LLM-based planners.", "questions": "- What is $c_{max}$ in Algorithm 1? If it is the refinement iterations, how many iterations are used in the paper?\n- In experiments, the method runs 4 times and the best result is used for evaluation. I'm curious what the statistics of the 4 runs look like. Does a \"magical seed\" lead to good results while the others fail?\n- The language descriptions are generated by a GPT-4 from ground-truth PDDL files. I will expect there might be missing items or hullicination. I wonder, do you manually check and fix the generated outputs?", "limitations": "See Weakness and Questions", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper studies the important problem of learning PDDL domains from embodied interaction for classical planning. This is a promising direction to enable long-horizon planning with formal guarantee. You may also find this recent paper [1] highly relevant.\n- In contrast to existing work that require human intervention, the paper boldly attempts to generate PDDL files automatically using feedabck from environment interaction. The proposed EW metric does make some sense to me.\n- In experiments, the method seems to be fairly capable in recovering valid PDDL files.\n\n[1] Han, Muzhi, Yifeng Zhu, Song-Chun Zhu, Ying Nian Wu, and Yuke Zhu. \"InterPreT: Interactive Predicate Learning from Language Feedback for Generalizable Task Planning.\" RSS 2024.", "weaknesses": "- My major concern is on the major contribution of the paper - the automatic mechanism that iteratively refines the generated PDDL files.\n  - **EW does not provide a sufficient objective.** The paper presents an Exploration Walk (EW) metric to provide feedback for the refinement process. The metric measures the difference of generated PDDL domain and ground truth environment by the feasible action sequences within. While EW=1 is the necessary condition for the generated domain to be valid, it is not the sufficient condition. I agree that EW can provide guidance at the initial stage, but in the end the objective in Equation (1) should be the one to optimize to produce a valid domain.\n  - **The exact feedback not explained enough.** The paper seems not to elaborate on the form of feedback provided to LLM for refinement - given the EW score which is a number. While the authors mention this briefly in the Appedix, I'm still don't fully understand how it works exactly. As this is the key part that makes the proposed approach possible, I would suggest the authors to provide more details in the main paper\n  - **The effectiveness of scalar-based feedback is doubtable.** Also, given the feedback is a number that provides little information on what the exact issue is (whether it's on problem file or domain file, whether it's on a precondition / effect term or on the predicate design, and which line), I doubt whether the LLM can perform reasonable refinement. I think it's highly possible that the iterative refinement process will goto nowhere.\n- Another important doubt is on the problem setup - where the natual language descriptions are translated line-by-line from the ground-truth PDDL files. \n  - **This setup is foundamentally different from what the problem of \"generating PDDL\" should be.** Under this setup, the challenge is no long generating PDDL files that requires exploiting environment interactions, but **translating** natural language into PDDL precisely without losing any information. More specifically, I believe the difficulty is to identify important predicates. Once the predicates are ready, the precondition & effect terms and initial & goal states should be relatively simple to be translated with GPT-4 with some prompt engineering.\n  - **The proposed approach seems to be unaligned with the challenge**. While I think utilizing something similar to EW metric is the way to go, it doesn't aligns well with the challenge posed by this problem setup.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. However, we respectfully disagree with the criticisms around our core contributions and problem setup, as we believe there are misunderstandings about our work. \n\nThe reviewer seems to believe that we only use the scalar feedback provided by the EW score and that the latter is insufficient. However, the scalar EW metric is only used to pick the best response from the LLM, and the textual feedback (as explained in Figure 2 and Appendix B.3) consists of the incorrect set of actions and a textual description of the last state. Please see the attached rebuttal pdf in the overall response for the exact form of feedback.  \n\nThe second group of criticism appears to try to redefine the problem that we set out to solve and concludes that our solution misaligns as a result. However, our goal is toward fully automatic planning in very complex problems from natural language description. To this end, all challenges surrounding automatic PDDL generation need to be resolved, including but not limited to the predicate identification problem raised by the reviewer, and we have made substantial progress in the overall problem, as evidenced by the benchmark results. \n\n---\nWe provide additional responses to each point raised by the reviewer below:\n\n**W1 about core contributions: EW does not provide a sufficient objective, the objective in Equation (1) should be the one to optimize to produce a valid domain.**\n\nWe agree that the final objective is Eq (1); however, this equation is not easy to optimize. We should note that it is common practice in machine learning to optimize a proxy objective instead when the primary objective is not easy to optimize. For instance, optimizing the cross-entropy loss as a proxy for accuracy or optimizing the L1 loss function as a proxy for sparsity. We believe the EW score serves a similar purpose as a proxy for Equation (1), which is hard to optimize.\n\n**W2 about core contributions: The exact feedback is not explained enough**\n\nWe will add an example of feedback for the Termes environment to our paper. Please see the attached rebuttal page for the example.\n\n**W3 about core contributions: The scalar provides little feedback.**\n\nPlease see the comment at the beginning of this response.\n\n**Weakness about problem setup: This setup is fundamentally different from what the problem of \"generating PDDL\" should be. The difficulty is to identify important predicates. Once the predicates are ready, the precondition & effect terms and initial & goal states should be relatively simple to be translated with GPT-4 with some prompt engineering.**\n\nOur goal is toward fully automatic planning in very complex problems from natural language description. To this end, all challenges surrounding automatic PDDL generation need to be resolved, and we have made substantial progress in the overall direction, as evidenced by the benchmark results.  We appreciate the reviewer's acknowledgment of the challenge in identifying important predicates, which we did target in this work. However, we would like to emphasize that determining the correct preconditions and effects is also more complex than it might seem. Our observations indicate that GPT-4 sometimes misses preconditions or effects. For instance, as shown in the attached rebuttal PDF, the LLM corrected an action's precondition only after receiving feedback from the environment about the action's illegality. We also note that the level of description in our setting is similar to that of prior works (e.g., [1]). Overall, we are excited to see future extensions of our work that rely more on environment interaction and less (or not at all) on natural language descriptions. Our work lays the groundwork for achieving this goal.\n\n---\n**Responses to questions**\n\n**Q: What is $c_max$  in Algorithm 1? If it is the refinement iterations, how many iterations are used in the paper?**\n\nIt is the maximum number of refinement iterations (or conversation turns). Following prior works on code generation [2, 3], we set a maximum of $c_{max}=4$ in our experiments.\n\n**Q: In experiments, the method runs 4 times and the best result is used for evaluation. I'm curious what the statistics of the 4 runs look like. Does a \"magical seed\" lead to good results while the others fail?**\n\nWe did not “optimize for the random seed” if the question hints at it. Furthermore, in a realistic setting, it is perfectly reasonable to try a few times and use the EW score to decide the final solution, as that only relies on environment feedback. Furthermore,  the Best@4 metric is similar to the Pass@K metric, which is commonly used in code generation literature (e.g., [3, 4]).\n\nAs for the statistics, it depends on the environment. For example, on Termes, all four seeds succeed in recovering the correct domain. On Grippers, three seeds succeed. On harder environments, such as hiking, two seeds succeed in recovering the correct domain PDDL, and Floortile only one seed succeeds. \n\n**Q: The language descriptions are generated by a GPT-4 from ground-truth PDDL files. I will expect there might be missing items or hullicination. I wonder, do you manually check and fix the generated outputs?**\n\nWe did manually check for hallucination, and observed that the hallucination cases for back-translation are very rare, though not zero, and fixed the cases that we detected. We will publicly release the data along with the camera-ready version of our paper.\n\nNote that this manual checking for hallucination is only for creating the natural language description dataset for the problem setup. For automatic PDDL generation, there is no human in the loop.\n\n---\n[1] Liu et al., LLM+P: “Empowering Large Language Models with Optimal Planning Proficiency”, 2023\n\n[2] Madaan et al., “SELF-REFINE: Iterative Refinement with Self-Feedback”\n\n[3] Chen et al., Teaching Large Language Models to Self-Debug, ICLR 2024\n\n[4] Chen et al., Evaluating Large Language Models Trained on Code, 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uNNmrjwizk", "reviewer": "Reviewer_cVUW", "summary": "The paper proposes an approach to leverage LLMs and environment feedback to automatically generate PDDL domain and problem description files without human intervention.  They do so by an iterative refinement approach that generates multiple PDDL problem and domain candidates based on feedback obtained from the environment.  The authors show their approach works experimentally in 66% of 10 PDDL domains that they have tried.", "questions": "1. 66% solve rate what does it mean? Does it mean 7 out of 10 problems are solved? Also what does that mean? Is the PDDL now correct? How many problems of that domain is the different approaches now solving correctly? Can you please further clarify. \n\n2. How do you know if you have a planning problem at hand, which can be turned into PDDL? Have you tried it on Alfworld (has a PDDL but its not ideal), or Mind2Web, or anything that does not have an existing PDDL. Maybe this relates to assumption 2. \n\n3. With regard to the refinement how do you know when to stop? Is there a threshold on the metric that would give you signals on when to stop the refinement?\n\n4. Can you say anything in regard to algorithm 1's soundness/correctness  (and the approach in general).  Also with respect to algorithm 1, how many times a LLM model should be called to come up with a reasonable PDDL domain, and a reasonable PDDL problem? Can you comment on the cost associated with that as well (any information beyond the token size would be great). \n\n5. In the notation section 3, why is the set of all possible actions A separate from D, isn’t A always part of D. \n\n6. Regarding planning error, what about unsolvable instances?  Do you assume all instances are solvable? \n\n7. Regarding equation 1, can you clarify that you generate multiple problem instances, but only one domain, or also refining the domain multiple times, each time generating multiple problem files? \n\n8. Can you clarify why you need the ground truth p and d, how do you use the ground truth q to validate the answers. Also what happens if you are not provided with the ground truth domain/problem. Does this mean that even though a ground truth domain/problem is given we are going to use LLMs to generate the domain/problem? \n\n9. Can you please comment on applicability of your approach if the PDDL environment is not known/given?", "limitations": "Assumption 2 may be limiting the scope and the level of impact of the work.  Also what about cases that the ground truth domain/problem is not known which is in most cases when it comes to real applications. Also having to rely on the PDDL environment is also limiting the scope and impact of the work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem addressed in the paper is an important and interesting problem. The proposed approach with regard to the EW metric is novel and promising.", "weaknesses": "Assumptions: Assumption 2 may not be realistic. Often times, people may not know exactly what is the right way to capture the domain knowledge, that is what kind of things they should have said to ensure the pre/effects/initial state all are captured. What about the case that the domain description is missing a constraint or precondition?\n\nEnvironment requirement: in regard to the applicability of the work, there is a dependency on the environment to do the refinement, and that also may limit the impact of the proposed solution as the environment may not always be available for all domains. \n\n\nNovelty: the authors claim to be the first to enable PDDL generation using LLM without human intervention. However, there exists at least two related work that does also generate PDDL domain and problem without human intervention:   \n\n1. Large Language Models as Planning Domain Generators ICAPS 2024 (https://github.com/IBM/NL2PDDL)\n\n2. AUTOPLANBENCH: Automatically generating benchmarks for LLM planners from PDDL PRL 2024 (https://github.\ncom/minecraft-saar/autoplanbench/tree/main.)\n\n\n\nThe paper presentation can be improved. See the question section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments. Due to character limit constraints, below we summarize and answer the main questions raised by the reviewer.\n\n**Q1: Assumption 2 may be unrealistic. What if the domain description lacks a constraint?**\n\nWe put the first steps towards using the environment as a source of refinement on top of natural language for domain PDDL generation. We have removed the biggest assumption/blocker, which is human feedback. While our framework, at its current state, may not provide high accuracy in the absence of natural language description, or partial natural language description, we are excited to see future extensions of our work that rely more on environment interaction and less (or not at all) on natural language descriptions. Particularly, partial information in NL can be addressed in future work through better prompting for LLM to propose and fill in missing information. This can be further enhanced by multiple guesses in an overall search tree-like reasoning setup that is on top of the \"search tree\" in our method. Our work lays the groundwork for achieving this goal.\n\n**Q2: There is a dependency on the environment to do the refinement, and that also may limit the impact of the proposed solution as the environment may not always be available for all domains.**\n\nWe understand the concern that the \"PDDL\" environment might not always be available, thus limiting the impact. Our framework is not limited to PDDL environments, although the current implementation is. The only scenario where our framework is not applicable is when environment feedback is significantly delayed or slow. In such cases, relying on fully automated agent planning and action execution may not be advisable.\n\n**Q3: Novelty: two other works generate PDDL domain and problem without human intervention.**\n\nWe thank the reviewer for bringing our attention to these very recent related works, and will add citations to our paper. However, these two papers do not really change the fact that our work is the first work to generate both domain and problem PDDLs end-to-end without human intervention. To elaborate, (1) the work of Oswald et al. despite generating domain PDDL from natural language and proposing heuristics for PDDL action domain comparison, differs in two significant ways: first, **they assume the predicates are given**, which is a too relaxing assumption, second, their work does not directly translate problem **and uses the ground-truth problem PDDL instance for comparing the compatibility of two domains.** (2) the work of Stein et al. is focused on translating **PDDL to natural language, which is the opposite** of what we seek to achieve: translating natural language to PDDL. \n\n**Q4: What does 66% solve rate what does it mean? What does 7 out of 10 domains solved mean?**\n\n66% solve rate means that out of 100 tasks (10 problems for each of the 10 environments), 66 of them were solved by finding a correct plan. Note that a task will be solved iff both problem and domain are translated successfully. We observe that despite the translation of the domain being completely successful, sometimes the LLM makes minor mistakes in translating one or more of the 10 problems of the same domain. As such, we consider a domain to be solved if more than 50% of its tasks are solved correctly.\n\n**Q5: Clarify the need for ground truth PDDL. What if the PDDL environment is not known or given?**\n\nWe thank the reviewer for noting the potential of our framework for general planning problems that do not have a PDDL environment. As we have noted the potential applicability in section 4.1, we leave the extension to future work.  Automatically generating pddl problem and domains without human intervention is a challenging enough problem that we will have to address these other questions in future work. We use the ground-truth PDDLs only to retrieve the list of possible actions, and their applicability, compatible with our Assumption 1. Hence, our framework, by design, is agnostic to the underlying environment. Therefore, as long as the action interface is expressible in PDDL, and Assumptions 1and 2 are met, our method is applicable to the underlying environment.\n\n\n**Q6: How do you know when to stop refinement?**\n\nFollowing prior works on code generation [2, 3], we set a maximum of $c_{max}=4$ conversation turns in our experiments. That said, if the exploration walk metric is 1.0 and the task in hand is solved, we stop early and do not continue the refinement.\n\n**Q7: In algorithm 1, how many times a LLM model should be called? What is the cost associated with that as well beyond the token size?**\n\nFor the P&D method with variables $n_p$ problem samples, $n_d$ domain samples, and $c_{max}$ conversation turns, the language model is called $n_p n_d c_{max}$ times, which in our experiments is $5 \\times 10 \\times 4 = 200$ times. Computing the EW metric for each domain-problem pair takes less than two minutes on a 64-core Server CPU.\n\n**Q9: Regarding planning error, what about unsolvable instances?**\n\nWe consider any unsolvable instance as a part of a planning error. In general, a planning error means that there exists no plan that achieves the desired goal from the initial state, whether it is domain-problem incompatibility or unsolvability.\n\n**Q10: Do you generate multiple problem instances and refine the domain multiple times?**\n\nFor each generated problem, we run a fresh instance of domain refinement. As a part of problem translation, some predicates will be generated, as such, the generated domain should also conform the the defined predicates, and should be generated from scratch.\n\n[1] Guan et al.,  Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning, NeurIPS 2023.\n\n[2] Madaan et al., “SELF-REFINE: Iterative Refinement with Self-Feedback”\n\n[3] Chen et al., Teaching Large Language Models to Self-Debug, ICLR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their constructive feedback. We are encouraged that the reviewers find the automated PPDL generation problem to be important (cVUW, wCSx) that our exploration walk metric to be novel and promising (cVUW, wCSx, zxRN, iKF8), appreciate the analysis we did in the paper (iKF8), and that our paper is well-written (zxRN, iKF8). 

We have provided individual answers to the questions raised by the reviewers. To incorporate the modifications requested by the reviewers, we will revise our manuscript with the following:

* Add an example of exact environment feedback (provided in the rebuttal PDF page). This example showcases the reasoning of LLM for domain refinement and provides a clear example of the refinement prompt.
* Add an example from the Barman environment in Appendix A.2 (in addition to the Gripper example that we already have) to emphasize the cruciality of predicate design.
* Clarify the terminology in the related work section.
* Provide more detailed information on the computational complexity of the EW metric.
* Add citations to recent related works suggested by reviewers.

If the concerns are well addressed, we kindly ask the reviewers to raise the ratings accordingly. Should there be any further questions, please do not hesitate to ask us during the discussion period.

Meta-review: The reviewers generally agreed on the fact that the paper is tackling an important problem. Additionally, there was a general consensus that the Exploration Walk (EW) metric is an interesting contribution and could spur follow-up work in the future. The fact that the authors have also focused on leveraging minimal environmental feedback is also a useful aspect of the work.

While some of the reviewers noted the accessibility and clarity of the writing, the discussions clearly pointed out that there the current exposition does lead to confusion about the contributions of the paper, the setting, and the assumptions made. The authors should make a point to incorporate these points that were brought up during the discussion. 

Other important points that were highlighted during the discussion include the existence of related work that also doesn’t need human input, what lack of human input even means here, and explicitly capturing the cost associated with learning the model. I would also advice the authors to include discussion connecting the proposed methods to the larger literature on learning PDDL models from interacting with environment.

With that said, in my opinion, the paper makes a meaningful contribution to the space of planning with LLMs. As such, my recommendation is to accept the paper.
Decision: Accept (poster)