Title: Leveraging Environment Interaction for Automated PDDL Translation and Planning with Large Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Planning

PDDL

Large Language Models
Abstract: Large Language Models (LLMs) have shown remarkable performance in various natural language tasks, but they often struggle with planning problems that require structured reasoning. To address this limitation, the conversion of planning problems into the Planning Domain Definition Language (PDDL) has been proposed as a potential solution, enabling the use of automated planners. However, generating accurate PDDL files typically demands human inputs or correction, which can be time-consuming and costly. In this paper, we propose a novel approach that leverages LLMs and environment feedback to automatically generate PDDL domain and problem description files without the need for human intervention. Our method introduces an iterative refinement process that generates multiple problem PDDL candidates and progressively refines the domain PDDL based on feedback obtained from interacting with the environment. To guide the refinement process, we develop an Exploration Walk (EW) metric, which provides rich feedback signals for LLMs to update the PDDL file. We evaluate our approach on $10$ PDDL environments. We achieve an average task solve rate of 66\% compared to a 29\% solve rate by GPT-4's intrinsic planning with chain-of-thought prompting. Our work enables the automated modeling of planning environments using LLMs and environment feedback, eliminating the need for human intervention in the PDDL translation process and paving the way for more reliable LLM agents in challenging problems. Our code is available at https://github.com/BorealisAI/llm-pddl-planning

Leveraging Environment Interaction for Automated PDDL Translation and Planning with Large Language Models

Abstract

Large Language Models (LLMs) have shown remarkable performance in various natural language tasks, but they often struggle with planning problems that require structured reasoning. To address this limitation, the conversion of planning problems into the Planning Domain Definition Language (PDDL) has been proposed as a potential solution, enabling the use of automated planners. However, generating accurate PDDL files typically demands human inputs or correction, which can be time-consuming and costly. In this paper, we propose a novel approach that leverages LLMs and environment feedback to automatically generate PDDL domain and problem description files without the need for human intervention. Our method introduces an iterative refinement process that generates multiple problem PDDL candidates and progressively refines the domain PDDL based on feedback obtained from interacting with the environment. To guide the refinement process, we develop an Exploration Walk (EW) metric, which provides rich feedback signals for LLMs to update the PDDL file. We evaluate our approach on 10 PDDL environments. We achieve an average task solve rate of 66% compared to a 29% solve rate by GPT-4's intrinsic planning with chain-of-thought prompting. Our work enables the automated modeling of planning environments using LLMs and environment feedback, eliminating the need for human intervention in the PDDL translation process and paving the way for more reliable LLM agents in challenging problems. Our code is available at https://github.com/BorealisAI/llm-pddl-planning

1 Introduction

Large language models (LLMs) have demonstrated remarkable success across various domains, including mathematics, coding, and even the bar exam [1]. These models excel at understanding and generating natural language, offering flexibility and adaptability to a wide range of tasks. However, when it comes to planning and long-horizon reasoning, LLMs have shown limited performance [8, 28], despite some promising results [3].

Planning is a crucial aspect of intelligence that involves reasoning to find a sequence of actions to achieve a desired goal state from an initial state. The Planning Domain Definition Language (PDDL) [18] is a widely used formalism for describing planning problems. PDDL provides a structured way

to define the domain, which includes the types of objects, predicates, and actions, as well as the problem instance, which specifies the initial state and goal conditions. PDDL enables the application of search-based algorithms, such as breadth-first search (BFS) or A ∗ search, which can guarantee to find a valid solution if one exists. However, the downside of PDDL is that it requires a well-defined and structured domain and problem definition, which can be challenging to create, especially for complex scenarios. Figure 1 showcases snippets of some PDDL problems and domain files along with an action plan produced by a classical planner.

Recent studies explored combining the strengths of LLMs and PDDL-based planning [15, 7, 9]. The idea is to leverage LLM for translation from natural language (NL) problem descriptions into PDDL formal descriptions, and then use a classical planner to solve the translated PDDL problem [9]. This hybrid approach could theoretically take advantage of the flexibility of NL input and the correctness guarantees provided by the classical planner. If the translation from NL to PDDL is accurate, the resulting plan is guaranteed to be valid.

Unfortunately, existing approaches have not been able to generate both PDDL problem and domain descriptions with reasonable success rates without humans in the loop, as we shall elaborate in Sec. 2. While translating PDDL problems is feasible given the domain PDDL description [15], generating domain PDDL from NL correctly is a more nuanced and challenging problem. To do so requires identifying causally relevant objects to design predicates, as well as their inter-relationships, in a way that accurately reflects the possible states and transitions of the environment. A small error, for example in predicate design, could lead to entirely incorrect domain description and failed planning (see Appendix A.2 for a real example). Guan et al. [9] take a step toward this goal relying on human-in-the-loop to detect and correct mistakes made by LLMs.

In this work, we develop a fully automated method for generating PDDL domain and problem definitions using LLMs and environment feedback without relying on human intervention. Intuitively, our method lets an LLM build hypothetical 'mental models' of the environment, in the form of proposed PDDL domain descriptions. The LLM then verifies and updates the 'mental model' by observing discrepancies between the feasibility of actions under its 'mental model' and the real environment. This method enables LLMs to use classical planners to solve complex planning problems whose solutions may require hundreds or thousands of steps that all need to be correct.

We first highlight the challenges of this task and then propose our solution. In particular, our contributions are as follows:

We demonstrate that even small modifications to PDDL domains can render plan search infeasible, limiting the feedback information for LLMs to perform in context update.

To address this, we introduce a new Exploration Walk (EW) metric, which is a smooth similarity measure between two domains by comparing the executability of random action sequences sampled from one domain on the other. Crucially, EW only requires access to the action interface and executability of the environments, not directly the ground-truth PDDL.

We propose an EW-guided tree search approach that leverages LLMs to generate and refine the PDDL domain and problem files iteratively and automatically.

We evaluate our method on 10 challenging PDDL domains, where a number of them are from the International Planning Competition, and show that it outperforms a baseline that generates PDDL files in a single attempt without refinement. Our method solves 7 out of 10 environments, achieving an average task solve rate of 66% and average EW score of 0.84, compared to 34% task solve rate and 0.53 EW score for the baseline, and 29% solve rate by GPT-4 ( gpt-4-1106-preview )'s intrinsic planning with chain-of-thought prompting.

To the best of our knowledge, this is the first work that enables modeling a planning environment via PDDL translation using LLMs and environment interaction, without the need for human intervention.

2 Related Work

LLMs and Classical Planning. There has been recent interest in integrating LLMs with PDDL [15, 28, 9, 7, 30, 23, 10, 20, 26], and more generally neural networks with PDDL [24, 2]. Silver et al. [25] leverage LLMs to take domain PDDLs and problem PDDL specifications, and synthesize a Python function to generate domain-specific plans, as a replacement for search-based planning. Liu

et al. [15] show that using LLMs to translate problem specification to PDDL, and using classical solvers results into a higher planning accuracy that using LLM directly as a planner. Dagan et al. [7] consider a similar setting, but assume that the list of objects is partially observable, and the LLM needs to interact with the world to observe the list of objects. All of the mentioned works, however, assume that a domain PDDL files is already provided. Oswald et al. [20] generate domain PDDL from natural language and propose heuristics for comparing PDDL action domains. However, their approach assumes that predicates are provided, whereas our work makes no such assumption. Additionally, Oswald et al. [20] rely on ground-truth problem instances for domain compatibility evaluation, whereas we directly translate problem PDDL without any such assumptions. Guan et al. [9] translate both Domain and Problem from natural language description but rely on human experts to correct mistakes in the domain translation before generating problem PDDLs. In this work, our goal is to lift the human-intervention assumption, and instead, use domain interaction for evaluation and verification. See Table 1 for a summary of related work comparison.

Direct Reasoning with LLMs. Recent research has explored eliciting direct reasoning capabilities within Large Language Models (LLMs). This reasoning can be either entirely direct [31, 29] or partially direct with the assistance of basic external tools [16]. However, the primary limitation of these approaches lies in the inherent tendency of auto-regressive LLMs to produce errors in longhorizon reasoning tasks [28]. Even a minor mistake in a single reasoning step can lead to cascading errors, ultimately resulting in an incorrect final answer [8]. When applied to classical planning, this approach delegates the entire plan generation process to an LLM instead of leveraging a dedicated classical planner. Studies have demonstrated that this strategy is suboptimal compared to generating PDDL code directly [9, 15], highlighting the importance of incorporating classical planning tools for faithful plan generation in classical planning tasks.

External Reasoning and Code Generation. This last line of work focuses on generating executable code from natural language instructions such as SQL or Python code generation [4, 19, 17, 5, 16, 32]. Here, the LLM often acts as a code translator, and the reasoning logic lies within the generated code. Chen et al. [4] show that LLMs are capable of Python code generation from docstrings to high accuracy. The authors also find that taking multiple code samples from an LLM and picking the best samples results in an accuracy boost. Later works show that iterative refinement of LLM responses improves the accuracy on the downstream task [17, 5], especially given external feedback such as unit tests or human feedback. Our work is related to code generation as we produce structured PDDL files. However, our setting presents three challenges: (1) there are two types of PDDL files, in contrast to a single Python script, and the two files need to be consistent with each other; (2) more importantly, getting external feedback and the evaluation of a generated PDDL code is not as easy as python unit tests, and as we show in Section 4.3, (domain generation) errors are abundant and hard to trace; (3) LLMs are trained with a lot more Python code compared to PDDL, as the later is much scarcer.

3 Notation and Background

Notation. We denote ✶ [ · ] as the indicator function. The notation 1 : N refers to the sequence of integers ranging from 1 to N . For a set A , we define A ∗ as the set comprising all possible sequences of elements drawn from A , and define 2 A as the power set of A .

PDDL. Planning Domain Definition Language (PDDL) is a formal language used to describe and specify planning problems for automated planning. Here, we have two types of PDDL files: (1) Domain PDDL , which defines possible predicates ( i.e. , states), and actions in the environment. Executing each action requires some precondition ( i.e. , a set of predicates to have a specific value), and the execution leads to some effect ( i.e. , a change in the values of some predicates). (2) Problem PDDL , which contains a set of initial predicates and a set of goal predicates.

The problem PDDL instantiates the domain definition PDDL to form a concrete environment. Together, the planning problem is fully defined and formalized. A classical planner takes in both files and searches for a plan based on the provided specification. A plan is a sequence of actions, starting from the initial state, leading to a state satisfying the goal conditions, with each action respecting the rules of the environment. Formally, let D , P , A be the set of all possible domains, problems, and actions, respectively. Then, given a domain d ∈ D and problem p ∈ P , a classical planner C : D×P → A ∗ ∪ {⊥} takes in domain d and plan p , and produces a plan q := C ( d, p ) which is either set of actions from A ∗ , or a planning error ⊥ . Aplanning error may be due to an infeasible plan search ( i.e. , plan not found), syntax errors, or incompatible domain and problem. A plan validator verifies whether a plan q is executable and achieves the desired problem goal given a domain PDDL d and problem PDDL p , i.e. , whether q solves the planning problem instance. The validator function, denoted as V d,p ( q ) : A ∗ →{ 0 , 1 } , is 1 if the plan is valid, and 0 otherwise. For convenience, we assume V d,p ( ⊥ ) = 0 . Similarly, we define plan execution checker E d,p : A ∗ → { 0 , 1 } , which only checks whether an action sequence is executable in a domain or not. Note that the difference between V and E is that the former checks for both plan executability and goal satisfaction, while the latter only checks for plan executability. We also define S as the set of all possible states. Function A d,p : S → 2 A delineates the set of legal actions given the current states ( i.e. , actions that would not immediately result in E d,p returning 0). The function S d,p : A×S → S denotes the state transition function ( i.e., S d,p ( a, s ) determines the subsequent state given the current state s and action a ). Finally, we denote the initial state induced by d and p to be s d,p, 0 ∈ S . See Table 3 in the Appendix for a summary of notations.

To illustrate the definitions with an example, consider the Grippers [13] environment with several rooms containing robots and boxes. Robots can move balls between rooms using their left and right grippers. Given an initial setting of robots and balls in different rooms, the main goal is to move specific balls to specific rooms using the robots.

Figure 1 shows an annotated example domain, problem, and plan for this environment. The domain determines predicates and actions. Predicates such as at-robby keep track of object states ( e.g. , whether a particular robot is in a particular room) and defining suitable predicates is a crucial part of domain design. The move action for moving a robot from one room to another has three parameters: robot r , departure room from , and destination room to . Each action has preconditions and effects, which comprise the main logic of the domain for determining the actionability of an action. In the case of the move action, the precondition is that the robot must be in the from room, and the effect is that it will no longer be in that room

Domain PDDL

Problem PDDL

Plan

(define (domain gripper-strips)

...

(:types room obj robot gripper)

(:predicates

(at-robby ?r - robot ?x - room)

(free ?r - robot ?g - gripper)

...

)

(:action move

:parameters (?r - robot ?from ?to - room)

:precondition (and (at-robby ?r ?from))

:effect (and (at-robby ?r ?to)

(not (at-robby ?r ?from))))

(:action pick

...

)

(:action drop

...

)

...

)

Predicates

Preconditions

Effects

(define (problem gripper-p)

...

(:objects

)

(:init

(free robot1 rgripper1)

...

)

(:goal

(and

(at ball1 room2)

...

)))

Object List

Initial State

(move robot2 room3 room1)

(pick robot2 ball2 room1 lgripper2)

...

Actions

and will be in the to room. A problem PDDL p specifies the initial state of robots, boxes, rooms, and the final goal. For instance, (at-robby robot2 room3) means that robot2 is initially at room3 . The predicate (at ball1 room2) specifies the goal condition that ball1 must eventually be moved to room2 . Aplan constitutes a sequence of actions to reach the goal. For instance, one action could be (move robot2 room3 room1) , moving robot2 from room3 to room1 . If robot2 is not already in room3 , this action is considered illegal, and the environment will produce an error. For a complete example of domain d , problem p , and plan q , see Listings 1, 2, and 7, respectively in the Appendix.

Large Language Models (LLMs). We assume access to a powerful language model LLM. LLM n ( X ) denotes sampling n responses from the LLM given prompt X . Following the prior works, we set a temperature of τ = 0 for sampling with n = 1 ( i.e. , greedy sampling), and a temperature of τ = 0 . 7 for n > 1 [5]. Whenever possible, we use zero-shot or one-shot chain-of-thought prompts [14, 29] for the LLM to reason before generating a response.

4 Method

Given an environment e , its domain NL description and a task NL description, the environment's object list and action interface, our goal is to model the environment by generating a domain PDDL ˆ d ∈ D and a problem PDDL ˆ p ∈ P , such that applying a classical planner C on the PDDL files produces a valid plan for the environment, i.e. , C ( ˆ d, ˆ p ) is a valid plan for e , i.e. , V d,p ( C ( ˆ d, ˆ p )) = 1 .

4.1 Setup

For evaluation, we assume there exists a ground truth domain PDDL d ∈ D , and a corresponding problem instance p ∈ P . However, the ground truth is not directly compared to generated ˆ d, ˆ p , but to validate the plan ˆ q := C ( ˆ d, ˆ p ) by executing the validator of the ground-truth environment, V d,p (ˆ q ) .

Formally, for each environment e with domain PDDL d ∈ D , and N tasks with their corresponding ground-truth problem PDDLs p 1: N := ( p 1 , p 2 , . . . , p N ) , p 1: N ∈ P N , our goal is to generate a domain PDDL ˆ d , and a sequence of task PDDLs ˆ p 1: N := (ˆ p 1 , ˆ p 2 , . . . , ˆ p N ) such that the average solve rate V is maximized:

Generating accurate ˆ d and ˆ p 1: N in one attempt is often impractical [9], and some form of feedback is required to refine the response. Guan et al. [9] leverage human expert feedback on ˆ d to correct the generated domain. However, human feedback may not always be reliable and is not scalable. Before introducing our method that relies on environment feedback instead, we first state our assumptions:

Assumption 1 (Environment access) We assume the list of objects and action interfaces are known. Furthermore, we assume that executability and verifiability of actions can be observed (through the functions E d,p and V d,p ).

Assumption 2 (Natural language description) We assume the natural language descriptions of the domain and task are both given.

The action interfaces are equivalent to APIs available to LLM agents. So it is reasonable to assume that the exact API call signatures are known. On the other hand, one may wonder why the object list, which appears in problem PDDLs as illustrated in Figure 1 needs to be assumed to be given, when the NL problem description should describe the objects involved in the planning tasks. This is because the NL description may not refer to the object instances using exactly the same label as the environment induced by d and p . If p refers to a robot as robot1 but the user specifying the natural language problem description calls it Jarvis , then the environment only recognizes robot1 and not Jarvis , so the LLM would have no way to correct this mistake due to trivial name mismatch. See Appendix A.1 for a detailed example of our assumptions on the Grippers environment.

Note that our assumptions do not require the underlying environment to be a PDDL environment, but it can be any environment as long as PDDL is expressive enough to capture the working mechanisms of the environment. For digital agents in virtual environments, the list of objects and action interfaces are just different data objects and APIs available. The assumptions could even hold true for physical agents in the real world, provided recognition and control are sufficiently accurate. In this work, we focus on PDDL environments only, although our framework is more general.

4.2 Difficulty of domain PDDL generation

Generating the correct domain PDDL is challenging, as small mistakes could make the plan search fail. To demonstrate this brittleness, we simulate random omission of k terms, where 0 ≤ k ≤ 10 , from the action precondition and effects of the original domain d . For instance, in the case of the Grippers (Figure 1), we may create a new synthetic domain by removing the (at robby ?r ?to) term from the effects of the move action. Namely, we define ˆ d k ∼ P k ( d ) , where P k ( d ) represents the uniform random removal of k terms. Then, for each generated ˆ d k , coupled with the ground truth task PDDLs, we compute whether the classical planner is able to find a plan without error and compute the Plan-Not-Found rate under k omissions, PNF k , of the environment.

0

1

2

3

4

5

6

7

8

9

10

Number of Removed Terms (

k

)

0.0

0.2

0.4

0.6

0.8

Plan-Not-Found Fraction

Average

(a)

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.2

0.4

0.6

0.8

1.0

Average EW Score

Storage

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.25

0.50

0.75

1.00

Average EW Score

Floortile

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.4

0.6

0.8

1.0

Average EW Score

Hiking

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.00

0.25

0.50

0.75

1.00

Average EW Score

Logistics

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.25

0.50

0.75

1.00

Average EW Score

Termes

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.25

0.50

0.75

1.00

Average EW Score

Driverlog

1.0

Depot

1.0

Rovers

1.0

Barman

(b)

0.8

0.8

0.25

0.8

Average EW Score

0

1

2

3

4

5

6

7

Term Difference

We empirically measure the value of PNF k using Monte-Carlo estimation on 15 environments. As shown in Figure 2a, PNF 1 has an average of 0 . 14 among different environments. This means that on average 14% of the terms in domain PDDLs are so critical that removing them results in a plan-not-found error. This situation is exacerbated for larger k : at k = 3 , the average PNF k reaches around 0 . 3 . In practice, the problem PDDL ˆ p i also needs to be generated, and the generated domain ˆ d may have extra terms, both of which may further increase the planning-not-found rate.

4.3 Domain alignment measure via Exploration Walk metrics

Whenever the plan search fails, absolutely no information is available to the LLM about which part of the problem or domain has issues. This is because the underlying search algorithm (such as BFS and A ∗ ) fails and as a result, it does not produce any output. For example, with BFS, it enumerates all paths (possibly several thousand paths or more), and finds none satisfy the goal conditions, leaving the plan search without any useful insights. As an alternative, we introduce the Exploration Walk (EW): a smooth feedback signal that provides incremental feedback for LLM in-context learning. EWboth provides a mechanism to gather richer feedback information that feeds into LLM context for PDDL refinement, as well as computing a smooth scoring metric that to compare multiple PDDLs and guide the refinement process forward.

Intuitively, the idea is to take legal random action sequences and verify their executability under LLM's "mental model" environment induced by an LLM-generated PDDL domain. This is analogous to the retrodiction step in scientific methodology, where existing observations and experimental data need to be explained by the existing model.

And in the other direction, EW takes executable random action sequences from an LLM-generated PDDL domain and verifies whether they are correct in the real environment. This is analogous to hypothesis testing in scientific methodology, where new predictions are verified experimentally.

We now describe the EW and EW metrics formally. We define an Exploration Walk of length T to be any action sequence sampled from a strictly positive distribution P d,p,T over executable T -step action sequences in A ∗ corresponding to domain d and task p . We assume the probability of non-executable action sequences to be zero under P d,p,T . In other words, ∀ q 1: T , P d,p,T ( q 1: T ) > 0 iff E d,p ( q 1: T ) = 1 .

For the rest of this paper, we use the simplest possible EW, with a uniform distribution over valid actions at each step. Note that to sample uniform random EW from the ground truth environment induced by d and p , we do not need direct access to the full d and p . We only need the list of objects in p and the action interface in d , and executability checker E d,p , consistent with our Assumption 1. At each step, running E d,p on all possible actions yields the legal actions at that step for EW.

Given an EW distribution, we define an EW metric using the fractions of executability of EW walks from one domain under another, averaged over all different lengths.

8

9

10

1.0

0.8

0.6

0.4

1.0

Average EW Score

0.8

0.6

0.4

1.00

0.75

0.50

0.25

0

0

0

1

1

1

2

2

2

Childsnack

3

4

5

6

7

Term Difference

Satellite

3

4

5

6

7

Term Difference

Zenotravel

3

4

5

6

7

Term Difference

8

8

8

9

9

9

10

10

10

Definition 1 (EW Metrics) Let p 1: N and ˆ p 1: N be problems in domain d and ˆ d respectively, such that the set of objects in p j and ˆ p j are consistent. We define the one-sided measure m d GLYPH<1> ˆ d and the symmetric one m d ↔ ˆ d for the degree of alignment between two domains d and ˆ d as:

where T max is the largest EW walk length.

m d GLYPH<1> ˆ d measures what fraction of EWs sampled from domain d are executable on the domain ˆ d . Then, m d ↔ ˆ d takes the harmonic mean of m d GLYPH<1> ˆ d and m ˆ d GLYPH<1> d to produce the final EW measure. This metric has two favourable properties: (1) it ensures that m d ↔ ˆ d = m ˆ d ↔ d , thereby providing a consistent measure of similarity regardless of the order of domain comparison. (2) the harmonic mean is resistant to trivial domain similarity inflation. By employing the harmonic mean rather than the arithmetic mean, the symmetric EW metric prevents domains that are overly permissive (e.g., domains where all actions are permissible without any preconditions) from being similar to more restrictive domains. For example, in a scenario where domain ˆ d allows all possible actions without restrictions, m d GLYPH<1> ˆ d = 1 . An arithmetic mean in this context would yield m d ↔ ˆ d ≥ 0 . 5 , overestimating the similarity. In contrast, the harmonic mean results in m d ↔ ˆ d = ϵ , where ( ϵ ≪ 1 ) for most cases.

Note that while the PDDL problems p 1: N and ˆ p 1: N appear in the definition of EW metrics, we only use the fact there are aligned object sets in them. We could also use an arbitrarily sampled object list to form an ˜ P and pair ˜ P with D and ˆ D for EW metrics. But since for PDDL generation, we already generate ˆ p 1: N , it is more convenient to use them.

Importantly, EW metrics can be computed without direct access to the full ground truth domain d and problems p 's. As established before, to sample uniform random EW, we just need access to the object list and action interface, plus the environment executability checker of the source domain. So even for m d GLYPH<1> ˆ d , where the EW action sequences come from d , we do not need more than what is available through Assumption 1.

To demonstrate the relationship between m d ↔ ˆ d and domain disparity, we use the same simulated random omission study setup from Sec. 4.2. For a pair of modified domains, we count the number of terms that differ, and inspect m d ↔ ˆ d as function of increasing number of differing terms in Figure 2b for six example domains (see Figure 4 in the Appendix for the full set). We observe that, on average, a greater discrepancy in the number of terms between two domains correlates with a reduced EW score m d ↔ ˆ d . This observation provides additional support to the use of the EW score as an effective measure for domain differences.

4.4 Leveraging LLMs to generate PDDL files

We now show our overall LLM-based method for PDDL generation using the EW score to guide and measure the progress of domain generation. To illustrate the process, we first focus on a domain d with a single task p . Recall that we are given NL description of the environment domain d NL and problem p NL (Assumption 2), as well as the object list in p and action interface from d (Assumption 1). Then, by using d NL, p NL, and access to environment action feedback, we seek to generate ˆ d ∈ D , ˆ p ∈ P .

Our method starts by initializing templated ˆ d (0) based on action interfaces and templated ˆ p (0) using object list. Example template ˆ d (0) and ˆ p (0) are shown in Listings 6 and 4 of Appendix A.1. We then use an LLM to improve the initial ˆ d (0) and ˆ p (0) .

Given that domain PDDL files are typically more complex than problem PDDL files, our strategy prioritizes the generation of a problem PDDL file ˆ p first, followed by the domain ˆ d . This approach enables us to assess the quality of the generated domain immediately. Moreover, prior works on code generation [4], tree-of-thought [31], and self-debug [5] have found that taking multiple samples from the LLM response and taking the best response leads to better performance. However, they often require an evaluation metric on the generated response (such as unit test cases, or execution traces). Here, we use the EW metric introduced in Section 4.3 to serve as an evaluator of the generated domain. These considerations lead to our proposed Algorithm 1. We emphasize again that the

GLYPH<1>

GLYPH<1>

0.2

1.0

0.1

0.4

0.3

0.2

0.2

1.0

Problem Translation

Domain Refinement

P&D Chain

P&D Tree

Classical

Planner

a 2

→ a 1  → a 3

a 2

→ a 4  → a 1

a 3

→ a 2  → a 2

Exp Walks

a 2

→ a 1  → a 3

a 2

→ a 4  → a 1

a 3

→ a 2  → a 2

a 2

→ a 4  → a 1

a 4

illegal

LLM

LLM

ground-truth domain and problem d, p are only used to take exploration walks and evaluate a plan through the environment in 1.

Note that each environment contains N > 1 problems, therefore, we need to translate all problem instances into PDDL. Similar to Liu et al. [15], given one problem p 1 NL and its generated translation ˆ p 1 , we translate the rest of the problems p 2: N NL in a one-shot manner. That is, we generate ˆ p i := LLM 1 ( p 1 NL , ˆ p 1 , p i NL ) as the final problem translation for problem i for all 2 ≤ i ≤ N .

5 Experiments

Dataset. We consider PDDL files from real environments, taking nine domains from a combination of domain PDDLs from Liu et al. [15] and Seipp et al. [22]. The LLM may have seen the mentioned domains in its pre-training data, which is a common issue for current benchmarks. To mitigate this issue, we also modify the original Grippers domain, and create a modified domain called 'Grippersood' domain, to ensure no LLM has seen it previously. We generate natural domain descriptions for

all PDDL files by back-translating them using GPT-4 and manually inspecting and modifying the translations for correctness. For each environment, we consider one domain PDDL d and N = 10 problem PDDLs p 1: N . We use one problem for domain translation and EW evaluation, and all problems for evaluating a final domain response. We reserve the Blocksworld environment as an incontext example for prompting the LLM. As such, we do not evaluate the Blocksworld environment itself in our evaluations. See Appendices A.1 and C for more details on dataset curation.

Feedback Format. The natural language feedback given to LLM is in the following form: [Action sequence] [State description] . That is, we first provide LLM with the sequence of actions taken from one exploration walk, up until one action fails. Then, we provide the environment state description from the last step. We show an example of environment feedback and LLM response for the Termes environment in Listings 9 in the Appendix. We deliberately choose a simple feedback format to maintain the general applicability of our framework.

Baselines and Metrics. We use GPT-4 [1] ( gpt-4-1106-preview ) as the LLM since models with lower capability may struggle with syntax errors [9]. We consider the following methods: (1, 2) Intrinsic Planning (CoT): where the language model generates a complete plan without the help of any external planning library, based on the given descriptions, both with and without chain-of-thought prompting. This baseline does not leverage any classical planner or PDDL translation. (3) P&D Chain: Our proposed method (Algorithm 1) with n d = n p = 1 . (4) P&D Tree: Our proposed method with multiple response generations ( n d = 10 , n p = 5 ). (5) P&D Tree + DomProp: Our proposed method with multiple response generations and domain proposals for each problem (see Appendix B.2). Following prior works [17, 5], we set a maximum conversation turns of c max = 4 .

We run each algorithm for four seeds and compute the Best@4 metric, which takes the highest score among the four seeds. We report two metrics: (1) tasks solved 2 , measuring the fraction of the N = 10 tasks successfully solved (Eq. (1)), and (2) EW score, comparing the final domain through running exploration walks on all N problems (Eq. (2) with T max = 10 ). We use the original fast-downward [11] library for planning, the modified fast-downward library from text-world [6] for python-compatible state explorations, and the V AL [12] library to validate plans.

Results. Table 2 shows the final results on various environments. We consider a domain generation to be solved if a method achieves > 0 . 5 solve rate since we observe the rest of the errors are problem translation errors rather than domain translation errors. Our proposed method solves 7 out of 10 domains, compared to 3 solved by the Intrinsic CoT baseline. We also generally observe the correlation of EW score with task solve rate. Particularly, even when the task solve rate is zero, the EW metric shows signs of progress, e.g. , in domains such as Barman and Childsnack where all task solve rates are zero, the EW metric shows a clear distinction between method performances. Moreover, when the EW metric is high, such as 1 . 0 , we observe a generated PDDL domain to be very close to the ground-truth domain, and differing in very few predicates. For instance, in the case of the 'Hiking' environment, the P&D Chain achieves zero solve rate, but a perfect EW score, which we observe perfect solution in the case of P&D Tree.

Computational Cost. For the results in Table 2 using the GPT-4 model, we used 12 . 40 million input tokens and 8 . 73 million output tokens. Computing the EW is relatively negligible compared to the cost of LLM inference. In our experiments, computing the EW score for a single domain-problem pair takes less than two minutes on a 64-core server CPU.

6 Conclusion

In this work, we present a novel approach for modeling planning environments via PDDL translation using large language models (LLMs) and environment feedback, without relying on human intervention. The key contributions include introducing the Exploration Walk (EW) metric to measure domain similarity and guide domain refinement, and an iterative method that leverages LLMs to generate and refine PDDL domain and problem files. Evaluation on 10 real-world PDDL domains demonstrates the effectiveness of the proposed approach, outperforming a baseline that generates PDDL files in a single attempt without refinement. The method solves 7 out of 10 environments, achieving an average task solve rate of 66% and an average EW score of 0.84.

The current limitations include potentially insufficient and efficient exploration caused by random EW. More sophisticated EW strategies could improve the success rate while lowering the cost in the future. For example, strategies from the reinforcement learning literature ( e.g. , [27, 21]) could be adapted to improve exploration efficiency and success rates. Another limitation is that we have only applied the framework to PDDL environments, despite it being applicable to digital or even physical environments. We hope this work will inspire further research at the intersection of language models and planning, enabling the development of more advanced and autonomous planning systems.

References

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

A. Ahmetoglu, M. Y. Seker, J. Piater, E. Oztop, and E. Ugur. Deepsym: Deep symbol generation and rule learning for planning from unsupervised robot interaction. Journal of Artificial Intelligence Research , 75:709-745, 2022.

S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.

X. Chen, M. Lin, N. Schärli, and D. Zhou. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations , 2024. URL https: //openreview.net/forum?id=KuPixIqPiq .

M.-A. Côté, A. Kádár, X. Yuan, B. Kybartas, T. Barnes, E. Fine, J. Moore, R. Y. Tao, M. Hausknecht, L. E. Asri, M. Adada, W. Tay, and A. Trischler. Textworld: A learning environment for text-based games. CoRR , abs/1806.11532, 2018.

G. Dagan, F. Keller, and A. Lascarides. Dynamic planning with a llm. arXiv preprint arXiv:2308.06391 , 2023.

N. Dziri, X. Lu, M. Sclar, X. L. Li, L. Jiang, B. Y. Lin, S. Welleck, P. West, C. Bhagavatula, R. Le Bras, et al. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems , 36, 2024.

L. Guan, K. Valmeekam, S. Sreedharan, and S. Kambhampati. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https: //openreview.net/forum?id=zDbsSscmuj .

M. Han, Y. Zhu, S.-C. Zhu, Y. N. Wu, and Y. Zhu. Interpret: Interactive predicate learning from language feedback for generalizable task planning. In Robotics: Science and Systems (RSS) , 2024.

M. Helmert. The fast downward planning system. Journal of Artificial Intelligence Research , 26:191-246, 2006.

R. Howey, D. Long, and M. Fox. Val: Automatic plan validation, continuous effects and mixed initiative planning using pddl. In Proceedings of the 16th IEEE International Conference on Tools with Artificial Intelligence , ICTAI '04, page 294-301, USA, 2004. IEEE Computer Society. ISBN 076952236X. doi: 10.1109/ICTAI.2004.120. URL https://doi.org/10. 1109/ICTAI.2004.120 .

IPC. International planning competition, 1998. URL https://www.icaps-conference. org/competitions/ .

T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zeroshot reasoners. In Advances in Neural Information Processing Systems , volume 35, pages 22199-22213, 2022.

B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone. Llm+p: Empowering large language models with optimal planning proficiency. ArXiv , abs/2304.11477, 2023. URL https://api.semanticscholar.org/CorpusID:258298051 .

Q. Lyu, S. Havaldar, A. Stein, L. Zhang, D. Rao, E. Wong, M. Apidianaki, and C. CallisonBurch. Faithful chain-of-thought reasoning. In J. C. Park, Y. Arase, B. Hu, W. Lu, D. Wijaya, A. Purwarianti, and A. A. Krisnadhi, editors, Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 305-329, Nusa Dua, Bali, Nov. 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. ijcnlp-main.20. URL https://aclanthology.org/2023.ijcnlp-main.20 .

A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark. Self-refine: Iterative refinement with self-feedback. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum? id=S37hOerQLB .

D. McDermott. The 1998 ai planning systems competition. AI Magazine , 21(2):35-55, 2000.

A. Ni, S. Iyer, D. Radev, V. Stoyanov, W.-t. Yih, S. I. Wang, and X. V. Lin. Lever: Learning to verify language-to-code generation with execution. In Proceedings of the 40th International Conference on Machine Learning (ICML'23) , 2023.

J. Oswald, K. Srinivas, H. Kokel, J. Lee, M. Katz, and S. Sohrabi. Large language models as planning domain generators. In 34th International Conference on Automated Planning and Scheduling , 2024. URL https://openreview.net/forum?id=C88wQIv0aJ .

S. Sagar, A. Taparia, and R. Senanayake. Failures are fated, but can be faded: Characterizing and mitigating unwanted behaviors in large-scale vision and language models. arXiv preprint arXiv:2406.07145 , 2024.

J. Seipp, Á. Torralba, and J. Hoffmann. PDDL generators. https://doi.org/10.5281/ zenodo.6382173 , 2022.

T. Silver, V. Hariprasad, R. S. Shuttleworth, N. Kumar, T. Lozano-Pérez, and L. P. Kaelbling. PDDL planning with pretrained large language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop , 2022. URL https://openreview.net/forum?id= 1QMMUB4zfl .

T. Silver, R. Chitnis, N. Kumar, W. McClinton, T. Lozano-Pérez, L. Kaelbling, and J. B. Tenenbaum. Predicate invention for bilevel planning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 12120-12129, 2023.

T. Silver, S. Dan, K. Srinivas, J. Tenenbaum, L. Kaelbling, and M. Katz. Generalized planning in PDDL domains with pretrained large language models. In AAAI Conference on Artificial Intelligence (AAAI) , 2024.

K. Stein and A. Koller. Autoplanbench:: Automatically generating benchmarks for llm planners from pddl. arXiv preprint arXiv:2311.09830 , 2023.

R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction . A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.

K. Valmeekam, M. Marquez, A. Olmo, S. Sreedharan, and S. Kambhampati. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023. URL https://openreview.net/forum?id=YXogl4uQUO .

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-ofthought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824-24837, 2022.

Y. Xie, C. Yu, T. Zhu, J. Bai, Z. Gong, and H. Soh. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05128 , 2023.

S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. R. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum? id=5Xc1ecxO1h .

T. Zhang, T. Yu, T. Hashimoto, M. Lewis, W.-t. Yih, D. Fried, and S. Wang. Coder reviewer reranking for code generation. In International Conference on Machine Learning , pages 41832-41846. PMLR, 2023.

A Dataset

A.1 Dataset Details.

Dataset Examples. We provide an example of each file for the Grippers environment: (1) The ground-truth domain d (Listing 1) of ground truth PDDL domain (2) One ground-truth problem p (Listing 2) (3) Domain natural language description along with a PDDL template for action interfaces d NL (Listings 5 and 6) (4) Problem natural language description along with a PDDL template with the list of objects (Listings 3 and 4)

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.2

0.4

0.6

0.8

1.0

Average EW Score

Storage

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.25

0.50

0.75

1.00

Average EW Score

Floortile

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.4

0.6

0.8

1.0

Average EW Score

Hiking

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.4

0.6

0.8

1.0

Average EW Score

Childsnack

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.00

0.25

0.50

0.75

1.00

Average EW Score

Logistics

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.25

0.50

0.75

1.00

Average EW Score

Termes

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.25

0.50

0.75

1.00

Average EW Score

Driverlog

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.4

0.6

0.8

1.0

Average EW Score

Satellite

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.2

0.4

0.6

0.8

1.0

Average EW Score

Depot

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.4

0.6

0.8

1.0

Average EW Score

Rovers

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.6

0.8

1.0

Average EW Score

Barman

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.25

0.50

0.75

1.00

Average EW Score

Zenotravel

0

1

2

3

4

5

6

7

8

9

10

Term Difference

0.25

0.50

0.75

1.00

Average EW Score

Parking

2

3

4

5

6

7 2. pick: This action allows a robot to pick up an object using one of its grippers. The preconditions are that the object and the robot must be in the same room , and the specified gripper must be free (not holding any object). The effect is that the robot is now carrying the object with the specified gripper , the object is no longer in the room , and the gripper is no longer free.

8

9

that can be used to pick up and drop objects. The goal is to move objects from their initial locations to the desired goal locations using the robots and their grippers.

The domain includes three actions:

move: This action allows a robot to move from one room to another. The precondition is that the robot must be in the starting room. The effect is that the robot is no longer in the starting room and is now in the destination room.

drop: This action allows a robot to drop an object it is carrying in a specific room using one of its grippers. The preconditions are that the robot must be carrying the object with the specified gripper and the robot must be in the specified room. The effect is that the object is now in the room , the gripper is free , and the robot is no longer carrying the object with that gripper.

10 0

10 1

10 2

10 3

10 4

10 5

Number of Lines

0

20

40

60

Frequency

A.2 Criticality of predicate design.

Here, we give an example on the delicacy of predicate design. Consider the Grippers environment, where each robot has two grippers: left gripper and right gripper. In our experiments, one of the main predicates that the LLM incorrectly generates is the free predicate (see Listing 8). This predicate keeps track of whether a gripper is free or not. Therefore, at first sight, (free ?g - gripper) seems a natural choice to show a particular gripper is not occupied and hence is capable of picking a ball. However, when designed this way, in contrast to (free ?r - robot ?g -gripper) (missing the robot argument), this small detail causes the final domain to be entirely wrong! The reason is that there would no longer be any association between a robot and its two grippers. Therefore, on the incorrect domain, one robot will be able to pickup an object with the gripper of another robot! In fact, we observe that this incorrect design for the free predicate, is the reason behind the failure of the 'P&D Chain' method in Table 2.

We provide one more example from the Barman environment, illustrating the criticality of predicate design. The Barman environment involves actions related to manipulating containers (e.g., shot glasses, shakers) to prepare and serve drinks using various ingredients. One of the key predicates used in the domain is (used ?c - container ?b -beverage) , which keeps track of which beverage has been used in a specific container. This is important for actions like refilling or cleaning, where knowing the specific beverage type is essential to ensure conformation to the environment rules ( e.g. , a container can be refilled only with the beverage that it already had, otherwise, it needs to be cleaned first). However, we have observed that when the LLM generates the domain, it sometimes mistakenly omits the beverage argument, simplifying the predicate to (used ?c - container) . At first glance, this might seem like a harmless simplification, as the container usage is still tracked. However, this change results in significant problems in the overall domain behavior. Since the beverage is no longer specified, the domain can no longer differentiate between containers used for different types of beverages. This leads to situations where a container that has already been used for one beverage can be incorrectly treated as if it can hold another beverage without requiring proper cleaning or resetting actions. Such a mistake can cause the final domain to generate invalid plans, as the planner will fail to ensure that containers are used properly with respect to their contents, leading to cascading errors in tasks like mixing drinks, cleaning containers, or pouring from shakers.

B Implementation Details

In this section, we explain our implementation details.

B.1 One-shot prompting

To generate PDDL files (problem PDDL and domain PDDL), we always include a one-shot example prompt from the BlocksWorld environment. This environment is concise easy enough to fit into context, and explanatory enough to demonstrate example to the LLM for better output steerability. This includes problem generation, domain proposal, and problem refinement. For instance, when prompting the LLM to generate problem translation from natural language, e.g. , LLM ( p NL ) , we also prompt the LLM with an example from Blocksworld.

B.2 P&D Tree with Domain Proposal

As discussed in A.2, predicate design is challenging. Therefore, in one variant of our method, which we call 'P&D Tree DomProp', we propose for the LLM to first draft a domain proposal, then generate a problem PDDL based on the predicates found in the draft. This way, the LLM first generates domain-aware predicates, then generates the problem PDDL. Formally, line one in Algorithm 1 will be changed to the following two lines:

where the problem PDDL is generated by first creating a domain proposal.

B.3 Domain Refinement Strategy

Refinement Interface. For the domain refinement stage, in our early experiments we observed that prompting the LLM to regenerate the domain results into redundant output generation and

0.2

1.0

0.1

0.4

0.3

0.2

0.2

1.0

0.3

1.0

0.4

0.1

0.6

0.3

Problem Translation

Domain Refinement

Domain Proposal Draft

P&D Chain

P&D Tree

P&D Tree + Domain

Proposal

more importantly, sometimes modifies incorrect parts of the domain. For instance, the LLM had a high tendency towards changing the action interface signature, despite the instructions explicitly mentioning not to change the signature. As such, we provide a python interface for the LLM to modify a domain. The interface provides the LLM with the following two functions:

The first function adds predicates to the list of already created predicates, and the second one modifies the preconditions and effects of a particular action. Guan et al. [9] use a similar approach where they generate the domain PDDL one action at a time, and gradually create predicates. However, our python function interface allows for more flexibility, such as more convenient implementation as well as enabling the LLM to modify an action several times, or introduce predicates in between reasoning steps.

Domain Rating. Our main domain rating originates from the EW metric. When generating domain refinement strategies, the LLM may make mistakes hence failing before even the EW metric could be computed. For instance, the modification may be invalid, containing syntax error, or failing to fill parts of the template. To facilitate incorporating these into the EW metric strategy, we create the following rating system for each domain refinement modification:

when the EW metric is perfect ( i.e. , equals 1 . 0 ), we also run the planning on the environment ( i.e. , evaluate V d,p ( . ) ) and stop early if the plan is valid.

B.4 Further experiment details

LLMcalls per task. For each task in Algorithm 1, the overall complexity of LLM calls is O ( n p × n d × c max + N ) . The complexity of domain generation is O ( n p × c max × n d ) . This is because at first, n p problem candidates are generated and for each problem candidate the algorithm goes through a refinement procedure (lines 1 and 2 of Algorithm 1). The refinement is a tree with depth c max (where c max is the maximum number of refinement turns) (line 5), and at each level of the tree, one node is expanded with n d children (where n d is the number of domain refinement candidates) (line 6), which leads to O ( n p × c max × n d ) complexity. Once the domain is ready, the complexity of task generation for N tasks is O ( N ) since for each task we only call the LLM once to get a problem translation.

Number of successful seeds. In Table 2, we report the results over four seeds. To provide further analysis, we report the number of seeds a domain was successful in successfully generating a correct domain. The number of seeds that succeed in generating correct domain for the Termes, Movie, Miconic, Grippers, Hiking, Grippers-ood, and Floortile, are 4, 3, 3, 3, 2, 1, 1, respectively.

C Natural Language Description Generation

To generate natural language description of domains, problems, and environment states, we use the following strategies:

Domain: We use a few-shot translation strategy. We first pick three diverse environments of 'Grippers', 'Childsnack', and 'Termes' to manually (with assistance of GPT-4) curate domain translation. Then, we use these three domains as three-shot in-context examples to translate the rest of domains. The example prompt is provided in Listing 10.

Problems: We use a similar few-shot translation strategy for problem translation. We first pick two diverse environments of 'Termes' and 'Satellite' for problem two-shot problem translation. Once one problem from a target domain is translated, we use that problem translation as in-context example to translate the rest of the problems. This step is crucial to ensure all problems from the same domain are translated in a consistent manner. The example prompt is provided in Listing 11.

Natural Language Predicate Description: To generate natural language description of states, we generate a python files for each domain, with one function to produce natural language description of predicates for state description. The example prompt is provided in Listing 12.

the sandwich must not already exist. The effect of this action is that the bread and content are no longer available in the kitchen , and a sandwich is now

451 3. move -down: This action allows the robot to move from a higher position to a neighboring lower position. The preconditions are that the robot is at the starting position , the starting position is a neighbor to the destination position , the starting position has a certain height , and the destination position's height is one less than the starting position's height. The effect is that the robot is no longer at the starting position and is now at the destination position.

452 4. place -block: This action allows the robot to place a block at a neighboring position , increasing the height of that position by one. The preconditions are that the robot is at a position next to the block position , both positions have the same height , the robot has a block , and the block position is not a depot. The effect is that the height of the block position is increased by one, and the robot no longer has a block.

453 5. remove -block: This action allows the robot to remove a block from a neighboring position , decreasing the height of that position by one. The preconditions are that the robot is at a position next to the block position , the robot's position is one height unit higher than the block position , and the robot does not have a block. The effect is that the height of the block position is decreased by one, and the robot now has a block.

454 6. create -block: This action allows the robot to create a block at a depot. The preconditions are that the robot is at the depot and does not have a block. The effect is that the robot now has a block.

455

destroy -block: This action allows the robot to destroy a block at a depot. The preconditions are that the robot is at the depot and has a block. The effect is that the robot no longer has a block.

1 Your task is to translate problem PDDL files into natural language. Ensure that the resulting description covers all initial state and goal conditions.

2 DO NOT be lazy in your response , be extremely precise in your descriptions such that all conditions are covered in your description and there is no ambiguity in your description.

3 If you do not find any common rule about some conditions , list all of them.

4 For the initial conditions , start with "Initially:", and for the goal conditions , start with "Your goal is to".

5

6

7

8

9

10

11

The

ALWAYS

Two wrap

your code

examples

Q:

Domain

Description:

'''markdown

The

Termes behavior

domain is

a

of robotic

move around ,

structures , pick

in provided

the appropriate

below.

planning domain

agents

(inspired up

blocks , and

includes remove

actions removing

depot.

actions blocks ,

defined markdown

that syntax.

simulates by

stack the

termites)

them blocks

for from

moving and

in the

creating this

to build

structures.

robot , and

domain

The placing

destroying that

can domain

and blocks

include:

12 1. move: This action allows the robot to move from one position to another at the same height. The preconditions are that the robot is at the starting position , the starting position is a neighbor to the destination position , and both positions have the same height. The effect is that the robot is no longer at the starting position and is now at the destination position.

13 2. move -up: This action allows the robot to move from a lower position to a neighboring higher position. The preconditions are that the robot is at the starting position , the starting position is a neighbor to the destination position , the starting position has a certain height , and the destination position's height is one less than the starting position's height. The effect is that the robot is no longer at the starting position and is now at the destination position.

14 3. move -down: This action allows the robot to move from a higher position to a neighboring lower position. The preconditions are that the robot is at the starting position , the starting position is a neighbor to the destination position , the starting position has a certain height , and the destination position's height is one less than the starting position's height. The effect is that the robot is no longer at the starting position and is now at the destination position.

15 4. place -block: This action allows the robot to place a block at a neighboring position , increasing the height of that position by one. The preconditions are that the robot is at a position next to the block position , both positions have the same height , the robot has a block , and the block position is not a depot. The effect is that the height of the block position is increased by one, and the robot no longer has a block.

16 5. remove -block: This action allows the robot to remove a block from a neighboring position , decreasing the height of that position by one. The preconditions are that the robot is at a position next to the block position , the robot's position is one height unit higher than the block position , and the robot does not have a block. The effect

are at

a

Listing 11: Domain back-translation prompt template, with domain natural language description and problem PDDL placeholders for each target domain.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims are supported by experiments.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In the conclusion section, we discuss limitations of our work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe our method in Algorithm 1, and Appendix B. The code will be made publicly available upon publication.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The code will be made publicly available upon publication. We explain our method in Algorithm 1. For the data, the PDDL files are publicly available, and we provide examples on how to obtain natural language descriptions.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Our method uses pre-trained large language models, and we provide examples in Algorithm 1 on how we prompt the LLMs.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Figure 2b and Figure 4 show standard error. Table 2 does not contain any confidence intervals as the experiments become computationally expensive.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In the experiments section, we reported the number of tokens used to create Table 2.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The setup used in our work does not have direct societal impact as it is centered around generating PDDL code.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work is a fundamental work and poses no such risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the sources from which we use library, data, and code.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The code will be made publicly available upon publication

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an approach to leverage LLMs and environment feedback to automatically generate PDDL domain and problem description files without human intervention.  They do so by an iterative refinement approach that generates multiple PDDL problem and domain candidates based on feedback obtained from the environment.  The authors show their approach works experimentally in 66% of 10 PDDL domains that they have tried.\n\nSTRENGTHS:\nThe problem addressed in the paper is an important and interesting problem. The proposed approach with regard to the EW metric is novel and promising.\n\nWEAKNESSES:\nAssumptions: Assumption 2 may not be realistic. Often times, people may not know exactly what is the right way to capture the domain knowledge, that is what kind of things they should have said to ensure the pre/effects/initial state all are captured. What about the case that the domain description is missing a constraint or precondition?\n\nEnvironment requirement: in regard to the applicability of the work, there is a dependency on the environment to do the refinement, and that also may limit the impact of the proposed solution as the environment may not always be available for all domains. \n\n\nNovelty: the authors claim to be the first to enable PDDL generation using LLM without human intervention. However, there exists at least two related work that does also generate PDDL domain and problem without human intervention:   \n\n1. Large Language Models as Planning Domain Generators ICAPS 2024 (https://github.com/IBM/NL2PDDL)\n\n2. AUTOPLANBENCH: Automatically generating benchmarks for LLM planners from PDDL PRL 2024 (https://github.\ncom/minecraft-saar/autoplanbench/tree/main.)\n\n\n\nThe paper presentation can be improved. See the question section.\n\nQUESTIONS:\n1. 66% solve rate what does it mean? Does it mean 7 out of 10 problems are solved? Also what does that mean? Is the PDDL now correct? How many problems of that domain is the different approaches now solving correctly? Can you please further clarify. \n\n2. How do you know if you have a planning problem at hand, which can be turned into PDDL? Have you tried it on Alfworld (has a PDDL but its not ideal), or Mind2Web, or anything that does not have an existing PDDL. Maybe this relates to assumption 2. \n\n3. With regard to the refinement how do you know when to stop? Is there a threshold on the metric that would give you signals on when to stop the refinement?\n\n4. Can you say anything in regard to algorithm 1's soundness/correctness  (and the approach in general).  Also with respect to algorithm 1, how many times a LLM model should be called to come up with a reasonable PDDL domain, and a reasonable PDDL problem? Can you comment on the cost associated with that as well (any information beyond the token size would be great). \n\n5. In the notation section 3, why is the set of all possible actions A separate from D, isn’t A always part of D. \n\n6. Regarding planning error, what about unsolvable instances?  Do you assume all instances are solvable? \n\n7. Regarding equation 1, can you clarify that you generate multiple problem instances, but only one domain, or also refining the domain multiple times, each time generating multiple problem files? \n\n8. Can you clarify why you need the ground truth p and d, how do you use the ground truth q to validate the answers. Also what happens if you are not provided with the ground truth domain/problem. Does this mean that even though a ground truth domain/problem is given we are going to use LLMs to generate the domain/problem? \n\n9. Can you please comment on applicability of your approach if the PDDL environment is not known/given?\n\nLIMITATIONS:\nAssumption 2 may be limiting the scope and the level of impact of the work.  Also what about cases that the ground truth domain/problem is not known which is in most cases when it comes to real applications. Also having to rely on the PDDL environment is also limiting the scope and impact of the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an approach that leverages LLMs to generate PDDL domain and problem files from natural language descriptions, and refine them iteratively based on environment interactions. In particular, it proposes an Exploration Walk (EW) metric that provides feedback signals to guide the iterative refinement process. In experiments, the proposed method successfully recovers PDDL files in 7 of 10 domains, outperforming LLM-based planners.\n\nSTRENGTHS:\n- The paper studies the important problem of learning PDDL domains from embodied interaction for classical planning. This is a promising direction to enable long-horizon planning with formal guarantee. You may also find this recent paper [1] highly relevant.\n- In contrast to existing work that require human intervention, the paper boldly attempts to generate PDDL files automatically using feedabck from environment interaction. The proposed EW metric does make some sense to me.\n- In experiments, the method seems to be fairly capable in recovering valid PDDL files.\n\n[1] Han, Muzhi, Yifeng Zhu, Song-Chun Zhu, Ying Nian Wu, and Yuke Zhu. \"InterPreT: Interactive Predicate Learning from Language Feedback for Generalizable Task Planning.\" RSS 2024.\n\nWEAKNESSES:\n- My major concern is on the major contribution of the paper - the automatic mechanism that iteratively refines the generated PDDL files.\n  - **EW does not provide a sufficient objective.** The paper presents an Exploration Walk (EW) metric to provide feedback for the refinement process. The metric measures the difference of generated PDDL domain and ground truth environment by the feasible action sequences within. While EW=1 is the necessary condition for the generated domain to be valid, it is not the sufficient condition. I agree that EW can provide guidance at the initial stage, but in the end the objective in Equation (1) should be the one to optimize to produce a valid domain.\n  - **The exact feedback not explained enough.** The paper seems not to elaborate on the form of feedback provided to LLM for refinement - given the EW score which is a number. While the authors mention this briefly in the Appedix, I'm still don't fully understand how it works exactly. As this is the key part that makes the proposed approach possible, I would suggest the authors to provide more details in the main paper\n  - **The effectiveness of scalar-based feedback is doubtable.** Also, given the feedback is a number that provides little information on what the exact issue is (whether it's on problem file or domain file, whether it's on a precondition / effect term or on the predicate design, and which line), I doubt whether the LLM can perform reasonable refinement. I think it's highly possible that the iterative refinement process will goto nowhere.\n- Another important doubt is on the problem setup - where the natual language descriptions are translated line-by-line from the ground-truth PDDL files. \n  - **This setup is foundamentally different from what the problem of \"generating PDDL\" should be.** Under this setup, the challenge is no long generating PDDL files that requires exploiting environment interactions, but **translating** natural language into PDDL precisely without losing any information. More specifically, I believe the difficulty is to identify important predicates. Once the predicates are ready, the precondition & effect terms and initial & goal states should be relatively simple to be translated with GPT-4 with some prompt engineering.\n  - **The proposed approach seems to be unaligned with the challenge**. While I think utilizing something similar to EW metric is the way to go, it doesn't aligns well with the challenge posed by this problem setup.\n\nQUESTIONS:\n- What is $c_{max}$ in Algorithm 1? If it is the refinement iterations, how many iterations are used in the paper?\n- In experiments, the method runs 4 times and the best result is used for evaluation. I'm curious what the statistics of the 4 runs look like. Does a \"magical seed\" lead to good results while the others fail?\n- The language descriptions are generated by a GPT-4 from ground-truth PDDL files. I will expect there might be missing items or hullicination. I wonder, do you manually check and fix the generated outputs?\n\nLIMITATIONS:\nSee Weakness and Questions", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work talks about generating PDDL domain and problem files with LLMs. Specifically, it improves existing frameworks, particularly Guan et al. [8], in terms of increasing the degree of automation & eliminating the need for human corrective feedback. The core contribution of this work is the EW score. To compute the score, it only requires access to a set of executable plans and an executability checker (which can be either a simulator or the actual environment). The EW score is used to select sampled domain models given by the LLMs.\n\nSTRENGTHS:\n- The paper is well-written, with precise and rigorous wording and formalism.\n- The attempt to reduce the need for human feedback in domain generation is a meaningful and useful step forward. \n- The introduction of the EW score not only forms the foundation of this work but also holds potential for future applications/research (e.g., for evaluation or as a heuristic)\n\nWEAKNESSES:\n1. The domain model sampling/generation is done in a relatively simple way. The feedback message could be more informative than just indicating the inexecutable step or action in a plan.\n\n2. Regarding the structure of the related work section, the distinction between \"intrinsic reasoning\" and  \"external reasoning\" seems unclear to me, especially given that \"with the assistance of basic external tools [14]\" is mentioned under the \"intrinsic reasoning\" subsection. Also, even for the task of PDDL generation,  certain degree of \"intrinsic reasoning\" is needed. Rather than saying \"intrinsic reasoning\", I guess the authors likely meant \"direct plan generation.\"\n\n3. It doesn’t make it clear how much knowledge is provided in the domain NL description. Clarifying on this helps readers understand whether this work leverages LLMs as a knowledge source or a \"translator.\" -- from the examples in appendix, it seems that LLMs are used as the latter in this work (note that I am not saying translation is trivial)\n\n4. While the EM score may be effective for selecting candidate domain models and guiding their generation, its suitability as an evaluation metric is questionable. We know there exists a \"ground-truth\" PDDL and our goal is to fully recover its functionality. This is a binary 0/1 problem. This is not like a generated model with 0.8 avg solve rate is more usable than a model with 0.2 avg solve rate.\n\n5. Also, I think it's important to mention that the avg solve rate can only serve as an approximate measure of the equivalency between two domain models. A 100% avg. solve rate doesn't guarantee model equivalency (but this seems to be an easier-to-compute measure)\n\n6. Assumption 2 is stated in a loose way. An NL description of a domain can be given at different degrees of detail (which correspond to different levels of difficulty in domain PDDL generation).\n\n7. Line 211: I think it's better to say \"as long as PDDL is expressive enough to capture the dynamic / working mechanisms of the environment\" rather than \"the env supports PDDL action interface.\"\n\n8. It's unclear what the takeaway of Sec. 4.2 should be. Firstly, \"plan-not-found\" only accounts for a certain fraction of consequences caused by removing a term or predicate. Other consequences, such as producing invalid plans, can also occur. Secondly, it is well known that obtaining a valid domain model is challenging, even for humans. The authors should better explain the connection between Sec. 4.2 and the other parts of the paper.\n\n9. The authors should give more information on the computational complexity/cost (e.g., time consumption) associated with the calculation of EW score per candidate model.\n\n------------------------\nOverall, I find this manuscript well-written, and the idea can be valuable to the community. Therefore, I am leaning towards recommending acceptance.\n\nQUESTIONS:\nSee the weakness section.\n\nLIMITATIONS:\nSee the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents an approach for modeling planning environments via PDDL generation using LLMs and environment feedback, without relying on human intervention. This is achieved by an Exploration Walk (EW) metric to measure domain similarity and guide domain refinement, and an iterative rectifying method that leverages LLMs to generate and refine PDDL domain and problem files. The evaluation of this method is performed alongside baselines on ten different standard planning domains from IPC.\n\nSTRENGTHS:\n1. The presentation of the work is quite clear and concise. \n2. The exploration walk method included in the PDDL file correction loop is a rather simple and unsophisticated way to obtain approximate scores for the domain generation process.\n3. A decent amount of experimentation and analysis have been performed and stated in the work.\n\nWEAKNESSES:\n1. Section 4.2 - demonstration of the brittleness of PDDL generation can be made more realistic such as additionally including hallucinated object identifiers or actions or symbols, which are highly probable with LLM-based code format generators.\n2. The PDDL generation with LLMs approach is not as novel and the exploration walk/ environment feedback approach may not be as useful in generating completely new domains from descriptions or making custom modifications to existing domain files. The method does not really differ for rectifying problem files.\n3. There are intrinsic problems with generating domain files from descriptions for domains such as Barman - where the levels of shaker and actions such as clean shot, empty shot do not translate well for LLM-based generation. More analysis and description in this line of argument are necessary.\n\nQUESTIONS:\n1. This work on generating programs for planning problems [https://arxiv.org/abs/2305.11014] may also need to be cited in related work.\n2. The authors may benefit from more sophisticated exploration approaches in this paper [https://arxiv.org/abs/2406.07145] that solve a different problem.\n\nLIMITATIONS:\nLimitations have been addressed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}