Generating Symbolic World Models via Test-time Scaling of Large Language Models
Authors: Zhouliang Yu
Yuhuan Yuan
Tim Z. Xiao
Fuxiang Frank Xia
Jie Fu
Ge Zhang
Ge lin
Weiyang Liu
Venue: Accepted by TMLR ()

Abstract: Solving complex planning problems requires Large Language Models (LLMs) to explicitly model the state transition to avoid rule violations, comply with constraints, and ensure optimality—a task hindered by the inherent ambiguity of natural language. To overcome such ambiguity, Planning Domain Definition Language (PDDL) is leveraged as a planning abstraction that enables precise and formal state descriptions. With PDDL, we can generate a symbolic world model where classic searching algorithms, such as A*, can be seamlessly applied to find optimal plans. However, directly generating PDDL domains with current LLMs remains an open challenge due to the lack of PDDL training data. To address this challenge, we propose to scale up the test-time computation of LLMs to enhance their PDDL reasoning
capabilities, thereby enabling the generation of high-quality PDDL domains. Specifically, we introduce a simple yet effective algorithm, which first employs a Best-of-N sampling approach to improve the quality of the initial solution and then refines the solution in a fine-grained manner with verbalized machine learning. Our method outperforms o1-mini by a considerable margin in the generation of PDDL domains, achieving over 50% success rate on two tasks (i.e., generating PDDL domains from natural language description or PDDL problems). This is done without requiring additional training. By taking advantage of PDDL as state abstraction, our method is able to outperform the current state-of-the-art methods on almost all competition-level planning tasks.

Published in Transactions on Machine Learning Research (05/2025)
Generating Symbolic World Models via Test-time Scaling of Large Language Models
Zhouliang Yul.2,, Yuhuan Yuan",, Tim Z. Xiao*, Fuxiang Frank Xia², Jie Fu", Ge Zhang?, Ge Lin Weiyang Liu
'The Chinese University of Hong Kong "Hong Kong University of Science and Technology
*HKUST (Guangzhou) 4Max Planck Institute for Intelligent Systems, Tübingen
Environmental Systems Research Institute "Shanghai Artificial Intelligence Laboratory 'SEDD, Bytedance
Reviewed on OpenReview: httppr/r/ppaaaneeea.eee///////rr
Abstract
Solving complex planning problems requires Large Language Models (LLMss) to explicitly model the state transition to avoid rule violations, comply with constraints, and ensure op- timality-a task hindered by the inherent ambiguity of natural language. To overcome such ambiguity, Planning Domain Definition Language (PDDL.) leveraged as planning ab- straction that enables prccise and formal state descriptions. With PDDL, we can generate a symbolic world model where classic searching algorithms, such as Ax, can be seamlessly ap- plied to find optimal plans. However, directly generating PDDL domains with current LLMs remains an open challenge due to the lack of PDDL training data. To address this challenge, we propose to scale up the test-time computation of LLMs to enhance their PDDL reasoning capabilities, thereby enabling the generation of high-quality PDDL. domains. Specifically, we introduce a simple yet effective algorithm, which first employs a Best-of-N sampling approach to improve the quality of the initial solution and then refines the solution in a fine-grained manner with verbalized machine learning. Our method outperforms ol-mini by considerable margin in the generation of PDDL domains, achieving over 50% success rate on two tasks (f... generating PDDL domains from natural language description or PDDL problems). This is done without requiring additional training. By taking advantage of PDDL as state abstraction, our method is able to outperform current state-of-the-art methods on almost all competition-level planning tasks.
1 Introduction
Enabling large language models (LLMs) [22] to plan in complex scenarios like Barman, Floortile, and Termes remains an open problem [45]. While recent LLMs like OpenAl-ol-mini and OpenAl-ol-preview [22] excel at complex reasoning tasks, including coding [9] and mathematics
[11], they still struggle with doductive reasoning and principled planning [10, 45, 43) that requires the consideration of optimality, constraints, and complex state transitions. This limitation persists in ol-preview even when using self- self-critique techniques and multiple answer re-sampling strategies (See Figure 2). A natural solution is translating the world abstraction from natural language into PDDL [27], which utilizes first-order logic (FOL) to explicitly describe states and relationships. Compared to natural language, the formal nature of PDDL simplifies verification and enables the precise specification of constraints and objectives, facilitating the seamless integration of off-the-shelf planning algorithms. However, it remains an open challenge to translate natural language descriptions into PDDL. domains with accurate syntax and semantics. Current LLMs perform poorly in this translation task due to two key challenges: the scarcity of high-quality PDDL training data. [61] and the complexity of
"Equal contribution. !Corresponding Authors.

---

Published in Transactions on Machine Learning Research (05/2025)
NoN Initialization
MVML
Thought (M)
Domain D (M)
Objective
0
Predicates::
Predicates:
On:
Thought )
Domain D
Explanation
LLM fopt.
Predicates:
Predicates:
Domain Generation
Actions:
On:
(Clear? x)
To D
Stack:
Explanation
k. (on tabless)
LLM
Explanatio
for all k
Actions:
Actions:
(k)
F!
Pickupp
Procendition
Explanattio
Effect
Stack:
Precondition
LLM S Sugebate
Effect
D
Task
Planning
NomaDomii
Pomain
Problem
Domain
You expert pdli domain generation. Please generate pódl domaiin for the following natural language description: (NL, description)
expert pddi generation. Please generate the corresponding PDDI domain for the following problem code: (problem, code)
(:damain blockswarld)
(defiine (prublem BW-rand-12)
(sobjects bl b2 b3 b4 ...)
(:init
Search Algorithm
(arm-cmply Kon-table...
(ggall
Figure An overview the proposed method. Our test-time compute scaling approach consists of two main steps: (1) Best-of-N Sampling for PDDL Initialization (see Section 3.2): We start by running parallel sampling process to generate multiple chain-of-thought responses that are composed of the formalized PDDL-based world model representation ( and the natural language thought T! (2) Closed-loop Iteration with iVML (see Section 3.3): We use iVML to iteratively improve the solutions. iVML incorporates: (1) An optimizer LLM fopt that evaluates the solutions from the previous iteration, and (2) A learner LLM fupdate that learns from the feedback and updates the PDDL-based world model D,, Here, N represents the total number candidate solutions generated, k is the index of the lop candidates retained for further optimization (with K indicating the total number of such candidates), and the index is used to denote the iteration step within the optimization procedure. The optimal PDDL-based world model will be used in the systematic search engine for planning.
maintaining logical consistency across predicates and actions [26]. Traditionally, the translation process has heavily relied on human expertise [17] or orale environment [37], making it difficult aulomate and scale.
To address these challenges, our work leverages LLMs to generate PDDL-based symbolic world models for task planning without requiring model fine-tuning. We achieve this through simple yet effective strategy to scale the test-time computation of LLMs. Specifically, start by generating multiple PDDL domains from the input text query using Best-of-N (BoN) sampling [46]. This step aims to generate diverse candidate solutions for initializing the verbalized optimization process using high-temperature sampling. Then we refine the best candidates by optimizing the generated PDDL domains with the proposed instance verbalized machine learning (iVML.) such that the generated PDDT. domain can gradually fit the input query (e.g., natural language descriptions or PDDL problems).
Our method is guided by the insight that effective test-time scaling elicit stronger reasoning capabilities over formal languages like PDDL, thereby compensating for the scarcity of high-quality PDDL training data. We start by applying BoN sampling to explore the solution space and select the best initial solution, and then iVML aims to exploit the solution space around this initialization such that the solution gets gradually improved. Verbalized Machine Learning (VML; [50]) is a test-time in-context training framework where the main idea is to paramcterize functions using text prompts. Such next-parameterisod functions can be evaluated through LLMs. To solve machine learning task, VML iteratively refines a learner LLM's text prompt based feedback from an optimizer LLM, which takes into account the training data and learn-
2

---

Published in Transactions on Machine Learning Research (05/2025)
ing objectives. The learner LLM's text prompt characterizes data patterns to perform inductive inference tasks such as regression and classification. In our paper, we apply VML. to PDDL domain refinement and propose the iVML framework that reformulates VML for instance learning. Unlike the original tasks in [50], PDDL domain refinement is an instance learning task without training data to produce verbalized gradients. Therefore, rather than requiring training data, we use LLMs to verily the validity of PDDL domains and generate critiques that serve as verbalized gradients. Specifically, the learner LLM receives an initial PDDL domain and generates a refined version based on critiques from an optimizer LLM, iteratively eliminating logical inconsistencies and grammatical errors. However, the performance of iVML is highly dependent on the quality of initial solutions, as poor initialization can result in slow convergence or suboptimal solutions. While BoN sampling emphasizes exploration by independently generating diverse solutions, it does not lever- age past predictions from LLMs, hence limiting its ability to exploit the solution space. iVML, in rontrast, emphasizes exploitation by iteratively refining solutions based on the optimizer LLM's feedback. To achieve a good balance between exploration and exploitation, we propose to combine these complementary strategies for generating high-quality PDDL domains. This hybrid approach leverages the strengths of both methods by applying iVML to refine solutions initially generated through BoN sampling. Our contributions include:
Scalable PDDL domain generation: We propose an effective test-time scaling approach for automatic and scalable PDDL domain generation without additional model training. Using Swwen2.5-Coder-7B as the base model, our approach achieves state-of-the-ae performance with an 85.2% success rate on the NT.2Domain task and 71.4% on Prob2Domain, substantially surpassing ol-mini's performance (41.7% and 33.7%, respectively).
Application of VML to instance learning. We introduce the iVML framework to adapt VML to instance learning, where no training data is available. We use LLMs to check the validity of PDDL domains and generate textual critiques as gradients to iteratively update PDDL domains.
Efficient test-time compute scaling: We enhance VML with BoN sampling initialization, effectively balancing exploration and exploitation to achieve faster convergence and obtain better solutions.
Robust planning through PDDL abstraction: We demonstrate that PDDL-based formal abstrac- tion enables more robust planning compared to direct LLM-based approaches. Our method successfully handles complex domains such as Barman and Termes, where existing LLM-based planners fail.
2
Related Work
2.1 LLMs for Task Planning
Recent advances in large language models (LLMs) (e.g., GPT-4 have shown promise in handling common reasoning tasks, such as GSM8K [11] and HumanFval [8]. The gain of reasoning capabilities of T.L.Ms can be attributed to (but not limited to) several factors: (1) Extensive training on reasoning datasets: [54] fine- tunes LLMs with distilled fhain-of-thought datasets to improve plausible reasoning, and recent models have scaled this approach using larger and more diverse datasets, which has enhanced performance on tasks such as mathematical reasoning, coding, and logical reasoning. However, such an approach raises concerns about whether the observed gains are attributable to data contamination [39, 28]. (2) Self-improvement during inference time: Some approaches incorporate verifiers that provide synthetic feedback during inference, using self-reflection [36], process reward models [57], or simple sparse objective reward [59] to guide improvement. However, [39] shows that imperfect verifiers increase false positive samples during scaling up reasoning at inference time. Recent advanced crasoning-focused LLMs (e.g., ol-mini and ol-preview) have adopted both strategies to train extensively on reasoning datasets [22] and leverage the long hidden-chain of thought [48] during inference, making them essentially different from vanilla LLMs (e.g., GPT-4). However, both GPT- 4 and ol-preview exhibit limitations in lackling complex planning tasks. As demonstrated by [44], neither GPT-4 nor ol-preview succeeded in solving any instances of very complex planning problems such as Termes, Barman, and Floortile. While ol-preview demonstrated marginally higher accuracy on relatively simpler planning tasks (e.g., BlockWorld and Tyreworld), they are observed making frequent hallucination by rule violation, inabilities to generate feasible plans, misinterpretation of goal state, etc.
3

---

Published in Transactions on Machine Learning Research (05/2025)
Natural Language Description of T'rmees
You control robot that can take the following actions to build complex structures. Move from position to another. The new position and the old position must be al the same height. block higher than old pasition. Move down from position znother, and height the
Move from position another, and the height the position position is one block lower than the old position. Place black at neighhoring position from the robot's current position. The robot must have block. The current height at the rohot's position and the block's position must he the same. A block cannot he placed at the depot. The height the block's position will be one block higher than the current height. the
current position. The robot robot's position must be the same the new height at the block's position. The new height the black's position will be one black lawer than the current beight. Create block at the depot. The
Remove a black neighhoring position
have a block. A cannot be removed from depot. current height at
block. Destro block the depos.
have black.
The robot grid with and columms.
pos-0-U pos-U-1 pos-0-2
pos-1-00 1-00 pos-1-1 pos-1-2
pos-2-0 pos-2-1 pos-2-2
pos-3-4l pos-3-1 pos-1-2
The rolott pos-2-0.
The depot for new blocks is at pas-2-0. O. The maximum height of blocks is 3. Your goal is to build blocks so that the beightt pos-1-2 is You cannot have an unplaced block at end. Remember, in each step, you have to make sure that you da not violare the rale.
L.M as Planner
Stute Estimation heights pos-ll 1-2
Z
Z
Z
2
X
X
al. Only dipot genentte remote bludis.
Initial State:
Step I:
At the depor (pos 2 0L, create block (robot now has Book!; Move from pos2 post 0, post
Step 2:
At the depet (pmn-2-0), enzrc block frobor now lax black). Move from post o then
b). Depot in pos-240
Place block pas-1-2 1-2 pas-1-1 (heights pas-1- and
pos-6-1 pos- (hoights both 0), increasing beight pos-O-3 I. Back the pool-0
Macce
pos-
d) Rack to the pos2-n
Vielatin: Canmot place frrm poselll to pos-
Rule Violation: Cannot place from pos pos directty, these positions are not nrighhoring Acconding the rales: a biock
diceclly, these positions are not neigthocing. According to the riloe a block must be plased at posiám pusianaljuunn aijuscii to the
are many emons canner get poal siste siccesfillyy
must be placed at possion adjacent the rabue's current location.
tobot's current lccusion.
Z
Z
Y
Y
3
1). depot (pos-2-00 2-0, block blocks,
Goal State: a). Build block that the height pos-1-2 is
up to pondhsl;
b). Move tiem poo2.0 so pos1-0, to pos move
5). Place the block pos 1-2 from pos 0-1 (heights of pos
x). depul (pue-200), meate bluck (rubut now has bluck);
<). Place the block at pos-1-2 from pos- (Leights at pos-O. D-1 and pos-1-2
b). Move from post to pos i-0, then post IJ then move up to pos O-1;
and pos-1-2 1-2 are both I). d). Budk the 2.5.2.0.
are both 2).
d). The heiglt pose is 3, matching the goul.
L.M as Critique
The robot successfully builds a height 3 at pos- while adhering stricrly to the given rules. The plan is optimal terms of both the number of steps and compliance, ensuring that the robor not have any unplaced blocks end.
Figure 2: OpenAL-ol-prentt prevewww plans for Termes: ol-preview frequently exhibits hallucination during the planning process. Specifically, in steps three and four, the LLM violates predefined rules when selecting and leveraging actions. Additionally, step four hallucinates the achievement of the goal, leading to incorrect or unrealistic outcomes. Even when using ol-preview itself evaluate the hallucinated plan, it incorrectly identifies the plan as valid.
Although vanilla LLMs can approximate state transitions [18], they do so by probabilisticall predicting subsequent tokens based on patterns learned from vast datasets rather than through logical deduction or structured inference [24]. According to [43], ol-preview similarly relies on approximate reasoning during planning, executing fuzzy pattern matching that stems from its reinforcement. learning training paradigm.
This approach, while effective for certain reasoning tasks, fundamentally differs from structured logical infer- ence [10]. Inspired by traditional model-based reinforcement learning approaches [2], which solve decision- making problems by predicting discrete world models for heuristic search, our method significantly enhances LLMs' planning capabilities through a two-stage process. First, we predict a symbolic Planning Domain Definition Language (PDDL)-based world model using an instance verbalized machine learning approach. This stage transforms the problem into a structured, symbolic representation, enabling more logical and systematic reasoning. Second, we leverage heuristic search methods, such A+, efficiently find optimal solutions within this structured framework.
4

---

Published in Transactions on Machine Learning Research (05/2025)
Methods
Synthesis Objective
Benchmarking
Technical Method
Each action separately within PDDL domains
3 domains:
[17]
Human experts
Household, Logisties, Tyreworld
Whole PDDL problems (initial state and goal)
2 domains:
[61]
Finetumed on large dataset
Gripper, Blockworld
10 domains:
Interact with oracles
[26]
Whole PDDL domains
Gripper, Blockworlds, Termes, etc (d.e., Require ground-truth environment)
283 IPC domains (NL2Domain)
Test-time scaling without model training & oracle interaction
Ours
Whole PDDL domains
332 IPC domains (Prob2Domain)
Table 1: Comparison between current PDDL syuthesis methods and ours.
2.2 World Model Generation
However, automatically generating scalable PDDL-based world models by LLMs is still challenging. Current LLMs rely heavily on either human-in--te-loop) extensive training data, or oracle environment to plausibly be superior in generating PDDL on limited scenarios. For example, [17] leverages multiple human experts to refine the logical error in individual PDDL action expressions generated by LLMs. [61] collect more than 132,027 SFT data to train LLM on limited simple planning scenarios, (e.g., BlockWorld and Grippers). Mean- while, [30, 26, 38] employ iterative refinement processos for generated PDDL domains. These approaches access oracle environments consisting of ground-truth PDDL problems and PDDL domains and validate the generated PDDL domain by the feedback that compares the resulting plans against ground-truth plans generated by the oracles. By contrast, our approach generates PDDL domains in an oracle-agnostic manner, focusing on scalability scenarios where rulc-basod ground-truth signals are unavailable. To compare with other PDDL-related works, our method eliminates the need for training data and does not rely on an oracle to improve the generated PDDL domain (See Table 1).
Several methods have also explored using LLMs to generate world model representations other than PDDL. For example, GIF-MCTS [12] and World-coder [40] translate natural language descriptions of world models into Gym-like Python code [6], and use pre-collected trajectories validate and provide feedback for wrong state-transition predictions iteratively. Our work, however, focuses on more general planning scenarios without pre-collected validation datasets to provide critique feedback.
2.3 Adaptation of LLMs
In the absence of pre-collected validation datasets, the adaptation of LLMs for the PDDL-based world model presents unique challenges. While parameter-cfffcient finctuning methods (e.g., [21, 33, 25, 13]) euable effective adaplation of LLMs to downstream lasks, they still require high-quality training data. If there is insufficient training data, in-context learning [7, 47, 14] offers an alternative adaptation approach. Prompt optimization techniques [60, 31, 52] further enhance adaptation performance by deriving improved instruction prompts from limited training data. Recently, [50] proposes verbalized machine learning (VML) to update modeelchharacterizing text prompts with I.I.Ms. Unlike conventional machine learning settings, functions in VML are parameterized within the language space, and the "model" in the text prompt is iteratively refined through the in-context learning mechanisms of LLMs. [31, 55] introduce the concept of textual gradients criteria. for updating text prompts. Although finetuning methods can achieve effective adaptation, they risk causing catastrophic forgetting in pretrained LLMs, potentially compromising their general instruction-following capabilities. Therefore, test-time adaptation of LLMs to downstream tasks has emerged as a practical solution. In our paper, we introduce a simple test-time adaptation method for generating PDDL-based world models, which scales efficiently with test-time compute.
5

---

Published in Transactions on Machine Learning Research (05/2025)
3 The Proposed Test-time Compute Scaling Approach
Natural language task planning faces significant challenges in state estimation, constraint-based plan gener- ation, and plan validation. One approach to address this issue is to generate explicit, unambiguous world modols (e.g., PDDL) beforc conducting planning accordingly. Our proposed methods aim to explore the key question: How con we effectively generale symbolic world model via LLMs?
Generating symbolic world models requires not only natural language understanding but also sophisticated deductive reasoning to maintain logical consistency across all model components. Without such formal modeling sophistication, models risk generating inconsistent state transitions and producing suboptimal plans. To address this challenge, we enhance LLMs' reasoning capabilities through an instance verbalized machine learning algorithm (see Section 3.3), initializing it with better candidates generated by the best-of-N sampling. See Figure for the overview of our method.
3.1 PDDL-based World Model Representation
Classical planning problems are inherently complex. Even determining plan satisfiability [34]-whether any solution exists for a given planning problem is NP-hard. Planning problems that involve optimization under constraints pose even greater challenges for natural language planners. Our approach, which utilizes the PDDL-based representation, offers several advantages: (1) PDDL employs a logical system to express atoms and predicates derived from STRIPS [16] (í.e., Stanford Research Institute Problem Solver), and therefore provides formal and unambiguous syntax for representing world models. PDDL-based world modeling employs explicit representations that not only enrich the description of actions and states but also ensure precise and straightforward validation, thereby eliminating ambiguily. (2) Natural language plan prediction often reduces to an n-gram task of ungrounded token generation. We provide the description of the Termes problem in both natural language (sce Figure 2) and PDDL (sce the text box below) to illustrate the difference. By adopting PDDL as representation, we transform this task into explicit classical planning. This formulation enables the use of search algorithms such as Av. With the help of the PDDT, representation, planning graph (e.g., Figure 6) can be used provide improved heuristic estimates when searching through the state space.
3.2 Best-of-N Sampling for PDDL Initialization
Our test-time scaling approach adopts a two-stage optimization process. The first stage is search for good initial solutions via best-of-N sampling, and the second stage is to iteratively refine the initial solutions via instance verbalized machine learning algorithm (See Section 3.3). For cach problem, the LLM generates many candidate solutions in parallel and retains the K samples with the highest log-likelihoods. This process involves three main steps: candidate generation, scoring, and selection. During sampling, temperature parameter 7 is used to scale the logits (i.e., the log-likelihood of raw score associated with a certain token), and in this case we can represent the probability of token u. at position L as:
logit(w("))
exp
pe(w!
EvEy exp
logit(e)
where the temperature adjusts the sampling distribution. Higher temperature (e.g. 2 0.7) increases randomness and diversity in the generated candidates by flattening the probability distribution, giving relatively more weight to lower-probabiliyy tokens, in contrast, a lower temperature (e.g. T < 0.3) sharpens the distribution, favoring high-probability tokens and producing more deterministic outputs. In the scoring phase, each candidate o; is assigned a score S, based on the sum of the log-likelihoods of its generated tokens:
(u(()) )
S log
Ei (1,2....,N) NN
where: Lę is the length of candidate c₁. In the selection phase, the top K candidates with the highest scores are chosen as the initialization points to be optimized by iVML.

---

Published in Transactions on Machine Learning Research (05/2025)
Termes in Planning Domain Definition Language
Domain:
[define (domain termes)
(rrguireeenni styping :negative preconditions)
(-ypas
numb object
position object)
( predicaates
(hright -p- position nunh)
position)
block]
(SUEC
7n2 mmmb
(MEGCHORR position)]
position]
action
parametere (Irom position Tto poxition 1 numb)
precondition (and (at 2from Thrm (to](hiight Thom hag(he Tto Th3)
effectt (and (not (at alrm 7to)
action move-up
paramsters (?rom- position 7hfrom numb Io position 7hto - numb)
prscondition land(at Hfom (MBIHHOOR From 7to [he[(highht Hrom 7hfrom X height (SUCC hto Tntram))
reffect [and] not (at Mrom [](a 2ta)))
( (raction move-down
acction place-block ...
block block block
Problem
[define (problem termes-
Eddomain termes)
termes-0003B--0
20000-0000--kkk---addoooooo..o tar_b3a-rrddom_twwerr____ 1at_too___oers__________ 1x3
Initial state:
RDO
Goal state:
Maximal
(Jobjetts
e0 rumb
pox-0-0 position...
(:init
(height pox-6-0 nl) ...... [at pax-2-0)
(SUCC m1 =0 .......
[NEIGHORR pos pos 1 -0) ........
(S-DEROT poa-2-0)
(height pox-0-0 m0)
(hae-black) }313)
3.3 iMLL Instance Verbalized Machine learning
With the BoN sampling to select the candidates the initial solution, introduce instance verbalized machine learning to refine both the generated PDDL domain and the natural language chain of thought (CoT). iVML is an adaptation of VMT [50] to the instance optimization setting, where the goal is to optimize and refine a single instance (i.e., PDDL domains in this paper). In iVML, the functions are parameterized using natural language rather than numerical values in conventional machine learning. Viewing an LLM as the inference engine, we can evaluate such a natural language parameterized function (f... via sope and optimize its model parameters in the natural language space (i.e., via. fupdote (-)). In the context of iVML, the optimization objective is to generate PDDL domain D that correctly aligns with the input prompt description G, as formulated below
D arg min L(G,D),
where L(-) is a loss function defining the closeness between 9 and D. Here, 9 is always within the language spaces for all the tasks we considered, for example, in NL2Domain task, & corresponds to a natural language description of the planning domain, for Prob2Domain, G represents a PDDL problem instance. Solving Equation (3) is difficult as both G and D are text, and C(-) is hard to define unless abstractly using natural approximately solve Equation (3) with an iterative algorithm that alternates between two natural language parameterized functions at the iteration i:
language. Using the iVML framework,
F; (C, 9, T.-1, D,-1), T. Dj fupdate (F,, T T1-1,D--1).
(4)

---

Published in Transactions on Machine Learning Research (05/2025)
where T₃-1 and D,-1 correspond to the current thoughts and the current PDDL domain, F, is the feedback from the optimizer function fopt()) (-), T3 and D; are the updated thoughts and PDDI. domain output from the update function (-) Here, T, is generated by applying CoT sampling at each step of the PDDL domain generation process, it comprises all non-PDDL components the outputs of fupdate()) exchuding the final structured PDDL domain itself. Both Sopt(.) and Supdate(-) are parameterized by natural language. The initial PDDL domain Do is initialized from the best-of-N sampling. These two functions are evaluated through separate LLMs calls. We show the prompt templates for sope(-) and supdato()) below:
Prompt template for (.pt(.)
You will be provided natural language description of planning domain, and its corresponding PDDL domain code with intermediate thoughts explaining each predicate and action. Your task to generate critical feedback the PDDL domain code based the natural language description. You should evaluate the grammar and logic of the PDDL domain codes, and the logic error in the intermediate thoughts. PDDL synthesis problem: (9)
natural language chain of thoughts: (T:-1)
Generated PDDL domain: (D:-1}
Prompt template for fupdate)()
You will be provided a PDDL domain code and critical feedback on the PDDL domain code based on the natural language description. Your task is to generate new PDDL domain code that is more consistent with the natural language description.
PDDL synthesis problem: (9)
Natural language chain of thoughts at the previous turn: (T:-1} Generated PDDL domain the previous turn: (D,-1)
The error of the PDDL domain (F:-1}
iVML with BoN initialization effectively balances exploration and exploitation. BoN employs broad exploration strategy, maintaining diverse candidate solutions to probe distinct regions of the combi- natorial search space. While this approach mitigates initialization bias, it suffers from diminishing returns: beyond a critical sample size, the probability of discovering novel valid solutions decays due to redundant model generations. In contrast, iVML performs verbalized in-context exploitation by iteratively refining BoN-sclectod candidates based on objectives defined natural language. This closed-loop process enables precise error correction (e.g., resolving precondition conflicts in PDDL actions) but remains susceptible to local minima-a fundamental challenge in non-convex optimization [35]. Our approach combines BoN and iVML to achieve effective balance between exploration and exploitation, addressing the limitations of each method alone through a two-phase optimization framework: (1) BoN initialization that generates mul- tiple diverse and high-quality initializations (D) solutions with verbalized machine learning.
( Ii and (2) iVML refinement that optimizes these initial
4 Experiments and Results
Section 4.2 examines state-of-the-arl LLMs' capabilities PDDL domain synthesis and evaluates whether test-time scaling approaches enhance PDDL generation quality. Section 4.3 analyzes the convergence patterns of BoN and iVML, highlighting the specific improvements iVML offers over BoN. Section 4.4 investigates iVML's sensitivity to initialization conditions and whether diverse candidate sets can boost iVML perfor- mance. Section 4.5 explores whether iVMLL's advantages in PDDL domain synthesis extend to other symbolic code generation tasks within the PDDL language (e.g., PDDL problem synthesis), demonstrating its broader applicability. Section 4.6 assesses our planning approach (generating PDDL domains via iVML followed by heuristic search- rearch-based based planning) against LLM-based planners on complex planning problems.
8

---

Published in Transactions on Machine Learning Research (05/2025)
Model
Parama
NLaDomain (%)
Problem2Domain (%)
Avg. (%)
Open-Source Models
Qwen2.5- In-structt
0.513
0.0
0.0
0.0
Innstrrtttt
1.5B
0.0
0.0
0.0
Qwen2.5-1mm 5-Instruct
3B
2.1
1.5
Qwen2.5 Qwen2.5-Instrun 5-Instruct
713
5.7
11.7
8.7
2wen2.5-Instruct 5-Instruct
14B
21.6
25.3
23.5
Qwen2.5.4 2.-Istrutt
31.8
Qwen2.5-Instrue 5-Instruct
72B
38.5
32.8
35.7
Qwen2. 5-Coder
1.5B
0.0
0.0
0.0
Qwen2.5- 5-CCderr
7B
21.9
Llama3 1-Instruct
8B
0.0
0.0
0.0
Llama3. 1-Instruct
70B
1.1
0.0
0.6
Yi-1.5-Chat
Yi-1.5-Chat
9B
6.7
9.3
8.0
Yi-1--Chatt
34B
12.0
8.7
10.4
Yi-Coder
1.5B
0.0
0.0
0.0
Yi-Coder
9B
9.9
14.5
12.2
Models
GPT--4o
5.3
50.0
27.7
ol-mini
41.7
33.7
37.7
o1-preview
55.8
52.4
54.1
Our Methods
BoN-8- BoN-88QQee2..5 Qwen2.5 Insttruut
0.5B
0.0 (+0.0)
0.0 (+0.0)
0.0
BoN-8- Qwen2.55-nstee
1.5B
2.1 (+2.1)
0.3 (+0.3)
1.2
1.N.s--wen2..--mme 6-Instrout
3B
11.7 (+9.5)
1.2 (+0.3)
6.5
BoN-8- S.t....ee..2// 5-Instreut
7B
9.2 (+3.5)
34.6 (+22.99
21.9
BoN-8- BoN-8-Qwen2. 2.en2..-Inntrron
6-Instrout
14B
51.6 (+33.00)
62.0 (+-36.7)
56.8
BoN-8-
32B
66.8 (+466.7)
71.1 (+39.55)
70.9
BoN-8- w...-/ee...5../
5-Instruct
72B
60.8 (+22.3)
73.8 (+411.0)
67.3
BoN-8-Qwen2.5-Coder
7B
73.1 (+51.2)
63.3 (+444.9)
08.2
Yi.1..-a
(+39.9)
39.8 (+30.5)
BoN-S-Tlamaa. I-Instruct
8B
0.7 (+007)
0.0 (400.0
0.4
0.5B
0.0 (+-0.0)
0.0 (+0.00)
0.0
VVML-5-BoN&&&QQQee222. 5-Instruct
1.5B
2.8 (+2.8)
(+0.3)
VMM 5t-nttruut
3B
18.7 (+16.6)
1.8 (40.3)
10.3
WMLL--S-BBoNN&&sSceeee22255 Insttruct
7B
21.9 (+16.2)
49.1 (+37.3)
35.5
14B
(+55.4)
80.4 (+55.1)
VM
32B
86.2 (+-2.2)
90.9 (4-59.3)
88.6
WMM--S-BBoNN&s..eeee2255//lieee Instrtt
72B
78.4 (+39.9)
86.4 (+53.6)
82.4
7B
85.2 (+63.3)
71.4 (4553.0)
78.3
VVML-S-BaN.&.Vi-l1.
9B
93.9 (+84.0)
93.0 (+78.5)
93.5
iVML-5-BoN-4 Laam-aa Insttruct
8B
2.8 (+2.8)
(+0.0)
1.4
Table 2: A comparison of performance in PDDL domain synthesis between the baseline models (including both open-source and closed-source models) and our methods. BoN-8 refers BoN sampling with 8 candidates, while WVML-5-BNN-8 denotes five iterations of iVML training initialized with BoN-8. In this experiment, the metric refers to the percentages of the generated PDDL domains that pass VAL validation without errors.
4.1 Experiment Setup
Evaluation tasks and datasets. We evaluate several methods on the International Planning Competition benchmark which encompasses diverse complex planning domains and problems. Our evaluation focuses on two key PDDL domain synthesis tasks, including (1) NL22omain, which aims to convert natural language descriptions to PDDL Domains; and (2) Prob2Domai, which aims to derive necessary PDDL domains from PDDL problems. The evaluation metric used here is the success rate of the generated PDDL domain passing the PDDL validation system [20].
Large language model settings. The backbone LLMs in our experiment include Qwen2.5 5-Insructt (0.5B- 72B parameters) [51), L.MMA3. (8B and 70B parameters) [15], and Vi-1.5-Chat. (6B, 9B, and 34B
http:///guhhub.cmm/poosssss//// http:///ppootssso/ppdmmaaaaeeeee
9

---

Published in Transactions on Machine Learning Research (05/2025)
parameters) [53]. We also incorporate specialized code-oriented LLMs, specifically Qwen2.5-Coder and Yi- 1.5-Coder: In addition to open-source I.LMs, we benchmark against OpenAT's proprietary models, including GPT-40, ol-mini, and ol-preview. We test our proposed test-time scaling methods on Qwen models in a zero-shot sctting without model finetuning.
Chain of thought prompting All baselines here utilize chain-of-thought (CoT) prompting by default. Current LLMs have been extensively trained on datasets that include step-by-step reasoning besides final answers [49]. This enables the models to generate better reasoning traces during inference. The detailed CoT prompt template for our method is provided in Appendix D.
Sampling hyperparameters. To generate diverse PDDL domain synthesis paths, we use temperature sampling for both the BoN and iVML algorithms, and the lemperature equals 0.7.
Evaluation Metrics. We clarify the evaluation metrics for the experiments as follows:
For PDDL domain synthesis experiments (í.e., from Section 4.2 to Section 4.4), we adopt the VAL [20] validation tool to examine the syntax and semantic correctness of the PDDL domains. This includes checking for issues such invalid predicate parameter types, incorrect predicate names/ /fones/foraas,, improper predicate usage. For methods generating PDDL domains, we mcasure the percentage of domains that pass VAL. validation without errors.
For PDDL problem synthesis experiment (i.e., Section 4.5), we use the percentage of correct generated problems (i.e., the generated symbolic goals matches with the ground-truth goals) as metric.
For the experiments in Section 4.6, we evaluate planning accuracy, defined as the percentage of synthesized domains that produce plans identical to the ground-truth reference plans.
4.2 Main Results in PDDL Domain Synthesis
Current LLMs perform poorly in PDDL domain synthesis. Despite advances in code and math reasoning, LLMs exhibit fundamental limitations in PDDL-based formal synthesis. For instance, in Table 2, Qwen2.5-Innttuel (72B) achieves only 38.5% and 32.8% accuracy in NI.2Domain and Prob2Domain tasks, respectively. The results suggest that existing LLMs still fall short in symbolic reasoning tasks.
Search-augmented reasoning enhances formal synthesis. In Table 2, among the closed-source models, ol-preview emerges as the top performer with an average accuracy of 54.1%, outperforming other models in both NL.2Dmain and Problem2Domain tasks. The ol-serics models, which integrate search-based rea- soning during inference [32, 56], demonstrate significant improvements over standard instruction LLMs. For example, GPT-40 achieves only an average accuracy of 27.7%.
Code-oriented models outperform genoral-purpose models. In Table 2, we observe that code- specialized models (e.g., Qwen2.5-Codd Coder and Yi-l.5-Coder) demoustrate superior performance than their general-purpose counterparts. For example, Qwen2.5-Codhe (7B) outperforms Qwen-Tnstruct (7B) by 16.2%
NL2Domain (i.e., 21.9% compared to 5.7%). We hypothesize that the improvements stem from: (1) Implicit formalization training: code datasets teach type systems and predicate logic, (2) Syntax-scnsitive decoding: token-wise likelihood aligns with PDDL/s Lisp~like structure, and (3) Autoformalization priors: the code datasets that interleave natural language comments with pieces of code are high-quality datasets for CoT reasoning and autoformalization.
Test-time scaling is helpful for LLM at almost all scales. The BoN sampling method demonstrates universal offectiveness across the Qwen model family (e.g., 1.5B to 72B paramcters), significantly improving PDDL domain synthesis accuracy. For example in Table 2, BoN sampling with 8 candidates (BoN-8) improves Owen2.5-Instrue. (14B) from 21.6% to 51.6% NL2Domain. Gains persist through Qwen2.5-Instrum (72B) with 22.3% improvement on NL.2Dmain. Test-time compute scaling requires no further training or architectural changes, making it a computationally efficient addition to scaling up LLM's parameter numbers.
iVML can provide a robust and consistent improvement. iVML delivers robust performance gains over BoN across multiple model scales, demonstrating the power of itcrative self-improvement in PDDL domain synthesis. As illustrated in Table 2, five iterations of iVML training with BoN-8 initialization (iVML- 5- B-Bo-88) enables Sowen2.5TTnstrunn (32B) to achieve 86.2% on NT.2Domain, outperforming base BoN-8 with
10

---

Published in Transactions on Machine Learning Research (05/2025)
Sampling
Instance Verbalized Learning (N-8)
Prob 20emein
Prob22omain
Domain
Prob2Domain:
NL2Domain
Nl2Domain: Tw0
Domain Paice VAU (%6
D Poss VAL
10-
20-
002
1266
9:32
113
130
Sampling Budgets (Qwen2.5-8--odde)
Epochs Number (Qwen2.5:788 1.70o225:78.Cdom
Figure 3: Left: The performance trend of BoN with increasing sampling numbers. Right: The performance trend of
IVL with increasing training epochs (Setting the BoN sampling number N = 8).
19.4% improvement. The results position iVML as scalable and efficient framework for enhancing LLM performance in formal synthesis tasks.
4.3 Convergence Comparison between BoN and iVML
Experiment settings. This section presents a comparative analysis of the convergence behavior of BoN and iVML in synthesis tasks of PDDL domains. The computational efficiency and synthesis success rates of these methods depend on two parameters: the sampling budget for BoN N, and the number of training epochs for iVML T. Through controlled experiments, we examine how parametric variations affect synthesis effectiveness and identify the conditions under which their performance converges. We use Qwen2.5-Com (7B) the backbone LLM. The initialization of iVML is based on BoN-8 if not otherwise specified.
Convergence of BoN and iVML. On the left of Figure 3, we observe two distinct phases in the perfor- mance of BoN sampling as the sampling budget N increases. Phase 1 (N < 32): Both Prob2Domain and NL2Domain demonstrate a steep increase in the percentage of domains passing validation as the sampling budget increases, demonstrating the efficiency of exploring the candidate sampling improve accuracy. Phase 2 (N > 32): Performance reaches saturation with observable degradation trends. At higher sampling budgets (32 and beyond), Prob2Domain and NL2Domain converge to around 90%. On the right of Figure 3, iVML exhibits monotonic performance inprovement up to T = 80, significantly surpassing the saturated success rate achieved by BoN. For example, at N 256, BoN becomes saturated, with NL2Domain's 's accu- racy falling below 95% and Prob2Domain not exceeding 90%. In contrast, iVML enhances the accuracy to over 95% for NL2Domain and surpasses 90% for Prob2Domain at T =0,
80, showing its superiority over BoN.
Case study. The qualitative case study is given in Table 3, where we show BoN often fails generate the correct code. For example, in TyreWorld, despite explicitly stating the precondition that the container is open, BoN still generates the invalid predicate "closed ?container". Unlike BoN's brute-force sampling approach, iVML leverages an in-conlext self-refinement mechanism to (1) identify constraint violations (e.g., illegal block stacking or invalid preconditions)) (2) generate counterfactual natural language feedback to guide revisions; (3) strengthen domain-specific rcasoning priors through itcrative updates. Consequently, iVML can fix the BoN error and generate the corrected predicate "not (lused?containerr))
Analysis. In Figure we observe BoN's early saturation and performance degradation. This observation aligns with inference scaling flaws [39], where the optimal number of resampling steps for code synthesis tasks (e.g., HumanEval [3] and MBPP [8]) is always finite and low. This phenomenon arises from the lack of in-context learning with feedback from verifier. However, iVML uses self-critique mechanism combined with in-context learning to create a synthetic curriculum, enabling the model to learn to avoid error patterns in PDDL synthesis progressively. In contrast, BoN struggles with such dependencies due to its reliance on static sampling and the lack of iterative refinement.
11

---

Published in Transactions on Machine Learning Research (05/2025)
Natural Language
Bo
BoNg8 + IVML@6
BlockWorld (stack): The arm can stack a block on top of another if it holds the top block and the bot- tom block is clear. Post-action, the arm is cmplyy the top block rests the bottom block, and the bottom block is no longer clear.
(:action stack
Incorvectly clear the bottom block, which should not hap- pen when stacking block on the bottom block.
:parameters (2top- block ?bnttom block)
(:actoon stack
sprecondition (and (holding ?top) (clear ?bottom)]
:parameters (?top- block ?bottom -llok)
sprecondition (and (holding ?top) (clear ?bottom))
seffect (and (not (holding ?top]) (on
seffect (and (arm-rmpty) (not (holding ?top)) (not (clear ?bottom)) (on ?top ?bottom)])
"top ?bottom) (clear ?bottom))
(ffteh)) The precondi- tion for this action is that the ob- ject is inside the coutainer and the container is open. The effect of this action is that the object is held by the agent and not inside the container.
(saction fetch
The logic the fatch action incorrect. Procondition: Incorrectly checks if the object is Inalde the container and
sparametera (7ob) - objeet "container container)
the container is open. Effect: Incorrect logic. The effect should be "(not (in Tobj ?container))" instead of "(in Tobj Tomntainer) [not (in Zobj ?oontainer))".
sprecondition (and (in ?obj ?con- tainer) (closed "containeer) seffect (and (in Tobj ?container) (not (in ?obj ?container])
(sacion fetch
:parameters (2obj object ?coutainer - container) sprecondition (and (in 7obj ?container) (not [closed "con- tainer)))
reffect (not (in 2obj Peontainer))
TsPrretur--gganstt: This action also completes the tour by allow- the
(saction return-against :parameters (leurrent-city - city ?start-city city)
The precondition for this action should have "(connected Teurrent-city ?start-city)" ) instead of "(connected ?start-
ing the agent to return starting city against the direction of a counection. The precondition is similar to the "return-along" ac- tion, except the city is not previ- ously visited. The effect is that the sgent returns to the starting point and the tour is marked as complete.
city ?curreut-city)" correctly reflect the travel direc- tion.
pprecondition (and (in ?eurrent-
city) (counected ?start-city Teurrent- (not)yoootooppeee
(:action return-sgainst
:parameters (2eurrent-city- city ?start-city city) :prerondition (and (in ?current-city) (comnorted Fourrent- city ?start-city) (not-complete)) :effect (and (in ?start-city) (complete)))
seffect (and (in ?start-rity) (com- plete)))
Termes( remooe-bbockk): Create a binck at the depot. The robot will have the block.
(:action remove- block
The effect of the cemove-block action does not correctly update the height. of the block's position. It should he "[height toPos 7h-down)" ", but it is currently "(7h-down)". (:action remove-block sparameters (fromllos Pos toPos)
:parameters (fromPas toPos)
pprecondition (and (height fromPos
7h) (SUCC ?h ?h-down)
(height toPas ?h-down) (at fromPus) (not (has-block)))
:prorondition (and (height fromPos ?h) (SUCC Th down)
reffect (and (has block) (height toPos ?h)))
(height toPos ?h-down) (at fromPos) (not (has-block)))
:effect (and (has-block) (height toPos ?h-down)]]
Table 3: The comparison highlights the differences between Best-of-N sampling (BoN) and iVML in synthesizing action-level PDDL code. The red text marks where BoN@8 produces logically incorrect code, while the blue text. shows how iVML detects these inaccuracies and applies the necessary corrections.
4.4 Ablation Study of Initialization Strategies
Experiment settings. This study aims to compare two initialization strategies for iVML: 1. Single-pass sampling by requesting the LLMs to generate only once as the solution initialization for the iVML process, and 2. BoN sampling by requesting the LLMs to generate diverse candidate solutions for initializing the optimization process using high-temperature sampling.
BoN vs. Single-pass sampling. We compare BoN sampling with Singlo-pass sampling, which BoN sam- pling, as initialization strategy, provides iVML with the dual advantages of accelerated convergence and improved solution quality. For example, in Figure 4, with BoN-8 initialization in NI.2Dmain,, Deepseek- Coder saturates earlier than its single-pass counterpart at T = 16, while achieving a higher accuracy per- centage (approximately 95% compared to fewer than 60% in sngle-pass). Unlike single-pass sampling, which is analogous to random initialization traditional optimization, e.g., stochastic gradient descent, BoN gen- crates a diverse set of initial candidate solutions. The solution diversity can effectively improve iVML with expanded exploration. By covering a broader range of the solution space, BoN helps to avoid early conver- gence to suboptimal solutions. This means that the algorithm is less likely to get stuck in local minima, which are common challenges in complex optimization problems [4]. By selecting high-quality initial solution candidates, BoN guides the optimization process of iVML toward better optimality.
Performance of weaker LLMs. From Figure 4, we observe that iVML with BoN-4 initialization can improve LLaMa's performance in the NL2Domain task, which yields around 10 correct domains (around 4%) compared to zero in single-pass mode. However, the performance is much worse than Qwen2.5-Codd (7B) and Deepseek-Codee Cooder-Instuuche (7B). We attribute this performance gap to L.LaMa's limited exposure
12

---

Published in Transactions on Machine Learning Research (05/2025)
IVML under various intializations (DDaaa))
under various initializations (Prab7Domain)
100
100
Coweerodeer_ Bol
prob, QueenCoder, Rott
60
N. OwennCoder Bohs
80
prob. Qwendoderr Bolls
-
1. Dowen angle
probi Qwentodder single
* dscoder BoMB
prob, oscoder Bolld
dwender single
decader single
d.lomo_BoMk BoN4
Jomb Bohd
single
20
20-
146
17
E:0
1:13
12.0
1:39
1122
1:25
1:28
11
14
1:30
1:13
1:15
1:39
1122
1:23
122
Num ittraaiona
Num liarations
Figure 4: The performance of iVLL NL2Domain
Figure 5: The performance of iVML Prob2Domain
lasks across different initialization senging.
lasks across different initialization seltings.
Model
Setting
Correct Rate (%)
Zero-shot
0.00
Gemma 1.1 II 2B
F Fine-tuned
94.21
Zero-shot
0.00
Gemma 1.1 IT 7B
Fine-tuned
98.79
Zero-shot
0.01
Mistral v0.3 Instruct 7B
Fine-tuned
99.00
GPT-4o
Zero-shot
35.12
Ours (Qwen2.5-Coder-77.
BoN-16
99.24
Ours (0wen2.5-Coder-777
iVML-I-BoN-16
99.60
Table 4: Performance comparison of different models on PDDL problem generation
structured logical reasoning during pretraining, a deficiency in its pretraining knowledge that test-time compute scaling methods (e.g., iVML and BoN) cannot effectively address.
4.5 PDDL Problem Generation
Experiment settings. This section investigates the effectiveness of our approach while generalizing it PDDL problem synthesis. In contrast the PDDL domain, which outlines the general framework environment defined for planning tasks, the PDDL problem defines specific instance of the planning task within that domain. This involves defining two main components (1) Initial state: the starting state of the world, defined by the predicates that are true initially, and (2) Goal state: The objective that the planner aims to achieve. We adopt the Planetarium [61] benchmark, which evaluates LLMs' capacity to generate precise PDDL problems from natural language descriptions. These tasks are challenging due to the lack of planning background knowledge and the complex context described by the problem. The evaluation methods outlined in [61] test LLMs in both zero-shot and fino-tuned settings. The baselines being evaluated include GPT-4, Gemma 1.1 IT models [41] with 2B and 7B parameters, well Mistral v0.3 Instruct (7B) [23]. Importantly, during PDDL problem generation, pre-existing PDDL domain is provided input only natural language descriptions guide the synthesis process.
Main results. The results are presented in Table 4. We adopt the correct rate defined in Planetarium as the evaluation metric. This metric defined as the percentage generated problems whose goals correctly match the ground-truth goals. Planctarium fine-tuned Gemma and Mistral on training dataset containing 132,027 examples from the two-class PDDL problem code dataset, potentially raising overfitting concerns Gemma's 's accuracy increased dramatically from near 0.0% to over 98.8%. Our method enhances the Qwen2.5-
13

---

Published in Transactions on Machine Learning Research (05/2025)
Model
Setting
Floortile Barman Tyreworld Grippers Termes Blockworld
LLM-as- MaM-as-Planner Methods
Pass@1
0.0
6.7
0.0
23.8
4.8
4.8
GPT-4o
self-critique
10.0
13.3
0.0
33.3
0.0
14.2
Pass@8
13.3
33.3
45.0
45.0
10.0
23.8
Pass@1
5.3
33.3
50.0
57.1
23.8
38.1
ol-mini
self-critique
5.3
33.3
35.0
61.9
23.8
47.6
Pass88
0.0
33.3
70.0
61.9
52.4
23.1
Pass01
0.0
13.3
33.3
38.1
0.0
4.7
ol-preview
self-critique
5.0
6.7
35.0
33.3
4.7
9.5
Pass@8
33.3
33.3
85.0
66.7
19.0
33.3
Oracle Interaction Methods
Qwen2.5-7B-Coder Oracle Interaction
0.0
0.0
100.0
100.0
Our Methods (PDDL as the Planning Representation)
Qwen2.5-7BB-Ca
BoN-4
0.0
0.0
100.0
81.0
9.5
Qwen2.5-7BB-Coder
BoN-16
100.0
100.0
100.0
100.0
100.0
0.0
Qwen2.5-7BB-Coder
3oN-4-iVML-5
0.0
100.0
100.0
100.0
100.0
71.4
Qwen2.5-7BB-Coder
BoN-16-IVMT-5
100.0
100.0
100.0
100.0
100.0
81.0
Table 5: PDDL as the planning representation methods vs. L.MM-as-Planner methods. The evaluation metric plan accuracy. Our PDDL-based method uses the Fast Downward system [19] for heuristic search and plan validation.
Coder (7B) model through test-time scaling techniques, achieving 99.24% correctness rate with BoN-16 sampling. This improves further to 99.60% when combining iVML-1 with BoN-16 for solution initialization.
Comparison between SFT and iVML. The results in Table provide a comparison between supervised finetuning (SFT) and iVML. This reveals three key advantages of our method (listed as follows). (1) Bet- ter preservation of pretrained knowledge: Unlike SFT's static alignment fixed data distributions, iVML enforces structured reasoning priors through in-context learning, enabling adaptation diverse problem constraints without triggering catastrophic forgetting of pretrained knowledge. (2) Refinement through in- context oplimizalion: Building upon BoN's high-quality initialization, iVML performs in-context instance optimization to correct subtle errors, elevating correctness from 99.24% (BoN-16) to 99.60% (iVML-I-BoN- 16). (3) Computational efficiency: iVML requires significantly less compute than SFT while demonstrating strong performance on complex reasoning tasks, including those in long-tail domains.
4.6 Comparison to LLM-as-Plannes Methods
In the previous sections, we demonstrated that iVML enhances LLMs' ability to generate high-quality PDDL- based world models. In this section, we compare our method, which utilizes synthesized PDDL domains as world models, with LLMs-as-a-Plaanet methods that use natural language for world modeling and planning. A detailed description of the tested planning cases is provided in Appendix
Experimental settings. LLMs often hallueinate for tasks such as generating feasible or optimal plans, understanding the planning problem, and strictly following the rules, particularly in complex planning prob- lems (e.g., Termes and Barman) [44]. To investigate whether these hallucinations arise from limitations in prompt engincering, we provide LLMs with explicit instructions on the rules they must follow and require them to verify rule compliance at each step of the planning process. Additionally, we employ two stralegies to reduce uncertainty in LLM-generated plans: (1) Introducing the Pass@8 metric to evaluate the probability that at least one of the top 8-generated plans is correct, and (2) Allowing L.Mss self-evaluate their plans and refine them based on these assessments. The bascline implementations of LLM-as- as-Plen methods are based GPT-4o, ol-mini, and ol-preview, respectively.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Discussion on LLM-as-Planner methods. The observations are follows: (1) rule violation: Despite explicitly informing LLMs to examine rule violation at each step, the generated plans still contain elements that inherently violate the predefined rules. For example, in the step 3 and 4 of Figure 2, ol incorrectly places the block from pos-O-1 to pos-1-2, mistakenly assuming that they are neighboring positions. This action violates the rule governing the placement of blocks. (2) incorvedl state transition estimation: The LLMs fail to accurately estimate state transitions. For instance, after moving the block from pos-1-1 to pos-1-2, ol incorrectly assumes that the heights of both positions are 0, reflecting an inability to track state changes correctly. (3) incorrect goal achievement estimation: ol attempts to achieve the goal in manner that disregards all constraints and relies on flawed state estimations, which results in plan that is not only incorrect but also violates the fundamental rules of the task. (4) incorvect self-evaluation: ol fails to identify errors in its planning process, as it consistently assumes that its own responses are correct. This prevents it from correcting mistakes, further compounding the inaccuracies in its generated plans.
PDDL-based method vs. HLM-as-Plannee We present the numerical results in Table 5. For this experiment, we use planning accuracy as the metric. Specifically, for a given planning problem (e.g., Termes), it measures the percentage of generated plans from the synthesized domains that exactly match the ground- truth reference plans. For classical planning problems, using natural language as a planning represcntation proves suboptimal. Even when equipped with self-critique capabilities, the ol-preview system achieves only 5.0 on Floortile, 6.7 on Barman, and 4.7 on Termes benchmarks. This limitation stems from natural language's inherent ambiguity and lack of formal precision required for precise planning representation. LLM- as-Planner methods do not perform planning: instead, they treat planning tasks as n-gram text completion problems, leading to plans that may lack feasibility, violate the constraints, the underlying problem structure. In contrast, our approach leverages PDDL reprosentations to explicitly model state transitions. Through BoN-16-iVML-5 generation of high-quality world models combined with heuristic search algorithms (i.e., A*), our method solves nearly all tested instances across Floortile, Barman, and Termes domains. We compare our mothod with a PDDL-based approach [26] that uses the oracle environment to refine the answers. By contrast, our approach generates PDDL domains in an oracle-agnostic manner, leveraging test-time scaling to enhance scalability in scenarios without rule-based ground-truth signals. We found that our method (e.g., BoN-4-iVML-5) achieves comparable performance to the oracle interaction method in the cases they considered, with both achieving 100.0 on Termes. However, our method oulperforms the oracle interaction method on Floortile and Grippers.
fail to accurately reflect
5 Concluding Remarks and Current Limitations
Our work introduces test-time scaling framework for automated PDDL synthesis that integrates best-of-N sampling with instance verbalized machine learning. This approach demonstrates that effectively scaling test-time computation of open-source LLMs can outperform statc-of-the-art closed-source LLM planners, including OpenAT's ol-mini. Our hybrid method employs a two-phase optimization paradigm: (1) BoN initialization which generates diverse candidate solutions to explore critical regions of the search space, addressing the cold-start problem, where iVML begin with poor initialization in formal language synthesis. (2) iVML refinement which iteratively improves the BoN initial solutions through self-critique and natural language feedback, resolving logical inconsistencies and syntactic errors. Leveraging BoN's stochastic search to initialize iVML's refinement process, our method achieves faster convergence and higher-quality PDDL domains. The effectiveness of iVML in PDDL problem synthesis even surpasses models specifically fine- tuned for this task. The results show that our proposed test-time compute scaling approach can enhance LLMs' formal rcasoning and planning capabilitics. By generating PDDI--based symbolic world models, we enable explicit model-based planning with classical search algorithms (e.g., A*), avoiding the error-prone state transitions inherent in direct LLM-as-planne approaches. Beyond PDDL synthesis, our work provides a general framework for scaling up test-time compute of L.MSs for formal language synthesis.
The limitations of our work include: (1) challenges in semantic verifieation for anloformalization: Consistent with prior work in PDDL synthesis (e.g., [17, 61, 42]), our evaluation relies on VAL [20] for syntax validation and plan verification. While VAL ensures syntactic correctness (e.g., prodicate arity, type consistency) and plan executability (e.g., action preconditions and effects), it cannot detect semantic inconsistencies that vio- late domain intent or commonsense logic. This limitation parallels broader challenges in autoformalization.
15

---

Published in Transactions on Machine Learning Research (05/2025)
where even formal mathematical proof [58] struggles to verify semantic alignment between informal speci- fications and formal outputs through compiler checking. (2) simulation assumplions: Our evaluation relies on an idealized simulation environment with two key assumptions. First, actions execute perfectly, with no execution misalignment. Second, the state space is fully observable, with no sensor noise or occlusions. These idealized conditions differ significantly from real-world robotic manipulation scenarios.
6 Broader Impact
The methodology presented in our paper carries signifieant implications for advancing the safe usage of human-centered AI agents. Unverifiable fully autonomous AI agents pose substantial risks to the security and safety of human society, as their capabilities are too strong while very difficult to control [5, 29). Re- cent discourse in AI ethics and safety emphasizes that fully autonomous AI agents should not be developed without rigorous safeguards [29]. Current Al systems, particularly those relying on LLMSs, exhibit significant hallucinations, such as hallucinations, rule violations, incorrect state transition estimations, and incorrect goal achievement assessments (sce Figure 2). These issucs stem from LLMs' reliance on n-gram text comple- tion to model the world, method that lacks logical coherence and a systemic understanding of real-world planning. On the other hand, self-verification mechanisms often fail to detect inherent errors in generated plans. In contrast, the scientist AI paradigm [5] advocates for a two-stage approach: first, learning a struc- tured world model, and second, deriving rationally grounded inferences from it. Our method is similar to scientist AT [5] by employing test-time scaling to construct a symbolic world model before applying heuris- tic search to generate verifiable, rulc-followed plans. By prioritizing symbolic reasoning over unstructured natural language generation, our approach mitigates risks associated with hallucinations and enhances the transparency of agent planning, We argue that this framework represents a critical step toward controllable, interpretable, and safe human-centered AI systems.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ai preprint "xXi:2333.08771, 2023. 3
[2] Forest Agostinelli and Misagh Soltani. Learning discrete world models for heuristic search. In Rein- forcement Learning Conference, 2024.
(3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint ar arXiv:2108.0777 07782, 2021. 11
[4] Kyri Baker. Learning warm-start points for ac optimal power flow. In International Workshop on Machine Learning for Signal Processing, 2019. 12
[5] Yoshua Bengio, Michael Cohen, Damiano Fornasiere, Joumana Ghosn, Pietro Greiner, Matt MacDer- mott, Sören Mindermann, Adam Oberman, Josse Richardson, Oliver Richardson, et al. Superintelligent agents pose catastrophie risks: Can scientist ai offer safer path? arXiv preprint orXiv:2502.1500 2025. 16
[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1606.00006 2016. 5
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NewrIPS, 2020.
[8] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained code. or Xie preprint 2xXx:22077033774 2021. 3, 11
16

---

Published in Transactions on Machine Learning Research (05/2025)
[9] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Dis- entangling computation from reasoning for numerical reasoning tasks. or orXiv preprint orXiu:2211 1.12888, 2022. 1
[10] Kewei Cheng, Jingfeng Yang, Haoming Jiang, Zhengyang Wang, Binxuan Huang, Ruirui Li, Shiyang Li, Zheng Li, Yifan Gao, Xian Li, et al. Inductive or deductive? rethinking the fundamental reasoning abilities of lims. arXiv preprint arXiv:2408.00 00.1014.... 2024. 1,
[11] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiihhrr Nakano, Christopher Hesso, and John Schulman. Train- ing verifiers to solve math word problems. arXiv preprint arXio:2110. 14168, 2021. 1,
[2] Nicola Dainese, Matteo Merler, Minttu Alakuijala, and Pekka Marttinen. Generating code world models with large language models guided by monte carlo tree search. Xixiv preprint arXiv:2405.1530 2024.
5
[13] Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-elliciet fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-236) 2023. 5
[14] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjig Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arXiv preprint arXiv:2301.0023 2022. 5
[15] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 2xXv:2407.2178. 2024. 9
[16] Richard E. Fikes and Nils J. Nilsson. Strips: A new approach to the application of theorem proving to problem solving. IJCAI, 1971. 6, 22
[17] Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. Leveraging pre- trained large language models to construct and utilize world models for model-based task planning. In NeurIPS, 2023. 2, 5, 15
[18] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305. 14992, 2023. 4
[19] Malte Helmert. The fast downward planning system. Journal of Artificial Intelligence Research, 26:
191 246, 2006. 14
[20] Richard Howey and Derek Long. Val's progress: The automatic validation tool for pdd12.1 used in the international planning competition. In ICAPS Workshop on "The Competition: Impact, Organization, Evaluation, Benchmarks", ; 2003. 9, 10, 15
[2]] Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022. 5
[22] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.167900 2024. 1,
[3]] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. Xiv preprint ar ar.ii:2310.000 06825, 2023. 13
[24] Subbarao Kambhampati. Can large language models reason and plan? Annals of the New York Academy of Sciences, 1534(1):15-115 2024. 4
17

---

Published in Transactions on Machine Learning Research (05/2025)
[25] Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, et al. Parameter-efficient orthogonal finetuning via butterfly factorization. In ICLR, 2024. 5
[26] Sadegh Mahdavi, Raquel Aoki, Keyi Tang, and Yanshuai Cao. Leveraging environment interaction for automated pddl translation and planning with large language models. arXiv preprint ur urXiv:2207. 129799 2024. 2, 5, 15
[27]
Drew McDermott, Malik Ghallab, Adele F. Howe, Craig A. Knoblock, Ashwin Ram, Manuela. M. Veloso, Daniel S. Weld, and David E. Wilkins. Pddl-the planning domain definition language. 1998. 1
[28]
Iman Mirzadch, Keivan Alizadch, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mchrdad Fara- jlabar. Gsm-symbolie: Understanding the limitations of mathematical reasoning in large language models. orXiv preprint arXiv:24 10.05229, 2024. 3
[29] Margaret Mitchell, Avijit Ghosh, Alexandra Sasha Luccioni, and Giada Pistilli. Fully autonomous ai agents should not be developed. arXiv preprint Xrxi:2200 02649, 2025. 16
[30] James Oswald, Kavitha Srinivas, Harsha Kokel, Junkyu Lee, Michael Katz, and Shirin Sohrabi. Large language models as planning domain generators. In Proceedings of the International Conference Automated Planning and Scheduling, volume 34, pp. 423-431 2024. 5
[31] Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with "gradient descent' and beam search. arxiv preprint arxw:23055003 2023.
[32] Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, al. O1 replication journey: A strategic progress apportpprr 1. arXiv preprint or 2i 10. 1898, 2024. 10
[33] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling lexi-to-image diffusion by orthogonal finetuning, In NeurIPS, 2023. 5
[34] Stuart J Russell and Peter Norvig. Artificial intelligence: a modern approach. Pearson, 2016. 6
[35] Elad Sharony, Heng Yang, Tong Che, Marco Pavone, Shie Mannor, and Peter Karkus. Learning multiple initial solutions optimization problems. arXiv preprint orXiv:2411.02158, 2024.
(36] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Sys- tems, 36:8634-8652, 2023. 3
[37] Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hansknecht, Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXim:2010.03768, 2020.
[38] Pavel Smirnov, Frank Joublin, Antonello Ceravola, and Michael Gienger. Generating consistent pddl domains with large language models. arXio preprint orXiv:240400700 2024.
[39] Benedikt Stroebl, Sayash Kapoor, and Arvind Narayanan. Inference scaling flaws: The limits of Ilm resampling with imperfect verifiers. arXie preprint 4rXiv:2411.17501 2024. 3, 11
[40] Hao Tang, Darren Key, and Kevin Ellis. Worldcoder, a model-based Ilm agent: Building world models by writing code and interacting with the environment. arXiv preprint arXiv:2402.0 12275, 2024. 5
[41] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Julictte Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint ar arXin:2403.0820 08295, 2024. 13
18

---

Published in Transactions on Machine Learning Research (05/2025)
[42] Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambham- pati. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In NewrIPS, 2024. 15
[43] Karthik Valmockam, Kaya Stochly, and Subbarao Kambhampati. Lims still can't plan; can Irms? a preliminary evaluation of openai's ol on planbench. arXiv preprint or urXi::2409. 13373, 2024. 1,
[44] Kevin Wang, Junbo Li, Neel P Bhatt, Yihan Xi, Qiang Liu, Ufuk Topeu, and Zhangyang Wang. On the planning abilities of openai's ol models: Feasibillity, optimality, and generalizability. arXiv preprint
arXiv:2409. 19924, 2024. 3, 14
[45] Kevin Wang, Junbo Li, Neel Pratik Bhatt, Yihan Xi, Qiang Liu, Ufuk Topcu, and Zhangyang Wang, On the planning abilities of openai's ol models: Feasibility, optimality, and generalizability. ArXin, has/2409.19924, 2024. URL https://appt htttps:/apl..emmtticccchee semanticscholar arg/CorrpuID::2728550
1
[66] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint wxXi:2203.11171, 2022.
47] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXi preprint or 0rx::
2022. 5
48)
Jason Wei, Xuezhi Wang, Dale Schnurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning large language models. In NeurIPS, 2022.
[9] Jason Weston and Sainbayar Sukhbaatar. System attention (is something you might need too). arXiv preprint or arXi::2311.11822 2023. 10
[00] Tim Z Xiao, Robert Bamler, Bernhard Schölkopf, and Weiyang Liu. Verbalized machine learning: Revisiting machine learning with language models. ariie preprinl urXiv:2406. 04340, 2024. 2, 3, 5, 7
[1] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv2440710671,, 2024. 9
[2]] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In ICLR, 2024. 5
53] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint ar Xrxiv:240S.00 04652, 2024. 10
54] Ping Yu, Jing Xu, Jason Weston, and Ili Kulikov. Distilling system 2 into system 1. arXiv preprint arXiv:2407.0600 2024.
55) Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic" differentiation' via text. arXiv preprint ar arXiv:240.007496 2024. 5
56] Zhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin, Bo Wang, Shimin Li, Yunhua Zhou, Qipeng Guo, Xuanjing Huang, and Xipeng Qiu. Scaling of search and learning: A roadmap to reproduce ol from reinforcement learning perspective. arXiv preprint or Xix:2224 12. 14135, 2024. 10
[57] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXio preprint Xix::22408. 15240, 2024.
3
[58] Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. Minifff: a cross-system benchmark for formal olympiad-level mathematics. arXiv preprint XrXi:22109.001 10, 2021. 16
19

---

Published in Transactions on Machine Learning Research (05/2025)
[59] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-ajudge with mt-bench and chatbot arena. In NeurIPS, 2023. 3
[00] Yongchao Zhou, Andrei Toan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engincers. arXiv preprint arXiv:2211001910 2022.
5
[61] Max Zuo, Francisco Piedrahita Velez, Xiaochen Li, Michael Littman, and Stephen H Bach. Planc- larium: A rigorous benchmark for translating text. to structured planning languages. ar Xiviv preprinl
arX0:2400..08221, 2024. 1, 13, 15
20

---

Published in Transactions on Machine Learning Research (05/2025)
Appendix
Table of Contents
A Planning Problem Formulation
22
B STRIPS Formulation
22
C Tasks in Case Study
23
Prompt Template for Chain-of-Thougee
E Generated Domains
25
Prompts for LLM-as-Plannes Methods
30
State-based graph for Termes
31
H The Prompt of Our Methods on Termes
32
Human in the Loop Experiment
21

---

Published in Transactions on Machine Learning Research (05/2025)
A Planning Problem Formulation
The classical planning problem [16] in artificial intelligence involves finding a sequence of actions that tran- sition an agent from an initial state to a desired goal state within a deterministic and fully observable environment. is formalized as a tuple (S, T, 80, G) where: S is the set of all possible states; is the set of all possible actions; T: S x A - S is the state transition function, specifying the outcome state resulting from applying an action in state; So E S is the initial state; G is the goal condition, predicate over states. The objective is to find sequence of actions a1 02,00,.... ................................ Un E A such that applying these actions successively transitions the system from 8o to state 8n satisfying the goal condition G:
Sp = T( an) = T(T(.... ((T a1).
am)
with
Sn G G
Previous works that adopt LLMs planners estimate state transitions implicitly within language latent spaces, lacking explicit representations of the stale space required for classical planning. This implicit representation can make it challenging to ensure consistency, validity, and completeness in the planning process. In contrast, our work leverages LLMs to generate explicit representations of the state space by creating PDDL domains. We utilize the generative capabilities of LLMs to produce formal PDDL models from high-level descriptions of the planning tasks. This approach bridges the gap between natural language specifications and formal planning models.
By generating PDDL domains using LLMs, we obtain: 1. explicit definitions of the set of states S through predicates and objects, 2. formal specifications of actions A, including their preconditions and effects, 3. deterministic state transition function T derived from the action definitions, 4. abilities to cooperate with clearly defined initial state so and goal condition G in PDDL syntax. Thus the new objective under this background is: Given high-level descriptions of planning tasks, our objective is to leverage LLMs to create PDDL domains that can represent state transitions explicitly. By generating these explicit representations, we enable classical planning algorithms to efficiently search for plans using the defined state transitions during the planning process.
B STRIPS Formulation
The states are expressed through set of predicates that describe the properties of objects in the environment. Each predicate represents a relationship or characteristic, such as on (A, B), which indicates that object A is on top of object B. A state can be represented as:
So = (on(A, B), clear(C))
This representation includes relationships that capture the positions and statuses of the objects within the environment. Actions in PDDL are tightly bound to the representation of states through the use of preconditions and effects. Each action is defined by specilying what must be true in the current state for the action to be applicable (preconditions), as well as what changes in the state when the action is performed (cffects). For examplc, consider an action move(A, B, C), indicating that move the object A from B to C. The preconditions and effects could be defined as follows: Preconditions: Pre(move(A, B, C)) = (A, B), clear(C)) clear()))and Effectt:: mofe A, B, C)) {on(A, (A, C), clear (B)). They indicate that for the action move (A, B, C) to be executed, object A must be positioned on B, and C must be clear of any objects. And after moving A from B to C, A is now C, and B is clear. The transition can be formulated as: A(S,A) S". State transitions occur according to defined sequence of actions that an agent may take, leading to new configurations of the world. For instance if the action move(A) (A, B, C)) on Su, the transition can be represented as:
S = T(S, move B, C)) + [on(A, C),, cle
By repeatedly exploring all possible actions, we can form a state transition graph consisting of nodes (states) and directed edges (actions) that connect these nodes based on the actions that lead to different states. We continue this process by expanding the graph from the newly added nodes until the goal state is reached and added to the graph, there more actions left to explore.
22

---

Published in Transactions on Machine Learning Research (05/2025)
C Tasks in Case Study
Barman, The main goal of the Barman domain is to simulate the task of a bartender who prepares and serves cocktails by manipulating ingredients, tools, and glassware within a bar setting, In this scenario, the agent is tasked with creating a specific cocktail by following a series of actions that involve: 1. Identifying and dispensing the necessary ingredients required for the cocktail from the available dispensers. 2. Using bar tools such as shakers and shot glasses effectively, ensuring they are clean and suitable for use. 3. Coordinating the use of both hands to pick up and handle objects while ensuring that hands are free when needed and that objects are properly placed surfaces like the bar counter when not in use. 4. Adhering to the proper sequence of steps for cocktail preparation, which includes dispensing ingredients into the shaker, mixing them, and then pouring the mixture into shot glass. 5. Maintaining the correct state of all objects involved, such as keeping the shaker and glasses clean and empty before use, and updating their states appropriately as actions are performed. 6. Successfully preparing the cocktail and having it contained in the shot glass, thereby fulfillngg the goal of serving the drink as intended.
Gripper. Gripper problem is designed to test the agent's 's ability to manage resources and plan actions in scenario involving manipulating multiple objects across different locations. The agent, represented by one more robots, must strategically perform the following tasks: Pick Up Balls: The agent must use its available grippers to pick up the balls from their initial locations. This requires careful hand management to ensure grippers are free and avaílable when ncoded. Transport Balls: The robot needs to navigate between rooms to move balls to their specified larget ions efficiently. This involves planning the correct sequence of moves and ensuring that the robot. in the correct room with the appropriate objects. Drop Balls: The agent must release the balls in the designated rooms. This requires ensuring that the robot's grippers are properly aligned and that the release actions are performed at the correct time. Manage Resources: Throughout the task, the agent must effectively manage both its grip and position within the environment, making sure that it follows constraints such as carrying capacity and room access.
Tyreworld. The main goal of the Tyreworld problem is to simulate the challenges of vehicle maintenance and tire management in a scenario where a vehicle may suffer random tire failures. The primary objectives include: 1. Replacing Flat Tires: The agent must effectively replace flat tires with intact ones on the vehicle's hubs. 2. Inflating Tires: The intact tires must be inflated before being mounted onto the vehicle. 3. Ensuring Secure Fastening: After replacing and inflating the tires, the nuts on the hubs must be securely tightened to ensure the wheels are safely attached. 3. Resource Management: The agent must manage limited resources (e.g., spare tires, tools like jacks and wrenches) strategically to minimize the risk of failure or being stranded. 4. Navigating Uncertainty: The agent must effectively plan actions while accounting for the possibility of tire failures and other uncertainties in the enviromment. 5. Reaching the Destination: Ultimately, the goal to ensure the vehicle is properly equipped with intact, inflated tires, allowing it to continue its journey successfully. 6. The Tyreworld problem serves to test an agent's planning, resource management, and adaptability in unpredictable scenarios related to vehicle maintenance.
The main goal of Floor-tile is to enable the robot to navigate the environment, manage its colors, and paint the tiles according to specific requirements, The robot must efficiently utilize its movements and actions to achieve its painting objectives while adhering to the constraints of tile occupancy and color availability. In Floor-tile, three object types are defined: robot, tile, and color. Initially, the robot is located on a specific tile while holding color. It can move up, down, right, or left with different costs: moving up costs 3, while moving down, right, left costs 1. The robot can paint a tile above or below for cost of 2, and changing its color incurs cost of 5.
Floor-tile.
Termes. In Termes, a robot operates in an environment to manage and manipulate blocks at different positions. The robot can perform several actions, including moving between adjacent positions, placing blocks onto stacks, removing blocks from stacks, and creating or destroying blocks at designated depot locations. Fach position on the grid has a specific height, and the robot must respect these height constraints when moving or manipulating blocks. The robot can only carry one block at time, and it must be at the same height level to move horizontally or at height difference of one to move vertically. The goal is to efficiently use these capabilities to achieve specific block arrangements or configurations within the environment, adhering to the constraints of adjacency, height, and block availability.
23

---

Published in Transactions on Machine Learning Research (05/2025)
D Prompt Template for Chain-of- Thoughht
Prompts for Our Methods
You be given natural language description planning problem. Your
translate description
This includes defining predicates and actions based on the information provided.
natural language description. Note that individual conditions preconditions and effects should be listed separately. For example, "object1 is washed and heated" should be considered as separate conditions "object1 is washed" and "objectl heated": Also, PDDL, two predicates cannot have the same name even if they have different parameters. Each and its parameters
Information
agent
provided
predicate PDDL must have unique define predicate names in an intuitive and readable way. Remember: Ignore the information that you think is not helpful for the planning task. You only responsible domain generation. Before you generate the concrete domain thought about the meaning of each variable, and the step-by-ste explaination of the domain code. Even if didn't provide the exact name of the predicates and actions, you should generate them based on the information provided the natural language description.
be explicitly defined in the predicate definition. is recommended to
you
first generate natural language
Template is:
### Thought:
predicatesl: the of predicatel, explanation predictatel
predicaten:
of predicaten, explanation predictaten actionl: the name of actionl, explanation of action
actionn: the of action, explanation action
<thought>>
#f Domain: "'pddl
The concrete pddl code for domain.pddl
Now your generate solution, you
provided
Descriptio:: Natural language description of the planning domain
24

---

Published in Transactions on Machine Learning Research (05/2025)
E
Generated Domains
Barman
Domain
[define (domain barman)
(crequirements :atrips :tsping]
heverage dispenser
object Ingrediest
beverage
(holding container)
(cnsable
container)
(handempty hand]
(empty container)
(contains Te container 76 beverege)
(clcan 76 container]
lused 7e container 7b bavarage)
(diapences 7 dispenser 71 ingredient] (shaker-empty-
To shaker 71 level]
(raction group
parametors [ostable (handompty 1h]] Te)) (not (handempty 75)) (holding 76
container)
fill
(raction rehill-shot
(taction ampty-shat
(raction clean-shat
(caction pour-shot- co--------------------------- shaker
(caction ---hot-to--ue--saake
(caction empty-shaker
(caction clean-shaker
(cationn shali
(1 (raction pour-sbaker-to-the coe.e
(problem
(tobjacts
sbakedd shaker left right hand shatl shet2 shot3 thate shat ingredient1
ingrediest2 ingredients ingradient cocktails cocktail2 cocktails cocktail disgenserl dispenser2 dispenser3 dispenser 10 11 level
(caii
(ontablc shakerl]
(ontable shoil)
(clesn shakerl) .....
(empty shakeri)
-parti ingredient1] 100
('goal
(contains shotl cocktaill) (contains cocktail3) [conszins shot3 cock:ail2)
ingrediett2 right dipersed] (pour-hott-to- lngeddent right disponer)) (pourshet-to shaker)) liresp shots)
(gropp
ingrediec22
ingredient? right)
lngredeent
ingrectentt right
(graes
(empty right
ingedienl right
hmpedien3 shukes (pourebaakertee to cocktall shoti right shaker (empby-shsker right shakers cocktail 10) shaker right shaker)) [lesow right shakert) (grasp right (pour- shat-to-clan shaker shose ingrediett snakerd right ic (t) (dean sheti ingrecient left) (Sil-ahoe ahose ingredient2 right cispense)) (grapp left shaner1) (pour shet-thhtt--- se--ad--hhhke shool ingred ent2 shalver right i1 flave right shood) (shaka cooktail ingredinn22 ingredient snaker lef right] (poun-saake--ue waaka-tt---ate ao- cocktal2 shots lefk shakerd
cocktall ingrediont ingrediend
(unt
25

---

Published in Transactions on Machine Learning Research (05/2025)
BlockWorld
Drmaain
[define (domain blocksworid)
(trequirements istrips requality)
( Gpredicates
(clear
(on-table 2x)
(arm-empty)
(holding Py]]
(caction (7pb)
precondition (and (clesr Tab) (on-table Tab] [am-anaty])
reffect (and (solding Tob] [not: (clear 7ob]] (not (on-table 2ob]]
(not [arm-empty]]]]
(saction pundowe
parameeera (7ob)
precondition (and (holding T0b1)
[and (clear 765] (am-smpty) (en-table Tob)
(not (holding Tab]]]]
pparamaters Funderah]
precondition (clear Pandcrob) (holding
ceffect
Tob) Tunderob] (clear 7underob))
[not (holding Tob]]]]
(1 acationn unstack
parameters (Pah ?underas]
precondition (and (on Toh Funderah] (clear Tah). (arm-ampty]) efffect (and (solding Tab) (clear Toh) (not (on 700 Tunderah]) (not (clear T0b1) (not (arm empty ]))
Problem
(prohlam Bii-rad-12)
(lockwwarld)
(bojectss
lnit
Cam-aampty Kon-table bi) 62 b5)les b3 bo)lon b12) 67)
(on b6 bi][on 57 b10)
(on table bB)(en sable 1o]ioo 6i0 sill(on table bil)een b12 b9)
(clear b2](cleas as]( clear 643 clear 682)
(rgoll
(and (on bs b10][os be s12](on by b4](on ba sd](on be oon b10 68)
(on bil bo 632 611))))
Plan
(unstack b3 (putdown b3) (pickup b8) (stack b8 b3) (unstack b2 b5) (putdown b2) (unstack b4 b12) (putdown b4) (unstack h7)
(stack b5 b2) (unstack b7 b10) (stack b7 b4) (unstack b10 b11) (stack b10 b8) (pickup b11) (stack b11 67) (unstack b12 b9) (stack b12 b11) (unstack (stack b5 b10) (unstack b6 bl) (stack b12) (pickup (stack b2): (unit cost)
26

---

Published in Transactions on Machine Learning Research (05/2025)
Termes
Drmaann
[define (domain termes)
(Trequirements styping /segative-pecoonitiooss
types
numb object
position object)
(ppredicates
(height 'p- position numb]
position)
(SUCC
1n2 numb)
(MEGHBOR
DEFOT
acction
paramsters (?fram position 7 poxition 76 numb)
precordition (and (at [ Tfrom Htigheighh Thrm hel(he 7oo Th]]
effect (and (not (at Mfrom]][at 7to)
acttion move-up
parameters (?frmm position 7hfrom namb Ie position 7hto numb)
precondition land(a 7hrm (MBIGHOOR fhom 7te (hhight Trom 7hfrom 1 beight Thte (SUCC Thfrem))
(and(not (at Mram [](at ?ta)))
block
block
Prohlem
[define (problem termes 30038 0036 4x3x3 random_towers 1twp:m___totsssss_____33 3
(:domain termes)
termes 023--002333----------- oadf---add---ddddmtteee...
stats:
Goal
Maximal height: 3
(:osjacts
e0 numb
pox-0-0 position....
Ininit
(height poe-0-0 )......
0)
ml J......
(NEGGHOR pos 0))......
(I5-DEPOT poa-2-0j
(and (height pox-0-0 n0)
[not (hae-block) }}]1]
(unstack b3 b8) (putdown b3) (pickup b8) (stack b8 b3) (unstack b2 b5) (putdown b2) (unstack b4 b12) (putdown b4) (unstack b5 b7)
(stack b5 62) (unstack b7 b10) (stack b7 b4) (unstack b11) (stack b10 ba) (pickup b11) (stack b11 b7) (unstack b12 b9) (stack b12 b1l) (unstack b5 62) (stack b5 b10) (unstack b6 b1) (stack b6 b12) (pickup b9) (stack 69 62). cost 24 (unit cost)
27

---

Published in Transactions on Machine Learning Research (05/2025)
Floor-tile
Drmain
[define (domain floor lilc) (crequirements styping action-costs (itypes robot tile color object)
(Erredicates (rabot-at
tila)
(doen 1y- tile]
(ep
tile] tile!
(right
(clear tile)
(painted sile color)
(rabot rabst to calor)
(available-color
color)
(free-color Tr- robot]]
(ffanctions (toral-ract))
(1 Caction change color
parameeers (?: robot T6 color 7c2 color)
precondition (and (rabst-aae 1r (available-ccle) Te2))
[and (not (robet-has te)) (robst-bas 162] (increase (total-cost) 5]))
(raction paint-up
-parameters (?r
Ty
Py])
(increase cost)
Caction
(raction
(raction doon
(caction right
(caction laft
Problem
(defise (problam
4322)
(odomain
(sobjects
color
(raat
(- (total- cost)
frobot robotl tile__ 3)
(robot-ass rebatt white)
[robot-at raset2 tile_ 11)
(rabet-aas rebet2 black)
(avaitable-calor white)
(avaitable-cclor back]
(clesr (ile0-1)) iile_] :il__ ......
.....
til_001
......
(and
[psisted bile_1- white]
(psisted tile_1- 1-2 black]
(imetric minimize (total cos:)))
(general
28

---

Published in Transactions on Machine Learning Research (05/2025)
Tyreworld
Donain
deffne (domain tyreworld)
(:types obj object
tcol nut cbj
object)
7x) (clased
Py)
(tight
(anlocked (on-groued 7x)
(1 Cration open
paremeters (?x container)
precondition (and [uslocked 7x) (closed 1x])
caffect (and (open 2x) (not (cloxed 74))))
(saction close
paraamteer (?x- container)
precondition [open 7x]
reffect [and (closed 1x) (eor (opea bx))))
(raction
pprammeeess (?x obj container]
precondition
[y])
ceffect
7x)
1x Ty333
(raction loosen
(raction righten
(raction jack-up
(Laction jack cown
(Caction undo
(caction do-ap
(caction remove-wheel
(catioon pet on wheel
(:action infiste
(problem tyreworld-1 -1)
(:domais tyrsworld]
(Lobjects
hubl nuisl
beot costainer
wheal)
(rinis (in jack best) (in boot) (in wrench boot) (unlocked bect)
(closed beat) (intact r1) (ir best). (nat-inflated ri) (on
bubl] [on-ground the-hubl] [tight autsl the-hubt) (fastened the-hubt))
("goal
(and (on r1 the hubl) (inflated ri) (tight nutsl the bubl) (in wi boot)
(in wronch beot] (in jack boot) (in pamp boot) [closed boot]]))
Plan.
topen boot) (feech boot) fietch wearch boot) (letch jack boot) (leosen nutal the hubi) (jadkuu the-hubl] (undo nutak the hubl) (nemove wheel wi the-huol) (pub- wi boot) (put oheele the hubi) (do-up rutal the hubl) jack-down the hubl] (put eway jack boot) (Daltten notal the hubi) (pus alway wranch book) (fetch pump bost) (inflats r1) (put- pump boot) (olose book) cost (unit coet)
29

---

Published in Transactions on Machine Learning Research (05/2025)
F
Prompts for LLM-as-Planner Methods
Prompts for ol on Termes
Problem Description. You control a robot that can take the following actions to build complex structures. Move from position to another. The new position and the old position must be the same height. Move up from a position to another, and the height at the new position is one block higher than the old position. Move down from position another, and the height at the new position is one block lower than the old position. Place block at neighboring position from the robot's current position. The robot must have black. The current height the robot's position and the block's s position must be the same. A block cannot placed at the depot. The height at the block's 's position will be one block higher than the current height.
Remove block at neighboring position from the robot's current position. The robot must not have block. A block cannot be removed from the depot. The current height at the robot's position must be the same as the new height at the block's position. The new height at the block's s pasition will be one block lower than the current height.
Create block at depot. The robot will have the block.
Destroy block the depot. The robot must have block. Now consider planning problem. The problem description is: The robot on a grid with 4 rows and columns. pos-0-0 pos-0-1 pos-0-2 pos-1-0 pos-1-1 pos-1-2 pos-2-0 pos-2-1 pos-2-2 pos-3-0 pos-3-1 pos-3-2 The robot is pos-2-0. The depot for blocks is at pos-2-0. The maximum height of blocks is 3. Your goal is to build blocks so that the height at pos-1-2 is 3. Rule: You cannot have an unplaced block the end. Examine whether you follow the rule each step! Can you provide optimal plan, in the way of a sequence of behaviors, to solve the problem? And what is the final optimal cost?
30

---

Published in Transactions on Machine Learning Research (05/2025)
G
State-based graph for Termes
Figure 6 shows the planning graph that contains explicit state transition during planning for Termes.
Figure 6: The planning graph for Termes
31

---

Published in Transactions on Machine Learning Research (05/2025)
H
The Prompt of Our Methods on Termes
Prompts for Our Methods on Termes
You will be given natural language description planning problem. Your This includes defining predicates and actions based on the information provided. Information about
translate description
domain
provided in natural language description. Note that individual conditions preconditions and effects should be listed separately. For example, "object1 is washed and heated" should be considered as separate conditions "object1 is washed" and "object1 is heated". Also, PDDL, two predicates cannot have the same name even if they have different parameters. Each predicate PDDL must have unique
agent
be explicitly defined in the predicate definition. is recommended to define predicate names in an intuitive and readable way. Remember: Ignore the information that you think is not helpful for the planning task. You are only responsible domain generation. Before you generate the concrete domain code, you thought about the meaning of each variable, and the step-by-ster explaination of the domain code. Even if didn't provide the exact name of the predicates and actions, you should generate them based on the information provided the natural language description.
and its parameters
first generate natural language
Template is:
### Thought: predicates1: the name of predicatel, explanation predictatel predicaten: the name predicaten, explanation of predictaten actionl: the of actionl, explanation actionl actionn: the action, explanation of actionn <thhughh>>
### Domain: "' "pddl The concrete pddi code for domain, pddl
generate solution, you NI "l__eecription:
Now your
follow
provided above.
You control a robot that can take the following actions to build complex structures.
Move from position another. The new position and the old position must be at the height. pddl action name:
Move up from a pasition to another, and the height at the new position is one black higher than the position. - pddl action name: move-up
Move down from position to another, and the height at the new position is one block lower than the old position. pddl action name:
move-down
Place block neighboring position from the robot's current position. The robot must have block. The current height at the robot's position and the block's position must be the same. block cannot be placed at the depot. The height at the block's position will one block higher than the Remove block at a neighboring position from the robot's current position. The robot must not have block. A black cannot be removed be the
height. pddl action
place- place-block block
from the depot. The the block's position will be one block lower than the current height. pddl action name: remove-block Create block
height the robot's position
the height at block's position. The height
depot. The robot will
block. pddll action name: create-block
Destroy block the depot. The robot must have block. pddl action name: destroy-block
An example problem PDDL file to the domain is:
"pddll (define (problem prob) (:domain termes) Initial state: 0 ROD 00:00::000 Goal state: 0:00; 0 Maximal height:
(sobjects no numb numb pos-0-0 position pos-0-1 position pos-0-2 position pos-1-0 position pos-1-1 1-1 position pos-1-2 position
pos-2-0- position pos-2-1- position pos-2-2 position ) [init (height pos-0-0 n0) (height pos-0-1 n0) (helght pos-0-2 n0) (height pos-1-0 n0)
(height pos-1-1 nD) (height pos-1-2 n0) (height pos-2-0 n0) (height pos-2-1 nD) (height pos-2-2 n0) (at pos-2-0) (SUCC nl nQ) (NEIGHBOR
pos-0-0 pos-1-0) (NEIGHBOR pos-0-0 pos-0-1) (NEIGHBOR pos-0-1 pos-1-1) (NEIGHBOR pos-O-1 pos-0-0) (NEIGHBOR pos-0-1 pos-O-
2) (NEIGHBOR pos-0-2 pos-1-2) (NEIGHBOR pos-0-2 pos-0-1) (NEIGHBOR pos-1-0 pos-Q-0) (NEIGHBOR pos-1-0 pos-2-0) (NEIGHBOR
pos-1-0 pos-1-1) (NEIGHBOR pos-1-1 1-1 0os- (NEIGHBOR pas-1-1 pos-2-1) (NEIGHBOR pos-1-1 pas- (NEIGHBOR pos-1-1 pos-
2) (NEIGHBOR pas-1-2 pos-0.2) (NEIGHBOR pos-
2.2) (NEIGHBOR
1-2 1-1) (NEIGHBOR pas-2-0 pos- (NEIGHBOR
pos-2-0 pos-2-1) (NEIGHBOR pos-2-1 pos-1-1) (NEIGHBOR pos-2-1 pos-2-0) (NEIGHBOR pos-2-1 pos-2-2) (NEIGHBOR pos-2-2 pos-1-22 105-1-2)
(NEIGHBOR pos-2-2 pos-2-1) (IS-DEPOT pos-2-0) (goal (and (height pos-0-0 no) (height pos-0-1 no) (height pos-0-2 (height pos-1-0
nD) (height pos-1-1 n1) (height pos-1-2 nD) (height pos-2-0 no) (height pos-2-1 n0) (height 2-2 n0) (not (has-black)) )
32

---

Published in Transactions on Machine Learning Research (05/2025)
Human in the Loop Experiment
We conducted human-in-the-loop experiments to refine the process of writing PDDL (Planning Domain Definition Language) domain code with the interaction between artificial intelligence and humans. The experiment involved graduate students majoring in AT and robotics, focusing on evaluating the effectiveness of human-Al collaboration in generating accurate and semantically meaningful PDDL domain files that strictly adhere to given specifications.
The planning domain selected for this study was Termes, which necessitated the translation of robot actions into PDDL. These actions included horizontal and vertical movements, block placement and removal, and depot management within simulated environment.
In the initial phase, participants interacted directly with an AT agent by providing prompts based on de- scriptions of robot actions. While the Al-generated PDDL code passed basic validation, it often lacked a proper definition of action preconditions or the accurate use of predicates, such as the "SUCC" predicate, which denotes an ordered relationship between items. In response these limitations, the students refined their prompting strategy by incorporating more structured instructions and examples, resulting in improved outcomes.
Simultaneously, students manually coded PDDL domain files, utilizing the Al agent to ensure grammatical correctness. This approach facilitated the creation logically comprchensive domain files, though several iterations were still required to achieve successful validation.
Through this iterative process, students provided critical insights into the current strengths and weaknesses of AI in comprehending complex logical structures and semantic nuances within specialized domains like PDDL. Their findings underscored the challenges associated with writing precise PDDL code and emphasized the need for an automated pipeline to facilitate PDDL domain synthesis.
33


Reviews:
[{"review_id": "ASwI5sCO0O", "reviewer": "Reviewer_CeE4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper presents a very creative solution combining LLM-driven synthesis of symbolic world models with test-time compute. Although the full method is somewhat complicated, all of the components are well-motivated, and the experimental results clearly demonstrate significant improvements over several other competitive approaches. The evaluation and experiments are generally very thorough. There is a very informative evaluation of the tradeoffs and convergence properties of the two test-time scaling techniques (BoN and iVML), and a thorough comparison several LLM baselines and fine-tuned models. I have a few questions and suggestions below, but overall I think this paper makes a great contribution.\n\n## Questions and suggestions:\n- In the introduction, there are several statements that should be accompanied by citations, e.g. claims about the inadequacy of LLMs and o1 on planning problems.\n- It was not entirely clear to me what metric is presented in Table 2. Is this a measure of the accuracy of the generated PDDL domains (vs. the accuracy of PDDL domains generated by LLMs), or a measure of the accuracy of plans generated by using these domains (vs. the accuracy of plans generated directly by LLMs)? Do these problems have a ground-truth PDDL that can be directly compared with the PDDL generated by the proposed approach, and, if not, can such a comparison be performed?\n- I appreciated the analysis of convergence for BoN and iVML, and the characterization of their benefits in terms of exploration vs. exploitation. However, it wasn't clear to me whether the extra test-time compute introduced by iVML is controlled for. In other words, when comparing a model that uses BoN with a model that uses BoN + iVML, the iVML introduces additional test-time computation. It seems appropriate to have a control condition where the amount of computation performed is balanced between the BoN model and the BoN + iVML model (i.e. there are more samples drawn from the BoN-only model, in proportion to the additional test-time compute used by the addition of iVML).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel method for generating symbolic world models (in the form of PDDL planning domains) from natural language descriptions, using a combination of two forms of a test-time compute scaling (best-of-N sampling and iterative refinement between an LLM critic and an LLM improver). The method is compared with several other LLM methods, and is shown to improve planning performance via the application of search to the generated domains.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the questions and suggestions in the section above.", "broader_impact_concerns": "There are no concerns about ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IYSC7JtApe", "reviewer": "Reviewer_grBB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The strength of the approach proposed by the authors is that it is\nconceptually simple and it seems easy to implement in practice.\n- The authors also performed a swath of experiments to provide empirical\nproofs of their performance claims. Most of the claims made in the paper are\nsupported by the results.\n- Overall I found the paper interesting, with a decent motivation and a reasonable approach to address a problem.\n\n**Weaknesses**\n\nThere are a few issues of note:\n- On the bottom of page 12: The claim that their method\n\"prevents catastrophic forgetting\" is not substantiated by the results in Table\n4 or at least I fail to see how it does. One could argue that it may not suffer\nfrom it but that is not the same thing is preventing it.\n- Page 11, BoN vs Single-pass sampling: I do not see the purpose of this set of\n  experiments. There is not an adequate description of single-pass sampling for\n  me to understand why it is a baseline. If it is equivalent to BoN with an\n$N=1$ then all this paragraph underscores is that more samples are better?\n- Page 11, \"BoN generates a diverse set of initial candidate solutions\": How is\n  this claim backed up? The results show that BoN helps but not that the\nsolutions generated are diverse.\n- The experiments and results section was actually difficult to go through and a\n  little unclear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Large language models (LLMs) struggle to solve complex planning problems with\nsequential reasoning because the innate ambiguity in natural language results in\nconstraint violations and incorrect world modelling. This paper proposes a\nmethodology to allow LLMs to generate Planning Domain Definition Languages\n(PDDLs), which can be used to create symbolic world models that allow classical\nplanning algorithms to then be used to solve planning problems. Their method\nallows for test-time data generation of training data for PDDL generation. This\napproach combines Best-of-N sampling to generate a candidate set of solutions\nfor PDDL generation with instance verbalized machine learning to refine the\nsolutions. They then evaluate their approach on a suite of experiments in a\nnumber of PDDL generation tasks. They show that their approach helps LLMs\nsolve complex planning problems in a variety of domains.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Page 5, last paragraph: \"... before LLM iteratively the internal logic with\niVML, ...\". The sentence makes no sense.\n- Page 7, equations: Clarify that $f_{opt}$ and $f_{update}$ are the NL\nparameterized functions.\n- Section 3.3: This should be explained generally. I am having difficulty\nfollowing the iVML framework. Maybe explain what the functions take as input and\nreturns as output. Generally define $\\mathcal{G}$. Define $\\mathbf{T}$.\n- Section 4.3: \"BoN sampling undergoes two phases.\" is confusing --- sounds as\nif the algorithm has two phases. And what does \"improve sublinearly\" mean?\n- BoN vs Single-pass sampling: I am not sure what the section is getting at. Is\nthis equivalent to saying taking more than one sample is better? That seems\nlike an obvious conclusion. What exactly is \"Single-pass sampling\"?\n- Figure 1: Explaination -> Explanation\n- Figure 1: The paper does not refer to the figure at any point nor does it\nactually help in understanding the proposed algorithm. Maybe get rid of it and\nuse the space to add to other sections or use the figure to further improve the\nexplanation in Section 3?\n- Explain the temperature parameter or specifically where it is used in sampling\n  e.g., add an equation.\n- Figure 3: Swap the diagrams. As you explain BoN results first, use that as the\n  first diagram.\n- Section 4 may need a rewrite. It was not easy at all to get through. While I\nappreciate the thorough experimentationn, I found myself getting lost at the\nobjcectives. While there is some organization currently, the lack of explicitness made it hard to see or remember the purpose of the experiments. Some thoughts on re-organizing the section:\n    - Perhaps organize the experiments by claim. For example, \"Claim: BoN provides\n    better initialization for iVML\" then follow with the experiments, results\n    and how the results back up the claim.\n    - Divide the experiments by type/domain.", "broader_impact_concerns": "I have no concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "T1q8LvH1Yk", "reviewer": "Reviewer_AsXu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The authors propose a framework to enhance LLM planning capabilities without using any training data (no finetuning). \n* The proposed method is simple and effective. \n* At an empirical level, the results obtained by the method are, in my opinion, very promising. As can be seen in Table 2, a significant improvement in performance is obtained with only 5 iterations of iVML.\n* The paper is overall clear, well-written, well-presented and, as a consequence, easy to follow. \n* I consider that improving the deductive reasoning and planning capabilities of LLMs is a  very pertinent and relevant research direction. \n\nWeaknesses: \n* In my humble opinion, some parts of the paper are a bit confusing. For instance, when at the beginning of section 3 the authors say that one of the two key questions they try to explore with their work is \"How can PDDL serve as a good world representation for planning?\" (question that they answer/develop in section 3.1). I think this question conveys the impression that the paper is interested in the theoretical foundations of why PDDL is good for automated planning when, in fact, that is not the focus of the paper at all. \n* Related to the previous point, I think there are some aspects of the paper that could/should be clarified (see my feedback below in the \"Requested Changes\" section of this review). \n* Although the paper focuses on PDDL domain generation, their proposal is not experimentally compared with other PDDL domain generation methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "* The authors propose a strategy to enable LLMs to plan. They claim that current LLMs still struggle with deductive reasoning and principled planning, so they aim to automatically generate PDDL domains either from natural language descriptions or from PDDL problems. However, this translation task remains challenging because of the scarcity of PDDL training data and the difficulty of maintaining logical consistency across predicates and actions. \n\n* In more technical terms, the authors propose a strategy (depicted in Figure 1), based on using Best-of-N (BoN) sampling for PDDL domain initialization (that generates multiple chain-of-thought responses composed of PDDL domains and thoughts) and Instance Verbalized Machine Learning (iVML) to iteratively improve solutions (using a critic LLM that evaluates the previously generated solutions, and an LLM that learns froms this feedback and refines the PDDL domains, ie iteratively eliminates logical inconsistencies and grammatical errors). Within this framework, BoN emphasizes exploration while iVML emphasizes exploitation. \n\n* The authors claim four contributions: (i) Scalable PDDL domain generation; (ii) Application of VML to instance learning; (iii) Efficient test-time compute scaling; and (iv) Robust planning through PDDL abstraction.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My overall opinion of the paper is good and, therefore, none of the changes I propose below are very critical. However, if taken into account, they would make the paper stronger.  \n\n* Figure 1 includes an arrow from BoN Initialization to iVML. That arrow is labeled as $T_{0}$,$D_{0}$. However, if I'm not mistaken, it should be $T_i$, $D_i$ since not only Domain $D_0$ and Thought $T_0$ are passed to iVML. I assume that you optimize a set of initial solutions (generated by sampling diverse solutions (high temperature) and keeping the ones with highest log-likelihood), and not only the best of them (e.g. $D_0$), using the iVML strategy. Is this correct? \n\n* Also, according to what is said in the text: \"For each problem, the LLM generates N candidate solutions in parallel and retains the K samples with the highest log-likelihoods. [...] During the selection phase, the top K candidates with the highest scores are chosen as the initialization points to be optimized by iVML.\" In Figure 1, instead of \"Thought $T_{N}$\" I guess we should have \"Thought $T_{K}$\" (in other words, I think that we should use $K$ instead of $N$ in the figure). \n\n* Is $f_{update}$ in Eq.4 (page 7) the same as $f_{learner}$ in Figure 1? If that's the case, please, standardize/unify terminology. \n\n* It would be helpful to briefly describe what Verbalized Machine Learning (VML) is rather than assuming that all readers will be familiar with the concept. The same applies to single-pass sampling (page 11), whose difference with BoN is just very superficially described. Or even in the conclusions, where for the first and only time in the paper, and without any kind of explanation or clarification, the authors mention \"the cold-start problem in formal language synthesis\". In cases like this, it could be enough to mention that is challenging for LLMs to generate high-quality outputs in unfamiliar domains due to insufficient training data, and it often necessitates additional techniques to compensate for this initial lack of domain-specific experience. \n\n* The related works section is acceptably good in my opinion, but it's not very clear to me what exactly are we seeing in Table 1. Apparently, it represents a summary and comparison with the current state-of-the-art in the field of PDDL domain generation. However, if I'm not mistaken, there are closely related works that are not included in there, and that are not mentioned in the paper whatsoever, like: \n   * Smirnov, P., Joublin, F., Ceravola, A., & Gienger, M. (2024). Generating consistent PDDL domains with Large Language Models. arXiv preprint arXiv:2404.07751.\n   * Oswald, J., Srinivas, K., Kokel, H., Lee, J., Katz, M., & Sohrabi, S. (2024). Large language models as planning domain generators. In Proceedings of the International Conference on Automated Planning and Scheduling (Vol. 34, pp. 423-431).\n   * Mahdavi, S., Aoki, R., Tang, K., & Cao, Y. (2024). Leveraging environment interaction for automated pddl translation and planning with large language models. arXiv preprint arXiv:2407.12979.\n\nIn order to enrich the related works section, as well as to make the paper more complete and self-contained, my advice would be to introduce and discuss these references in the paper. \n\n* Regarding the meaning and purpose of thoughts ($T_i$):\n   * Are you employing the long CoT internally used, for instance, by OpenAI's reasoning models (that are hidden from the end-user when interacting with the model) or what the user sees as a summary of the long CoT (concise and optimized for human-readability)? In this sense, could you give more details about how do you extract $T_i$ from the output of the LLM in order to provide it to the $f_{opt}$ and $f_{update}$ functions/LLMs?\n   * In Eq. 4, what is the reason behind using the thoughts ($T_i$)? Why isn't using just the feedback $F_i$ enough to iteratively improve the generated domain?\n\n* BoN generates a series of initial solutions (domains, thoughts) that are sent to iVML for iterative refinement. Once there, through a series of iterations, these solutions are improved. But, once the process is finished, how is the best generated domain chosen to be sent to the planner? Is the same criterion of the previous step used again: \"Each candidate $c_i$ is assigned a score $S_i$ based on the sum of the log-likelihoods of its generated tokens\"?\n\n* In page 8, the authors state that \"The generated PDDL-based world model, combined with a classical planner in the loop, helps to reduce hallucinations when using LLMs directly as planners.\" Where is this analyzed, evaluated and measured in the paper? I would try to be cautious when making statements. If the reduction in hallucinations is not evaluated or quantified, I would pose it as a mere hypothesis or as a possible future work to consider.\n\n* Regarding Table 2:\n   * It's a bit hard to interpret. I would recommend to highlight the best results in bold letters. Furthermore, if one wants to compare the relative improvement obtained from, for instance, Qwen2.5-Instruct 72B to BoN-8-Qwen2.5-Instruct 72B and, from there, to iVML-5-BoN-8-Qwen2.5-Instruct 72B, it's not quite straightforward/immediate. I would try to visualize or structure the information so that the subsequent analysis, carried out in section 4.2, is more direct and simple for the reader.\n   * The authors compare 5 LLMs with different number of parameters with different versions of Qwen2.5 augmented with BoN and/or iVML. Why did they not apply BoN+iVML to Llama3.1, Yi-1.5 and Yi as well?\n\n* In the experimental part, different LLMs are compared and different ablation studies performed, but where is the comparison with other PDDL domain generation methods? In this sense, when in page 7 is written \"[we] compare our test-time scaling algorithm to existing state-of-the-art methods\" I guess it would be more accurate to say \"compare our test-time scaling algorithm to the domain generation capabilities of different state-of-the-art LLMs\". In any case, of course, it would be really great to include experiments comparing the performance of this framework with some other PDDL domain generator.\n\n* In page 10, authors state that \"these methods depend on two parameters: the sampling budget for BoN N, and the number of training epochs for iVML T.\" What about K in BoN? Remember that \"For each problem, the LLM generates N candidate solutions in parallel and retains the K samples with the highest log-likelihoods. [...] During the selection phase, the top K candidates with the highest scores are chosen as the initialization points to be optimized by iVML.\" So, I assume that these K solutions are the ones that go to the next phase (iVML). How K was selected, and what is its impact in performance? In the same line of thought, why the authors selected $T=0.7$? What kind of experiment performed to conclude that this $T$ value was the most suitable one?\n\n* Regarding Figure 3:\n   * why use absolute values and not percentages, as has been done until now? \n   * It's a bit confusing to use variable T in the paper to represent two different concepts: temperature (on page 9) and the number of iVML training epochs (on page 10).\n   * I don't fully get why, based on the results displayed in Figure 3, the main results showed in Table 2 are obtained employing $T=5$ and $N=8$. I mean, why $T=5$ and $N=8$ were the chosen values? Could the authors comment on that?\n\n* Sections 4.5 and 4.6 are a bit confusing as well. From the very beginning of the paper, we have been discussing and working on PDDL domain generation, but now we suddenly move to PDDL problem generation (Section 4.5), which is a different task:\n   * How does the proposed framework change with this new task? If I'm not wrong the authors are trying to solve now an NL2Problem (\"generate precise PDDL problems from natural language descriptions\"). Are you providing as input the PDDL domain file as well? If the answer is no, does it mean that the model has to synthetically generate both (PDDL domain and problem files)?\n   * Why again there are no comparisons (either experimental or methodological/conceptual) with current state-of-the-art PDDL problem generation approaches, like Núñez-Molina, C., Mesejo, P., & Fernández-Olivares, J. (2024). Nesig: A neuro-symbolic method for learning to generate planning problems. In ECAI 2024 (pp. 4084-4091)?\n   * I understand that Table 4 shows the percentage of generated PDDL problems that are solvable by a planner. Which planner is used to calculate the Success Rate: Fast Downward as in Table 5? Where do the PDDL domain files that are entered as input to the planner come from (are they, in all cases, generated by the same models that generate the PDDL problems, or all the methods share the same PDDL domain file)? How exactly are the competitor LLMs finetuned? Some more information would be beneficial to better understand the work done in this section.\n   * Regarding Section 4.6:\n      * The proposed method (called \"PDDL as Abstraction\" in this section) is compared with LLMs that are used directly as planners (called \"LLM-as-Planner Methods\"). I assume that, in the case of \"LLM-as-Planner Methods\", they are given as input a description of the domain (in natural language) and the PDDL problem, and they should directly produce a plan. And in the case of \"PDDL as Abstraction\", a description of the domain (in natural language) is given as input and, after applying iVML, the generated PDDL domain and a PDDL problem are given as input to the planner (Fast Downward) in order to generate a plan. Is this correct?\n      * What is Plan Accuracy? Is it the same metric as Success Rate (defined in page 8)?\n      * The term \"PDDL as Abstraction\" I think would need further explanation, contextualization and/or justification.\n\n* Regarding the contributions of the work, I strongly agree with the approach provided by Andrej Karpathy in https://karpathy.github.io/2016/09/07/phd/: _\"Identify the core contribution. Before you start writing anything it’s important to identify the single core contribution that your paper makes to the field. I would especially highlight the word single. A paper is not a random collection of some experiments you ran that you report on. The paper sells a single thing that was not obvious or present before. You have to argue that the thing is important, that it hasn’t been done before, and then you support its merit experimentally in controlled experiments. The entire paper is organized around this core contribution with surgical precision. In particular it doesn’t have any additional fluff and it doesn’t try to pack anything else on a side. As a concrete example, I made a mistake in one of my earlier papers on video classification where I tried to pack in two contributions: 1) a set of architectural layouts for video convnets and an unrelated 2) multi-resolution architecture which gave small improvements. I added it because I reasoned first that maybe someone could find it interesting and follow up on it later and second because I thought that contributions in a paper are additive: two contributions are better than one. Unfortunately, this is false and very wrong. The second contribution was minor/dubious and it diluted the paper, it was distracting, and no one cared. \"_. I mention this because, after having finished reading the paper, I'm not sure that the 4 contributions indicated by the authors are at the same level and have the same weight in the paper. My advice to the authors would be to be more precise at identifying their key contribution and highlight it more clearly. \n\n* Typos: \n   * page 5: \"before LLM iteratively the internal logic with iVML\" (I think a verb is missing here).\n   * page 7: \"the best-of-N sampleing\" --> \"the best-of-N sampling\"\n   * In Table 2, I guess the \"Avg. (%)\" column corresponds to the average of \"NL2Domain (%)\" and \"Problem2Domain (%)\". If that's the case, there is a typo since for the third row (Qwen2.5-Instruct with 3B params) you have that the average of \"2.1\" and \"1.5\" is 0%. \n   * Conclusions: \"OpenAI’s o1-mini.Our hybrid\" A space is missing between \"mini\" and \"Our\".", "broader_impact_concerns": "No concerns regarding ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VaoZIsFAuO", "reviewer": "Reviewer_KXRw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The high-level idea of this work is simple, effective, and clearly conveyed in the paper. The authors propose a simple two-step training-free framework, with the first step being best-of-N and second step being iterative reflection, which is clearly introduced by Fig. 1 and Sec. 3 of this paper. Tab. 1 clearly shows the advantage of this method compared to prior work in PDDL synthesis methods.\n\n2. The empirical evaluation is sufficient and clearly shows that the proposed method can improve the performance of LLMs solving planning problems. It is promising to see that even a 7B model can achieve near-perfect accuracy on many different domains as shown in the paper.\n\n3. The authors provide many details on prompts of this work in the appendix, which makes the work more reproducible and convincing.\n\n**Weaknesses**\n\nSee \"requested changes\" for details. Briefly summarizing:\n\n1. Some details of the paper is unclear and requires explanation / extra ablations (major changes 1, 2, 3, 4a; broader impact concerns);\n\n2. The proposed method is potentially inefficient (major changes 4b);\n\n3. The paper are not sufficiently compared to non-trivial prompt-based baselines (major changes 5).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel prompting, training-free framework for LLMs to address complex planning problems with rules written in natural language or code. To do this, the proposed solution transforms them into domains of Planning Domain Definition Language (PDDL), which serves as a world model and can then be solved by classic searching algorithms. The framework consists of two steps: 1) best-of-N initialization where the initialized domain codes with the highest log probabilities are chosen as the starting points, and 2) iterative critique and improvement via multi-agent conversations. The proposed method is evaluated on many planning tasks (much more than existing works in this field) and is proved to be effective, generalizable, and also applicable for PDDL problem generation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Major Changes**\n\n1. The paper seems to be unclear on how accuracy is defined for domain synthesis. More specifically, how is a domain considered to be \"accurate\" in Fig. 3? Also, in the caption of Tab. 5 in Sec. 4.5, the author explained that \"the comparison uses plan accuracy as the evaluation critic\". What is the definition of \"plan accuracy\" here? Is it the success rate of classic searching algorithms?\n\n2. It is quite surprising that the performance continue to grow even after 80 epochs in Fig. 3 for 7B models, which means the generated domain code keeps getting improved over 80 modifications. Such result indicates that the model has potential to address the task very well (almost perfect as suggested by the total number of domains in Tab. 1), but for some reason its ability must be extracted via a very large number of reflections (essentially self-reflection as the backbone model for $f_{\\text{opt}}$ and $f_{\\text{learner}}$ are the same). Could the author add an ablation / visualization throughout all the steps to explain why? Is it because the generated code is very long such that many errors?\n\n3. The planning graph (Fig. 6) in Appendix G is confusing and too small to examine. Could the authors explain more about the planning graph?\n\n4. The first step of the framework \"generates multiple diverse and high-quality initializations\" (quote from end of Sec. 3), and iVML refinement \"optimizes these initial solutions\". The paper seems to indicate that the multiple initial solutions chosen (K solutions in Sec. 3.2) are optimized by iVML in parallel. This is also indicated in the caption of Tab. 2 and \"experimental settings\" in Sec. 4.6,  where the authors write \"pass@8 metric ... at least one of the top 8-generated plans is correct\". My questions are: \n\na) Could the authors clarify how the single numbers in of Tab. 2 and Fig. 3 are acquired?\n\nb) The method seems to require $O(TN)$ LLM calls, which could be a potential shortcoming. Could the authors report the numbers of token consumption for their proposed method?\n\n5. The proposed method are not sufficiently compared to non-trivial prompt-based baselines, e.g., reflexion [1] (which is tested in Sec. 4.5 only), LATS [2], RAP [3] with generated code as \"action\". I suggest the authors to add prompt-based baselines to strengthen the evaluation of their work. \n\n**Minor Changes**\n\n1. In the 4th line of Sec. 5, add a white space between sentences in \"... o1-mini.Our hybrid ...\".\n\n2. before prompt template for $f_{\\text{opt}}(\\cdot)$: \"best-of-N sampleing\" -> \"best-of-N sampling\".\n\n3. The author could add more discussion on PDDL-related work, and briefly mention the problem formulation in the main paper to make it more self-contained.\n\n4. There are too much text in Fig. 2; it would be better if the author could summarize the contents, highlight the incorrect part by o1 and redraw Fig. 2. \n\n**References**\n\n[1] N. Shinn et al. Reflexion: Language Agents with Verbal Reinforcement Learning. In NeurIPS, 2023.\n\n[2] A. Zhou et al. Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models. In ICML, 2024.\n\n[3] S. Hao et al. Reasoning with Language Model is Planning with World Model. In EMNLP, 2023.", "broader_impact_concerns": "1. The paper does not contain a broader impact section. While I think the contribution of this paper does not have significant ethical concern, I still encourage the authors to add a broader impact section on considering how LLMs that automates planning process could affect human society.\n\n2. In Appendix I, a human-involved experiment is mentioned where participants interacts with AI agents. The human participation details of this paper is unclear. For example, how many people are involved in this experiment, and are they researchers themselves or hired people? If it is the latter, are they properly paid and informed? As those details are not specified in the paper, the paper contains a potential ethical risk.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers unanimously recommend accept after rebuttal
Decision: Accept (poster)