Simulation to Rules: A Dual-VLM Framework for Formal Visual Planning
Authors: Yilun Hao
Yongchao Chen
Chuchu Fan
Yang Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Vision Language Models
Planning
PDDL
LLM Tool Use
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose VLMFP, a Dual-VLM-guided framework that can autonomously generate both PDDL problem and domain files for formal visual planning.

Abstract: Vision Language Models (VLMs) show strong potential for visual planning but struggle with precise spatial and long-horizon reasoning, while Planning Domain Definition Language (PDDL) planners excel at formal long-horizon planning but cannot interpret visual inputs. Recent works combine these complementary advantages by translating visual problems into PDDL. However, while  VLMs can generate PDDL problem files satisfactorily, accurately generating PDDL domain files, which encode planning rules, remains challenging and typically requires human expertise or environment interaction.
We propose VLMFP, a Dual-VLM-guided framework that autonomously generates both PDDL problem and domain files for formal visual planning. VLMFP combines a SimVLM that simulates action consequences with a GenVLM that generates and iteratively refines PDDL files by aligning symbolic execution with simulated outcomes, enabling multiple levels of generalization across unseen instances, visual appearances, and game rules.
We evaluate VLMFP on 6 grid-world domains and demonstrate its generalization capability. On average, SimVLM achieves 87.3\% and 86.0\% scenario understanding and action simulation for seen and unseen appearances, respectively. With the guidance of SimVLM, VLMFP attains 70.0\%, 54.1\% planning success on unseen instances in seen and unseen appearances, respectively. We further demonstrate that VLMFP scales to complex long-horizon 3D planning tasks, including multi-robot collaboration and assembly scenarios with partial observability and diverse visual variations. Project page: https://sites.google.com/view/vlmfp.

Published as a conference paper at ICLR 2026
SIMULATION TO RULES: A DUAL-VLM FRAMEWORK FOR FORMAL VISUAL PLANNING
Yilun Hao
Yongchao Chen
MIT yi gilluhace@itteeu
MIT / Harvard University
yongchacchhenf fas waw. hawvard.
Chuchu Fan
Yang Zhang
MIT
MIT-IBM Watson Al Lab Yang 2hang20 ilm. com
chuchu@mit.edi
ABSTRACT
Vision Language Models (VLMs) show strong potential for visual planning but struggle with precise spatial and long-horizon reasoning, while Planning Do- main Definition Language (PDDL) planners excel at formal long-horizon plan- ning but cannot interpret visual inputs. Recent works combine these comple- mentary advantages by translating visual problems into PDDL.. However, while VLMs can generate PDDL problem files satisfactorily, accurately generating PDDL domain files, which encode planning rules, remains challenging and typi- cally requires human expertise or environment interaction. We propose VLMFP, a Dual-VLM-guidee framework that autonomously generates both PDDL prob- lem and domain files for formal visual planning. VLMFP combines a SimVLM that simulates action consequences with GenVLM that generates and itera- tively refines PDDL files by aligning symbolic execution with simulated out- comes, enabling multiple levels of generalization across unseen instances, visual appearances, and game rules. We evaluate VLMFP on 6 grid-world domains and demonstrate its generalization capability. On average, VL. achieves 87.3% and 86.0% scenario understanding and action simulation for seen and unseen appearances, respectively. With the guidance of Sim VLM, VLMFP at- tains 70.0%. 54.1% planning success on unseen instances in seen and unseen ap- pearances, respectively. We further demonstrate that VLMFP scales complex long-horizon 3D planning tasks, including multi-robot collaboration and assem- bly scenarios with partial observability and diverse visual variations. Project page:
INTRODUCTION
Although Large Language Models (LLMs) have shown strong performance in solying text-based planning problems (Wei et al., 2002 Yao al. [022]] Raman al.] [2022] Yao et al.] (2024), many real-world planning tasks, such as robot assembly, drone navigation, and autonomous driving, are inherently visual, making the reliance on carefully engineered text inputs impractical and limiting. This gap motivates the shift toward VLM- based planning, where visual inputs provide more di- rect and intuitive basis for reasoning. However, current VLMs lack precise spatial understanding and long-ho onng-horizon reasoning, which constrains their ability to address complex, multi-step planning problems that involve intricate spatial relationships among multiple objects (Wu al. 2024).
On the other hand, Planning Domain Definition Language (PDDL) (McDermoti 2000 is a formal language designed to describe planning problems and domains in structured, machine-interpretablle way. PDDL has enabled numerous automated planners to derive long-horizon solutions. However, although PDDL-based planners excel at reasoning over structured domains, they depend on correctly structured PDDL domain and problem files and cannot directly interpret visual inputs. Constructing accurate PDDL definitions is non-trivial and requires expert knowledge, which often inaccessible to non-expert users, limiting the broader adoption of PDDL planners in real-world scenarios.

---

Published as a conference paper at ICLR 2026
Image Observation
Domain Description
SimVLM
Initaal States
PDDL validator
Randam Actions
Execution Result & F'eedback
Problem
Reached?
Description
Plan
Sumilario compure
Domain PDDI
Probiem PDDL
Problem
Execution
Presereening
Domain PDDL
Domain PDDL
Rating & Feedback
.nDI Planner
Updating
GenVLM
Figure 1: An overview of VLMFP. Sections in orange are the inputs. Sections in green represents ViM and its outputs. Sections in blue denotes the Gen and its outputs. Sections in gray provided. VLMFP takes in the image and domain description, describe the scenario, generate and validate PDDL files, compare their execution results with VI.,, and update the PDDL files.
Recent works have explored combining the advantages of language models and PDDL planners through various approaches. (Liu et al.) [023) Xie et al. 2023) employ LLMs as translators to convert natural language scenario descriptions into PDDL problem files. (Mahdavi et al. (024) leverages environment interactions to enable file generation for both problem and domain. How- ever, these methods require either textual scenario descriptions, environment access, pre-defined PDDL files, which cannot be directly achieved through visual inputs. More recently, vision-language models (VLMs) have been used to extract scenario information from visual inputs and generate cor- responding PDDL problem files (Shrri et al. 2024 Dang et al. 2025). However, since current VLMs lack the ability accurately generate the domain PDDL, these approaches also assume cess to ground truth domain PDDL files, without which PDDL planners cannot produce any results.
In this paper, address the challenge of visual long-horizon planning by introducing novel framework, VIMM-Gudde Formal Planning (VLMFP, illustrated in Fig. []. a Dual-VL.M-uidde framework that autonomously generates both problem and domain PDDL files for visual planning. VLMFP integrates two specialized VLMs: a fine-tuned Sim VLM that perceives the scenario from visual inputs and simulates action outcomes, and a large GenVLM that generates and iteratively re- fines PDDL files by aligning their execution with Sim VLM's simulations. Generating both problem and domain PDDL from visual inputs requires object recognition, spatial understanding, reasoning, and PDDL knowledge. We fine-tune small VLM as VIM to strengthen its spatial reasoning, while using a large model as Gen for general reasoning and extensive PDDL knowledge. This formulation is agnostic to enviromment dimensionality, allowing the same perception- torsoono-------iiiee pipeline to extend beyond
settings to visually grounded 3D planning tasks.
Importantly, VLMFP achieves multiple levels of generalizability. A generated domain PDDL can be reused across all instances of the same domain, while problem PDDL files can be adapted efficiently for new instances as in-context examples. The framework also transfers well to unseen appearances and even altered environment rules. We evaluate VLMFP on six grid world domains, showing that Sim VLM reliably describes scenarios, simulates actions, and determines goal achievement, while Gen GenVIM VLM, guided by VIMVI.M feedback, generates valid PDDL. files that enable planners solve both seen and unseen problems. Beyond grid-world benchmarks, we further examine the scalability of VLMFP on complex long-horizon 3D planning tasks, including assembly and multi- robot collaboration scenarios under partial observability.
In summary, our key contributions are:
We construct a large-scale dataset of 430k action sequence simulations with reasoning and feed- back across 6 grid-world domains of different map sizes, appearances, and game rules. We fine- tune VL.-72 with the dataset, and our finetuned model demonstrates strong generalization to unseen instances, appearances, and game rules.
We propose VLMFP, Dual- Miaal-VLM-aa -ugided framework that autonomously generates PDDL domain and problem files for visual planning, which, to our knowledge, is the first framework to leverage visual inputs generate both PDDL. files without human feedback or direct environment access.
2

---

Published as a conference paper at ICLR 2026
By combining SimVLM (for perception and action simulation) with GenVLM (for symbolic rea- soning and file refinement), VLMFP achieves robust, reusable domain generation and efficient problem instantiation. VLMFP notably achieves 70.0% and 54.1% success rates with GPT-4o as the Gen VLM, outperforming the best baseline Cod by 39.3% and 21.8% for unseen instances in seen and unseen appearances, respectively for 6 grid-world domains.
We demonstrate that VLMFP scales beyond grid world benchmarks to complex long-horizon 3D tasks with partial observability and diverse visual variations, with an average of 86.4% and 79.8% success rates for unseen instances in seen and unseen appearances, respectively.
RELATED WORKS
LLM and VLM Plamning Although LLMs have achieved impressive performance on text-based planning tasks (Wei et al.) 2022, Yao et al., 2022 Raman et al. 2022 Yao et al., (024), many real- world planning problems are inherently visual. Relying solely on carefully constructed text inputs is often impractical and cannot capture rich spatial details in visual environments. This limitation has motivated growing interest in VLM-based planning, where visual observations provide a more direct and intuitive foundation for reasoning (Driess al., 2023 Huang al. 2023 Zhang et al. 2023 Nasiriany et al. 2024). However, current VLMs still fall short in precise spatial understand- ing (Wu et al. 2022). restricting their effectiveness on complex planning tasks involving multiple objects and intricate spatial relationships. Recent works explore Visual Chain-of- Though (Li et al.) 2025 /Zhao et al. 2025), using images as part of the reasoning process while generating plans. How- planning
ever, this process introduces computational overhead and is hard to apply to tasks. In this work, we fine-tune VLM to enhance its visual-spatial understanding and reason- ing, enabling it to guide the generation of PDDL files for solving monghoorzzon planning problems.
LLM and VLM PDDL Since existing LLMs lack the ability to reliably perform long-horizen reasoning in complex tasks (Achiam et al. 2023a Valmeekam et al.] 2022 [2023] Kambhampati
al. 2024). recent works have explored combining LLMs with external solvers to augment their
reasoning and planning capabilities (Wu et al. 2022,
He- Yueya et al. 2023 Pan et al. 2023 Ye
al. 2024, Li et al.) 2023 2024 Hao al. 2022a b). Among such sol solvers combining L.MSs with
PDDL-based planners a powerful option Silver et al. 20222] Liu et al. 2023
Stein & Koller 2023
Xie et al. 2023; Stein & Koller 2023 Guan et al. [2233 Oswald et al. 2024 Mahdavi et al. 2024).
as PDDL designed for precise symbolic reasoning over long- morizon problems. However, these works either require predefined PDDL domain files, human corrections, or environment access to provide feedback while generating PDDL files. In addition, further enable visual interpretation, many works combine VL.Ss with PDDL planners (Shirai et al., 2024 Dang et al., 2025) by gen- erating PDDL Problem files based image observation. However, similarly, as generating PDDL. domain files is complicated, these works assume access to predefined domain files. In our work, we use
framework autonomously generate both PDDL problem and domain files, without human guidance or environment access.
3 VLMFP
3.1 PROBLEM FORMULATION AND CHALLENGES
In our setting, a visual planning problem is defined by two pieces of information: A domain de- scription Ra, which is natural language deseription of the general rules, domain settings, available actions, and the planning objective of the problem; and problem setup image ip, which is an image showing the layouts, initial states of the problem instance.
Figure 2 top left shows an example domain description and problem setup image of the planning problem FrozenLake. As can be observed, the domain description elaborates the rules of the game (e.g., no stepping onto an ice hole) and available actions of the player (i.e., moving up, down, left, and right). The problem setup image is a map showing the initial positions of the player, the destination, and frozen likes.
The goal of VLMFP is, given the problem definition pair (na, ip). to come up with an action quence to achieve the goal. Considering the complementary strengths of VLMs in image perception and natural-languaa interpretation, and symbolic methods such PDDL in formal planning, our
3

---

Published as a conference paper at ICLR 2026
Domain Description
Problem Domain PDDLs valid PDDL
GenVLM Updates: Updhte Reseoring:
The grd world consists that
files, preserenning passed.
Undatel Domsin PDDL:
ground
Start Rating & l'redback
sce holes. starss at one cell and must reach the gozl cell by moving down, left,
player
Rating & Feedhack: Round
The precendition of maveriggtt
Caction move-down
iperametens Tho)
Random Action Seguetee
down, move right
insuficiiot. Since agent is not allowed to ento holes, (not (iee-hole should added dosini PODL fil.
pprceondition (and
ento ice hole.
right, without stepping
Step I- Action mave down. Elveutioon Ressoning: The agent
SimVI M Excention Rosult Ferdlback
tsot fiec-hole settect
lal "Nam)
(down fhom ?to))
VL M generated scenarto description:
from pos-I-1. Since
of the grid woeld is The agont pos- I-1.
there ise hole cell pos-2-1, it sucoessfillly moves pos-2-1.
(cot (st ?[oons))
lat 'to)
latles at poo-1-2 1-2, pan--4, pou-2-d pay- 3-1, pone--3, pus--4.
The gont pos-4-4.
Essestionn Resul: Successful.
PODI, filer updated, stortnng new round of rating Rating & Feedbbckk: Rouad 2
Step 2 Action move right...
Your good is ochinve the following configuration
Generated PDDI Exceution Resolt
agrust poa-444
The are
generated environment
All randome action executions hav remais, score
Gea VLM generated PDDI, Files:
Action sequeeec is exccutabble both environments
(detine (prublem FL)
Probiem PDDL:
Domain PDDL:
(defiine (domain F1.)
Random Action Sequetee 2. move right
Plan Found:
Soarching for plas using PD
SimVLM Exceution Recult & Feedlback
dows pos- por2
Dobjects
Opeedicates
Step Action right:
(ensse-right por-1o1 pos2-1)
pos-1 1-1 pos 12... ...
(x)
Eleecstion Ressoning:
(nooe down pas 2.2 por 2)
The agant tries to move right from pos-1-1 to pas-1-
down pos- 3-2 port-
C-mii
Caction move-right
2. Since there is an ice linle in cell pas-1-2, -2, the agent
(mosse-right poo-4-2
pprameters ("fiom (to)
falls execution unsuccessful.
hole,
ends and
(museerright 3 puu4-4)
(toe hole pos- 1-2)
precondition
(iloes)
Clocking for plan valsdanerr
(right pos- 1-1 pos- 2)
tright 'iro (to)
Euecution Resul:: Unsuccesffll
effbet (and
Vim Execution Result
Plo)
Generated PDDL Executioa Result
Gool Reaching: Successfill.
(gnell
(not (ax (ftoo)))
The actions are exscutable generated environment
The plan is valid
pos-4-4)
Execution discrepancy exists (scure Updase PDDL files
Probion and Domain PDDL files
returned the oumpor
Figure 2: An example of how VLMFP tackle a Frozenlake instance.
framework integrates these advantages to transform visual observations into formal specifications that can be solved symbolically.
However, translating visual inputs into PDDL specifications using VLMs is an error-prone process. PDDL provides a standardized framework for representing automated planning problems, consisting of two files: 0 PDDL domain file fa. which formally encodes rules by defining predicates and actions with preconditions and effects; and @ PDDL problem file fp. which specifies objects, the initial state, and the goal for concrete instance. fa and fp can be viewed as formal translations of the domain description n. and problem setup image ip, respectively. See Appendix E for examples.
The translations of these two files using VLMs come with their respective challenges. On one hand, translating the problem setup from the image ip to the PDDL problem file fp requires precise understanding of the spatial relationship the image, but existing VLMS tend to make mistakes in spatial perception tasks such as counting rows/columns, measuring distance etc (Wu et al., 2024). On the other hand, the PDDL domain file fa should be general enough to work for all problem instances under the same task (e.g., different maps of the same FrozenLake game), but generating the PDDL domain file using VLM alone, without human input or constant access to the environment, can easily fail accurately model all constraints and dynamics(e.g.. to move left, the goal cell must not be ice hole, and must be at left of the origin cell).
3.2 VLMFP OVERVIEW
To tackle the aforementioned challenges, VLMFP introduces dual-VIM framework, consisting of a SimVLM (Simulation VI.M) and Gen (Generation VLMM). SiM takes the domain description ng. the problem setup image ip. and list of action the FrozenLake game), and performs three tasks: Describing the spatial relationship in the ip in natural language, denoted np. @ reasoning and predicting the consequence of the proposal actions (e.g. whether the player hits a wall/ice hole), and @ determining whether executing æ successfully achieves the problem goal. GenVLM generates both PDDL files with the help of SimVLM
[e1r]] (e.g., move left in
The two VLMs should have different comparative advantages. VLM should be stronger at pre- cisely understanding the spatial relationship in images, and have a superior capability in simulating
4

---

Published as a conference paper at ICLR 2026
action consequences. GenVLM should possess stronger general reasoning and question-answenine capabilities, and have richer knowledge in PDDL.
With these complementary strengths, SimVLM supports GLn in two key ways. First, it pro- vides a structured description of the spatial relationships in the problem setup image ip. compensat- weaker spatial perception. Second, SimVLM serves as execution-checking oracle: by comparing its simulated action outcomes with those derived from the generated PDDL model, any discrepancies signal potential errors and are fed back to GenVLM for refinement.
ing for
Based on these rationales, VLMFP generates PDDL files via the following four steps (as in Figure
Candidate PDDL files generation. Given the domain descriptions and problem setup image, (na, fp), the SimVLM first generates a description of the spatial relationships (7p) in the problem setup image, which is then fed to the GLn generate candidate PDDL files, namely
Step 1:
np = Vs(na, ip),
f(0) fp = Va(nn, mp).
(1)
where Vs and VG represent the forward passes of VLM and Gen respectively.
Step 2: Prescreening. The generated PDDL files checked against syntactic correctness and semantic consistency.
Step 3: Simulation consistency checking. Random action sequences are executed in the PDDL environment based on the generated PDDL files, whose results are compared against the simulation results by Sim VL. Inconsistencies are summarized as feedback, denoted
Step 4: PDDL files updating. Vcn refines the generated PDDL files based on the feedback:
fa Jp Va ip. nyi 8, f( (tt-1)
Sp f((-1)).
VLMFP iterates over steps 2-4 until consistency is achieved and a plan is found by the PDDL planner. Figure gives an example of this process. In the following, Section 4.1 will discuss the implementation details of the two VLMs; Section 4.2 will provide further details of the above steps.
4 IMPLEMENTATION DETAILS
4.1 VLM DETAILS
VLM. As discussed in Section 3.2 SimVLM tasked with precisely describing the problem setup image ip and predicting action consequences. Since existing VLMs are generally weak in spatial relationship reasoning, we fine-tune Qwen2-VL-72 VL-7B to accomplish these tasks. Specifically, collected six grid world domains of varying complexity. For each domain, collect data across map sizes ranging from 3 to 8 with varying obstacle probabilities, and create 5-6 distinct visual appearances. In total, this yields dataset of 430k datapoints.
The fine-tuning process enables VLM to better align visual observations with spatially grounded reasoning. In particular, SimVLM learns to generate concise natural language narratives of the initial scenario and to simulate the outcomes of action sequences within these domains. By grounding visual inputs in structured textual descriptions and action simulations. SimVLM serves as reliable intermediate model that bridges raw perception and the formal representation required for PDDL generation. We show examples of API-based large VLMs' failure to accurately describe the scenario and simulate actions in Appendix B.4
VLM. While SimVLM tailored for perception and structured simulation, generating PDDL. domain and problem files requires advanced reasoning to capture action dynamics and precise understanding of PDDL syntax and conventions to ensure valid domain specifications. To address this, we leverage large VLM API, GPT-4o (Achiam et al.) (023b), referred to as GenVLM, which has broader reasoning capacity and linguistic knowledge needed for reliable PDDL construction. Guided by the scenario descriptions and simulated outcomes provided by SimVLM VLM gen- erates initial PDDL files and iteratively refines them to resolve inconsistencies. In this way, Gen- VLM complements VLM by bridging high-level reasoning with formal symbolic representation.
5

---

Published as a conference paper at ICLR 2026
4.2 ALGORITHMIC DETAILS
Prescreening. In this step, VLMFP generates and verifies whether the generated initial PDDL problem and domain files are structurally valid before moving to further evaluation. A PDDL domain or problem is valid if it is syntactically correct and semantically consistent, meaning the domain's 's actions, predicates, and types well-defined and the problem's objects, initial state, and goals align with that domain.
For example, as shown in Fig. 2 in the Frozenl domain, the problem PDDL defines ice hole positions using predicate ce-hole, but if the Domain PDDL does not define i ce-holl under the (:predicates section, these two files would be identified prescreening. We set the maximum rounds of file regeneration to be 5. This stage is crucial because filters out syntactic or structural errors at an early step, thereby ensuring that only valid PDDL files progress to the consistency checking process and that subsequent comparisons are meaningful.
invalid and cannot pass the
Simulation consistency checking. In this step. VLMFP evaluates the fidelity of the generated PDDL. files by comparing their execution results with those of Sim VLM. Random executable action sequences are sampled in both environments and executed in the other environment: SimVIMM sim- ulates the outcomes and provides reasoning step by step, and the PDDL environment checks action exccutability under the generated domain.
For walk length T, let Psim. be distribution over SimVLM-executable T-step action sequences, where Efarf (4) {0, 1} indicates whether q executable in the generated PDDL environment (fas fp). Similarly, let Pfa.,..T be distribution over T-step walks executable in (fa: fp). and let
Estm (g) E {0, 1} indicate VLM executability. Following (Mahdavi et al. 2024), we define the Exploration Walk (EW) score as:
((,))=2((T))))
Tima
mew mew(d,p)=1 22
P (B +
max EppP (Bsim (4))
T
maax
(3)
The EW score compares bi-directional similarity. A higher EW score indicates stronger align- ment between the generated and reference environments. For example, in the FrozenLake domain, Sim VLM predicts that moving right from pos-1-1 to pos-1-2 should fail due to an ice hole, whereas the generated PDDL environment instead allows the move. This discrepancy lowers the EW score and signals VLn to refine the domain file. Importantly, this stage not only enables VLMFP to detect execution mismatches and rate generated files using the EW score, but also produces natural language feedback on the incorrect actions to guide further file updates.
PDDL files updating. In this step, VLMFP refines the generated PDDL files based on the discrepan- cies identified in the last step. Using the feedback that describes the incorrect action and its expected outcome, Gen VLM systematically inspects all critical components by listing and reasoning whether the objects, object types, init states, goal states, predicates, actions are valid and coherent, identifies which file(s) are erroneous, and applies targeted modifications, such as adding missing objects or correcting action preconditions, to regenerate updated PDDL specifications.
EXPERIMENTAL RESULTS
5.1 DOMAINS
We finetune SimVLM and test on six grid world domains: Frozenlake, Maze, Sokoban, Package, Printer, and Overcooked (Towers et al., 2024, (Silver & Chitnis 2020 Jin al. 2023; Wu al.) 2021]. For each domain, we collect data across map sizes ranging from 3 to 8 with varying obstacle probabilities, and create 5-6 distinct visual appearances. In total, this yields a dataset 430k datapoints. By default, we use this dataset for SimVLM fine-tuning, which is used to evaluate framework VLMFP. We also evaluate on two more complex 3D embodied tasks, MultiRob and Assembly (Feng et al. 2025 Ji et al. 2025), to assess generalization across environment complexity and visual variability. Domann descriptions, visualizations of different scenario appearances, domain complexity analysis, and dataset statisties included in Appendix A.
A.2
A.3 and B.5

---

Published as a conference paper at ICLR 2026
Frozenlake
Maze
Sokoban
Package
Printer
Overcooked
$
Seen
Unseen
Figure 3: Visualizations of some seen and appearances for six grid world domains.
5.2 SIMVLM PERFORMANCE AND APPEARANCE GENERALIZATION
We fine-tune a Qwen2-VL.-7B Wang et al. 2024 model as Sim on datasets from six grid world domains. Given domain description, an image, and action sequence, SimVLM outputs (1) a natural language problem description, (2) step-by-step reasoning and outcomes, and (3) judgment on goal achievement. Appendix
Appendix|Bpprovided proviies fine-tuning details, sample inputs and outputs. Before integrating VLM into VLMFP, we evaluate its performance on these tasks individually. Since each datapoint has a fixed-format output, we measure performance by exact string matching between the model's predictions and the ground-truth outputs. The four outputs are referred to as Task Description, Execution Reason, Execution Result, and Goal Reach.
To assess Vim'LM's generalization ability, we consider two settings: Seen (S) and Unseen (U) appearance. S evaluates images with visual styles present in training, while evaluates novel ap- pearances under the same rules. This measures how well Sim VLM fits the training distribution and how robustly it transfers to new visual variations of the same task. For each domain, we evaluate on 1000 random datapoints for both S and U, and report the average string matching rate.
Table 1: String matching rate (%) for 4 SLM output types on 6 grid world domains.
Output
Frozenlake
Maze
Sokoban
Package
Printer
Overcooked
Average
S
U
U
S
U
S
U
U
Task Descrip.
92.3
99.1
89.8
74.9
51.6
100
99.7
99.9
96.3
99.2
65.6
95.5
82.6
Exec. Reason
96.3
96.0
85.3
97.1
76.6
84.6
89.8
88.4
87.4
86.8
78.9
75.9
85.7
88.1
Exec. Result
96.2
95.9
85.3
96.8
75.2
83.4
89.8
88.4
87.4
86.8
78.8
75.7
85.5
87.8
Goal Reach
94.0
93.7
80.3
94.7
74.2
83.2
87.8
86.0
84.1
84.6
73.9
71.2
82.4
85.6
Average
96.6 94.5
87.5
94.6
75.2
75.7
91.9
90.6
89.7
88.6
82.7
72.1
87.3
86.0
Results and Analysis. We summarize SimVLM's performance on six grid-world domains under both seen (S) and unseen (U) appearance settings in Table There are three key takeaways:
First, SimVLM achieves strong accuracy across tasks on seen visual appearances, On average, reaches 95.5%, 85.7%. 85.5%. 82.4% in the seen setting across all domains for the four outputs, respectively. This highlights the effectiveness and reliability of SimVLM.
Second, Sim generalizes well to unseen visual appearances. On average, it reaches 82.6%, 88.1%. 87.8%, 85.6% the unseen setting across all domains for the four outputs, respectively. The gap between S and U is minimal (average drop of 1.3% across all domains and outputs), showing that the fine-tuned model not overfitting training appearances and can transfer to new visual styles while maintaining stable output quality. This robustness is crucial for scaling to real-world environments where visual variation is common.
Third, enrors in the Task Description do not necessarily affect action simulation. While the Task Description output under drops by 12.9% on average compared to S, the execution simulation outputs remain equally accurate. This suggests that failures in Task Description typically involve only a small subset objects, which often do not impact the sampled action sequences. For instance, the Sokoban task contains multiple object types: box, goal, wall, agent, floor, and the appearances

---

Published as a conference paper at ICLR 2026
of these objects are all changed. When the appearances of the objects in domain change, since SimVLM is not familiar with the new look of objects, it could fail to recognize or localize some objects. However, when the misidentified objects are not directly involved in the sampled action sequence, execution reasoning and results remain correct. This separation highlights that Sim VLM robust in reasoning about action dynamics, even when its initial object recognition is imperfect.
Together, these results validate that VimVLM can produce reliable structured outputs from visual inputs and maintain strong generalization new appearances, making it a suitable foundation for guiding PDDL generation in VLMFP.
5.3 VLMFP PERFORMANCE
We evaluate VLMFP on six domains using GPT-4o (Achiam et al. 2023b). Each problem instance comes with domain description and image observation, from which VLMFP generates both PDDL problem and domain files. The problem file is then used as an in-context example to prompt VLn to generate PDDL problem files for 100 new instances, with the PDDL Planner validating the plans. We repeat this process for 15 different input instances and calculate the average success rate of finding correct plans, that is, the average success rate of 1500 trials. We follow the same procedure for unseen appearances. Appendix C provides all prompts and failure-case analysis.
Baselines We compare VLMFP against 1) Direct: VLM direct plan generation with an in-context example. 2) CoT: chain-of tfain-of-thought prompting (Wei et al. 2022) with an in-context example by asking LLMs reason before generating the final answer, 3) CodePDDL: prompts LLM generate PDDL files, given problem descriptions generated by VLM For all baselines, we use GPT-4o and also include Direct and CoT baselines with GPT-5 (OpenAl 2025). All baselines have the same domain description and instance image observation inputs. For Direct and CoT, since they do not generate PDDL files, we directly apply them to generate plans for the 100 problem instances.
Table 2: Success rate (%) comparison of VLMFP with baselines on grid world domains.
Method
Frozenlake
Maze
Sokoban
Package
Printer
Overcooked
Average
U
S
U
S
U
S
U
U
S
U
S
U
Directarr-to
7.0
8.0
1.0
2.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
1.3
1.7
Directerr-s
30.0
24.0
10.0
15.0
15.0
9.0
5.0
3.0
11.0
9.0
0.0
0.0
11.8
10.0
CoTarT-o0
9.0
10.0
1.0
2.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
1.7
2.0
CoTGPTS
36.0
28.0
14.0
16.0
14.0
13.0
2.0
3.0
10.0
12.0
0.0
0.0
12.7
12.0
CodePDDLePt-n
88.1
77.1
87.5
74.9
0.0
0.4
0.0
19.0
7.2
22.1
1.1
0.0
30.7
32.3
VLMFP GPT-40
95.2
81.1
88.7
82.8
55.8
25.1
75.2
53.4
58.7
61.0
46.2
21.3
70.0
54.1
Results and Analysis We present VLMFP performance in Table
There are three key takeaways:
First, VLMFP achieves strong performance, significantly outperforms all baselines. Across six do- mains, VLMFP achieves an average success rate of 70.0% (S) and 54.1% (U), while the strongest baseline (CodePDDLaaPT.d) reaches 30.7% (S) and 32.3% (U). Note that CodePDDL-aPT.4 has access to the SimVLM generated scenario descriptions, which substantially aid the problem gener- ation. Direct plan generation and thain-of-though prompting perform very poorly, confirming that without explicit PDDL grounding, LLMs struggle to maintain consistency in long-horizon plan- ning. These results highlight the advantage of plan generation with formal PDDL files guided by Sim VLM. Importantly, the results demonstrate that, after seeing one problem instance in the do- main, VLMFP can reliably aid the planning of problem instances not seen by VLMFP. This clearly proves VLMFP to be effective, generalizable, and efficient framework.
Second, VIMFP generalizes well to unseen appearances. On average, VI.FFP can solve 54.1% of the unseen problem instances in unseen appearances, which still substantially outperforms the baselines. This demonstrates that the strong perception capability of SimVLM well adapted by VLMFP, enabling VLMFP generalize better across different visual styles.
Third, domain complexity influences the relative gains of VLMFP. In simpler domains such as Frozenlake and Maze, both VLMFP and CodePDDL achieve high success rates, though VLMFP still maintains a clear margin. In complex domains like Sokoban and Printer where reasoning about different object types and multiple complicated actions is required, VLMFP delivers strong perfor-
8

---

Published as a conference paper at ICLR 2026
mance gain over CodePDDL: 55.8% 0.0% (U) in Sokoban, and 58.7% 7.2% (U) in Printer. This shows that the rating and updating processes are crucial for capturing complex domains.
Together, these findings validate that VLMFP is not only effective and efficient in solving unseen problem instances, but also robust to variations in visual appearances and domain complexity.
5.4 VLMFP EFFECTIVENESS OF VLMFP COMPONENTS
We validate each stage of LLMFP with ablations domains. We examine the effectiveness of Pre- screening, Feedback, and Update by removing them from VLMFP one at time. Similarly, VLMFP generate PDDL files based on one problem instance and test with 100 other problem instances.
Table 3: Success rate (%) when removing key components of VLMFP on 6 grid world domains.
Method
Frozenlake
Maze
Sokoban
Package
Printer
Overcooked
Average
No Preservening
94.3
62.5
23.9
56.9
37.3
10.3
47.5
No Feedback
95.5
84.9
38.0
58.9
53,4
35,9
61.1
No Update
88.1
87.5
0.0
0.0
7.2
1.1
30.7
VLMFP
95.2
88.7
55.8
75.2
58.7
46.2
70.0
Removing any of the three components negatively impacts per- formance across domains. While prescreening and feedback contribute steady improvements, the updating stage is essential. Success rates reduce to near zero in complex domains without updating. These demonstrate the effectiveness of all three stages for VLMFP achieve robust performance.
We report ablation results in Table 3
5.5 GENERALIZATION ON GAME RULES
Beyond generalization unseen instances and visual appearances, we further evaluate whether the framework can adapt to changes underlying environment rules. FrozenLake, we construct 15 rule variants by modifying core mechanics such ice-hole behavior and movement dynamics (see Appendix D for rule descriptions). A Sim model is fine-tuned with these variations and eval- uated on five unseen rules using 100 problem instances. The unseen rules introduce progressively more complex dynamics, ranging from reworded known behaviors and modified transition effects to novel mechanies that influence multi-step execution (Appendix DD.. We evaluate whether Sim VLM can correctly predict execution reasoning and outcomes for actions involving these unseen rules.
Table 4: Success rate (%) when testing Sim VLM unseen rules for Frozenlake.
Output
Rulel
Rule2
Rule3
Rule4
Rule5
Exec. Reason & Result
94.2
99.0
76.1
59.2
71.1/0
Results are summarized Table 4. SimVLM achieves strong performance on Rules 1-4, reaching accuracies of 94.2%, 99.0%, 76. 16.1% %, and 59.2%, respectively, demonstrating robust generalization to altered rule formulations and modified environment dynamics. These results indicate that Sim VLM can successfully interpret and apply previously unseen rule variations when their underlying struc- ture remains related to training dynamics. Performance drops on Rule 5, which introduces a com- pletely novel freezing mechanic not observed during training. Although SiM correctly explains the rule behavior (i.e., the next action should be skipped) with 71.1% % success rate, it fails to apply this constraint during execution, resulting in correct reasoning but unsuccessful action simulation. This highlights key limitation when adapting to entirely new dynamics, while also suggesting that Sim VLM retains partial semantic understanding of unseen rules.
Overall, these results demonstrate that VLM exhibits promising generalization to modified and unseen environment rules, supporting the adaptability of VLMFP beyond fixed planning dynamics.
5.6 VLMFP FOR COMPLEX 3D PLANNING TASKS
While previous experiments focus controlled grid-world domains for systematic evaluation, VLMFP not restricted to such settings and naturally extends to complex 3D continuous envi- ronments. We further evaluate framework on two complex long-horizon 3D tasks: an assembly

---

Published as a conference paper at ICLR 2026
task (Feng et al. 2025) and multi-robot collaboration task (Ji et al. 2025) shown in Figure The domains are challenging because (1) the tasks are fully 3D with no natural-languaa goal, requiring Sim to infer spatial relationships and relational dependencies directly from images, and (2) all block attributes (number, color, shape, position, state), table textures and colors, robot and back- ground colors, and dependency orders randomized with frequent occlusions, producing highly variable, partially observable configurations. We collect 100k datapoints for each domain and fine-
Multi Robot
Assembly
Perspective 1
Perspective
Initial configuration
Goal configuration
Figure 4: Visualization of MultiRob and Assembly.
tune Qwen2-VL-7B as our SimVLMMexampll inputs and outputs are provided in Appendix B.7 Appearance variation is introduced by randomizing robot, table, background, and object colors, lighting conditions, and camera viewpoints in MultiRob, and by varying block color, position, or- der, and number under different random seeds in Assembly. Unseen appearances from a different distribution together with unseen random seeds. Following our evaluation protocol, evaluate Sim VLM 1,000 datapoints each from seen (S) and unseen (U) appearance distribu- tions, and report the end-to-end success rates of VLMFP over 1,500 trials for each setting.
colors sampled
Results and Analysis As shown in Table 5 Sim achieves consistently strong performance across all four evaluation metrics, with an average of 96.2%, 96.0% for Assembly and 99.5%, 99.5% for MultiRob for and unseen appearances, respectively. This strong SimVILM performance greatly supports PDDL files generation, with VLMFP 's end-to-end performance: the overall plan- ning success rates are high across both seen and unseen settings for both tasks, achieving 92.4%, 85.9% for Assembly and 80.3%, 73.6% for MultiRob, respectively.
Since SimVLM can already reason over initial and goal images infer object dependencies, as well as multi-view observations to address occlusions, the framework naturally supports further ex- tensions. For example, incorporating temporal image sequences could enable reasoning over scene dynamics and support planning in evolving environments. Extending our framework to more di- verse and realistic 3D tasks is an important direction for future work, particularly given the growing availability of scrge-scale real-world vision and robotics datasets.
Table 5: SiMVLM String matching rate (%) and VLMFP Success rate (%) 2 3D domains.
Domain
Task Descrip.
Exec. Reason
Exec. Result
Goal Reach
Average
VLMFP
S
U
S
U
S
U
S
U
S
U
S
U
Assembly
86.9
84.4
99.1
99.1
99.2
99.1
99.4
99.3
96.2
95.5
92.4
85.9
MultiRob
99.9
99.5
99.5
99.4
98.6
99.0
100.0
100.0
99.5
99.5
80.3
73.6
CONCLUSION
We introduce VLMFP, a Dual-VI -LMguuLd framework that autonomously generates PDDL domain and problem files from visual observations for long-horizon visual planning. By combining a fine- tuned VLM for visual understanding and action simulation with a large GenVLM for symbolic reasoning and iterative refinement, VLMFP removes the need for pre-defined domain files, human supervision, environment access. Experiments on six grid-worke domains show that Sim SimVLM reliably produces scenario descriptions and action simulations, while VLMFP consistently generates valid PDDL files enabling planners solve diverse tasks. Results demonstrate strong generalization to unseen instances, appearances, and game rules, highlighting the value of integrating perception and symbolic reasoning for scalable formal planning. Beyond 2D benchmarks, we further show that VLMFP extends to complex long-horize 3D planning tasks under partial observability and diverse visual variations, demonstrating the scalability of the perception-to-symoolie planning paradigm.
10

---

Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
Our study uses only synthetic grid-worid environments and does not involve human subjects, pri- vate information, sensitive data. As such, there no associated risks related to privacy, security, or legal compliance. While our framework advances automated planning by combining vision and symbolic reasoning, we acknowledge the potential misuse of planning systems in harmful applica- tions. To mitigate such risks, our experiments are confined to safe benchmark domains, and we will release all code, models, and datasets support transparency and reproducibility.
REPRODUCIBILITY STATEMENT
For reproducibility, we provide detailed domain descriptions Appendix A.I1 dataset statistics in Appendix B.5 fine-tuning parameters in Appendix B and the details and prompts of VLMFP in Appendix C The code is included in the Supplementary Material. Upon acceptance, we will release code, model, and dataset publicly under open-source license.
LARGE LANGUAGE MODEL USAGE FOR WRITING
In this paper, we employed large language models specificall ChatGPT- as writing aids. Draft passages were submitted to the model to improve grammar and refine structure, and the results were carefully reviewed and edited by the authors. The use of the tool was strictly limited text polishing; it was never applied to generate original content or references.
10 ACKNOWLEDGMENT
This work was supported by ONR under Award N00014-22-1-2478 and MIT-IBM Watson AI Lab. However, this article solely reflects the opinions and conclusions of its authors.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023a.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.088: 087744 2023b.
Xuzhe Dang, Lada Kudláčková, and Stefan Edelkamp. Planning with vision- language models and use case in robot-assisted teaching, arXiv preprint 2rXiv:2501.17665 2025.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied mul- timodal language model. 2023.
Yunhai Feng, Jiaming Han, Zhuoran Yang. Xiangyu Yue. Sergey Levine, and Jianlan Luo. Reflec- tive planning: Vision-language models for multi-stage long-horizon robotic manipulation. arXiv preprint ar Xixv:22502.100 16707, 2025.
Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. Leveraging pre- trained large language models to construct and utilize world models for model-based task plan- ning. Advances in Neural Information Processing Systems,
2023.
Yilun Hao, Yongchao Chen, Yang Zhang, and Chuchu Fan. Large language models can plan your travels rigorously with formal verification tools. arXiv preprint arXiv:2404. 1891, 2024a.
Yilun Hao, Yang Zhang, and Chuchu Fan. Planning anything with rigor: General-purpose purpose zero-shot planning with llmebased formalized programming. arXiv preprint arXiv:2410. 12, 2024b.

---

Published as a conference paper at ICLR 2026
Joy He- Yueya, Gabriel Poesia, Rose E Wang, and Noah D Goodman. Solving math word problems by combining language models with symbolic solvers. arXiv preprint arXiv:2304.09102, 2023.
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint 3oXYv:2307..09733. 2023.
Jiabao Ji. Yongchao Chen, Yang Zhang, Ramana Rao Kompella, Chuchu Fan, Gaowen Liu, and Shiyu Chang. Collision-and reachability-aware multi-robot control with grounded IIm planners. arXiv preprint arXiv:2505.20573 2025.
Emily Jin, Jiaheng Hu, Zhuoyi Huang, Ruohan Zhang, Jiajun Wu, Li Fei-Fei, and Roberto Martín- Martín, Mini-hehavior: A procedurally generated benchmark for long-horizon decision-making in embodied ai. arXiv preprint arXiv:2310.01824, 2023.
Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Kaya Stechly, Mudit Verma, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. Llms can't plan, but can help planning in llm-modulo frameworks. arXiv preprint arXiv:2402.01877 2024.
Beibin Li, Konstantina Mellou, Bo Zhang, Jeevan Pathuri, and Ishai Menache. Large language models for supply chain optimization. arXiv preprint arXiv:2307.03887. 2023.
Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulié, and Furu Wei. Imagine while reasoning in space: Multimodal visualization-of-tuee -f-thuugttt. arXiv preprint 2rXiv:2501.07542. 2025.
Zelong Li. Wenyue Hua, Hao Wang, He Zhu, and Yongfeng Zhang. Formal-llm: Integrat- ing formal language and natural language for controllable Ilm-based agents. arXiv preprint arXiv:2402.00798, 2024.
Bo Liu, Yuqian Jiang. Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Lim+ P: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304. 1..304/11777 1477, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://
llava -lava-vl.gar wlw.aa/llittuubb..o/lllo//// wttp://..oooooog///oeea//// 01144-01-30 30-30-
Sadegh Mahdavi, Raquel Aoki, Keyi Tang, and Yanshuai Cao. Leveraging environment interaction for automated pddl translation and planning with large language models. Advances in Neural Information Processing Systems, 37:38960-39008, 2024.
Drew M McDermott. The 1998 ai planning systems competition. Al magazine, 21( 2000.
Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowl- edge for vlms. arXiv preprint arXiv:240.007872. 2024.
OpenAl. Introducing GPT-5, Blog post, 2025. URL https://opeaai.com/mnneo
introducing gpt- 57| Accessed 2025-08-07.
James Oswald, Kavitha Srinivas, Harsha Kokel, Junkyu Lee, Michael Katz, and Shirin Sohrabi. Large language models as planning domain generators. In Proceedings of the International Con- ference Automated Planning and Scheduling, volume 34, pp. 423-431, 2024.
Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. Logic-Im: Empower- ing large language models with symbolic solvers for faithful logical reasoning. arXiv preprint 2rXi::230S.12295 2023.
Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. Planning with large language models via corrective re- re-prompting. In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022.
12

---

Published as a conference paper at ICLR 2026
Keisuke Shirai, Cristian C Beltran- Hernandez, Masashi Hamaya, Atsushi Hashimoto, Shohei Tanaka, Kento Kawaharazuka, Kazutoshi Tanaka, Yoshitaka Ushiku, and Shinsuke Mori. Vision- language interpreter for robot task planning. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 2051-2058. IEEE, 2024.
Tom Silver and Rohan Chitnis. Pddlgym: Gym environments from pddl problems. arXiv preprint arXiv:2002.06433, 2020.
Tom Silver, Varun Hariprasad, Reece S Shuttlewrtth, Nishanth Kumar, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. Pddl planning with pretrained large language models. In NeurIPS 2022 foundation models for decision making workshop. 2022.
Katharina Stein and Alexander Koller. Autoplanbench:: Automatically generating benchmarks for Ilm planners from pddl. arXiv preprint arXiv:231 1.09300 2023.
Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shanghang Long, al. Paligemma 2: A family of versatile vlms for transfer. arXiv preprint arXiv:2412.03355. 2024.
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032. 2024.
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large lan- guage models still can't I plan (a benchmark for Ilms on planning and reasoning about change). In NeurlPS 2022 Foundation Models for Decision Making Workshop, 2022.
Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Informa- fion Processing Systems, 36:75993-76005. 2023.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing Aisionllanguag model's 's perception of the world at any resolution. arXiv preprint 1eXXix:2409.22991, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Qiucheng Wu, Handong Zhao, Michael Saxon, Trung Bui, William Yang Wang, Yang Zhang, and Shiyu Chang. Vsp: Assessing the dual challenges of perception and reasoning in spatial planning tasks for vims. arXiv preprint Xrxiv:2407.00 403.00/6..,6.. 2024.
Sarah A Wu, Rose E Wang, James A Evans, Joshua B Tenenbaum, David C Parkes, and Max Siiiman-Weiner Too many cooks: Bayesian inference for coordinating multi-agent collabora- tion. Topics in Cognitive Science, 13(2)4414432, 2021.
Yuhuai Wu, Albert Qiaochu Jiang, Wenda Li, Markus Rabe, Charles Staats, Mateja Jamnik, and Christian Szegedy. Autoformalization with large language models. Advances in Neural Informa- tion Processing Systems, 35:32353-32368. 2022.
Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating natural lan- guage planning goals with language models. arXiv preprint arXiv:2302.05128. 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 2022.
Shunyu Yao. Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
Xi Ye, Qiaochu Chen, Isil Dillig, and Greg Durrett. Satlm: Satisfiability- language models using declarative prompting. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2026
Xiaohan Zhang, Yan Ding, Saeid Amiri, Hao Yang, Andy Kaminski, Chad Esselink, and Shiqi Zhang. Grounding classical task planners via vision-language models. arXiv preprint 0rX4v:2304.08587, 2023.
Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, et al. Cot-vla: Visual chain-of-thoughe reasoning for vision- language-action models. In Proceedings of the Computer Vision and Pattern Recognition Confer- ence, pp. 1702-1713, 2025.
14

---

Published as a conference paper at ICLR 2026
SIMULATION TO RULES: A DUAL- VLMM FRAMEWORK FOR FORMAL VISUAL PLANNING
Introduction
Related Works
VLMFP
3.1
Problem Formulation and Challenges
3.2
VLMFP Overview
Implementation Details
VLM Details
4.2
Algorithmic Details
5
Experimental Results
Domains
5.2
Sim Performance and Appearance Generalization
5.3
VLMEP Performance
8
5.4
VLMFP Effectiveness VLMFP Components
5.5
Generalization Game Rules
5.6
VLMFP for Complex 3D Planning Tasks
Conclusion
10
Ethics Statement
11
8
Reproducibility statement
11
9
Large Language Model Usage for Writing
11
10
Acknowledgment
11
Domains
17
A.1 Domain Descriptions
17
A.2 Scenario Appearance Visualizations
18
A.3 Domain complexity comparison
19
B Sim VLM Details
20
baselines
20
B.2
Sim VLM base model ablation
21
B.3
Sim VLM performance across different seeds
22
B.4 Failure case for GPT-5 to accomplish the task
23
15

---

Published as a conference paper at ICLR 2026
B.5 Dataset Statistics
24
B.6
Fine-tuning Implementations
24
B.7
Input and Output Examples
24
VLMFP Details
27
C.1 Direct and COT Major Failure Cases
27
VLMFP Major Failure Modes and Cases
27
VLMFP Implementation Details and Prompts
28
D Genelization to Game Rules
35
D.I Game Rule Variation Descriptions
35
Ground truth PDDL files examples
36
16

---

Published as a conference paper at ICLR 2026
A DOMAINS
A. DOMAIN DESCRIPTIONS
We test on six grid world planning domains, Frozenlake, Maze, Sokoban, Package, Printer, and
Overcooked (Towers et al. 2024 Silver & Chitnis 2020; Jin et al. 2023 Wu et al. 2021
Frozenlake In the scenario, you have a girdworld, where each cell can be either normal ground ice holes. The player starts at cell, and there is a goal position in cell. The goal is to move the player to the goal position. You can move up, down, left, right, but you cannot move into the border, and stepping into the ice hole will fail the game.
Maze In the scenario, you have maze gridworld with walls and goal positions. The player can move up, down, left, or right. If the player hits a wall or performs an invalid action, the action fails. The goal is reach the goal cell.
Sokoban In the scenario, you have gridworld with walls, boxes, and goal positions. The player can move, push the box to goal, and push the box to other position. The player can only push the box forward, not toward other directions. If the player hits wall or performs invalid action, the action fails. The goal is to push all boxes to reach the goal cells.
Package In the scenario, you have a gridworld with some closed packages in it. The player can turn-left, turn-right, move, pick-up, drop-down, open, close the packages. If the player hits border or performs an invalid action, the action fails. The goal is to open all packages.
Printer the scenario, you have a gridworld with a desk region and printer. The player can turn-left, turn-right, move, pick-up, drop-down, toggle-on, or toggle-off the printer. The player cannot move into the desk region. If the player hits border, desk, or performs an invalid action, the action fails. The goal is to pickup and drop the printer in desk region and toggle it on.
Overcooked In the scenario, you have gridworld cooking game with counters, ingredients, chop- ping boards, and a goal position for delivery. The players can move, chop, pick, drop, merge- ingredient, put- put-plate, deliver. If the player moves into the counter or performs an invalid action, the action fails. The goal is to cook the salad my merging chopped ingredients, put merged ingre- dients into plate, and put the plate to the delivery position.
We also test on 3D domains Assembly and MultiRob (Feng et al. 2025 Ji et al. 2025
Assembly the scenario, there is puzzle consisting of a board and several blocks with different colors the table. The goal is to assemble the puzzle to reach the goal configuration with the robot arm. The robot arm can pick up. put down, insert, or reorient the block. The blocks have dependence on other blocks; that is, to achieve the goal, the blocks should be inserted after the blocks they depend on are inserted.
MultiRob In the scenario, there a grid-like environment consisting of robot arms and objects with different colors the table. The goal is to move all objects to their specified target locations. The robot arm can pickup and putdown the block. Each robot arm can only reach its surrounding four grids.
17

---

Published as a conference paper at ICLR 2026
A.2 SCENARIO APPEARANCE VISUALIZATIONS
We create six different scenario appearances for Frozenlake, Maze, Sokoban, Package, Printer, and five scenario appearances for Overcooked. We include the visualizations of different appear- ances in Fig. 5 For Frozenlake, Maze, Sokoban, we change the appearances for every object in the scenario, such as the frozen ice hole, the box, the wall, the agent, etc.. The change objects have various looks. For example, the agent can have the look of a person, a car, or bike. And the goal can also have the look of present, home, target, or bag of money. For Package and Printer, we change the color of the background and the desk. For Overcooked, we not only change the color of the background, look the agents and objects, but also change the ingredients from lettuce and tomato to tomato and onions. The differences in appearances not only bring challenges for the Sim VLM to recognize unseen objects, but also to understand the game rule and reason about the objects in the scenario (lettuce onion?).
Frazenlake
Maze
Sokoban
Package
Printer
Overcnoked
Seen
Unseen
Figure 5: Full visualizations of various appearances for 6 grid world domains.
18

---

Published as a conference paper at ICLR 2026
A.3 DOMAIN COMPLEXITY COMPARISON
Frozenlake actions (move up, move down, move left, move right); Mechanism: no stepping into ice holes; Object types: grid positions
Maze 4 actions (move up, move down, move left, move right); Mechanism: cannot move into walls: Object types: grid positions
Sokoban 3 actions (move, push-to-goal, push-to-nongoal); Mechanism: movable objects; can only move/push to clear position; agent, box, and goal positions need to be three consecutive positions in one direction; Object types: grid positions, direction, box
Package 5 actions (turn-left, tum-rrghtt, move, open, close); Mechanism: orientation; interactive objects; multiple objects to visit; Object types: grid positions, direction, package
Printer actions (turn-left, turn-right, move, pick, drop, drop desk, toggle on, toggle off); Mech-
anism: orientation; interactive objects; Object types: grid positions, direction
Overcooked 9 actions (move, pick-ingredient, drop-ingredient, pick-plate, drop-plate, chop, merge-ingredient, put-plate, deliver): Mechanism: multiple players; interactive objects; object different states; multiple objects to visit; Object types: grid positions, direction, player, ingredi- ent, plate, shopping-hoare delivery-point
Assembly 6 actions (pickup, putdown, reorient_to.ruo uppriggh reorient horizontall,.... insert, unstack); Mechanism: 3D; object dependency; interactive objects; object different states; multiple objects visit; no natural language goal description
MultiRob 2 actions (pickup, putdown); Mechanism: 3D; multiple robots; occlusion; reachability; multiple perspectives; interactive objects; multiple objects to visit; Object types: grid positions, robot, object
When the domains become complex, without feedback and updating, it is easy for LLMs to ig- nore some preconditions/effects directly generate correct domain and problem PDDL files. This significant performance difference showcases the importance of the feedback and updating modules when handling complex domains.
actions or fail to assign every object a type, making it hard to
19

---

Published as a conference paper at ICLR 2026
B SIMVLM DETAILS
B.1 SIMVLM BASELINES
We compare SimVLM with baselines: 1) GPT4o-Direct: using GPT4o to direct accomplish the task, given the expected format; and 2) GPT4o-InContext: using GPT4o to accomplish the task, given the expected format and in-context example. Note that we use the same four metrics (Task Description, Execution Reason, Execution Result, Goal Reach) for comparison. However, since Execution Reason, we do not expect API-based VLMs to output exact formatted output as ground truth output, did not exact string match. Instead, use sentence-transformees transformers library with all-MiniLMM-L-- model to check the Semantic Similarity for Execution Reason, which is a less strict evaluation than Sim VLM.
The results are shown in 6 GPT4o-Direct and GPT4o-InContext can only accomplish the four tasks with average of 2.5% and 3.3 (.3t(mxyyy low because this is not binary judgement), 51.6% and 76.5 (semantic similarity), 35.7% and 67. (binary judgement), 81.4% and (biarry judgement), comparing 82.6%, 88. 1% 87.8%, 85.6% of SimVLM. We can observe that the API- based large VLM lacks precise spatial understanding and long-horizon reasoning capabilities to directly accomplish the task of SimVLM, thus fine-tuning specialized model is necessary.
Table 6: String matching rate (%) comparison across four metrics 6 grid world domains.
Task Description
Method
Frozenlake
Maze
Sokoban
Package
Printer
Overcooked
Average
Directorr-40
0.0
0.0
0.0
12.7
2.0
0.0
2.5
InContextoPT-to
0.0
0.0
0.0
12.0
8.0
0.0
3.3
SimVLM
92.3
89.8
51.6
99.7
96.3
65.6
82.6
Execution Reason
DirectorT-4n
34.6
63.4
21.7
55.6
63.4
70.7
51.6
InContextapr-4n
47.8
82.7
82.9
79.5
82.4
83.8
76.5
SimVLM
96.0
97.1
84.6
88.4
86.8
75.9
88.1
Execution Result
Directorr-40
29.1
42.9
16.9
23.9
56.6
44.8
35.7
InContextarT-4o
32.1
63.0
73.4
82.8
83.6
68.9
67.3
Vim
95.9
96.8
83.4
88.4
86.8
75.7
87.8
Goal Reach
Directorr-40
72.0
68.8
80.8
79.2
91.4
96.0
81.4
InContextepT-4o
83.8
74.4
96.8
83.8
89.0
99.2
87.7
SimVLM
93.7
94.7
83.2
86.0
84.6
71.2
85.6
20

---

Published as a conference paper at ICLR 2026
B.2 SIMVLM BASE MODEL ABLATION
We selected Qwen2-VL-7BB (Wang et al. 2024) as our base model because it is one of the small VLMs that achieves state-of-the-art performance across wide spectrum of multimodal bench- marks. We add an ablation study that evaluates SimVLM performance comparing Qwen2-VI. -L- 7B with LLaVA-NeXT-7B (Liu et al. 2024) and Google PaliGemma22 (Steiner The results are shown in Table 7 and Table Both models performed reasonably well, but they showed slightly lower accuracy, higher variance, and are less generalizable unseen appearances than VL-72- Importantly, although the choice of Swen2-VL-77B reflects empirical stability, our framework does not depend on Qwen2-VL-7B specifically. VLMFP remains model-agnostic, and substituting the simulator backbone is straightforward.
2024).
Table 7: String matching rate (%) for 4 SimVLM output types on 6 grid world domains, with LLaVA-NeXT-7B as the base model.
Output
Frozenlake
Maze
Sokoban
Package
Printer
Overcooked
Average
S
U
U
S
U
S
U
$
S
U
U
Task Descrip.
99.3
87.2
99.5
30.7
85.9
61.2
99.0
98.6
99.7
80.7
97.0
43.0
96.7
66.9
Exec. Reason
89.2
86.2
74.5
91.2
64.2
81.2
79.5
79.2
75.3
75.7
68.5
68.5
75.2
80.3
Exec. Result
89.3
86.4
74.2
91.3
64.9
82.3
79.6
79.2
75.4
75.8
68.8
68.8
75.4
80.6
Goal Reach
89.6
87.6
73.9
91.2
65.9
83.5
80.1
79.7
75.9
76.3
69.2
69.2
75.8
81.2
Average
91.8
86.8
80.5
76.1
70.2
77.0
84.5
84.2
81.6
77.1
75.9
62.4
80.8
77.3
Table 8: String matching rate (%) for 4 SimVLMM output types on 6 grid world domains, with PaliGemma2-10B the base model.
Output
Frozenlake
Maze
Sokoban
Package
Printer
Overcooked
Average
S
U
S
S
U
U
Task Deserip.
99.8
88.5
99.6
13.2
76.7
50.6
95.2
94.4
88.1
85.8
99.7
42.5
93.2
62.5
Exec. Reason
90.9
87.2
77.5
94.0
68.5
84.7
83.0
82.8
78.9
78.8
72.5
72.8
78.6
83.4
Exec. Result
90.9
87.2
77.6
94.1
67.3
86.4
83.2
83.0
79.0
78.9
73.3
73.5
78.6
83.9
Goal Reach
90.9
89.7
77.2
94.0
71.0
87.2
83.1
82.8
80.0
79.7
74.1
74.1
79.4
84.6
Average
93.2
88.1
83.0
73.9
70.9
77.2
86.1
85.8
81.5
80.8
79.9
65.7
82.4
78.6
21

---

Published as a conference paper at ICLR 2026
B.3 SIMVLM PERFORMANCE ACROSS DIFFERENT SEEDS
Without an external oracle, absolute correctness of SimVLM cannot be guaranteed. However, we conduct experiments evaluate Sim VLM across random seeds to measure its stability. We show the mean and standard deviation of the string matching rates for metrics across 3 seeds in Table 9 and Table 10 Across the domains, the standard deviation for all four metries is very small, indicating that Sim Sim's perception, step-wise reasoning. and action-result predictions are highly stable and SimVLM does not hallucinate unpredictably.
Additionally, we emphasize that the SimVLM metrics are computed over 1000 evaluation data- points, and VLMFP results are averaged over 1500 planning trials for each domain. These large- sample evaluations already demonstrate strong empirical stability despite the possibility of halluci- nation, and the resulting performance substantially exceeds all baseline models.
Table 9: The mean of string matching rate (%) for SimVLM output types on grid world domains across 3 seeds.
Output
Frozenlake
Maze
Sokoban
Package
Printer
Overcnoked
Average
$
U
U
$
U
U
$
U
U
Task Descrip.
99.9
91.9
99.2
89.9
74.0
51.4
100.0
99.9
99.8
96.6
99.2
64.9
95.3
82.4
Exec. Reason
94.9
94.8
83.6
96.9
72.4
83.4
88.5
89.0
85.7
85.4
76.9
75.8
83.7
87.5
Exec. Result
94.9
94.7
83.6
97.0
72.1
83.0
88.5
89.0
85.7
85.4
77.0
75.9
83.7
87.5
Goal Reach
93.5
93.5
80.3
94.7
72.7
83.1
86.8
86.6
83.6
83.7
73.9
73.2
81.8
85.8
Average
95.8
93.7
86.7
94.6
72.8
75.2
91.0
91.1
88.7
87.8
81.8
72.4
86.1
85.8
Table 10: The standard deviation of string matching rate (%) for 4 SimVLM output types 6 grid world domains across 3 seeds.
Output
Frozenlake
Maze
Sokoban
Package
Printer
Overenoked
Average
U
S
U
$
U
Task Descrip.
0.1
0.5
0.1
2.1
0.0
0.2
0.1
1.1
0.2
Exec. Reason
1.5
1.3
1.5
0.2
3.8
2.0
2.3
1.0
2.2
2.0
1.8
3.7
2.0
1.3
Exec. Result
1.4
1.3
1.5
0.2
3.6
0.6
2.3
1.0
2.2
2.0
1.6
3.7
2.0
1.0
Goal Reach
0.5
0.2
1.7
0.0
1.4
0.1
1.1
0.7
0.5
0.8
1.9
2.4
0.7
0.2
Average
0.8
0.7
0.7
0.0
2.2
0.6
1.4
0.7
1.2
1.1
0.9
2.5
1.1
0.6
22

---

Published as a conference paper at ICLR 2026
B.4 FAILURE CASE FOR GPT-5 TO ACCOMPLISH THE SAME TASK
For the following image observation as input, we show the GPT-S generated task description and action execution results for the sequence [move left, move down, move down, move up, move up, move right]. From the output, we can observe that GPT-5 fails to recognize the locations of ice
Figure 6: The input image GPT-5 the image observation of problem instance.
holes. In addition, since it mistakenly thinks pos-2-1 is ice hole, the action execution also fails. The agent should successfully move to pos-2-1 on Step 2, while GPT-5 marks it as moving into the ice hole, which results in game failure. We include the correct output for this instance from VLM in Appendix B.7
From the image Initially:
observe the: Initial State Description:
The size of the gridworld is 4x4.
The agent is at (pos-1-1).
The goal is (pos-4-4).
The ice holes are at (pos-1-2), (pos-1-3), (po8-2-1), (pos-2-2), (pos -3-2), (pos-3-3).
Your goal is to achieve the follewing configuration: The agent is (pos-4-4)
Stop 1 Action move left: Exccution Reasoning: The agent tries to move left from pos-1-1, This action is not valid because it would move into the border, the agent stays pos-l-i.
Execution result: Unsuccessful.
Step 2 Action move down:
Execution Reasoning: The agent tries to move down from pos-1-1 to
-2-1. Cell pos-2-1 is an ice hole, the agent falls in and the game ends. Execution result: Unsuccessful.
Step 3 Action move down:
Execution Reasoning: The agent has already failed the game, this execution is invalid.
Execution result: Invalid.
Step 4 Action move up: Execution Reasoning: The agent has already failed the game, this execution is invalid. Execution result: Invalid.
Step 5 Action move up: Execution Reasoning: The agent has already failed the game, so this execution is invalid.
Execution result: Invalid.
Step 6 Action move right: Execution Reasoning: The agent has already failed the game, so this execution is invalid.
Execution result: Invalid.
Goal reaching: Unsuccessful.
23

---

Published as a conference paper at ICLR 2026
B.5 DATASET STATISTICS
Table 11: Dataset and Task statistics for six grid world domains.
Task
FrozenLake
Maze
Sokoban
Package
Printer
Overcooked
Grid Sizes
3-8
5-8
5-8
4-8
4-8
5-8
# #Appearances (S/U)
5/1
5/1
5/1
5/1
5/1
4/1
Action Types
4
3
7
Data per Appearances
20000
20000
20000
5000
5000
19739
B.6 FINE-TUNING IMPLEMENTATIONS
We utilize four H100 80GB GPUs for fall-parameter tin-itnning of the Qwen2-VL-7B model. The model is trained for epochs in the SFT stage. The learning rate set to 1 10 We batch size of 2 for training. We use 10% of the training data for validation, and we select the model with the lowest validation loss. We perform the inference of SimVLM using single H100 80GB GPU.
B.7 INPUT AND OUTPUT EXAMPLES
Example Input of the Frozenlake domain:
<image>Given a sequence of actions and an image observatien the initial setup of a scenario, your goal is to decide the execution result after executing given actions. To achieve this task, you will first describe the setup you observe from the image, give the reasoning that explains how the player moves and the success failure reason for each single action, then rate whether the overall execution is successful, and eventually describe whether the actions achieves the task's s goal.
Task Description:
the scenaric, you have girdworld, where each cell can be either normal ground or ice holes. The left upper corner is pos-1-1) for (row, column) representation. The player starts at a cell, and there is goal position in cell. The goal is to move the player the goal posi position. You move up, down, left, right, but you cannet move into the border, and stepping into the ice hole will fail the game.
Action Sequence:
1: move left
2: move down
3: move down
4: move up
5: move up
6: move right
Example Output of the Frozenlake domain:
From the image we can observe the: Initial State Description:
Initially:
size the gridworld is 4x4.
The agent is at (pos-1-1).
The goal is at (pos-4-4).
The ice holes are at (pos-1-2), (pos-1-4), (pos-2-4), (pos-3-1), (pos -3-3), (pos-3-4), (pos-4-1).
Your goal is to achieve the follewing configuration:
The agent is at (pos-4-4)
Step Action 1eft:
24

---

Published as a conference paper at ICLR 2026
Execution Reascning: The agent tries action is not valid as is moving into the border. the execution is unsuccessful and the agent stays at the original location. Exccution result: Unsuccessful.
left from pos-l-1. This
Step 2 Action
down:
Execution Reasoning: The agent tries to move down from pes-1-1, so will move to pos-2-1. There is no ice hole in cell pos-2-1 and the agent is not moving to the border, no invalid action is executed to pos-2-1.
and the agent successfully Execution result: Successful.
Step 3 Action move down:
Execution Reasoning: The agent tries to move down from pos-2-1, it will move to pos-3-1. Since there is ice hole in cell pos-3-1, the agent will fall into the ice hole, so the game ended and the exccution is unsuccessful. Exccution result: Unsuccessful.
Step 4 Action up: Execution Reasoning:
The agent has already failed the game, so this execution is invalid. Execution result: Invalid.
Step 5
Action up:
Execution Reasening: The agent has already failed the game, so this execution is invalid. Execution result: Invalid.
Step 6 Action
right:
Execution Reascning: The agent has already failed the game, this execution is invalid. Execution result: Invalid.
Goal reaching: Unsuccessful
Example Input of the Assembly domain:
<image><imageeGivven a sequence of actions, an image observation of the initial setup of a scenario, and an image observation of the goal contiguration, your goal is to decide the execution result after executing given actions. To achieve this task, you will first describe the setup reasoning that explains how the agent moves and the success or failure reason for each single action, then rate whether the overall execution is successful, and eventually describe whether the actions achieves the task's s goal.
goal you observe from the image, give the
Task description: In the scenaric, There is a puzzle consisting of a board and several blocks with different colors on the table. The geal is to assemble the puzzle with the robot arm. The robot pickup, putdown, insert, or reorient the block. If the robot performs an invalid action, the action fails. The goal is to assembly the blocks to reach the goal configuration. In the goal
configuration, some blocks have dependence to other blocks, that is, achieve the goal, the blocks should be inserted after the blocks they depend on are inserted.
Action Sequence:
1: reorient red nail
2: insert blue block
3: pick up red nail
4: insert yellow block
Example Output of the Assembly domain:
From the images we can observe that: Initially:
25

---

Published as a conference paper at ICLR 2026
There 7 blocks.
blue block is upright; is not inserted.
The brown block is upright; it is not inserted.
The orange block is upright; it is inserted.
The purple block is upright; it is not inserted.
The yellow block is not upright; it is not inserted.
The pink nail is upright; it is inserted.
The rea nail is upright; it is not inserted.
The blue block depends orange block and purple block. The brown block depends on blue black and yellow block.
The yellow block depends on orange block and purple block.
The pink nail depends on brown bleck.
The rec nail depends orange block.
Your goal to achieve the follewing configuration:
All blocks are inserted.
Step 1 Action reorient red nail:
Execution Reasoning: The robot arm tries reorient red nail, but it cannot recrient object red nail since it is not in hand. Execution result: Unsuccessful.
Step 2 Action insert blue block:
Execution Reasoning: The robot arm tries to insert blue block, but cannot insert object blue block since it is not in hand.
Execution result: Unsuccessful.
Step 3 Action pick up red nail:
Execution Reasoning: The robot arm tries to pick up red nail. Since no invalid action is executed, the agent successfully pick up red nail. Exccution result: Successful.
Step 4 Action insert yellow block: Execution Reasoning: The robot arm tries to insert yellow low block, but it cannot insert object yellow block since it is not in hand. Execution result: Unsuccessful.
Goal reaching: Unsuccessful
26

---

Published as a conference paper at ICLR 2026
C VLMFP DETAILS
C.I DIRECT AND COT MAJOR FAILURE CASES
For all domains, one major failure case for Direct and CoT is because the failure to correctly recog- nize and locate the objects in the scenario due insufficient image recognition and spatial reasoning capability. In addition to this, some other failures result from the misunderstanding of actions. For example, in the Package, Printer, and Overcooked domain, when the agent stands in cell, it can perform actions, such as open and close, to objects in adjacent cells. But the VLMs often think the agent should stand in the same cell the object operate it. Including in-context examples is beneficial, but could not stop all failures like this.
C.2 VLMFP MAJOR FAILURE MODES AND CASES
Sim VLM perception errors and GenVLM generation errors are the major failure modes. We looked into the failure reasons of the 15 input instances for both seen and unseen appearances for all do- mains and summarized them in Table
Table 12: Error type distribution of VLMFP for input instances domains. errors/total failures for each category.
Values show
Error Type
Frozenlake
Maze
Sokoban
Package
Printer
Overcooked
S
U
S
U
S
U
S
U
S
U
S
U
Perception
0/0
0/1
0/1
1/3
2/5
5/8
0/4
0/7
0/5
0/5
0/8
5/10
Generation
0/0
1/1
1/1
2/3
3/5
3/8
4/4
7/7
5/5
5/5
8/8
5/10
From the table, we can observe that for visually busier scenes(sokoban) and scenarios with unseen objects (onions in Overcooked U), the perception error becomes common. While in other cases where the domain mechanisms are complex with various constraints, failing to generate and update correct action specifications, necessary predicates, or initializing objects within limited rounds the major failure case.
More specifically, for failure in generating correct PDDL problem files, one major failure case for all domains is that it fails to generate all states that describe directional relationships between cells. For example, 'move-dir pos-1-1 pos-1-2 right' defines 'pos-1-2 is right to pos-l-1'. Although the problem description is correct, the Gen VLM sometimes fails to include all available directional relationships, so that some actions cannot be performed. addition to this type of failure, for complex domains, there are often different types of objects. Sometimes, GenVLM fails to assign types for all objects.
For failure in generating correct Domain problem files, the major failure case is failure describe correct action preconditions and effects. For example, in the Package domain, the agent can only open the package faces. The definition for the open action is:
(:action open
:parameters (?pkg package ?pes position ?pkgpos position ?dir direction)
:precendition (and
(at ?pes) (package-at ?pkg ?pkgpos)
(package-closed ?pkg)
(facing ?dir)
(move-dir ?pos ?pkgpos ?dir)
:effect (and
(not (package-closed ) (package-open ?pkg)
27

---

Published as a conference paper at ICLR 2026
This 'facing ?dir' precondition and the '(move-dir ?pos ?pkgpos ?dir)' easy neglect. For CodePDDL, has similar failure cases as VLMFP. Without the updating process, the failures cannot be corrected.
believe that more techniques could be implemented to mitigate both perception errors and gen- eration errors. We observe during development of VLMFP that the generalization capability of improved as it sees more diverse domains, map sizes, appearances. and rules. Thus, there great potential to develop VLM base models to increasingly diverse environments. Additionally, the errors of GenVLM could decrease with more PDDI--targetd prompting and checkers, updating rounds, and stronger VLM reasoning capabilities.
generalized simulation 'world model' when it exposes stronger
There remain many promising directions, both ingineering-orienter and research-oriented, that could further strengthen the framework. We leave these explorations to future work and emphasize that VLMFP is, the best of our knowledge, the first framework generate both PDDL domain and problem files directly from visual inputs, without human feedback or environment access. We believe that establishing a reliable and generalizable visual-to-symbolic planning pipeline sub- stantial contribution to VLM--ased long-horizon planning and opens numerous avenues for future research.
C.3 VLMFP IMPLEMENTATION DETAILS AND PROMPTS
Prescreening During prescreening, GenVLM first generates the problem PDDL file based the domain description, problem description, and image observation. The descriptions include the prob- lem and domain PDDL template, which only includes the available object types, action names, and action variable names. In the prompt, a non-task-specific example included. The example a 3x3 grid world with obstacles. The prompt is:
Given a natural language descripticn of a planning problen. and an image observation of the initial scenario, your task is generate complete PDDL problem instance that is equivalent language description and image observation, contain all necessary predicates as described in the description, and is thorough and complete. Consider all predicates that are relevant and neccesary. Example:
its natural
Domain Description:
(domain_nl_wwappa
Domain PDDL Template:
''pddl
(define (domain grid)
(: (trequirements :strips)
(:predicates)
(:action move-up
:parameters (?from ?to)
:precendition ()
:effect ()
(:action move-down
:parameters (?from ?to)
:precondition ()
:effect ()
(:action move-left
:parameters (?from ?to)
:precondition :sffect ()
(:action move-right
:parameters (?from ?to)
28

---

Published as a conference paper at ICLR 2026
:precendition
:effect ()
Problem Description:
"markdown
You are tasked with manipulating an agent to reach the goal without colliding into the wa--. The position representation is (row, column) representation. For example, (pos-4-3) represents the position in fourth row and third column. The left upper corner (pos-1-1). You can perform four actions: move-up, move-down, move-left, and move- right.
Initially:
The agent is at (pos-2-1).
The goal is at (pos-3-3).
Your goal is to achieve the follewing configuration:
The agent is (pos-3-3)
...
Problem FDDL Template: 'ppdal
(define (preblem grid)
(:domain grid)
(:objects
(:init )
(:goal (and
Problem PDDL:
"'mrrkownn
(define (problem grid) (:domain grid)
(:objects
pos-1-1 position
pos-1-2 position
pos-1-3 position
pos-2-1 position
pos-2-2 position
pos-2-3 position
pcs-3-1 position
pos-3-2 position
po3-3-3 position
(:init
(at pos-2-1)
(is-goal pos-3-3)
(move-dir-left pos-1-2 pos-1-1)
(move-dir-leftt pos-1-3 pos-1-2)
(move-dir-left pos-2-2 pos-2-1)
(move-dir-leftt pos-2-3 pos-2-2)
(move-dir-left pos-3-2 pos-3-1)
(mov--di-leeft pos-3-3 pos-3-2)
(mov--di-rright pos-1-1 pos-1-2)
(move-dir-right pos-1-2 pos-1-3)
(move-dir-right pos-2-1 pos-2-2)
(move-dir-right pos-2-2 pos-2-3)
(move-dir-right pos-3-1 pos-3-2)
(move-di--right pos-3-2 pos-3-3)
(move-dir-up pos-2-1 pos-l-1)
(move-dir-up pos-2-2 pos-1-2)
(move-dir-up pos-2-3 pos-1-3)
(move-dir-up pos-3-1 pos-2-1)
29

---

Published as a conference paper at ICLR 2026
(move-dir-up pos-3-2 pos-2-2)
(move-dir-up pos-3-3 pos-2-3)
(move-dir-down pos-1-1 pos-2-1)
(move-dir-down pos-1-2 pos-2-2)
(move-dir-down pos-1-3 pos-2-3)
(move-dir-down pos-2-1 pos-3-1)
(move-dir-down pos-2-2 pos-3-2)
(move-dir-dow) pos-2-3 pos-3-3)
(:goal (at pos-3-21)
...
Domain Description:
Songge__cmaii__te [n]
Domain PDDL Template:
Problem Description: [target_problom_n]
Problem PDDL Template:
After generating the problem file, GenVLM continues to generate the domain file based on the ex- ample the grid, domain, and problem descriptions, image observation, and the generated problem file. The prompt for this is:
You are given natural language description of a planning problem the domain (target_domain _ along with one problem instance in FDDL format and an image observation of the initial setup. Your task is to generate a PDDL domain for the target domain aaarget_domiin_aae that is equivalent to its natural language description and is compatible with the provided problem instance.
Starting from PDDL domain template, you are allowed to medity the template using the following two pythor function interfaces:
''yython
add_or_updated jeoo_r_uppaate_eedccctttsss (predicates: List [str])
modi modify_action (action_name: new_preconditions: [ist(str), new_effects [ist(str])
An example of above functions applied to an example PDDL domain template is as follows:
Example Demain Description:
mmarkoown
The robot has four actions: move-up, move-down, move-left, and move-righ ght This donain models agent navigating in grid world. The goal is to reach a target location. The world consists of set discrete positions (e.g., grid cells). The position representation is (row, column) representation. For example, (pos-4-3) represents the position in fourth (pos-4-3). Between each two positions, the directional links between them design how they can move between each other. For example, the agent can move-left from pos-3-2 to pos-3-1 because pos-3-1 is at left to pos-3-2, but cannot move-left from pos-3-2 to pos-3-3.
and third column, and (pos-4-4) is right of
Example Problem PDDL: ''pdal
(define (problem maze) (:domai1 maze)
30

---

Published as a conference paper at ICLR 2026
(:objects
pcs-1-1 position
pos-1-2 position
pos-1-3 position
pes-2-1 position
pcs-2-2 position
pcs-2-3 position
pos-3-1 position
pos-3-2 position
pos-3-3 position
(:init
(at po8-2-1)
(is-goal pos-3-3)
(move-dir-left pos-1-2 pos-l-1)
(move-dir-left pos-1-3 pos-1-2)
(move-dir-left pos-2-2 pos-2-1)
(move-dir-eeft pos-2-3 pos-2-2)
(move-dir-eeft pos-3-2 pos-3-1)
(move-dir-left pos-3-3 pos-3-2)
(move-di-rright pos-1-1 pos-1-2)
(mov--dir-right pos-1-2 pos-1-3)
mmoe--ir-righh pos-2-1 pos-2-2)
(move-ai-rright pos-2-2 pos-2-3)
(move-di--right pos-3-1 pos-3-2)
(move-dir-right pos-3-2 pos-3-3)
(move-dir-up pos-2-1 pos-1-1)
[move-dir-up pos-2-2 pos-1-2)
(nove-dir-up pos-2-3 pos-l-3)
(move-dir-up pos-3-1 pos-2-1)
(nove-dir-up pos-3-2 pos-2-2)
(move-dir-up pos-3-3 pos-2-3)
(move-air-down pos-1-1 pos-2-1)
(move-dir-down pos-1-2 pos-2-2)
(move-dir-down pos-1-3 pos-2-3)
(move-dir-down pos-2-1 pos-3-1)
imove-dir-down pos-2-2 pos-3-2)
-move-dir-down pos-2-3 pos-3-3)
(:goal (at pos-3-21)
Example FDDL Template: ''pddl
(define (preblem maze)
(:domain maze) (tobjects
(:init )
(:goal (and )
Example Completion:
'pyyhhon
add or htt__uuatt__prddeeee predicates (I
(move-dir-up ?y)
(move-dir-down 7x ?y)',
(move-dir-left 7x ?y)',
(move-dir-right ?x ?
(at ?x)
(is-gcal ?x)
1)
modify actiyaactinn// ('nove-p''
" (at ?from)",
31

---

Published as a conference paper at ICLR 2026
1, "(movedirr-u ?from ?to) "
(not (at ?from) )
(not (clear ?to))" (at ?tc)",
})
modify aatlyyy_cciinn ("ivonnn"oeedoooo move-down', ((a ?irom) ", (move-dir-dowa ?from ?to)"
},
(not (at ?from) >
(at ?tc)",
1)
action foctii_ n'move-left', (at ?from) (move-dir-left ?from ?to) "
1,
" "(not ?from)
(at ?tc)", ,
})
modify action moveerright',
((t ?irem)
[move-dir-right ?from ?to)
}, (not (at ?from))
(at ?tc)",
1)
Target Domain Description: (target_aemain_nl]
Target Problem PDD1:
(target_prablem_pdd]
Now, your task is to complete the follewing PDDL template by generating necessary predicates and action preconditions and effects:
Target FDEL Template: [target_cooaann_eeppaattodddlll
You must never modify action parameters, and you are only allowed to use the follewing two function interfaces to modi fy the template.
PDDL files updating During updating. GenVLM takes in the descriptions, image observation, pre- viously generated problem and domain PDDL files, and execution inconsistency feedback. Gen- VLM is prompted to reason about which part of the files is problematic and then update the file(s). The prompt for this is:
Here are your generated PDDL problem and domain. Please reason about the issue with your generated problem file generated code for domain generation.
Problem Description:
(target_problem_n))
Domain Description: (target_Gomain_nl)
Domain PDCL Template:
32

---

Published as a conference paper at ICLR 2026
Generated Problem PDDL: [problem_pddl]
Gencrated Domain PDDL: {domain_pddl}
Execution Feedback: lexeoution_feedbcck
In your respense, please
1. reason about if you want to update your problen em file and domain file to fix the issue by answering
Q1: List all predicates. Describe how they are related to objects and actions and how they should be updated.
Q2: List all actions in domain. Describe, explain, and compare action's definition in given natural Language description with its inputs, preconditions, and effects in domain file one by one. No two actions should have same preconditions and effects. If not, please correct in domain file.
Q3: List all objects in problem.
Q4: Check the :objects section. If fixed 'typing' exists, list the type of each object and if it is EXPlICITlY written as format [object_name typel in problem file. Explicit typing means the exact string type> must appear after eacn object typing. any object is listed without a type, then the problem file is incorrect and must be modified.
not
implicit
Q5: List all init states in problem. Predicates used in demain should be initialized in problem file. And they should describe every aspect of the problem, and be enough execute actions. When defining directional links of a grid, the init states should exhaustively enumerate all adjacent positions for every applicable cell in this grid, from first row and column to last row and column, no matter it is valid novement contains obstacles or not. If not, please correct problem file.
Q6: list all goals in problem.
Q7: Based on previous answers, summarize the errors one by one, and explain which (sle(s) should be updated and which parts of them should be updated in detail.
2. update the problem file to be correct and complete according to your analysis, if needed (Your modified PDDL will be directly used, so return only complete and valid PDDL: no comments, TODOs, placeholders or omissions);
3. generate the new code for domain generation according to your analysis if needed.
If you want to modify the new domain file, you are only allowed to use the following two function interfaces to modity the template.
''python
add lrs_____updat__preeiieee (predi List [str]] modify_action (action_name: str, new_preconditions: [ist( new_effects : List[str]]
An examplc of above functions follows:
'''ythhn
add_or__u modify_action// ('move', (is-robet ?x)"], (is-robet ?y) []
(I' (is-rcbet ?x) (])
...
Please give your answer strictly in the following format:
33

---

Published as a conference paper at ICLR 2026
Problem file and Domain file update reasoning: New problem file: [fill in N/A if not needed] New domain file: [fill in N/A if not needed]
The updating loop is skipped when the EW score reaches 1.0 (abs(ew-rating 1.0) < 1e 9) and
the plan generated using current PDDL files is valid. This means, the generated PDDL files returned when they are verified by SimVLM to be able to solve the current instance and can achieve the same results as VLM for all explorations.
The iteration repeats for 4 times, and if EW score cannot achieve 1.0 or the generated plan is not valid, the generated PDDL files are returned. However, this does not necessarily imply that generated files are entirely incorrect. For example, when the problem PDDL file is incorrect but the domain PDDL is correct, a well-specified domain PDDL generalizes across instances, enabling potential successful solutions for other instances.
We collected the convergence rates for all domains, each is evaluated on problem instances, with the maximal round of 4. On average, 77.1% and 64.8% problem instances converged for seen and unseen appearances, respectively. These results show that the EW-guided refinement loop effective across diverse domains and appearance variations. We find that convergence tends to be fastest in domains with simpler transition rules (e.g.. FrozenLake, Maze) and more gradual in domains with richer multi-object interactions (e.g., Sokoban, Overcooked), which is consistent with our millure-mode analysis in Appendix C.2
Table 13: Convergence Rate (%) of VLMFP for 15 input instances on domains.
Convergence
Frozenlake
Maze
Sokoban
Package
Printer
Overcooked
Average
S
S
S
U
U
U
U
S
U
VLMFP
100.0 86.7
93.3
80.0
66.7
33.3
73.3
53.3
66.7
66.7
40.0
33.3
77.1
34

---

Published as a conference paper at ICLR 2026
D GENELIZATION TO GAME RULES
D.1 GAME RULE VARIATION DESCRIPTIONS
We designed 15 rules and collected data for these rules under the default Frozenlake appearance, with size 3-8, and different ice hole probabilities. For each rule, we collect 20k data. Here's ' the detailed rule descriptions for the new rules.
RI: Ice holes not result in failure. Instead, the agent must step on one ice hole once as precondition for game completion.
R2: Variant of R1. The agent must step two ice holes as a precondition for game completion.
R3: Ice holes function as teleports. There are two ice holes in the scenario that allow teleportation between each other.
Ice holes function as teleports. Any ice hole allows the agent to teleport back to the origin position.
R4:
R5:
the agent steps on an ice hole, must execute the same action twice actually execute it.
R6: The agent has two lives. It does fail when stepping on the first ice hole, but fails the game when stepping on the second one.
R7: Stepping on an ice hole unlocks a lucky rocket, allowing the agent to step forward two steps in the same direction.
R8: Ice is slippery. If the agent steps ice and the next cell in the same direction is also ice, the agent slips to the second ice position, continuing to slip until reaching a non-ice position.
R9: The agent can only step into an ice hole if entering from above. Stepping in from other directions is invalid.
R10: After stepping on an ice hole, the agent must always execute actions twice to actually execute them.
R11: Variant of R10. The agent must execute actions three times instead of twice.
R12: If the agent steps onto ice, it slides in that direction until hitting a wall.
R13: Variant of R12. The agent bounces back until reaching wall instead of sliding forward.
R14: Stepping on an ice hole swaps the goal and origin positions.
R15: The agent can only move up or down when on ice holes.
We test on unseen rules:
Rulel: Since ice is wet, stepping on ice hole causes the agent to step forward steps in the same direction.
Rule2: Ice holes are teleports to pos-2-2;
Rule3: If you step on an ice hole, you have execute the same actions three times to actually execute it;
Rule4: Stepping on an ice hole unlocks a lucky rocket, where you can step forward three steps in the same direction;
Rule5: If you step ice hole, you freeze and your next action is skipped.
35

---

Published as a conference paper at ICLR 2026
E GROUND TRUTH PDDL FILES EXAMPLES
Here we show ground truth PDDL Problem and Domain file examples for the Frozenlake and Pack- age domain. They represent very different sets of actions in grid world domains. In Frozenlake, the agent does not have orientations; that is, can freely move to any adjacent cell. However, in the Package domain, the agent has orientation and can only move in the direction it faces. Thus, the move action definition is different, and the turn action also required in the Package domain. In addition, the Package domain also shows some manipulation actions, which common in complex grid world domains.
Frozenlake Sample PDDL Problem File:
(define (problem FL-rand)
(:domain frozenlake)
(:objects pos-1-1 pos-1-2 pos-1-3 pos-1-4 pos-2-1 pos-2-2 pos-2-3 pos -2-4 pos-3-1 pos-3-2 pos-3-3 pos-3-4 pos-4-1 pos-4-2 pos-4-3 pos -4-4)
(:init
(at pcs-1-1)
(ice-hole pos-1-3)
(ice-hole pos-1-4)
(ice-hole pos-2-2)
(ice-hole pos-3-3)
(ice-hole pas-3-4)
(ice-hele pos-4-1)
(up sup_iiictiinn pos-2-1 pos-l-1)
(up direction pos-2-2 pos-1-2)
(up cireciinn pos-2-3 pos-l-3)
(up_cirection pos-2-4 pos-1-4)
(up_cirection pos-3-1 pos-2-1)
(up_cirection pos-3-2 pos-2-2)
(up_cirection pos-3-3 pos-2-3)
(up_direction pos-3-4 pos-2-4)
(up_direction pos-4-1 pos-3-1)
(up_cirection pos-4-2 pos-3-2)
(up_cirection pos-4-3 pos-3-3)
aup_airection
pos-4-4 pos-3-4)
(down_directioon pos-1-1 pos-2-1)
(co=n_direction pos-1-2 pos-2-2)
(cown_direction pos-1-3 pos-2-3)
(cown dcom_diirettion pos-1-4 pos-2-4)
(down_direction pos-2-1 pos-3-1)
(down
pos-2-2 pos-3-2)
(cown_dircction
pos-2-3 pos-3-3)
(cown_dircction pos-2-4 pos-3-4)
(cown_diroction pos-3-1 pos-4-1)
(cown_direction pos-3-2
pos-4-2)
(cown_direction pos-3-3
pos-4-3)
(cown_directian pos-3-4 pos-4-4)
(Left_direction pos-1-2 pos-1-1)
(Jeft_direction pos-1-3 pos-1-2)
Lleft_direction pos-1-4 pos-l-3)
(left_direction pos-2-2 pos-2-1)
(left_direction pos-2-3 pos-2-2)
(left_direction pos-2-4 pos-2-3)
(left_direction pos-3-2 pos-3-1)
dlfft_direction
pos-3-3 pas-3-2)
difttiiirection pos-3-4 pos-3-3)
(left_ddrection pos-4-2 pos-4-1)
(left_direction pos-4-3
pos-4-2)
(left_direction pos-4-4 pos-4-3)
(right_directiin pos-1-1 pos-1-2)
(right_directtoo pos-1-2 pos-l-3)
(right_directinn
pos-1-3 pos-1-4)
36

---

Published as a conference paper at ICLR 2026
Fiightddirettion pos-2-1 pos-2-2)
(right_direction pos-2-2 pes-2-3)
(right_diirettio pos-2-3 pos-2-4)
(right_direction pos-3-1 pes-3-2)
(right__direction pos-3-2 pos-3-3)
(right_ddirection pos-3-3 pos-3-4)
(right_direction pos-4-1 pos-4-2)
(tight_direction pos-4-2 pos-4-3)
tright_direction pos-4-3 pos-4-4)
(:goal (and (at pos-4-4)
Frozenlake PDDL Domain File:
(define (domain frozenlake)
(:requirements :strips)
(:precicates (at ?x) (down direetino ?from ?to) (ice-hole ?x)
(
Left seft_dirrection 7from ?to) sigghtddreetion ?irom ?to) {
direction ?from ?to))
(: (:action move-down
:parameters (?from ?to)
:precondition (and (at ?from) (down_direction ?from ?to) (not ice-hole ]from)))
:effect (and (at ?to) (net (at ?from)
(:action move-left
:parameters (?from ?to)
iprccondition (and (at ?frem) (left_direction ?from. ?to) (not
ice-hole ?
:effect (and (at ?to) (not (at {from)))
(raction move-right
:parameters (?from ?to)
:precondition (and (at ?irem) (right (right_direed direction ?from ?to) (not ice-hole ?
:effect (and (at ?to) (not (at ?from)
(:action move-up
:parameters (?from ?to)
:precondition (and (at ?frem) (up_directin ?from ?to) (not (ice- hole ?from) )
reffect (and (at ?to) (not (at ?from) )
Package Sample PDDL Problem File:
(define (problem package)
(:domain package)
(:objects
Fositions in 4x4 grid
pos-1-1 pos-1-2 pos-1-3 pos-1-4 pos-2-1 pos-2-2 pos-2-3 pos-2-4 pos -3-1 pos-3-2 pos-3-3 pos-3-4 pos-4-1 pos-4-2 pos-4-3 4-s-4-4
position
Packages
pkg-1 pkg-2 package
37

---

Published as a conference paper at ICLR 2026
; Directions
up down left right directicn
(:init
1 Agent initial position and crientation (at pos-3-3)
(facing up)
; Package locations and states (package-at pkg-1 pos-l-3) (package-closed pkg-1) (package-at pkg-2 pos-4-1) (paskage-closed pkg-2)
; Turn relations
(left-turn up left) (left-turn left down) (left-turn down right) (left-turn right up)
(right-turn up right) (right-turn right down) (right-turn down left) (right-turn left up)
Grid adjacency relationships
(move-dir pos-1-1 pos-1-2 right)
(move-dir pos-1-2 pes-1-1 left)
(move-dir pos-1-2 pos-1-3 right)
(move-dir pos-1-3 pes-1-2 left)
(move-dir pos-1-3 pos-1-4 right)
(move-dir pos-1-4 pas-1-3 left)
(move-dir pes-2-1 pos-2-2 right)
(move-dir pos-2-2 pes-2-1 left)
(move-dir pes-2-2 pos-2-3 right)
(move-dir pos-2-3 pos-2-2 left)
(move-dir pes-2-3 pos-2-4 right)
(move-dir pos-2-4 pes-2-3 left)
(move-dir pas-3-1 pos-3-2 right)
(move-dir pos-3-2 pes-3-1 left)
(move-dir pos-3-2 pos-3-3
right)
(move-dir pos-3-3 pas-3-2 left)
(move-dir pos-3-3 pos-3-4 right)
(move-dir pos-3-4 pos-3-3 left)
(move-dir pos-4-1 pos-4-2 right)
(move-dir pos-4-2 pas-4-1 left)
(move-dir pes-4-2 pos-4-3 right)
(move-dir pos-4-3 pes-4-2 left)
(move-dir pos-4-3 pos-4-4 right)
(move-dir pos-4-4 pas-4-3 left)
(move-dir pes-l-1 pos-2-1 down)
(move-dir pos-2-1 pes-l-1 up)
(move-dir pos-1-2 pos-2-2 down)
(move-dir pos-2-2 pos-1-2 up)
(move-dir pes-1-3 pos-2-3 down)
(move-dir pos-2-3 pos-1-3 up)
(move-dir pos-1-4 pos-2-4 down)
(move-dir pos-2-4 pos-1-4 up)
(move-dir pos-2-1 pos-3-1 down)
(move-dir pos-3-1 pas-2-1 up)
(move-dir pos-2-2 pos-3-2 down)
(move-dir pos-3-2 pas-2-2 up)
(move-dir pos-2-3 pos-3-3 down)
(move-dir pos-3-3 pas-2-3 up)
(move-dir pes-2-4 pos-3-4 down)
(move-dir pos-3-4 pos-2-4 up)
(move-dir pos-3-1 pos-4-1 down)
(move-dir pos-4-1 pos-3-1 up)
(move-dir pos-3-2 pos-4-2 down)
(nove-dir pos-4-2 pos-3-2 up)
(move-dir pos-3-3 pos-4-3 down)
(move-dir pos-4-3 pos-3-3 up)
(move-dir pos-3-4 4-s-4-4 down)
(move-dir pos-4-4 pos-3-4 up)
(:goal (and
(packagc-open pkg-1)
(package-open pkg-2)
Package PDDL Domain File:
38

---

Published as a conference paper at ICLR 2026
(define (domain package)
(:requirements :strips :typing)
(:types
positicn package direction
(:predicates
agent is at pesition (package-at ?pkg package ?pos position] package is at position
(at ?pos position)
(package-open ?pkg package)
package open
(package-clased ?pkg package)
package is closed agent is facing direction
(facing ?dir direction)
(left-turn ?from direction ?tc direction)
(right-turn ?from direction ?tc direction)
(move-dir ?posl position ?pes2 position ?dir direction) ; positions are move-dir in direction
(:action turn-left
:parameters (Pourrent-dir direction ?new-dir direction)
:precondition (and (facing ?current-dir)
(left-turn ?eurrent-dir ?new-dir)
:effect (and
(not (facing ?ourrent-dir))
(facing ?new-dir)
(: tacction turn-right
:parameters (Peurrent-dir direction ?new-dir direction) :precondition (and
(facing ?current-dir)
(right-turn ?current-dir 2new-dir)
:effect (and
(not (facing ?eurrent-dir))
(facing ?new-dir)
(:action move
:parameters (?from position ?to position ?dir direction) :precendition (and
(at ?from) (facing ?dir)
(move-dir ?from ?to ?dir)
:effect (and (not ?from))
(at ?tc)
(:action open :parameters (?pkg package ?pos position ?pkgpos position 7dir direction) precendition (and
(at ?pos) (package-at ?pkg ?pkgpos)
(package-closed ?pkg)
(facing ?dir)
(move-dir ?pos ?pkgpos ?dir)
39

---

Published as a conference paper at ICLR 2026
:effect (and
(not (package-closed ?pkg)) (packagc-open ?pkg)
(:action close :parameters (?pkg package ?pcs position ?pkgpos position ddir direction)
:precondition (and
(at ?pos) (package-at ?pkg ?pkgpos)
(package-open ?pkg)
(facing ?dir)
(move-dir ?pos ?pkgpos ?dir)
:effect (and
(not (package-open ?pkg))
(package- ?pkg)
40


Reviews:
[{"review_id": "HmKTj7QZF8", "reviewer": "Reviewer_TrTP", "summary": "This paper proposes a dual-VLM framework that generates and iteratively refines PDDL from visual inputs using SimVLM as a world model; the idea is simple, technically sound, and the experiments are well-designed. The approach is well-motivated and potentially generalizable.\nHowever, the  evaluation is limited to grid-worlds, leaving scalability to continuous/3D/POMDP settings unclear. Convergence is under-specified: EW score threshold, iteration limits, failure handling when the score doesn’t reach 1.0, and reproducibility under stochastic simulations. In adition, one risk is hallucination and safety—SimVLM-consistency is not correctness without an external oracle, so hallucinations can yield confidently wrong PDDL.", "questions": "**SimVLM Architecture Choices:** Why was Qwen2-VL-7B chosen specifically? Have you experimented with other VLM architectures? An ablation study comparing different base models would be valuable.\n\n**Failure Analysis:** Can you provide more detailed analysis of failure modes? When VLMFP fails, is it typically due to SimVLM perception errors, GenVLM generation errors, or convergence issues?\n\n**Model Hallucination Issue:** If SimVLM hallucinates and creates inaccurate transitions, could the dual-VLM framework potentially end up with worse generation results? Is there any way to guarantee correctness?\n\n**EW Score and Iterations:** Please see the questions outlined in the weakness section above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This work proposes a dual-VLM framework that can generate more accurate PDDLs given visual observations and raw language prompts. The idea is simple yet effective: leverage an additional VLM as a world model (SimVLM), which generates and iteratively refines PDDL files by comparing their execution with SimVLM's simulations.\nOverall, the paper makes meaningful contributions to PDDL generation from visual inputs.\nThe dual-VLM architecture is well-motivated, addressing the complementary weaknesses of spatial reasoning and PDDL generation. This idea of dual-VLM self-refinement can potentially be extended to VLM-based planning frameworks to improve planning domain knowledge generation.\nHowever, there are some concerns about (1) task generalizability, (2) convergence analysis, and (3) hallucination and safety issues.", "weaknesses": "### Limited Tasks\nThe evaluation is restricted to grid-world domains. Real-world planning often involves continuous spaces, 3D environments, and more complex dynamics. The paper doesn't discuss how the approach would scale to such scenarios. For example, SimVLM may have issues for cases involving POMDPs.\n### Hallucination Issues Without Guarantees\nThe framework fundamentally assumes SimVLM provides accurate ground truth for action simulations. However, this framework does not provide a proof or verification step against an external oracle; it guarantees syntactic validity and SimVLM-consistency, not alignment with real dynamics. When SimVLM hallucinates, the entire framework can produce incorrect PDDL files that match the hallucinated behavior rather than actual dynamics. The paper provides no mechanism to detect or correct SimVLM hallucinations, creating a critical failure mode where the dual-VLM framework could perform worse than simpler baselines. This is particularly concerning given recent work showing VLMs' propensity for confident hallucinations in spatial reasoning tasks.\n### EW Score and Iterations Unclear\nThe paper is unclear about the refinement process. Specifically: (1) The threshold for acceptable alignment is never explicitly defined (only implied to be 1.0 from figures), (2) There is no discussion of what happens when convergence fails after multiple iterations (e.g., when the EW score cannot achieve 1.0), (3) Since SimVLM uses ChatGPT-4o which may have randomness in simulation, how is reproducibility ensured? To clarify these details, it would be better to provide: explicit convergence criteria, iteration limits, failure handling procedures, and empirical analysis of convergence rates across different domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CN815k9Hip", "reviewer": "Reviewer_8x3c", "summary": "This paper introduces VLM-Guided Formal Planning (VLMFP), a novel framework for the autonomous generation of PDDL files from visual inputs.\n\nExisting VLM-based approaches in visual planning are limited in their ability to accurately generate domain PDDL and often rely on access to ground-truth domain files. \nTo overcome these limitations, VLMFP employs a dual-VLM architecture: a fine-tuned SimVLM that perceives environmental scenarios from visual inputs and simulates action outcomes, alongside a large generative VLM (GenVLM) that generates and iteratively refines PDDL files by aligning their execution with SimVLM’s simulations. \n\nEvaluated across six grid world domains, VLMFP demonstrates strong generalization capabilities to unseen instances, visual appearances, and game rules.", "questions": "My main questions regarding the evaluation are as follows:\n\n1. In Table 1, the proposed SimVLM component is not compared against any baseline methods. \n\n2. In Table 2, the CodePDDL baseline shows comparable performance to VLMFP on the Frozenlake and Maze domains but performs poorly on others. What is the underlying reason for this significant performance disparity across domains? An explanation is needed to interpret these results correctly.\n\n3. Several relevant papers mentioned in the text (Line 52-79) are not included as baselines in Table 1 or 2. Including these comparisons is crucial for a fair and thorough evaluation of VLMFP's performance against contemporary works.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed dual-VLM framework is well-justified.\nIt effectively allocates distinct reasoning roles to each model: SimVLM, a fine-tuned smaller VLM, specializes in spatial reasoning from visual inputs, while GenVLM, the larger model, leverages SimVLM’s output to reason over and generate PDDL content.\n\n2. The paper is clearly structured.\nThe introduction and Section 3.1 offer a coherent presentation of the problem formulation, related work, and the motivation behind the proposed approach.", "weaknesses": "The paper addresses a highly specific problem, which may limit the broader applicability of the proposed approach. \nThe evaluation, confined to six grid-world domains, lacks demonstration on more diverse PDDL scenarios. \nFurthermore, the requirement for in-domain training data for SimVLM in each new domain could hinder the framework's practical scalability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oQ7B1MhgSe", "reviewer": "Reviewer_puu1", "summary": "This paper investigates translating 2D navigation planning environments into formal symbolic PDDL representations. The task is meaningful because accurate translation enables invoking PDDL solvers for automated planning. The proposed approach uses an iterative refinement process: a VLM first generates draft PDDL files, then executes random actions in the environment to verify whether the outcomes align with simulation results. Experiments show that the method achieves a reasonable success rate (70% on average).\n\nOverall, the experimental results are generally sufficient, but the writing is poor and difficult to follow. The paper requires a major revision to improve clarity and readability before it can be considered for publication.", "questions": "See Weaknesses", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- This is an interesting and novel research direction, to the best of my knowledge.  \n- Based on the experimental results, the proposed method appears to significantly improve translation accuracy compared to existing approaches.", "weaknesses": "- The writing can be improved. The paper is dense and difficult to follow, focusing heavily on implementation details while lacking a clear, high-level conceptual framework. It is hard to identify the core scientific research question amid the technical descriptions. The method section is overly detailed, and the ideas could be presented more clearly by abstracting the core principles. For example, the iterative refinement process could be framed as learning a formal world model (domain files) and structured state representations (problem files). A clearer separation between methodology and implementation would significantly improve readability.\n\n- It is unclear whether dividing the process into two VLMs, SimVLM and GenVLM, is necessary. In principle, a single large VLM with self-refinement capabilities could fulfill both roles. Using a smaller model for specific subtasks makes sense from a computational efficiency perspective, but this seems more like an engineering optimization than a conceptual requirement. If the authors wish to emphasize cost-effectiveness as a contribution, this should be highlighted in the abstract and introduction, and supported with experiments comparing computation cost and accuracy between small and large VLMs.\n\n- I recommend adding a discussion on extending this framework to more complex and realistic domains, such as robotic control. A common criticism of PDDL-based approaches is their limited flexibility; this framework could address that issue. It would strengthen the paper to explicitly discuss how VLMFP might generalize beyond grid-world settings.\n\n- Some suggested related works:\n    - *Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction*\n    - *Planning with Reasoning using Vision-Language World Model*\n    - *Predicate Invention for Bilevel Planning*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PIfbfrDkdG", "reviewer": "Reviewer_rVGA", "summary": "This paper presents VLMFP, a framework that uses two Vision-Language Models (VLMs) to automatically convert visual planning problems into formal PDDL domain and problem files. SimVLM is fine-tuned for visual understanding and action simulation. GenVLM generates the PDDL files and refines them based on SimVLM's feedback. The method works by iteratively generating PDDL files, checking their consistency against SimVLM's simulations, and updating them to correct errors. Evaluated on grid-world domains, VLMFP successfully generates valid plans for unseen problem instances and visual appearances, removing the need for predefined domain files or constant access to the environment.", "questions": "1) The framework is demonstrated on 2D grid-worlds. What are the most significant technical hurdles you foresee in scaling this approach to more complex, realistic 3D environments, such as a simulated kitchen or a robot navigation task?\n\n2) The current method relies on discrete positions (e.g., pos-1-1). How could the approach be adapted to handle continuous state spaces, which are common in real-world robotics and control problems?\n\n3) The current setup assumes a full, top-down view of the entire state. How would VLMFP need to be modified to handle partially observable environments where the agent's view is limited?\n\n4) SimVLM required a massive, fine-tuned dataset (430k samples) for the grid-world domains. Is this level of domain-specific data a fundamental requirement, or do you see a path towards a more general-purpose SimVLM that could be applied to novel domains with minimal fine-tuning?\n\n5) The grid-worlds have a fixed set of object types with clear visual representations. How would the object recognition and spatial reasoning capabilities of SimVLM need to improve to handle novel, real-world objects with diverse and often ambiguous appearances?\n\n6) The iterative refinement process involving multiple VLM calls and PDDL simulations seems computationally expensive. What is the latency of the full VLMFP pipeline, and is it feasible for any real-time decision-making scenarios?\n\n7) Could the \"simulation consistency checking\" step be made more sample-efficient? Are there smarter strategies for generating the action sequences used for comparison, rather than random sampling, to identify logical flaws faster?\n\n8) The results show a sharp performance drop on the completely unseen \"freezing\" rule (Rule 5). Does this indicate a fundamental limitation in the system's ability to reason about truly novel action dynamics, as opposed to parametric variations of known rules?\n\n9) How robust is the iterative refinement process to persistent or cascading errors? If SimVLM itself makes a systematic error in simulation, could this lead GenVLM to converge on an incorrect but self-consistent PDDL domain?\n\n10) In your view, what single advancement—whether in model architecture, training data, or the core algorithm—would be the most critical for bridging the gap from these compelling grid-world results to a useful real-world application?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "It autonomously generates both PDDL domain and problem files from visual input, eliminating the need for human-predefined domains—a key bottleneck in prior work.\n\nThe separation into a specialized simulator (SimVLM) and a general-purpose generator (GenVLM) effectively leverages the strengths of different models.\n\nThe feedback loop that compares PDDL execution with VLM simulation is a reasonable mechanism for catching and correcting errors in the generated artefacts.\n\nThe framework demonstrates robust performance across unseen problem instances, visual appearances, and even some modified rules within the tested grid-world environments only.\n\nIt provides a preliminary basis for bridging the gap between high-dimensional visual perception and precise, symbolic reasoning.", "weaknesses": "The evaluation is confined to synthetic 2D grid-worlds. It remains unproven whether the approach can scale to realistic 3D environments, cluttered scenes, or tasks requiring complex physics reasoning.\n\nThe method likely relies on the discrete, structured nature of grid cells and positional predicates (e.g., pos-1-1). Translating continuous, real-world spaces into such a formalism is a major unsolved challenge.\n\nSimVLM was fine-tuned on a massive, domain-specific dataset (430k samples). Curating a similar dataset for every new real-world domain (e.g., kitchen manipulation, navigation) would be prohibitively expensive.\n\nThe iterative process of generating, executing, and comparing action sequences for refinement is computationally intensive and may not be feasible for real-time applications.\n\nWhile it handled some rule variations, performance dropped significantly on complex, novel rules (e.g., the \"freezing\" mechanic), indicating that reasoning about entirely new dynamics is still a limitation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)