InstructFlow: Adaptive Symbolic Constraint-Guided Code Generation for Long-Horizon Planning
Authors: Haotian Chi
Zeyu Feng
Yueming Lyu
Chengqi Zheng
Linbo Luo
Yew-Soon Ong
Ivor Tsang
Hechang Chen
Yi Chang
Haiyan Yin
Venue: NeurIPS 2025 poster ()
Keywords: Code generation; Task planning; Large language models
Primary area: applications

Abstract: Long-horizon planning in robotic manipulation tasks requires translating underspecified, symbolic goals into executable control programs satisfying spatial, temporal, and physical constraints. However, language model-based planners often struggle with long-horizon task decomposition, robust constraint satisfaction, and adaptive failure recovery. We introduce InstructFlow, a multi-agent framework that establishes a symbolic, feedback-driven flow of information for code generation in robotic manipulation tasks. InstructFlow employs a InstructFlow Planner to construct and traverse a hierarchical instruction graph that decomposes goals into semantically meaningful subtasks, while a Code Generator generates executable code snippets conditioned on this graph. Crucially, when execution failures occur, a Constraint Generator analyzes feedback and induces symbolic constraints, which are propagated back into the instruction graph to guide targeted code refinement without regenerating from scratch. This dynamic, graph-guided flow enables structured, interpretable, and failure-resilient planning, significantly improving task success rates and robustness across diverse manipulation benchmarks, especially in constraint-sensitive and long-horizon scenarios.

InstructFlow: Adaptive Symbolic Constraint-Guided Code Generation for Long-Horizon Planning
Haotian Chil
Zeyu Feng²
Yueming Lyu² Chengqi Zheng² Linbo Luo
Yew-Soon Ong2.
Ivor Tsang Hechang Chen Yi Chang! Haiyan Yin
School Artificial Intelligence, Jilin University, China
CCFAR and IHPC, Agency for Science, Technology and Research STAR), Singapore 'Engineriing Research Center of Knowledge-Driven Human-Machine Intelligence,
Ministry of Education, Jilin University, China
*Nanyang Technological University (NTU). Singapore, 'Xidian University, China chiht21 0eailmmmll....... jlu. edu. cn {chenhc, yichang) @jlu. edu. cn (feng_zeyu, 1yu_yueming, ivor_tsang san_haiaannaa star edu. ag zhen01440e .tt. edu. sg saysongentu.edu.. 1bluoOxidian edu. cn,
Abstract
Long-horizon planning in robotic manipulation requires translating under-specified, symbolic goals into executable control programs that satisfy spatial, temporal, and physical constraints. However, existing language model-based planners of- ten struggle with decomposing long-horizon tasks, enforcing constraints robustly, and adapting effectively to execution failures. We introduce InstructFlow, a multi-agent framework that establishes a symbolic, feedback-driven flow for code generation in robotic manipulation. InstructFlow comprises three coordinated agents: InstruetFlow Planner that constructs and traverses hierarchical in- struction graph to decompose goals into semantically grounded subtasks; a Code Generator that synthesizes executable code snippets conditioned on this graph; and a Constraint Generator that analyzes execution feedback to induce symbolic constraints when execution failures occur. These constraints are propagated up- stream to refine the instruction graph and guide localized code revision without full regeneration. This graph-guided, dynamic flow enables structured, inter- pretable, and stilure-resilien planning, yielding substantial improvements in task success rate and robustness across diverse manipulation benchmarks, particularly in constraint-sensittve and long- morioorizon scenarios. The implementation is available at https://giihub.com//hhh// Innssitcclo//
Introduction
Large language models have emerged central paradigm for robotic code generation, translating natural language instructions into exccutable control programs [4] 19 27]. Despite their versatility, they struggle with failure recovery. In a manipulation task such as placing an object into a bowl, an LLM--baed planner may detect that grasp fails due collision with stacked objects. However, it lacks the capacity to reason beyond the immediate failure and infer its structural cause. Existing approaches often resort blind retries or ad-hoe replanning, generating similar trajectories that reproduce the same error. These behaviors reveal a critical gap: LLMs can recognize execution failures but lack the structured reasoning required interpret and repair them.
task decomposition, reliable constraint enforcement, and adaptive
Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

This limitation arises from the representational nature of language itself. Natural language instructions are inherently under-specified. ambiguous, and difficult to ground in the physical world. When tasked with generating full execution plans directly, L.Mss frequently produce syntactically correct but physically infeasible code, and they rarely recover once execution errors occur. Recent methods have sought to bridge this gap by generating grounded skill sequences with continuous parameters (30 or synthesizing end-to-end executable programs [14]. Yet these approaches remain flat and reactive, lacking the hierarchical structure and feedback integration required for consistent reasoning under dynamic, constraint-rich conditions.
Recent works have begun exploring bredback-drivens planning, where language models incorporate environmental signals to improve execution reliability. This exemplified by PRoC3S [3. which introduces a two-phase pipeline that separates plan generation from constraint checking, enabling baiiure-triggerd replanning when constraint violations occur. While PRoC3S strengthens robustness through feedback, it still treats failures at the surface level, reacting to violations without interpreting their deeper structural causes. This limitation stems from language- based plans being under-specified and unstructured, thereby limiting causal reasoning and systematic plan refinement. Addressing this gap demands a framework that can induce symbolic knowledge from failure and use it to guide targeted, compositional repair rather than wholesale regeneration.
To address these challenges, we introduce InstructFlow, a modular, multi-agent framework that establishes symbolic. feedback-drin driven information flow for adaptive task planning and code genera- tion. InstructFlow operates beyond surface-level feedback: it interprets execution failures, induces symbolic constraints that capture their causal structure, and propagates this knowledge through hierarchical instruction graph that organizes task goals into composable subplans. Its constraint induction mechanism abstracts execution traces into human- human-interpretablee predicates that encode spatial, relational, and physical dependencies across tasks. By feeding these symbolic constraints back into the planning hierarchy, InstructFlow performs targeted repair instead of full regenera- tion, transforming reactive trial-and-error into deliberate, compositional reasoning and achieving interpretable, failure-resilient planning in long-horizon, constraint-sensitic robotic environments.
We highlight three key contributions of this work: modular multi- agent framework that establishes a symbolic. feedback- flow of reasoning for adaptive and interpretable code generation in robotic planning: (ii) a symbolic constraint induction mechanism that abstracts execution failures into causal, reusable predicates, enabling targeted repair rather than full regeneration; and (iii) extensive empirical validation across drawing, block stacking, and YCB packing benchmarks, demonstrating substantial gains in success rate, robustness, and recovery efficiency over strong LLM-based baselines.
2 Related Works
LLM-Based Code Generation for Robotic Manipulation Recent advances highlight the potential of LLMs as general-purpon purpose planners for robotic manipulation through code generation. CaP [14] pioneers the use of LLMs to synthesize Python-based reactive controllers, integrating perception modules and control primitives. RoboScript ( proposes a unified interface for deploying such code
across simulation and real robots, focusing deployability and modularity. L.LM (3 integrates task and motion planning with LLM-driven failure reasoning for robust code generation in dynamic environments. Instruct2Act [9] and VoxPoser [10] combine LLMs with VLMs, grounding language instructions into actionable code conditioned on perceptual inputs. RoboCodeX [19] introduces tree-structured multimodal reasoning framework, decomposing language commands into object- centric manipulation code. Recent systems like OctoPack (20) and RobotCode (13) further enhance generalization and reliability by combining LLM- LLM- generated programs with skill libraries. While these approaches demonstrate the promise of LLMs when equipped with structured APIs, affordance models, and perceptual grounding, they continue to struggle with handling execution failures and adapting plans in constraint-senssiivee or long-horizon tasks.
Symbolic Abstraction Planning Symbolic representations remain crucial for long-horizor and constraint-sensitiv manipulation. Classical TAMP systems [11] 4 7 (6 integrate symbolic task planning with motion controllers but depend on domain-specific predicates. Traditional robotics
planning relies on hard-coded symbolic world models 12). Hybrid methods bridge language and symbolic planning, such as L.L.+PP [66, which translates instructions into PDDL for optimal
symbolic planning. VisualPredicatoo [15] learns neuro-symbolic predicates from visual inputs, while

---

Code Generator
InstructFlow Flanner
Environment
Instrucion Graph:
Checker
Bubgoal
Pask Promgi
Kinematic
Subgoal
Grasp
Placement
Subgoal in
Chilged
Plerrigg rode
Reasoring rode
Symbolic Constraint Generator
Output
entiystcal
legum
Foedback
Cheell
propec and exters
the
280 occureancer: eur Shep 6, Action
pieki Violation: [Twin] Collision
gripper object_
Figure 1: Overview of the InstructFlow framework for multi-agent, symbolic, and feedback-driven code generation in robotic manipulation. The system comprises three coordinated agents: (a) InstructFlow Planner: parses the task prompt and constructs multi-level instruction graph that organizes goals into spatial and reasoning subgoals; (b) Code Generator: synthesizes executable code segments and samples parameter domains instantiate the plan; (c) Constraint Generator: analyzes execution feedback and induces symbolic constraints that refine the instruction graph for targeted code correction.
ViLaIn 2 extracts scene-level symbolic representations from vision-language models. Other works like PlanBench 28 create symbolic abstractions for plan feasibility analysis; P3IV 32 and RLang 21 focus on domain-agnostic symbolic representations aligned with LLM reasoning. Despite their use of symbolic abstractions, these approaches lack mechanisms for dynamic symbolic reasoning to abstract task-relevan information or diagnose failures for plan recovery, which is gap method explicitly addresses by integrating symbolic reasoning into the code generation and planning loop.
Feedback- -Drive Failure Recovery Recovering from execution failures remains a central chal- lenge for LL---drieen robotics, Most approaches treat LLMs or VLMs as success detectors [5] 18 29
while recent works explore feedback-driven plan repair. REFLECT [17] and AHA (6) leverage LLMs and VLMs for multi-modal failurre explanation, enabling language-guided correction. RoboRe- pair (22) and DoReMi [8] integrate LLMs and VLMs for execution misalignment detection and proactive repair. However, these methods primarily rely fine-tuning models for failure understand-
ing. To avoid fine-tuning, LLM (30]. ProgPrompt [24], and CLAIRify (25 propose failure-aware prompting and runtime verification to guide iterative plan repair. PRoC3S [3] further introduces a hybrid approach that combines .LM--generated partial programs with post-hoc constraint opti- mization. Building this foundation, our approach integrates symbolic constraint induction and graph-guided plan repair directly into the code-generation loop, achieving interpretable and adaptive failure recovery that generalizes across tasks.
Methodology
begin by formalizing the LLM-based code generation paradigm robotic manipulation. We then introduce InstructFlow. modular multi-agent framework built around a hierarchical instruction gruph and symbolic constraint induction mechanism. An overview of the system architecture provided in Figure
3.1 Overview
The problem of code generation for robotic manipulation involves translating natural language instructions into executable programs that operate reliably in robotic environments. Given task descriptions and the initial state of the environment, an LLM is instructed to generate parameterized
3

---

action plans that invoke low-level control routines to solve the task. Following a general paradigm the LLM produces two functions per task:
get plan: sequence of high-level actions conditioned on free parameters and environment state.
get_ domain: the feasible ranges for those parameters (e.g., spatial offisets, defining the search space for plan instantiation.
At the task level, the goal is generate code that completes the instructed objective without violating physical constraints. Execution feasibility is assessed by a continuous constraint satisfaction program (CCSP) module, which enforces four environment-leve checks: kinematic reachability, collision avoidance. grasp stability, and placement validity. The LLM must reason not only over a series of discrete action choices, also over continuous numerical parameters, which is a nontrivial requirement. Beyond sequencing skills, it must produce long-horizon code grounded in geometry, dynamics, and task task-specific semantics. The core challenge lies the lack of grounding from language prompts to low-level control logic that adheres to physical and dynamic constraints. When execution fails, LLMs often repeat or compound errors, exhibiting limited ability to adaptively repair code.
Our work aims to improve the failure recovery ability of LLM-based planners through structured planning and symbolic constraint-dre drivee code repair.
We propose InstructFlow, a modular multi-agent framework for symbolic, (cedback-drivel code generation in robotic manipulation. The core idea is introduce an instruction graph that hierar- chically decomposes high-level task prompts into semantically structured subgoals, and couple this representation with symbolic constraint induction mechanism for effective plan repair. Instruct- Flow modularizes the code-generan generation pipeline into three cooperative agents, each with a specialized role and local reasoning context (see Appendix A for the prompts used our three Agents):
Parses the task prompt and constructs hierarchical instruction graph capturing semantic and spatial dependencies. Each node encodes a typed subgoal grounded the robot's s skill space.
InstructFlow Planner:
Translates each subgoal into executable Python code, producing symbolic control routines along with parameter domains for sampling feasible continuous values;
Code Generator:
Constraint Generator: Monitors execution failures and induces symbolic constraints that explain the cause. These constraints guide context-specific graph and prompt revisions, enabling targeted subgoal repair without full plan regencration.
3.2 Instruction Graph Construction and Update
Instruction Graph Semanties central architectural contribution of InstructFlow is hierarchical instruction graph that enables structured task decomposition and adaptive symbolic reasoning. At each interaction round the InstructFlow Planner constructs an instruction graph G. (Vr, &g). conditioned on the task goal, initial state, and symbolic feedback. This graph acts as a typed, declarative scaffold for both task decomposition and constraint- awwrre refinement. The node is partitioned as:
V1 = Vplan Vreason:
Vplan O Vreason = 0,
with edges & C V2 x V/ capturing symbolic or temporal dependencies. Each edge (v,, v2) denotes directed flow of information, allowing parent nodes to influence the semantic context of their children.
(1) Planning nodes pllan E Vptan define grounded subgoals directly translatable into robot-executable code. Each is instantiated as a symbolic prompt:
plan (goal, state) + subgoal A E {pick, place,
These nodes form the plan's 's executable backbone and anchor structural code generation.
(2) Reasoning nodes u4 Vreason perform typed symbolic transformations that enrich planning with task- abstraction and constraint resolution. These typed modules abstract reusable domain knowledge, enabling modular plan revision: Tactect
IT, OT, where ΣT, O7, denote structured symbolic fields. Outputs propagated to downstream planning nodes, injecting symbolic knowledge
rcassnn

---

such as spatial udjacency or parameter tuning. We instantiate five core reasoning modules:
Tspuial S - Rel (Objects, Adjacency)
(spatial relation inference)
Taenity
S Rel (Objects, Density)
(local clutter estimation)
Tacenn: (G, 9) (Obect(Objects)
(target object selection)
Tpann (G, S, Φ) Order(Actions)
(task logic inference)
Tprrmm (G, 4) + RefiineePramDDmaai
(parameter range refinement)
Here, G denotes the high-level task goal, S represents the initial state, and O captures symbolic constraints induced from prior failures. These inputs are used by reasoning nodes extract task- relevant abstractions for plan refinement.
Drriven Graph Update core capability InstructFlow is ability to revise the instruction graph G. based on symbolic constraint feedback and failure diagnostics. At initialization (constrainto = 0), the graph contains only planning nodes. Upon failure (e.g., collisions, instabil- ity). the planner inserts reasoning nodes upstream of affected subgoals, dynamically composing symbolic stack tailored to the error mode:
G. Plannery.MM (g state,, constraint:-I).
This mechanism supports coarse-to- fine symbolic planning by injecting only the reasoning needed to refine or repair the faulty part the task.
InstruetFlow-Guice Guided Code Generation InstructFlow translates symbolic plans into executable code by composing structured prompts along the instruction graph Gt. Each planning node of Yplan generates a prompt:
instr (t)
Encode
((ttamm(t)) j=1
ziPlan
code (t) = LLM (instr(())
(3)
Task Prompt
Generated Code
where Encode(.) integrates the subgoal with symbolic refinements from reasoning nodes, such as spatial relations, parameter ranges, or action dependencies. (u))smm(())] denotes the number of reasoning nodes providing contextual information to yplam, including types such spatial reasoning, object selection, and other reasoning nodes as defined above.
This symbolic conditioning guides the LLM to produce context-aware and physically valid code. When failures occur, InstructFlow selectively updates the relevant subgoals and reasoning nodes impacted by the constraint violations, avoiding unnecessary recomputation of unrelated parts of the plan. By structuring prompt construction around the symbolic instruction graph, InstructFlow achieves interpretable, constraint-complia code generation, significantly enhancing sample efficiency and robustness in morizon, constraint-sensitiv tasks.
The Role of "Flow" While InstructFlow introduces multiple agents and hicrarchical instruction graph, the key distinguishing feature lies in the flow of symbolic information and feedback throughout the entire code generation loop. Unlike static prompting approaches, InstructFlow treats the prompt construction itself as a dynamic, graph-guided flow, where high-level goals, reasoning outputs, and failure-induced constraints are progressively injected into task- prompts at each planning node. This flow-centric prompt composition ensures that each code snippet is generated in a context- aware, failure-resilient, and constraint-complian manner, enabling efficient plan repair without full regeneration. The flow mechanism thus operates at two intertwined levels: Graph-level symbolic flow: From reasoning nodes to planning nodes. Prompt-level information flow: From task goal, through symbolic reasoning and feedback, into structured, adaptive prompts.
3.3 Symbolic Constraint Induction from Execution Failures
LLM-based robotic planners often lack structured mechanisms for failure recovery, relying instead on naïve re-prompts or implicit retries. We introduce a Constraint Generator that diagnoses execution failures, and abstracts them into logical constraints. These constraints serve symbolic corrections that guide graph restructuring and prompt refinement, enabling interpretable, efficient, and generalizable plan repair.
5

---

Failure Diagnosis Workflow. The symbolic constraint induction follows a four-stage reasoning workflow: (i) Failure-relevant entities retrieval, which identifies failure-relevant entities from the failure trace Ft and executed code Pr: (ii) Code-level reasoning, which instantiates involved variables and reasoning symbolic predicates that reflect the physical feasibility; (iii) Diagnostic reasoning, which compute geometric or geometric diagnostics based on predicates, such as collision proximity, path clearance, and placement stability; and (iv) Symbolic constraint induction, which abstracts diagnostic findings into declarative symbolic constraints that encapsulate the feasibility conditions violated by the current plan. This structured workflow transforms grounded execution failures into symbolic rules that guide prompt regeneration and enable interpretable, plan repair.
Physical Predicate as an Induction Basis. To enable interpretable failure diagnosis and structured symbolic constraint induction, we ground physical feasibility reasoning on set of declarative physical predicates. These predicates abstract task-specific physical interac- tions into reusable logical representations, serving as the foundation for symbolic reasoning across diverse manipulation scenarios. We categorize predicates along four functional components:
asnereeees tre collieles decected
Fallure-rolevatt retrloval
eISeCL - triee
steee blick
(i) Entities (E): Rather than pre-defining entities rigidly, Instruct- Flow dynamically abstracts task-relevant entities into functional roles based on the evolving task context and feedback, such as ?target (manipulated object), ?neighbor (potential obsta- cles), ?surface (supporting structures), and ?gripper (robot end- effector); (ii) Relations (R): Symbolic relations are flexibly instan- tiated to capture emergent spatial and semantic interactions during task execution and diagnosis, such as On (?a, ?b) for support/- contact, or Clear (?a, ?b) for proximity constraints, enabling contextual adaptation rather than relying on static domain rules; (iii) Physical Functions (F): InstructFlow leverages a set of physical diagnostics as interpretable abstractions, such as Dist (?a, ?b), SupportArea
W/LLE (223 4. 6.)
Code-leral reazcning
repplatatioon
crenites
Diagnoatic reasening
Cleaets (gripper, object_5)
Cleard (gripper, cbjact__
Symbolic constraint induction
(aobreaa()))))) (?obj), and COMDeviation (?ob)), which are dy- namically evaluated in response to execution feedback, guiding the symbolic reasoning process without hard-coded thresholds; (iv) Threshoids (B): Task-specific feasibility bounds, such as deaee for clearance margins, and Mmin for support stability ratios,, which can be tuned or inferred based on the environment state and failure modes, allowing InstructFlow to generalize beyond fixed rule specifications.
Figure 2: Failure Diagnosis
Workflow of Symbolic constraint generator.
These symbolic forms are grounded by diagnostics over physical basis, but are interpreted and manipulated as logical components of the instruction graph, which isolate the physical root causes of failure and ground them in explicit task parameters, forming the basis for symbolic abstraction.
Symbolic Constraint Induction. We formalize the symbolic constraint & as a conjunction over two complementary modalities of failure correction: relational structure and physical feasibility:
C, where C(E,RFF,B) = [Ra(En, )n.)) (,,) @ Tj)
(4)
FFF.)
Relational Construints
Physical Constraints
Here, for each relational constraint [e,,, eb,), Ca; and es, are entity instances (c.g., block, bowl) participating in the relation Rg For physical constraints, each term f, f,(O,) BT, represents a feasibillty condition, where: θ denotes the variables involved (e.g., poses, offsets), @ is comparison operator or (e.g., 2, =),, Ty E B is a sask-specific threshold (e.g., maximum allowable clearance).
This formulation allows each constraint & to capture both high-level task semantics and low-level physical requirements within a unified logical form, which the Constraint Generator can compose into logical constraints & for plan repair. For instance:
Opicck == ProximitySafe ?neighbor) A SeschCeear(ggrippe ?objeel),
Oplace == Dist(?pose, ?neighbor) 2 date A Stable0n(?objeed, ?surface)

---

Stace pfenicclatt
Heregee
Forr oh 933
blccina
aitha lapert durade
andsing diraae
gier Lew
Saccorytw sigets
(a) Drawing
(h) Arrange-Blocke Blocks
(c) Arrange- YCB
Figure 3: Illustration of tasks in our simulated environments, along with corresponding goals.
Notably, these symbolic constraints act as structured priors for graph refinement and generalize across task instances and environments, enabling not just plan repair but modular, interpretable priors that can be reused across planning episodes. By treating failure correction as symbolic program refinement, this representation integrates seamlessly into our instruction graph and enables feedback- driven, structurally grounded prompt generation (see Appendix B.2.2 for the symbolic constraints we summarized from the experimental results).
Experiments
4.1 Experimental Setup
We adopt the same environments, evaluation metrics, and protocol as PRoC3S 3 to ensure fair comparison, while extending its core planning pipeline with symbolic reasoning and constraint-guided repair. All experiments are conducted in the Ravens (31] simulation environment, using a 6-DoF URS arm with Robotiq 2F-85 gripper in tabletop workspace. Physies-based execution and constraint checking are handled via PyBullet. Simulations run on CPUs with 32GB RAM, with all baseline implementations integrated into unified evaluation framework.
Domains and Tasks. We evaluate our approach on three simulated domains, each designed to test planning with parameterized skills and physical constraints:
different aspects of
(1) Drawing:
The robot equipped with a draw __nnennn primitive that generates 2D trajectories to render geometrie and symbolic shapes a surface, while avoiding randomly placed objects. These tasks require precise parameter coordination under tight spatial constraints.
draw_line
(2) Arrange-Blocks:
The robot stacks and arranges colored blocks and bowls to form pyramids, lines, centered clusters. This domain tests stability, spatial accuracy, and planning under clutter and occlusions.
(3) Arrange-YCa YCB:
The robot manipulates complex objects from the YCB dataset (e.g., banana, meat can) perform packing and stacking. Irregular geometries introduce challenges grasping, placement feasibility, and collision avoidance.
Across all domains, generated plans are evaluated against a set of physical and geometric constraints that reflect real-world robotic limitations: (1) Kinematic constraints ensure that the robot's inverse kinematics solver produces a reachable end-effector pose, rejecting infeasible motions; (2) Collision constraints climinate plans that lead to unintended contact between the robot, environment, or other objects, allowing only expected contact such as during grasps; (3) Grasp constraints verify that the gripper properly encloses the object and maintains stability during lifting, rejecting grasps that cause slippage or collision; (4) Placement constraints require that, upon release, the object remains upright and stationary, i.e., any post-placement drift instability signals failure of physical feasibility.
Constraints.
Baselines. We compare our approach against three baselines:
7

---

(1) PRoC3S B|:
The original two-phase LLM- based planner that separates plan generation and constraint satisfaction using sampling- based solver with feedback.
(2) LLM [30] recent method which the LLM directly outputs grounded skill sequences with continuous parameters.
(3) Code-as-Policies (CaP) 14
A program synthesis-based strategy that uses LLM pro- duce complete Python programs encoding the action sequence and continuous parameters for execution.
Execution Details. Each approach is evaluated over 10 randomized seeds per simulated task. We use maximum budget of 1000 samples per trial (10000 for drawing tasks). We limit the number of feedback iterations to 5. All methods are queried via OpenAl's GPT-4o unless otherwise stated. A task considered successful if the final robot state satisfies the goal condition without violating any constraints (see Appendix B.1 for more details on experiment settings).
4.2 Benchmark Experiments
We benchmark InstructFlow against PRoC3S, LLM and CaP across three domains. As shown in Table across drawing. block arrangement, and YCB manipulation tasks, InstructFlow outperforms prior methods by 20 40% in task success rate.
Drawing
Arrange Blocks
Arrange YCB
Star
Arrow
Letters
Enclosed 50%
Pyramid 0%
Line
Packing
Unstack
Packing
Stacking
LLM
40%
40%
80%
40%
30%
0%
0%
10%
CaP
10%
0%
40%
30%
20%
20%
20%
10%%
30%
10%
PRoC3S
90%
80%
80%
60%
70%
50%
60%
30%
40%
InstructFlow (Ours)
100%
80%
100%
100%
90%
100%
90%
90%
60%
70%
Table Task success rates (%) across drawing, block arrangement, and YCB manipulation domains. Bold indicates top-performing results.
This improvement arises from InstructFlow's s ability to perform structured symbolic reasoning over task-specific failures, enabling targeted plan corrections multiple levels: (i) refining parameter domains to satisfy geometric constraints, (ii) inducing symbolic relations (e.g., adjacency, clearance) prevent repeated failure modes, and (iii) revising subgoal sequences based on environment feedback. For instance, in the Pyramid and Line tasks, baseline methods (e.g.. PRoC3S) frequently fail due improper block spacing, leading unstable stacks. InstructFFlow detects these failures and augments the instruction graph with symbolic adjacency constraints (e.g., Adjacent (?block_ i, ?block_j)), which guide the adjustment of offset ranges and enforce tighter placements, improving stability without exhaustive re-planning.
Similarly, in Packing tasks involving YCB objects with irregular geometries, InstructFlow leverages symbolic constraints over object proximity to guide precise placement corrections. When initial plans result in collision-pee pronn configurations, the system identifies violated ClearOf constraints and refines placement parameter ranges to balance object clearances and workspace boundaries. This targeted adjustment enables feasible, collision-fres placements without exhaustive resampling, a capability that flat prompt-baa based methods notably lack due to their absence of structured, feedback-drin repair mechanisms. (See Appendix B.2.1 for more experiment results on VLM)
Drawing
Arrange Blocks
Arrange YCB
Star
Arrow
Letters
Enclosed 100%
Fyramid
Line
Packing
Unstack 90%
Packing
Stacking
InstructF low
100%
80%
100%
100%
90%
60%
70%
InstruciFlow
Planner Agent
50%
90%
40%
InstructFlnnw
Constraint Agent
100%
80%
100%
80%
40%
100%
60%
60%
30%
40%
Table Ablation study results (% task success) highlighting the contributions of the InstructFlow Planner and Symbolic Constraint Generator.
Ablation results in Table 2 highhlihht the distinet roles of symbolic planning and constraint induction in InstructFlow's performance. Without the Planner, the system loses its ability to structure tasks

---

hierarchically, resulting in brittle plans and severe failures in multi-step spatial tasks (e.g., Pyramid, Packing, with up to 50% drops). Removing Constraint Induction disables feedback-dris driven repair, forcing the model into blind retries that struggle with physical feasibility, leading to 30-40% degrada- tion cluttered and precision-sensitee tasks. These results confirm that InstructFlow 's robustness stems from the synergy of symbolic task decomposition and failure-informed constraint refinement.
4.3 Robustness to Real-World Uncertainties
While our study is based on simulation, we explicitly model two key real-world uncertaintics, sensor noise and feedback ambiguity, to assess the robustness of InstructFlow under imperfect information, a common challenge in physical robotic systems.
Perceptual Noise. To simulate sensor inaccuracies. inject zero-mean Gaussian noise into object poses. As shown in Table 3 InstructFlow maintains high performance even under severe noise (a 0.02, roughly 50% of object size). For instance, success rates remain above 70% across most tasks, demonstrating strong perceptual robustness.
Pyramid
Line
Packing
Unstack
YCB-Packing
YCB-Stacking
Avg. Drop
No noise
90%
100%
90%
90%
60%
70%
0.005
90%
100%
90%
90%
60%
70%
0%
0.01
80%
100%
80%
80%
50%
60%
8.3%
0.02
70%
90%
70%
70%
40%
50%
18.3%
Table 3: Performance under varying levels of perceptual noise.
Feedback-Layer Noise. We further evaluate the system's 's resilience to feedback ambiguity by simulating imperfect feedback, including (1) incorrect object references and (2) incomplete traces with missing object IDs or causal descriptions. As summarized in Table InstructFlow sustains near- baar-baelline performance across tasks, despite corrupted or partial feedback, with average success rate drops limited to 11-16%
Pyramid
Line
Packing
Unstack
YCB-Pack
YCB-Stack
Avg. Drop
Ideal Feedback
90%
100%
90%
90%
60%
70%
Incorrect Feedback
60%
80%
70%
60%
30%
40%
16%
Incomplete Feedback
70%
90%
70%
70%
40%
50%
11%
Table 4: Performance under incorrect and incomplete feedback traces.
These results demonstrate that InstructFlow remains effective without perfect state estimation, key characteristics for real-world deployment. While real-robot experiments are an important direction for future work, these robustness evaluations offer strong empirical evidence of the system's readiness for real-world uncertainty.
4.4 Case Study
We take the Unstack task InstructFlow. The goal is to place the green block into the green bowl, but the task poses hidden challenges: the green block is often buried beneath stack, making direct access infeasible. Naive pick attempts cause collisions with blocks above, violating the constraints and leading to failure.
study to illustrate the effectiveness of
Existing methods, such as PRoC3S, can detect execution failures and make localized repairs, like inserting obstacle removal steps. However, they struggle with multi multi-layered occlusions. When the green block is buried under multiple stacked objects, PRoC3S lacks structured mechanism to reason about the cor- rect removal order. As result, often generates plans with invalid sequences actions that reintroduce collisions, ultimately failing complete the task.
Figure 4: Illustrative image the environ- ment for the Unstack.
Symbolic Constraints In the first round of code generation, InstructFlow receives the goal: place the green block into the green bowl. It constructs initial instruction graph with two planning nodes:

---

pick object_4 4 and place object_7. Correspondingly, the code attempts to pick object_4 at its current pose and place it at the target location.
Execution feedback, however, reveals repeated collisions during the pick action:
[Error Message]: "250 occurences: Step Action pick, Violation: [Twin] Collision detected between object_5 object gripper ringer" and "250 occurences: Step 0, Action pick, Violation: [Tein] Collision dotected between object_6 object gripper finger"
Given the failure feedback and generated code, the Constraint Generator localizes the root cause to the pick action on object_4 4, identifying the involved variables: the manipulated object, its pose, and the interfering objects (object_5, object_6) By analyzing the spatial configuration, InstructFlow infers that the gripper's approach vector intersects with the stacked obstacles, violating collision constraints. This reasoning leads to the generation of explicit symbolic constraints:
Ounstack ClearOf object_5)) ClearOf (gripper, object_56
These constraints distill raw collision feedback into symbolic predicates that express the essen- tial condition: the gripper must reach the target without obstruction. This abstraction transforms a low-level failure into a reusable, structured sig- nal for graph updates and targeted code repair.
geo, plasilattien: Ancerinltef.
dinar2i
plan
undapsal Pack aggen (Lar pelion Lieck)
plan Appand Jaces2 Ppice. bloces, pers potreo)
rabgual 2: Flaze sodect at a close Jocacion
Dynamic InstructFlow Graph Update. Given the symbolic constraints, InstructFlow Planner dynamically updates the instruction graph with reasoning nodes, which model the spatial and logical dependencies in the scene.
biooks LOLESRo segecca) "Naeess (%)
planaapees
inbggal Paze obfect_5 cimsr
Dxcclesss, y.eleaz,, Steret 1
irbggaa Fitt obfect (SSa blodoo
blacst dedeccal Nebees
Astszes bloses. polnz))
Specifically, the updated graph includes spatial reasoning node Tspatial that ana- lyzes vertical stacking relations between objects, inferring that object_5 and object_ 6 are stacked above object_4 via symbolic below relations. Based on this output, an object selection on reasoning node uresssn seeeection
boet? Eutssa..d8jeeci Paay
snaggaa Place caject Lato
bouil
planaapess pastioo Ppilcse",
sccend, vtttant, 4130
zetum plas
Figure 5: A code snippel illustrating how InstructFlow repairs the Unstack plan by intuitively injecting tar- geted object removal routine automatically derived from InstructFlow's structural reasoning.
identifies object_ and object_6 6 as obstacles to be removed according to the induced ClearOf reason constraints. A logic reasoning node Tlogic then determines the action sequence that satisfies these constraints, ensuring the objects are unstacked top-down. These reasoning nodes collectively refine the instruction graph by introducing new planning nodes to first move object_6 (yellow), then object_5 (teal), and finally pick object_ 4 (green), reflecting the inferred symbolic dependencies.
Code Repair. Based on the updated instruction graph, Code Generator regenerates the executable code satisfy the induced symbolic constraints. Unlike black-box retries, the code repair process is explicitly guided by InstructFlow's ''s graph structure, ensuring that prerequisite actions (e.g., obstacle re- moval) are correctly sequenced before the primary task. As shown in Fig. 5 the repaired code respects both spatial constraints (via clear placement of obstacles) and temporal dependencies (via correct unstacking order), demonstrating InstructFlow's 's ability to produce interpretable, constraint-compllant programmatic policies. We provide more case studies across different tasks in Appendix BB2.3
Conclusions
We presented InstructFlow, a symbolic and beclbback-riivee framework for robotic code generation that introduces an instruction graph to decompose tasks and enable interpretable, constraint-aware planning. By integrating structured symbolic reasoning and reusable constraint vocabulary, In- structFlow supports targeted plan repair, avoids full-plan regeneration, and significantly enhances robustness in manipulation tasks. Empirical results across challenging benchmarks validate the system's ' ability to handle fong-horizon, constraint-sensitee and sample efficiency. Looking ahead, we plan to extend InstructFlow to incorporate visual grounding and multi-modal constraint induction, enabling even richer symbolic reasoning from unstructured feedback in the physical world.
scenarios with improved success rates
10

---

Acknowledgments
This research is supported by the National Research Foundation, Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative, Career Development Fund (CDF) of the Agency for Science, Technology and Research (A "STARR (No: C233312007, No: C243512014) and the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award and, in part by the National Natural Science Foundation of China (No. U2341229, No. 624761 10); the National Key R&D Program of China (No. 2023YFF0905400, No. 2021ZD01 12500); the Key R&D Project of Jilin Province (No. 20240304200SF); the Key R&D Program of Shanxi Province, China or recommendations expressed this material are those the authors and do not reflect the views of the National Research Foundation, Singapore, and Infocomm Media Development Authority.
No: N-GG-NMLP-20240000
Any opinions, findings and conclusions
References
[1] Junting Chen, Yao Mu, Qiaojun Yu. Tianming Wei, Silang Wu, Zhecheng Yuan, Zhixuan Liang, Chao Yang, Kaipeng Zhang, Wenqi Shao, Yu Qiao, Huazhe Xu, Mingyu Ding, and Ping Luo. Roboscript: Code generation for free-form manipulation tasks across real and simulation. CoRR, abs/2402. 14623, 2024.
[2] Aidan Curtis, Xiaolin Fang, Leslie Pack Kaelbling, Tomás Oozano-Pérez, and Caelan Reed Garrett. Long-horizon manipulation of unknown objects via task and motion planning with estimated affordances. In 2022 International Conference on Robotics and Automation, ICRA, pages 1940-1946, 2022.
[3] Aidan Curtis, Nishanth Kumar, Jing Cao, Tomás Jozzano-Pérzz and Leslie Pack Kaelbling. Trust the proe3s: Solving long-horizon robotics problems with Imss and constraint satisfaction. In Conference on Robot Learning, volume 270, pages 1362-1383, 2024.
[4] Neil T. Dantam, Zachary K. Kingston, Swarat Chaudhuri, and Lydia E. Kavraki. Incremental task and motion planning: A constraint-based approach. 2016.
[5] Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision-language models as success detectors. In Conference on Lifelong Learning Agents, 22-25 August 2023, volume 232 of Proceedings of Machine Learning Research, pages 220133, 2023.
[6] Jiafei Duan, Wilbert Pumacay, Nishanth Kumar, Yi Ru Wang, Shulin Tian, Wentao Yuan, Ranjay Krishna, Dieter Fox, Ajay Mandlekar, and Yijie Guo. AHA: A vision-language-mond model for detecting and reasoning over failures in robotic manipulation. In The Thirteenth International Conference Leurning Representations, ICLR, 2025.
[7] Caelan Reed Garrett, Rohan Chitnis, Rachel M. Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tomás Lozano-Pérn Integrated task and motion planning. Annu. Rev. Control. Robotics Auton. Syst., 4:265-293, 2021.
[8] Yanjiang Guo, Yen-Jen Wang. Lihan Zha, and Jianyu Chen. Doremi: Grounding language model by detecting and recovering from plan-execution misalignment. In IEEE/RSI International Conference Intelligent Robots and Systems, IROS, 2024.
[9] Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li. Instruct2act: Mapping multi-modality instructions to robotic actions with large language model. CoRR, Abs/2305.11177 2023.
[10] Wenlong Huang, Chen Wang. Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In Conference on Robot Learning, CoRL. 2023, volume 229 of Proceedings of Machine Learning Research, pages 540-562, 2023.
[11] Leslie Pack Kaelbling and Tomás Lozano-Péree Hierarchical task and motion planning in the now. In IEFE International Conference on Robotics and Automation, ICRA, pages 1470-1477, 2011.
11

---

[12] George Konidaris. On the necessity of abstraction. Current opinion in behavioral sciences, 29:1-7, 2019.
[13] Jingyao Li, Pengguang Chen, Sitong Wu, Chuanyang Zheng, Hong Xu. and Jiaya Jia. Robocoder: Robotic learning from basic skills to general tasks with large language models. CoRR, abs/2406.03757,, 2024.
[14] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence. and Andy Zeng. Code as policies: Language model programs for embodied control. In /EEE International Conference on Robotics and Automation, ICRA, pages 9493-9500, 2023.
[15] Yichao Liang, Nishanth Kumar, Hao Tang, Adrian Weller, Joshua B. Tenenbaum, Tom Silver, João F. Henriques, and Kevin Ellis. Visualpredicator: Learning abstract world models with neuro-symbolic predicates for robot planning. The Thirteenth International Conference on Learning Representations, ICLR, 2025.
[16] Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. LLM+P: empowering large language models with optimal planning proficiency. CoRR, abs/2304.11477. 2023.
[17] Zeyi Liu, Arpit Bahety, and Shuran Song. REFLECT: summarizing robot experiences for failure explanation and correction. In Conference on Robot Learning, CoRL 2023, volume 229, pages 3468-3484, 2023.
[18] Yecheng Jason Ma. Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: towards universal visual reward and representation via value-implicit pre- training. In The Eleventh International Conference on Learning Representations, ICLR 2023.
[19] Yao Mu, Junting Chen, Qinglong Zhang, Shoufa Chen, Qiaojun Yu, Chongjian Ge, Runjian Chen, Zhixuan Liang, Mengkang Hu, Chaofan Tao, Peize Sun, Haibao Yu, Chao Yang. Wenqi Shao, Wenhai Wang. Jifeng Dai, Yu Qiao, Mingyu Ding, and Ping Luo. Robocodex: Multimodal code generation for robotic behavior synthesis. In forty-first International Conference Machine Learning, JCML, 2024.
[20] Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. In The Twelfth International Conference on Learning Representations, 2024.
[21] Rafael Sännigu Benjamin Adin Spiegel, Jennifer Wang, Roma Patel, Stefanie Tellex, and George Konidaris. Rlang: declarative language for deseribing partial world knowledge to reinforcement learning agents. In International Conference on Machine Learning, ICML, 2023.
[22] Claire Schlesinger, Arjun Guha, and Joydeep Biswas. Creating and repairing robot programs in open-world domains. CoRR, abs/2410. 18893, 2024.
[23] Keisuke Shirai, Cristian Tanaka, Kento Kawaharazuka, Kazutoshi Tanaka, Yoshitaka Ushiku, and Shinsuke Mori. Vision- language interpreter for robot task planning. IEEE International Conference on Robotics and Automation, ICRA, pages 2051-2058, 2024.
HeitrannHerrandzz, Masashi Hamaya, Atsushi Hashimoto, Shohei
[24] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In IEEE International Conference Robotics and Automation, ICRA, pages 1152-111300. 2023.
[25] Marta Skreta, Naruki Yoshikawa, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen, Kourosh Darvish, Alán Aspuru-Guzik, Florian Shkurti, and Animesh Garg. Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting. CoRR, abs/2303. 14100, 2023.
12

---

[26] Siddharth Srivastava, Eugene Fang, Lorenzo Riano, Rohan Chitnis, Stuart Russell, and Pieter Abbeel. Combined task and motion planning through an extensible faaner-indeeendee interface layer. In 2014 IEEE International Conference on Robotics and Automation, ICRA, pages 639-646, 2014.
[27] Hao Tang, Darren Key, and Kevin Ellis. Worldcoder, a model-based LLM agent: Building world models by writing code and interacting with the environment. In Advances in Neural Information Processing Systems 38: Anmual Conference on Neural Information Processing Systems 2024, NeurlPS, 2024.
[28] Karthik Valmeekam, Matthew Marquez, Alberto Olmo Hernandez, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, 2023.
[29] Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang. Gensim: Generating robotic simulation tasks via large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024.
[30] Shu Wang, Muzhi Han, Ziyuan Jiao, Zeyu Zhang, Ying Nian Wu, Song-Chun Chun Zhu, and Hangxin Liu. LIm Large language model-based task and motion planning with motion failure reasoning. In IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, pages 12086-12092, 2024.
[31] Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, and Johnny Lee. Transporter networks: Rearranging the visual world for robotic manipulation. In 4th Conference on Robot Learning, CoRL 2020, volume 155 of Proceedings of Machine Learning Research, pages 726-747, 2020.
[32] He Zhao, Isma Hadji, Nikita Dvomik, Konstantinos G. Derpanis, Richard P. Wildes, and Allan D. Jepson. p3iv: Probabilistic procedure planning from instructional videos with weak supervision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, pages 2928 2938. IEEE, 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately describe the main contributions the paper, including the proposed method and experimental results, which are consistent with the content in the main body.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the method, are discussed Section 4.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [NA]
Justification:
We don't t have assumptions and proof in this work. a complete (and correct)
proof?
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Experimental setups and details are fully documented in Section 4, ensuring reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We provide anonymous version of full runnable prompts and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ "cipps.c/puullc//guu /tt.e//uuudeeooooouuummiiiiiii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Detailed descriptions are included in Section and Appendix Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Detailed descriptions are included in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
16

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
show in tables
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detailed descriptions are included in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research fully complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The proposed method aims to improve the robustness and interpretability robotic manipulation systems, which may have positive societal impacts in terms of safer and more reliable automation. Potential risks, such misuse of autonomous repair systems in critical applications without proper safety verification, are acknowledged as future considerations.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets and code libraries used properly cited with version numbers and licenses indicated (see Section 4 and References).
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The paper describes the usage of LLMs as important and original component of the proposed method. Specifically, LLMs are integrated into the core code generation process, where they are conditioned by the dynamically updated instruction graph and symbolic reasoning modules to produce context-aware and constraint-complian robotic manipulation code. The LLM is not used for writing, editing, or formatting purposes, but serves central component of the proposed feedback-drive planning and repair framework, contributing directly to the scientific methodology and originality of the work.
Guidelines:
The answer NA
that the method development in this research does involve L.MMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

Appendix
This appendix provides detailed information about our method and experimental setup, is organized as follows:
In section A we first describe the prompting scheme used by different agents in InstructFlow, including both the shared initialization prompt and agent-specific templates for the planner, code generator, and symbolic constraint generator.
In section B.I we provide additional details on our experimental setup. including how programs instantiated and executed within the simulation pipeline. We also identify critical limitations in the original PRoC3S evaluation protocol and introduce a VLM-basedd semantic check to address them.
In section B.2.
we present extended experiment exploring the effect of visual inputs on planning, followed by a summary of symbolic constraints induced across different manipulation tasks in section B.2.2
In section B.2.3 we include several representative case studies that illustrate how In- structFlow recovers from planning failures through symbolic reasoning and instruction graph-guided code repair.
21

---

A InstructFlow Prompting Details
Here we provide details on the prompting scheme used for each agent in InstructFlow. The prompting
template consists of two parts: shared initialization template and agent-specific prompts. The complete structure is illustrated as follows:
Shared Prompt Templates:
[[tsystem.prompt)]}
[[idonain__etuu_codee
[[(skill_reeface))
[[[domain_skills))
Agent-Specific Prompt Templates:
{[(plamer__ole))}
[[[code- sereedeggeeeattttroolll] [Etoosttaant.geeerttoooooooll]]
A.1 Shared Prompt Templates
All agents share common initial prompt structure, consisting of: (a) system system_prompt, (b) domain_settin setus__codeeee__e____e_______see (c) skill_preface, (d) domain fainisssklss... This shared context is constructed following the initial prompt setup introduced in PRoC3S []. where detailed environment and skill specifications are defined. We adopt the same structure without modification, and refer readers to the original paper for full specification details.
A.2 Agent-Specific Prompt Templates
we have three primary agents: the InstructFlow Planner, which decom- poses the high-level goal into structured subgoals through instruction graph construction; the Code Generator, which translates the instruct graph into executable code snippets; and the Constraint Generator, which analyzes execution feedback to induce symbolic constraints for graph-guided code repair.
As outlined in Section 3
A.2.1 InstructFlow Planner
Instruction Graph Construction Prompt
<imputs>
[task_goal]
(initisl_stater
</inputs>
You are responsible for constructing adaptive instruction graph that serves as intermediate reasoning structure for rehotic task planning. Your to generate a sequence of base planning uodes that decompose task subgoals. These nodes form the initial executable backbone of the plan. Do include any description of the subgoal, which can represent either:
nodes this Each node should include semantic
concrete manipulation action (e.g- pick, place), a prerequisite operation g. selecting target object, removing an obstacle).
Ensure subgoals are:
logically vedered.
collectively sufficient to achieve the goal.
Node Type: Planning Node
"id": 'nl".
"type": "action".
'inpots": ("goal", "initial state"),
"output": < e'natural language subgoal'>>)
22

---

Instruction Graph Revision Prompt
<inpute?
(task_goal)
(initial_ctate)
paatt_instruction.geaaa
etyybolic_conatrane
</inputs>
Your lask is revise the instruction graph respanse to symbalic comsiruints inferred from execution failures, This process involves constructing reasoning, flow that incrementally generates refined instraction to update downstream planning nodes. You need follow these steps:
Step I:
Reasoning Node Selection
You nood to insert one more reasoning nodes upstream affected planning nodes guiding by symbolic constraints. Each reasoning node should perform symbolic transfornation relevant failure, and oulpui intermediale instruction frugment.
2: Instruction Flow Constraction Soquentially process cach reasoning node, using its output as an instruction fragmeut that incrementally updates the evolving description. These fragments directed information flow. end the reasoning node, concatenate instruction fragments to form single composite instruction that encodes the full reasoning chain and be used to update downstream planning nodes.
Step 3: Instruction Graph Update
Use the composite instruction update the planning nodes. Replace original subgoal with revised version, which incorporates both the task intent and refinements. 'hh updated subgouls of instruction graph should then be passed as the final instruction
generator.
We deline five types of reasoming nodes, each with its specilic functionality and structure as described below:
Thee spatial relation rensoning node analyzes pairwise spatial relationships between all visible objects to generate a spatial relation graph:
# Node Type: Spatial Relation Reasoning Node
"id": "n2".
"typs": "spatial_rclation relaticnnnnnnneeo rwlatiiiooccasonngggnnoooo _oodii
"input": l'initial state").
"bupptt: <'sputial_ relation, graph">)
-The object density reasoning node estimate the local spatial donsity around each object to reflect how crowded its surroundings are. Node Type: Object Density Reasoning Node "id": "n3".
"type' "ubject derestt_eeiity.aee analysis node".
"input": ["initial_ state"].
"output": <'object_ sebjject_densstt..o deensity_mupp.
The object selection rensoning node combines the goal, the outpur of the spatial perception node, the ourput of the object density analysis
the inputs which manipulated object(s) should be sclected accomplish the goal.
# Node Type: Object Selection Reasoning Node
"id": "n4".
"type": "object, selection, reasoning node".
'input": ['goal", "spatial_ relations graph", "object_ ceaitdddnntt__mmeeee "symbolic constraint"). "output:: <' <'munipulated, objjectsss)
-The plan logic Reasoning Node combines the goal, the outpot of object selection reasoning node, and the symbolic predicate as the imputs to inter the correct execution order among the manipulated targets.
Node Type: Plan Logic Reasoning Node
"id": "nS".
"type": "plan logic_renn wwwaseeinggnaaeeeeee node",
"input": ['goal", "manipulated_do objects", "output": < inccutiio_order"""
-The parameter range reasoning node combines the goal and the symbolic predicate adjusts the ranges plan parameters bused on explicit rules defined in symbolic predicates, and outputs instructions indicating whether to expand or shrink the ranges requirements.
# Node Type: Parameter Ronge Reasoning Node "id": 'n6".
"type": "parameter__ai raage haapiie__rangddddssteeeeeee
adjustment_ node". mode"".
"inputs": l"goal", 'symbolic, constraint")
"output": adjusssmnn,,metttr... dajjustmenr_instmetttoo'""
23

---

A.2.2 Code Generator
Code Generator Prompt
<inputa>
[Cask_gosl]
(imitial_state)
(aubgalls_instructionn)
</inpats>
You are ende generation agent in rootiic planning system. Your goal is to generate two things
First, generate python function named "ea_ppann that take discrete continuous inputs. No list inputs allowed and relurn the entire plan with all steps included where the parameters to the plan depend on the inputs. The plan should be generated hased the initial high-level level computation graph, which is composed of sequence of subgnals, Fach subgoal corresponns either a manipulation action or prerequisite operation.
Second, generate a python function 'gen_ domaii'' that returns set of bounds for the continuous or discrete inpor parameters. The number of bounds in the gencrated domain should exactly match the number of inputs the function excluding the state input.
The function you give should always uchieve the goal regardless of what parameters from the domain are passed as input. The plan' function therefore defines family of solutions the problem. Explain why the function will always satisfy the regardless of the input parameters. Make sure your function inputs allow for as much variability in outpur plan as possible while still achieving the goal. Your function be general as possible the function.
any corrcct
corresponds to input parameters
The main function should be numed EXACTLY pen_'plan and the domain of the function should named EXACTLY "gen clomain'. : Do not change the names. Do not create any additional classes or overwrite any existing omes. Aside from the inital inputs the 'gen_plann' function MUST NOT be type List Dict. List and Dict inputs 'gen_plan Additionally, the input to 'gen_domain" must be exactly the RavenBelif argument, even if this isn't explicilly used within luunction!
allowed.
A.2.3 Symbolic Constraint Generator
Constraint Generator Prompt
<inputs>
(task_goal)
(imitial_state)
(ealbura_readbac)
(generated_code
</inpute>
You are symbolic reasoning agent tusked with diagnosing execution failures in rohotic manipulation tasks. Your goal to induce generalizable symbolic constraints explain failure and can guide future plan currection. You lask perform following reasoning steps to generate symbolic constraint(s) for plan repair:
Step I. Failure-relevant retrieval:
Identily the exact code segment responsible for triggering failure. In failure feedback,, "Step N refers to the N-th action generated by code (zero bused Speei).Spesificlllyy it corresponds to the N appeepp( call in the code. For example: "Step I, Action: place" refers to the second action in the plan (index I).
Step 2. Code-level reasoning:
Extract variables relevant this failure, incloding:
MMnnipllated objects.
-Ohject poses and spatial relations.
-Action parameters (c.g. offsets).
-Domain ranges parameters.
Step 3. Diagnostic reasoning:
Based on the enviromment and extracted variables, analyze the geometric physical of failure. You must consider multiple possible causes, including but not limited lo:
Geometric violations (c.g.. collisions, unstable placement, path obstruction).
Temporal incensistencies (e.g.. incorrect subgoal ordering. premature actions).
Symbolic logical crrors (e.g., wrong object selection, missing preroquisite conditions).
For physical compute diagnostic metrics, incloting not limited to:
Proximity distances between relevant entities.
Existence of collision-free Stability metrics (e.g.. center of mass projection).
For symbolic or logical causes, analyze:
Whether the current action respects task-specific symbolic constraints.

---

Dependencies between subguals us delined instruction graph.
Whether preconditions for the current subgoul satisfied,
Provide concise diugnosis explaining why
occurred, explicidly stating:
The type of cause (physical, temponi, symbolic). The reasoning process leading to this conclusion.
You are not allowed to asuume the cause bused solely on failure deseription. All conclusions must be verified through concrete reasoning environment lask semantics, diagnostic metrics.
Step Symbolic Constraint Induction:
Given the diagnosis and variables, formulate symbolic constraint that abstracts the failure into a general, reusable rule. You must express the constraint using standard symbolic predicates, including but not limited to:
Spatial relations: On, ClearOf, Aligned, StableOn...
TTmpooaaooodrrin: Before, Precondition, Order..
Semantic affordances: Reachable, Occludes, Graspable, Affords...
-Physical feasibility; Proximityafe, PathClear, Placemen ForceStable...
The comstraint must express logical conditions involving task entities, their relationships, or parameter ranges, and be parameterized with thresholds bounds when applicable.
B Experiment Details and Additional Results
B.1 Experimental Setup Details
B.11 Program Instantiation and Simulation Pipeline
Our framework follows the same two-stage code generation and execution process used in PRoC3S 3 Specifically, after generating the code that defines a task plan via LLM, the resulting Python function consists of two components: get_plan() () and pee__dmaiin() ()
The get_plaat () function encodes a sequence of symbolic actions with continuous or discrete parameters, corresponding to the subgoals decomposed in the instruction graph. The get jet__ommann () function specifies the sampling bounds for each parameter from a predefined sampler. These samplers (e.g., Continuousei inuousSampler. GraspSampler) generate candidate values for plan parameters without awareness of environmental constraints such as collisions or instability.
To ground the abstract plan into an executable one, we adopt the same strategy as PRoC3S: sample n parameter instantiations from the domains defined in jet__doainnd O) and for each instantiation, evaluate the resulting plan in physics-based simulator. If the plan violates any constraints (e.g.. Kinematic, collisions, grasp, placement constraints), the simulator reports detailed constraint violation feedback. Once constraint-free plan is found in simulation, it is deployed in the real environment.
B.1.2 Fixing Protocol Limitations in PRoC3S Evaluation
While our experimental setup builds directly on the original PRoC3S 3 framework and reuses its environment, skill library, and simulation interface, we identified structural limitations in its evaluation protocol. Specifically, PRoC3S treats any execution that does not explicitly violate simulator constraints successful, regardless of whether the task goal has been semantically achieved.
Incorrect but vonstraint-free Plan Misclassified Successful For example, plan may place an object in an incorrect position, fail to form the required structure (e.g., a pyramid), or manipulate the wrong object altogether. As long collisions instability triggered in simulation, such plans are incorrectly classified as successful. Figure 6 illlstrate several cases where the task goal was clearly unmet, yet feedback generated to initiate replanning.
Fixing the Evaluation Protocol via VLM Check To address this fundamental evaluation gap, we introduce a semantic-level verification step using a languaeee model (GPT-4o). After executing each plan, we render the final scene and prompt the VLM to assess whether the natural language goal has been achieved. If not, the failure is recorded and propagated, triggering replanning cycle. Specifically, we explicitly instruct the VLM to evaluate structured visual conditions and provide clear success criteria along with positive and negative examples, enabling it to produce consistent and grounded success judgments.
25

---

(a) Pyramid: the top block does not make contact with the lower-left block to form pyramid.
(b) Unstack: an incorrectly colored block was placed into the target bowl.
(c) Packing: there is a object left outside the designated container
Figure 6: Task failures caused by incorrect plans that are accepted successful by PRoC3S, despite violating the task goal, because they do not trigger any constraint violations.
This fix does not modify the core PRoC3S planning mechanism, but augments the evaluation logic with reliable, goal-aware success signal. It enables all paselines-includie PRoC3S and InstructFlow-tte be assessed under consistent, semantically meaningful criterion. While this change may lower the reported success rates of prior methods, we consider it essential for fair and rigorous comparison, especially in tasks with under-specified goals ambiguous execution semantics.
Goal: Stack Pyramid out any three blocks
VLM-baased Success Check Prompt
<inputs>
(task_goal)
penritive_exampl_
(negative_exampl
(tinal_ecene)
</inputs>
You given goal instruction and an image showing the final scene after executing a robotic plan. Your task is to determine whether the final scene satisties this goal.
"No, the blocks are not arranged in pyramid formation"
Feedback
Success criteria (Pyramid: stack pyramid out of any three blocks): a valid pyramid consists of two blocks placed side by side as base, and a black placed on top, centered across the base blocks. The top block must visibly rest on and be supported by both base blocks. Small misalignments are acceptable as long as the overall structure clearly forms pyramid.
Answer with one of the following:
Answer:
Feedback
Answer: No
If No, provide brief explanation.
"Yes".
Figure 7: VLM--base success check two execution plans for the Pyramid task. The upper plan produces a non-standard pyramid where the top block does not contact the left base block. The VLM detects this semantic error and returns "No" with reason. The lower plan yields correct pyramid structure, and the VLM responds with "Yes". This demonstrates the importance of incorporating VLM check as a complementary layer evaluation protocol.
B.2
Additional Experiment Results and Cases
B.2.1 Additional Experiment
To assess the potential of vasion-language models (VLMs) for improving task planning, we conduct an additional experiment where our existing GPT-4o-based reasoning agent receive both the goal and an image of the initial scene, enabling multimodal visual reasoning.
26

---

This vision vision-augmentt augmented setup improves the planner's ' ability to reason under partial observability or ambiguous symbolic states. As shown in Table we observe performance gains in several visually grounded tasks, including Packing (Blocks), Unstack, and Packing (YCB), where early-stage visual cues help the system make better object selection or ordering decisions.
Drawing
Arrange Blocks
Arrange YCB
Star
Arrow
Letters
Enclosed
Pyramid 0%
Line
Packing
Unstuck 0%
Packing
Stucking
LLM" (w/ vision)
40%
40%
80%
50%
40%
30%
0%
10%
CaP (w/ Vision)
10%
0%
40%
30%
20%
20%
20%
20%
40%
10%
PRoC35 (w/ vision)
90%
80%
80%
5056
80%
60%
8058
40%
40%
InstruciFlow (w/ Vision)
100%
80%
100%
100%
90%
100%
90%
100%
60%
70%
Table 5: Task success rates (%) across drawing, block arrangement, and YCB manipulation tasks
using VLM-based visual reasoning. Bold indicates the best-performing results.
To further evaluate the planning efficiency of InstructFlow, we compare it with the baseline system PROC3S in terms of (1) average number of feedback queries required per task and (2) end-to-end wall-clock latency.
Feedback Efficiieccy. As shown in Table 6 InstructFlow significantly reduces the number of feedback-driven repair cycles. Tasks that require 2-5 full regenerations in PROC3S are typically com- pleted 1-3 symbolic repair steps in InstructFlow, resulting in a reduction of feedback interactions by approximately 37% on average.
Drawing
Arrange Blocks
Arrange YCB
Arrow
Eaclosed
Pyramid
Packing
Unstack
Packing
Stacking
PRoC3S
0.000.20
10080.20
0,6080.00
0.5081.50
L90+2225
1.2041 32
L8042.10
2.8042.22
3.108304
2,9022700
InstructFlow
0.00+0.00
10080,20
0.1040.20
0.0080.00
1.000882
0,200030
1.001222
19081182 1282
2.30+2.12
2,00=2.00
Table 6: Average number of feedback queries per task (mean # std).
Wall-clock Latency. Despite employing multi-agent planning architecture, InstructFlow does not incur higher computational cost. As shown in Table it achieves an average end-to-end latency reduction of approximately 4. 4.7%. This improvement is attributed to symbolic constraints that prune infeasible code paths and reduce retry cycles in LLM sampling.
Blocks
Amow
Facking
Saakkng
PROCES
Table 7: Walk-clock latency (in seconds, mean + std) per task.
These results show that the modular design of InstructFlow not only improves feedback efficiency but also retains low computational latency, making it scalable for complex tasks involving multiple agents and symbolic constraints.
B.2.2 Table Symbolic Constraints Discovered
Table summarizes the symbolic constraints that were automatically induced based on failure feedback across different manipulation tasks. From content perspective, highlight the following key properties:
Structural Consistency. All induced constraints conform to the symbolic constraint formulation presented in Section 3.3 Each constraint instance can be expressed either as a relational predicate R,(ea, eb) or physical threshold condition (+) @ r. This ensures that all constraints are logically composable, interpretable, and grounded in the formal symbolic space defined by ø.
Diverse Coverage of Constraint Types. The constraint set spans wide range of task-relevant constraint categories, including:
Spatial safety: ProximitySafe, ClearOf, PathClear...
Placement feasibility: PlacementFeasible. Aligned, StableOn...
Geometric parameters: Distance, Offset, CenterOfMass...
Temporal logic: Before, Order
27

---

These categories capture both physical feasibility and symbolic reasoning failure modes, supporting broad range of corrective strategies.
Notably, the table does not include constraints for the Drawing tasks. This is because the drawing tasks are comparatively simpler in structure and were typically solved by the planner in single attempt without triggering any failure-driven refinement. As result, no symbolic constraint induction process was invoked for these tasks, and they are therefore excluded from this table.
Task
Symbolic Constraints
StableOn(Yblock top,
ProximitySafe
PathClear("grippe "block):
beforreplacetbllock place((block_bottom))
Pyramid
Distance( "block bottom) E (U, 0.04
Aligned("block bottom) On(block top. 7block bottom);
(rder(Placo(block bottoml). Place(block bottom2), Place(block (oo))
Contact(block top, "block_bottonn A CloseTogether (bblock_ bottom))
CenterOfMaastblock top) F SupportArea("block bottom)
StableOn(1bloc) ?table);
Line
Al Aligunent("block)
Ali Alignment,
seaamentttolernnce tolerance C 0.01,0001]
Proximi tysafeihhySaffhhloekkk Phoundary);
wew.emeenneaaalleeeeaeeeaeaeeeee square_ regioon;
Packing (Blocks)
Withiinoounss(bloc" region_ center. 2"blcok_size);
ProximitySafeeblecte region center)
Wi WithinDDstaance("lock region center, ?max distance)
StableOn (block. Ybowl);
Proximi eerriiittySafe(grrpppep ?block);
('eripff (opripper./oobsssaaeee /obslacle);
Unstack (Blocks)
www.eeeenneessabeegrerreeoe block, green_
Inside (nreee(green, block,
Aligned(block, centerr bowl_center);
grren block, (obstacle) A offset_z 0.03
Proximi ProximitySafeele e(riierrrr Pobject);
GraspFeas sareiille(ggrrgas !object_pose);
FroximitySafen (object, table_centeri.
Packing (YCB)
Distance( ('objcct, table tablc__cener) < 0.06;
Placement Feasenleeessiibeieeeeeeeee center, threshold) A threshold = 0.06;
Graspable (2gras.. Pobject);
CollieeoonPree("rrippo,, Tobject)
OnToplobject_at a. object_b), b);
ClearOf surface);
ClearSurfacelobjet. b);
Stacking (YCB)
baacementteasilllllcbeeee. object b); Stable@n(object. a, object b):
Al LiigeeForStackinnee Graspable (oobect,, "grusp) A (olliss)onFree((djccc))
object
Table 8: Inducted constraints for each task across Arrange-Blocks and Arrange- YCB domains
B.2.3 Case study
In addition to the Unstack task discussed in section 4.4, we present several representative manip- ulation tasks as case studies to further demonstrate the effectiveness and generality of our method. For each task, we analyze the core planning difficulties, the symbolic constraints induced during execution, and how InstructFlow dynamically updates the instruction graph to recover from failures.
The goal is to stack a Pyramid out of any three blocks, which means the robot need Pyramid.
to construct pyramid-like structure by selecting any three available blocks: two as the base and one stacked on top. The key challenge lies in both object selection and precise spatial configuration. Specifically, the lateral distance between the two base blocks must be carefully chosen to ensure that the top block can be stably placed across them. Furthermore, this task is especially sensitive to execution noise: even plan passes all physical constraint checks in simulation, the same stack may collapse in the real environment due to minor perturbations such as control inaccuracy or object pose
28

---

estimation errors. PRoC3S [3] lacks feedback mechanism to detect sost-simulation failures. Once plan passes simulation, it executed directly without verifying whether the real-workd outcome satisfies the goal. Our method addresses this by introducing VL.M-based validation step: the executed scene rendered and checked against the original goal, with replanning initiated the structure is incorrect.
In the first round of code generation, InstructFlow receives the goal: "Stack a pyramid out of any three blocks." The InstructFlow Planner constructs an initial instruction graph consisting of seven planning nodes, each representing subgoal in the pyramid assembly process. Guided by this graph. the code generator produces executable plan. However, after executing the plan, VLM-base semantic validation reports a failure, indicating the following issue:
[Error Message]:
"The blocke
arranged
pyremid formation"
Given the failure feedback and generated code, the Constraint Generator localizes the issue to the code block responsible for pyramid construction, specifically the subgoal corresponding to subgoa15 in get_plan () and the offset_x parameter defined in get_domain( (). By examining the sampled parameter values, InstructFlow infers that the current range of offset_x x is too wide, causing the base blocks to be placed too far apart. As a result, the top block either falls during execution or forms a configuration that is not recognized as pyramid by the VLM. This reasoning leads to the generation of an explicit symbolic constraint:
dist Distance Distaacc((block_botttt (7block_bottom) e 0.04]
Here, 0.04 corresponds to the environment-defined block size, ensuring that the top block can span both base blocks without falling or misalignment.
Given the symbolic constraint, InstructFlow Planner dynamically updates the instruction graph by introducing reasoning nodes that refine the value range of offset__ Specifically, the updated graph incorporates a parameter range refinement node t param. whose output is an instruction to narrow the sampling bounds of offset_x.
reasson
Based on the updated instruction graph, the Code Generator regenerates the executable code to satisfy the induced symbolic constraints. As shown in Figure 5, the regenerated code modi- fies the jet_domain(( () function by narrowing the sampling range of offset_x from the original (BLOCK_STIC SIZE, BLOCK_STE SIZE 2) (0, (LOCK_SIZE). This adjustment results in tighter ar- rangement of the base blocks, enabling the top block to rest stably and form a recognizable pyramid structure. The final scene passes VLM validation, and the task successfully completed.
Stacking (Arrange-You) YCB). This task requires the agent to stack one object on top of another, selected from diverse set of YCB objects with varied shapes, sizes, and physical properties. The primary challenge lies in object selection: due to irregular geometries and asymmetric mass distributions, not object pairs are feasible for stable stacking. Therefore, selecting an appropriate pair is critical to the success of the task. A second challenge arises from the lack of feedback handling PRoC3S. Due to shape mismatch or small execution disturbances, the stacked configuration may fail in the real environment but without replanning.
In the first round of code generation, InstructFlow receives the goal: "Stack any object on any other object". The InstructFlow Planner constructs an initial instruction graph consisting of three planning nodes, corresponding selecting two objects, picking the first object, and placing on top of the second. Guided by this structure, the Code Generator generates an executable plan. However, after execution, VLM- semantic validation reports a failure, indicating that the chosen object pair did not result in valid or stable stacked configuration.
[Error Messagel: "No objects are stacked on top of each other"
Given the failure feedhack and generated code, the Constraint Generator localizes the issue to the code block responsible for selecting object pairs. Specifically, the analysis reveals that the object
29

---

der gen_plan(Lattial SavenBellet,
base flaat, fleat, offeet
plan
I Bubqeal 1: Belect three blecka for the przarid
biooks [ot] foz
roleets. 14 ob3. category blockl, block2, block3 blocks[:3]
"block":
Bubgesl
tlest
for
pyranid
plan. apanaapped (Action ""ick"y blockl pockl.pose.ppee pose.pponct)
I subgoal 3: Place the rizst bleck at the base position appeea ""pLace,,
base_y, bloext.pose.per por.pp.ooo.poot]...)1
accond plan. apppad ("pick", block2 block2.pose.. pose.potati)
Bubqeal
pysanid
19
I subgoal 5: Place the secend block next to the first block at the base position, ensuzing alignnent offset_x, base_y, block2. boekk..pose.peet www..poooc.ppeee.... (21))
plan.appcod (tction ("pacce,,
Bubqeal
top pyzamid (hation (Action ("pck", block3.posc.pes.
third
plan. appcad
25
subgoal 7. Place the thixd block top of the firat and aecond blocks to complete the pyranid append thtpeaaniiitii..coo. ("plass"", [base_x offeat_N base_y, plackt..sse.poiatte $12B))
return plan
dct gon, (initail RaveaBelict):
zeluen
"Eape y"s CoacinoousBaroler SCUNDS
BOUNDS [1] (11,1
*cttset x"1 contiseessSamierreBla. STEE, BLOCK SCEB 2), "afcoet_a"
coatisue (0, BAGCX_ BL8E1,
Figure 8: A code snippet illustrating how InstructFlow repairs the Pyramid task plan: to correct the code, InstructFlow incorporates a parameter range refinement node into the instruction graph based on the induced symbolic constraint, which guides the regenerated code to reduce the sampling range of offset x in the get domain function. This adjustment reduces the spacing between base blocks, enabling stable top placement and successful pyramid construction.
pair was selected via an environment- Di DiscreteSampler O) which randomly samples two objects without considering any factors that influence stackability. As a result, the constraint generator induces the following symbolic constraint to guide future selection:
< AlignedForstacckinn(fjfett.. object_b) PlacementFeasttbleecjeecc....p
Given the symbolic constraint, InstructFlow Planner dynamically updates the instruction graph by introducing reasoning nodes that guide the selection of a physically compatible object pair for stacking. Specifically, the updated graph includes
node to analyze relative object positions and an bacill_eeaaato__eeseeeeee
reason Tspotial renson
density to evaluate the local clutter surrounding each object. These outputs are fed into TTellect> which selects object_2 and object_4 an fcasible stacking pair based on the symbolic constraint SelectStackablePaa
resson
A node Tpaaan then determines the appropriate execution order, generating a subgoal sequence: "Pick object_2" followed by "Place object_ 2 on top of object_4."
Based on the updated instruction graph, the Code Generator regenerates the executable code to satisfy the induced symbolic constraints. As shown in Figure 6, the regenerated code eliminates the use of unconstrained object sampling in gen_domain (), replacing it with fixed object assignments (object_2 2 and object_4 determined through reasoning nodes. This ensures that the selected and PlacementFeasile
object pair adheres to the Ali AlignedForr www.gneoottaakiiii plan simplifies the domain by sampling only the grasp for the manipulated object and directly encodes the stacking intent in the plan logic. With this targeted refinement, the resulting scene passes VLM-based check, and the stacking task successfully completed.
constraints. The new
30

---

qun plan Havenbelies, ubjses, purals als, gropli Savenhhopp
gen pasect_nare:;
plan
Mobbagea to Pict the firet de selecced object
plan, p.an. appund (oct.. [sbject pundi, quspi]]]
Mobagea
poss the ascond object stack on top object pose dejeeta tebject nased ppoee
plan.append coetion
Inljest nase'
RavenPose (ensecond oefect_peee.w.
gezeernd objject_peaee zesecond pean. suock, (126)
Mubagoa is batine the cojects to be used in the plan
dbject narel tobject agple
object_paee passed fobjact_s* pokac drill
Mutbagea
object [gpple]
plan,
[sbject__padd
soubagea
sscond
drill] stack
secund object
initial.objen oojjeta fubject, nanez)
ASubocca 4: place the dieas abject (apole) on tap of the secend obfect rocwee drill) p' an, agpend (bet on itgl wost, Inegent nawel' graep, Ravenfose jessecond object object_pote.ix pore.s, yeaucand sbject_pcansy, assecond sbject poasue succk, (128)
sucaro plan
sevenolliuft:
list cinitiol obferrs............
return
*object miset" iy it cone-esemplertoojjee (d0),
"graepl SraepSaeplerr..
*objuct, pasi2" auuceaattappe (ebjectt
24
"graep?"; GrappSampeery)
cotuzn
Figure 9: A code snippet illustrating how InstructFlow repairs the Stacking task plan: to correct the code, InstructFlow updates the instruction graph based on the induced symbolic constraint. This update adjusts the subgoals from randomly selecting stacking targets to using specifically determined object pair, guiding the regenerated code to modify the corresponding logic in get _llan and remove the random sampling from get domain. This adjustment ensures the physical feasibility of the stacking operation.
31


Reviews:
[{"review_id": "IMHqvd9a5t", "reviewer": "Reviewer_xtF7", "summary": "The paper introduces InstructFlow, a modular, multi-agent framework for robotic code generation. It uses a hierarchical instruction graph and symbolic constraint reasoning for task planning. The model takes advantage of dynamically feedback from execution failures, which it integrates into symbolic constraints.", "questions": "Questions: \n\n- Would InstructFlow generalize across robotic domains beyond tabletop manipulation, e.g., mobile navigation, assembly?\n\n- How much doe sthe method depend on the underlying language model (e.g., GPT-4 vs. smaller models)?\n\n- Can the symbolic constraint vocabulary be learned or adapted online instead of being manually defined?", "limitations": "No dedicated section on this.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Combines probabilitisc and symbolic comnstraints for task planning, which I believe is novel.\n\n- Sizable improvements across multiple domains compared to strong baselines like PRoC3S and LLM3: 20–40%\n\n- Careful ablation analysis\n\n- Interpretability of the produced plans\n\n- Allows recovery from bad initial plans\n\n- Detailed description of the experimental setup, including task domains, constraints, baselines, and compute resources, which should enable reproducisbility.\n\n\nWeaknesses:\n\n- All results are based on simulations; however, nowadays in robotics, it is standard to have also real-world experiments with actual robots.\n\n- Limited kinds of tasks: table top manipulation\n\n- No Limitations section (even though some limitations are discussed in the text).\n\nMinor:\n\n- The authors refer to intuition in a couple of places; this is not a scientific method.", "was_revised": true, "final_justification": "I have read the author response. I would suggest that they add the additional results and clarification in the final version, if accepted. I am raising my score, as the authors have done a lot of extra work.", "rebuttal": "We are truly grateful for your generous evaluation and insightful feedback. We have reviewed your feedback thoroughly and respond below with clarifications and revisions made accordingly.\n\n### Weakness 1:  \n\n> “All results are based on simulations; however, nowadays in robotics, it is standard to have also **real-world experiments** with actual robots.”\n\n\n**Response**:\n\nWe agree that real-robot validation is valuable. While our study is simulation-based, we explicitly model two key real-world uncertainties: sensor noise and feedback ambiguity, to evaluate robustness:\n\n1. **Perceptual noise:** We inject Gaussian perturbations into object poses before symbolic abstraction to simulate sensor inaccuracies. As shown in **Table 1**, InstructFlow maintains strong performance even under severe noise (σ = 0.02, 50% of object size), with success rates remaining above 70% on most tasks.\n\n\n|    Table 1   (Reviewer-xtF7)   | Pyramid | Line | Packing | Unstack | YCB-Packing | YCB-Stacking | Averge Drop    |\n| --------------------- | ------- |:----:| ------- | ------- | ------- | -------- | --- |\n| InstructFlow          | 90%     | 100% | 90%     | 90%     | 60%     | 70%      |   -  |\n| InstructFlow $\\sigma =$ 0.005  | 90%     | 100% | 90%     | 90%     | 60&     | 70%      |   0%  |\n| InstructFlow $\\sigma =$ 0.01   | 80%     | 100% | 80%     | 80%     | 50%     | 60%      |  8.33%   |\n| InstructFlow $\\sigma =$ 0.02   | 70%     | 90%  | 70%     | 70%     | 40%     | 50%      |  18.33%   |\n\n2. **Feedback-layer noise:** We evaluate robustness under incorrect and incomplete failure traces, simulating real-world ambiguity.  (1) incorrect feedbacks with wrong object references, and (2) incomplete feedbacks with missing object IDs or causes. Across 10 tasks (Table 2 below), InstructFlow consistently achieves near-baseline performance despite missing or wrong feedback information.\n\n| Table 2   (Reviewer-xtF7)    | Star | Arrow | Letters | Enclosed | Pyramid | Line | Packing | Unstack | YCB-Packing | YCB-Stacking | Average Drop    |\n| -------------------------------- | ---- |:-----:| ------- | -------- | ------- | ---- | ------- | ------- | ------- | -------- | --- |\n| InstructFlow                     | 100% |  80%  | 100%    | 100%     | 90%     | 100% | 90%     | 90%     | 60%     | 70%      |  -   |\n| InstructFlow w/incorrect feedback  | 100% |  80%  | 100%    | 100%     | 60%     | 80%  | 70%     | 60%     | 30%     | 40%      |   16%  |\n| InstructFlow w/incomplete feedback | 100% |  80%  | 100%    | 100%     | 70%     | 90%  | 70%     | 70%     | 40%     | 50%      |   11%  |\n\nThese results show InstructFlow operates reliably without perfect state or supervision，a key requirement for real-world deployment. We view hardware validation as important future work and believe these robustness results provide strong evidence of real-world readiness.\n\n---\n\n### Weakness 2: \n\n> “Limited kinds of tasks: **table top manipulation.** ”\n\n### Question 1:\n\n> “Would InstructFlow **generalize across robotic domains beyond tabletop manipulation**, e.g., mobile navigation, assembly?”\n\n**Response**:\n\nWe appreciate the reviewer’s question on generality. While our experiments focus on tabletop manipulation for benchmarking consistency (e.g., PRoC3S), **InstructFlow is domain-agnostic by design**.\n\nInstructFlow operates over **textual instructions** and **symbolic predicates**, enabling generalization beyond tabletop setups. Its key components, symbolic instruction graph planning, modular node composition, and failure-driven constraint repair, reason over abstract subgoals without relying on domain-specific assumptions (e.g., fixed workplace or rigid objects). \n\n1. In **mobile navigation**, pfailures like deviation or blocked paths naturally yield predicates such as `PathBlocked` or `DeviationTooLarge`, enabling symbolic route repair.\n2. In **assembly domains**, misalignment or insertion failure can trigger constraints like `AlignedForInsertion` or `GripTorqueInsufficient`, driving targeted graph refinement.\n\nThese examples illustrate how InstructFlow’s **planning-repair cycle generalizes to other robotic domains**, and we view broader deployment (e.g., mobile or deformable tasks) as promising future work.\n\n---\n\n### Weakness 3: \n\n> “No **Limitations section** (even though some limitations are discussed in the text).”\n> \n\n**Response**:\n\nThank you for pointing this out. While we discussed key limitations throughout the paper, we agree that a dedicated section would improve clarity.\n- As noted in **`Lines 365–367`**, InstructFlow currently assumes structured symbolic inputs and limited sensory grounding. We are actively exploring multimodal extensions with visual grounding and constraint induction from unstructured feedback.\n- As discussed in **`Appendix B.2.3`**, performance degrades in scenarios with **irregular object geometries** and **tight spatial coupling** (e.g., Packing-YCB, Stacking-YCB), due to the increased difficulty of reliable constraint induction.\n\nWe will add a dedicated Limitations section in the final version to consolidate these observations.\n\n---\n\n### Question 2:\n\n“How much does the method depend on the underlying language model (e.g., **GPT-4 vs. smaller models**)?”\n\n\n**Response**:\n\nThank you for raising this point. We conducted additional ablation experiments usign **GPT-3.5-turbo** in place of GPT-4o across all and individual modules.\n\n\n| Table 3   (Reviewer-xtF7)    | Star | Arrow | Letters | Enclosed | Pyramid | Line | Packing | Unstack | YCB-Packing | YCB-Stacking |  Average Drop   |\n| ---------------------------------------- | ---- |:-----:| ------- | -------- | ------- | ---- | ------- | ------- | ------- | -------- | --- |\n| InstructFlow w/**GPT-4o**                | 100% |  80%  | 100%    | 100%     | 90%     | 100% | 90%     | 90%     | 60%     | 70%      | -  |\n| InstructFlow w/**GPT-3.5-turbo**         | 90%  |  70%  | 90%     | 100%     | 60%     | 80%  | 70%     | 60%     | 40%     | 50%      |   17%    |\n| Code generator w/**GPT-3.5-turbo**       | 90%  |  70%  | 100%    | 100%     | 70%     | 80%  | 70%     | 60%     | 50%     | 60%      |    13%   |\n| Constraint generator w/**GPT-3.5-turbo** | 90% |  80%  | 100%    | 100%     | 70%     | 90% | 90%     | 70%     | 50%     | 60%      |   8%  |     |\n| Planner w/**GPT-3.5-turbo**              | 90% |  80%  | 100%    | 100%     | 80%     | 100% | 90%     | 80%     | 60%     | 60%      |   4%    |\n\n1. **Code generator is model-sensitive**: Substituting GPT-4o with GPT-3.5 in this module results in the largest performance drop (~**13%**), highlighting its reliance on strong LLM reasoning for reliable code synthesis, especially in long-horizon, constraint-sensitive tasks.\n2. **Planner and Constraint Generator are robust:** These modules maintain competitive performance with GPT-3.5 (**4–8% drop**), demonstrating that symbolic abstraction and structured reasoning mitigate dependence on large-scale models.\n\nThese findings suggest that InstructFlow’s **core effectiveness stems from its agentic design**, not LLM size. It remains effective with smaller models in key components, making it practical for resource-limited deployments.\n\n\n---\n\n### Question 3:\n\n> “Can the **symbolic constraint vocabulary be learned or adapted online** instead of being manually defined?”\n> \n\n**Response**:\n\nThe constraint vocabulary in InstructFlow is **not manually defined nor fixed**. Instead, **symbolic predicates are dynamically induced online** from failure traces using spatial analysis, physical diagnostics, and semantic reasoning (**`Section 3.2`**).\n\nFormally (`Lines 266`, **`Eq. 4`**), $\\mathcal{C}(\\mathcal{E}, \\mathcal{R}, \\mathcal{F}, \\mathcal{B}) = \\{ R_i(e_{a_i}, e_{b_i}) \\} \\cup \\{ f_j(\\Theta_j) \\oplus \\tau_j \\}$, the Constraint Generator constructs task-specific constraints $\\mathcal{C}$ from:\n1. **Relational constraints** $R_i(e_{a_i}, e_{b_i})$ over entity pairs (e.g., reachability, proximity)\n2. **Physical constraints** $f_j(\\Theta_j) \\oplus \\tau_j$ capturing measurable task variables (e.g., alignment thresholds, support stability).\n\nThese predicates are constructed per execution context without reference to any pre-defined library. As illustrated in `Appendix B.2.2 (Lines 611-612, Table 4)`, InstructFlow generates diverse symbolic constraints across tasks, supporting adaptability and generalization. \n\nWe view learning a more generalized or transferable constraint space across tasks as an exciting direction for future work.\n\n---\n\nWe sincerely thank you for your encouraging evaluation and thoughtful comments. Your recognition of our work’s contributions is highly motivating, and your suggestions have helped us further strengthen the clarity and rigor of the manuscript.\n\n---"}, {"review_id": "YzwcBGLCvE", "reviewer": "Reviewer_MeGT", "summary": "This paper presents InstructFlow, an LLM-based system that combines symbolic planning and constraint-guided code repair for long-horizon robotic manipulation. The central idea is to represent tasks via a hierarchical instruction graph, which is refined using symbolic feedback when execution failures occur. The system shows improved success rates on simulation benchmarks compared to PRoC3S, CaP, and LLM^3.", "questions": "What is the concrete benefit of using an instruction graph over simply sequencing high-level actions with reactive refinement?\nCan you provide compelling examples where the graph becomes non-linear (e.g., has branches, merges, loops), and this structure is essential for solving the task?\nWould a flat plan with dynamic constraint-aware sampling perform similarly, especially if paired with a constraint checker?\nIf you are using the same test suites in PRoC3S, why are the reported numbers different from the original paper? If the variance of the performance is high, maybe you need more trials?\nWhy is the generated constraint ``reusable’’? (line 331) The constraint is specific to a particular task setting/block positioning, why would it be reusable?\nHow would the proposed system generalize this to real-world tasks that cannot provide systematic feedback in textual format?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe separation between planner, code generator, and constraint agent is conceptually clean and enables modular plan repair.\nBy using declarative constraints and spatial relations, the method enables interpretable and targeted adjustments.\nThe system outperforms baselines across several task domains and includes ablation studies to isolate component contributions.\n\nWeaknesses:\nThe rationale behind structuring tasks as a symbolic instruction graph is not well-justified. It’s unclear why this representation is fundamentally superior to flat sequence planning or dynamic reactive strategies. The benefit seems incremental.\nThe paper primarily demonstrates simple, linear subgoal sequences. There’s no evidence that the instruction graph needs to go beyond a list of ordered subtasks. Are there any realistic scenarios where the graph structure becomes nonlinear (e.g., with branching, loops, or dependencies)?\nThe method relies on accurate symbolic representations of the scene and object poses. It is unclear how it would generalize to perceptual uncertainty or sensor noise, common in real-world settings. \nAll experiments are in simulation. No attempt is made to show robustness in hardware or even perception-grounded simulation with visual inputs.\nWhile symbolic constraint induction is novel, the system appears to rely on a hand-designed set of predicates, limiting generality and scalability.", "was_revised": true, "final_justification": "The proposed method is technically sound and demonstrates better performance than the SoTA prior work on benchmark tasks. However, the type of feedback required is unrealistic for real applications on physical platforms.", "rebuttal": "We sincerely thank you for your careful review and thoughtful understanding of our work. Below, we provide detailed responses to your comments and describe the corresponding revisions made.\n\n### Weakness 1:\n\n> *“The **rationale behind structuring tasks as a symbolic instruction graph** is not well-justified. It’s unclear why this representation is **fundamentally superior to flat sequence planning** or dynamic reactive strategies. The benefit seems incremental.”*\n\n### Question 1:\n\n> *“What is the concrete **benefit of using an instruction graph** over simply sequencing high-level actions with **reactive refinement**?”*\n\n**Response**:\n\nUnlike flat or reactive planners, **our symbolic instruction graph supports structured reasoning and localized repair, which are crucial for *long-horizon, constraint-sensitive* tasks.**\n\n1. **Flat plans lack structural insight:** Flat/reactive methods re-sample or regenerate plans without understanding why failures occur. InstructFlow induces symbolic constraints and injects reasoning nodes (e.g., spatial analysis, object selection) to refine only the affected subgoals, enabling efficient, interpretable correction. **As formalized in `Eq. (3)`,** $\\text{instr}^{(t)} = \\text{Encode}(\\{ v^{\\text{reason}}\\_{\\mathcal{T}\\_j} \\}_{j=1}^{|\\mathcal{V}\\_{\\text{reason}}^{(t)}|},\\ v\\_{\\text{plan}}^{(t)})$, and $\\text{code}^{(t)} = \\text{LLM}(\\text{instr}^{(t)})$ , each code generation prompt is composed by integrating the outputs of upstream reasoning nodes in the instruction graph. This structured composition enables rich, context-aware prompting that flat sequences cannot support, leading to more grounded and accurate code synthesis.\n\n2. **The benefit is significant:** As shown in `Table 2 (lines 299–300)`, removing the instructflow planner leads to 30–50% drops in complex tasks like Pyramid and Packing, indicating fundamentally different planning behavior.\n\n---\n\n### Weakness 2: \n\n> *“The paper primarily demonstrates simple, linear subgoal sequences. There’s no evidence that the instruction graph needs to go beyond a list of ordered subtasks.”*\n\n\n### Question 2: \n\n> *“Can you provide compelling examples where the **graph becomes non-linear** (e.g., has branches, merges, loops), and this structure is essential for solving the task?”*\n\n**Response**:\n\nWe agree that the final execution plan is often sequential, but the **instruction graph is not an execution trace, it is a symbolic reasoning graph that encodes how subgoals are derived, refined, and repaired.**\n\nReasoning nodes often branch to multiple subgoals. When failures occur, we inject new reasoning nodes that modify only the affected subgraph, enabling structural repair, which flat lists cannot express.\n\n---\n\n### Weakness 3: \n\n> *“It is unclear **how it would generalize to perceptual uncertainty or sensor noise**, common in real-world settings.”*\n> \n\n**Response**:\n\nTo address this concern, we added a controlled experiment simulating **perceptual uncertainty**. We **inject Gaussian perturbations into object poses** before symbolic abstraction. We test InstructFlow on Arrange and Arrange-YCB tasks across noise levels $σ\\in\\{0.005, 0.01, 0.02\\}$, given that the average block_size is 0.04 (i.e., σ = 0.02 simulates severe, 50% object size noise). This setting reflects real-world errors and allows us to investigate robustness under increasing uncertainty. \n\n|  Table 1 (Reviewer-MeGT) | Pyramid | Line | Packing | Unstack | YCB-Packing | YCB-Stacking | Averge Drop    |\n| --------------------- | ------- |:----:| ------- | ------- | ------- | -------- | --- |\n| InstructFlow          | 90%     | 100% | 90%     | 90%     | 60%     | 70%      |   -  |\n| InstructFlow w/0.005  | 90%     | 100% | 90%     | 90%     | 60&     | 70%      |   0%  |\n| InstructFlow w/0.01   | 80%     | 100% | 80%     | 80%     | 50%     | 60%      |  8.33%   |\n| InstructFlow w/0.02   | 70%     | 90%  | 70%     | 70%     | 40%     | 50%      |  18.33%   |\n\n---\n\n\n### Weakness 4: \n\n> *“No attempt is made to show **robustness in hardware** or even **perception-grounded simulation with visual inputs**.”*\n> \n\n**Response**:\n\nIn robotic manipulation, perception primarily serves to estimate object-level states (e.g., poses, identities), which downstream planners operate on. Our simulation provides these directly, allowing us to focus on planning.\n\nThat said, we include a perception-grounded experiment in **`Appendix B.2.1 (Lines 574-575, Table 3)`**, where symbolic inputs are derived from visual observations via a VLM. This confirms that InstructFlow can operate with visual grounding and improves performance.\n\nFurther, as noted in response to Weakness 3, we **simulate noisy perception** via pose perturbations (Table 1), and InstructFlow remains robust, supporting its potential for real-world deployment.\n\n---\n\n### Weakness 5: \n\n> *“While symbolic constraint induction is novel, the system appears to **rely on a hand-designed set of predicates**, limiting generality and scalability.”*\n> \n\n**Response**:\n\nInstructFlow does not rely on a fixed or hand-designed predicate set. **Instead, symbolic constraints are dynamically induced from execution feedback via spatial analysis, physical diagnostics, and semantic reasoning.**\n\nAs detailed in **`Section 3.2`**, the Constraint Generator constructs symbolic predicates directly from failure contex, without referencing a predefined library. This process is formalized in Equation (4): $\\mathcal{C}(\\mathcal{E}, \\mathcal{R}, \\mathcal{F}, \\mathcal{B}) = \\{ R_i(e_{a_i}, e_{b_i}) \\} \\cup \\{ f_j(\\Theta_j) \\oplus \\tau_j \\}$\n\nHere, the constraint set $\\mathcal{C}$ is constructed from:\n\n1. **Relational constraints** $R_i(e_{a_i}, e_{b_i})$, which capture spatial and logical relations between entities ;\n2. **Physical constraints** $f_j(\\Theta_j) \\oplus \\tau_j$, which encode quantitative properties like distance, stability, or alignment thresholds.\n\n**`Appendix B.2.2`** `(Lines 611-612, Table 4)` further illustrates  diverse predicates automatically generated across tasks.\n\n---\n\n### Question 3: \n\n> *“Would a **flat plan with dynamic constraint-aware sampling** perform similarly, especially if paired with a constraint checker?”*\n> \n\n**Response**:\n\nWhile flat plans with constraint-aware sampling (e.g., PRoC3S) improve robustness over naïve outputs, **they lack the ability to explain failures or perform targeted plan repair.** InstructFlow introduces two key enhancements:\n1. **Symbolic diagnosis:** Failures trigger constraint induction (e.g., `ClearOf, Aligned`) to identify root causes.\n2. **Graph-guided repair:** Only affected subgoals are updated via reasoning nodes, enabling localized and interpretable refinement.\n\nAs shown in `abaltion study(lines 299–300, Table 2)`, removing either module leads to **30–50%** drops in success across complex tasks, confirming that symbolic abstraction adds essential value beyond flat planning.\n\n---\n\n### Question 4: \n\n\n> *“If you are using the same test suites in PRoC3S, why are the **reported numbers different from the original paper**?”*\n> \n\n**Response**:\n\nWhile we use the same environments and codebase as PRoC3S, its **original evaluation protocol** considers an execution successful if no simulator-level constraint is violated, **even when the task goal is not actually achieved** (e.g., misaligned stacks or wrong object placements). \n\nThis issue is documented in `Appendix B.1.2 Fixing Protocol Limitations in PRoC3S Evaluation`, where we clearly provide example cases when PRoC3S consideres success (pass constraints) but actually fail (task goal) in `Figure 6 (Lines 560-561)`. **Same issue has been discussed in prior papers such as AHA [2]**.\n\nTo address this gap, **we introduce a VLM-based semantic check that verifies whether the final scene satisfies the natural language goal.** `Figure 7 (Lines 565-566)` showcase our VLM-based success detector implementation. This evaluation protocol applies uniformly to all methods.\n\nAs a result, **our reported success rates may be lower than PRoC3S's original numbers, not due to task changes, but due to stricter, goal-consistent evaluation for fairer comparison.**\n\n[1] Trust the proc3s: Solving long-horizon robotics problems with llms and constraint satisfaction. (CoRL'2024).\n\n[2] AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation (ICLR'2025).\n\n\n---\n\n### Question 5:\n\n> *“Why is the **generated constraint reusable**? The constraint is specific to a particular task setting/block positioning.”*\n\n**Response**:\n\nBy “reusable,” we refer to the **symbolic form** of constraints (e.g., `ClearOf(gripper, ?object`), `Below(?a, ?b)`), which capture **task-agnostic physical relations** rather than scene-specific instances.\n\nAs shown in `Appendix Table 4`, predicates like `ClearOf`, `PlacementFeasible`, and `Aligned` recur across tasks such as Unstack, Packing, Stacking, and Pyramid, reflecting shared structural demands like collision avoidance, stable placement, and alignment.\n\n---\n\n\n### Question 6:\n\n> *“How would the proposed system generalize this to **real-world tasks that cannot provide systematic feedback in textual** format?”*\n> \n\n**Response**:\n\nOur method does not rely on text format per se, but on identifying task-level violations through perceptual or geometric cues. The Constraint Generator uses available feedback to locate relevant code segments, then performs semantic reasoning over object states and task goals to infer symbolic constraints.\n\nAs described in  `Appendix B.1.2 `, our evaluation already incorporates VLM-based semantic verification (GPT-4o) that checks goal satisfaction from images. This demonstrates that our system **can operate on unstructed perceptual feedback**.\n\n---\n\nThank you again for your detailed and constructive feedback. We greatly appreciate the opportunity to address these points and improve the clarity and robustness of our work.\n\n---"}, {"review_id": "fuQQuJgVCu", "reviewer": "Reviewer_fD4v", "summary": "The paper addresses the brittleness of large-language-model planners on long-horizon robotic manipulation tasks, where naive code generation often produces physically infeasible plans and, upon execution failures, resorts to blind retries without diagnosing root causes. To overcome this, the authors propose InstructFlow, a multi-agent framework that constructs a hierarchical instruction graph to decompose high-level goals into semantically typed subgoals, embeds a Constraint Generator that analyzes failure traces (e.g., collisions) to induce precise symbolic predicates and inject them back into the graph, and orchestrates a targeted, graph-guided code-generation loop that updates only the affected subgraph rather than regenerating the entire plan. Empirical results on drawing, block-stacking, and YCB object-manipulation benchmarks demonstrate marked improvements in success rates and robustness compared to existing LLM-based planners.", "questions": "Writing issues\n- Line 105 in the PDF reads \"a LLMis instructed...\", missing a space (\"LLM is\").\n- In Figure 1's caption, \"Phsical Diagnosis\" is a typo, should be \"Physical Diagnosis.\"\n- In Figure 1 and Figure 2, \"griper\" is misspelled; it should read \"gripper\".\n\nQuestions\n- How sensitive is InstructFlow's performance to the clarity and structure of the user-written instruction scaffold (e.g., decomposition prompts, action templates)? Have you evaluated how variations or errors in those instructions impact success rates and repair effectiveness?\n- Constraint Generator assumes clean, structured failure traces (e.g. precise collision logs), but real‐world feedback is often noisy or incomplete, how does InstructFlow handle ambiguous or incorrect feedback data?", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Pros\n- InstructFlow achieves substantial gains in task success(20-40 percentage points over strong baselines)across drawing, block-stacking, and YCB benchmarks, demonstrating its robustness in long-horizon planning under physical constraints.\n- The hierarchical instruction graph cleanly decomposes high-level goals into typed subgoals, providing clear semantic and spatial structure for both initial plan synthesis and targeted repair.\n- Its automatic symbolic constraint induction converts raw failure traces (e.g. collisions) into reusable predicates that guide focused code refinement, no per-task hand-crafting of constraints is needed.\n\nCons\n- The multi-agent, graph-guided pipeline incurs extra LLM calls and bookkeeping (planning, code generation, constraint induction), leading to higher computational overhead and latency compared to flat planning methods.\n- Setting up the domain requires defining action templates, object/region vocabularies, and API mappings-a nontrivial engineering effort that may slow adoption in new environments.\n- In scenarios with simple, short-horizon tasks, the marginal benefit of structured repair can be limited, suggesting diminishing returns when failures are rare.", "was_revised": true, "final_justification": "The revision demonstrates meaningful progress in addressing key concerns, particularly around logical coherence and experimental support. However, the scope of comparison remains somewhat narrow.", "rebuttal": "We would like to express our sincere respect for your insightful review! In the following, we systematically respond to your concerns with additional analysis and empirical results where appropriate.\n\n\n### Weakness 1:\n>*The multi-agent, graph-guided pipeline incurs **extra LLM calls and bookkeeping** (planning, code generation, constraint induction), leading to **higher computational overhead** and **latency** compared to flat planning methods.*\n\n\n**Response:**\n\nWhile our pipeline introduces modular components, InstructFlow achieves **lower latency** and **fewer LLM calls** compared to flat planners like PRoC3S[1]:\n\n1. **Efficiency**: InstructFlow reduces feedback rounds by **37%** on average (Table 1), often converging in **1–3 ** vs. **2–5 ** regenerations in PRoC3S across complex tasks.\n\n|Table1 (Reviewer-fD4v )  | Star |Arrow | Letters | Enclosed | Pyramid | Line  |  Packing   |  Unstack   |   YCB-Packing  |  YCB-Stacking   |\n| ------------------ | ----------- | ---------- |:-----------:| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |\n| PRoC3S             | 0.50±0.20  |1.00±0.20 | 0.60±0.30 | 0.50±1.50 | 1.90±2.25 | 1.20±1.32 | 1.80±2.10  | 2.80±2.23 | 3.10±3.04 | 2.90+2.70 |\n| InstructFlow(ours) | 0.00±0.00      |     1.00±0.20        |    0.10±0.20      |      0.00±0.00        |      1.30+0.82       |      0.20+0.30       |       1.50+1.22      |       1.90+1.82      |     2.30+2.12        |      2.00+2.10       |\n\n2. **Wall-clock time**: As shown in Table 2 below, InstructFlow reduces latency by **~4.7%**, benefiting from symbolic constraints that narrow the search space and reduce code sampling needs.\n\n| Table2 (Reviewer-fD4v)   | Star |Arrow | Letters | Enclosed | Pyramid | Line  |  Packing   |  Unstack   |   YCB-Packing  |  YCB-Stacking   |\n| ------------------ | ----------- | ---------- |:-----------:| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |\n| PRoC3S             | 22.39±3.65  | 24.18±3.77 | 31.16±5.90  | 39.23±53.62 | 152.79±124.54 | 52.64±40.97 | 197.14±210.38  | 142.82±142.34 | 988.23±940.72 | 61.36±19.78 |\n| InstructFlow(ours) |    20.22+4.12     |       24.20+3.58     |   26.27+4.10      | 36.21+49.25            |      140.93+131.37       |      48.21+37.53       |       204.76+227.24      |     124.30+121.44        |      892.18+880.84       |      63.27+25.51       |\n\n[1] Trust the proc3s: Solving long-horizon robotics problems with llms and constraint satisfaction.(CoRL'2024)\n\n---\n\n\n### Weakness 2:\n\n> *Setting up the domain requires **defining action templates, object/region vocabularies, and API mappings**-a nontrivial engineering effort that **may slow adoption in new environments.***\n> \n\n**Response**:\n\nInstructFlow follows the same setup convention as PRoC3S, a one-time domain-level definition reused across tasks. Action templates (e.g., pick/place) are generic and map easily to common robotic APIs, making the system modular and portable to new domains with minimal overhead.\n\n---\n\n### Weakness 3:\n\n> *“In scenarios with simple, **short-horizon tasks**, the **marginal benefit of structured repair can be limited**, suggesting diminishing returns when failures are rare.”*\n> \n\n**Response**: \n\nWe would like to clarify that **InstructFlow is designed to adapt to task complexity,** and does not incur unnecessary overhead in simple, short-horizon tasks. Specifically:\n1. **Failure-driven activation:** The structured repair mechanism (constraint generator)  is only triggered when failures occur. In simple or successful cases, it remains dormant, and behaves like a flat planner, incurring no extra overhead and avoiding diminishing returns.s like a flat planner with negligible overhead.\n2. **Additional benefits:** Even when no failure occurs, InstructFlow’s hierarchical instruction graph provides clear semantic structure to the plan, improving **interpretability** (what subgoals are being pursued), and **debuggability** (where and why a plan might go wrong if it ever does). This structured representation facilitates downstream analysis, monitoring, and integration with safety checks，even in short-horizon tasks.\nIn summary, InstructFlow avoids overhead when unnecessary and provides benefits when needed, ensuring that its structural design remains practical across both simple and complex tasks. We will clarify this design behavior more clearly in the revised manuscript.\n\n---\n### Question 1:\n\n> *“How **sensitive** is InstructFlow's performance **to the clarity and structure of the user-written instruction scaffold** (e.g., decomposition prompts, action templates)? Have you evaluated how variations or errors in those instructions impact success rates and repair effectiveness?”*\n> \n\n**Response**: \n\nTo clarify, InstructFlow does not require users to write any instruction scaffold, decomposition prompt, or action template. **The “instruction scaffold” in our setting refers to a few-shot prompt provided to the Planner Agent,** which contains domain-level decomposition examples. These are fixed across evaluation and selected to be representative of the domain without overlapping with evaluation tasks.\"\n\nIn response to your valuable suggestion, we conduct an ablation study in which **the few-shot prompt is changed from same domain tasks to out-of-domain examples** (e.g., using examples from `Arrange YCB` to guide `Arrange Block`). As shown in Table 3, performance drops moderately but remains competitive, indicating that InstructFlow is reasonably robust to prompt variation. \n \n|  Table3 (Reviewer-fD4v)            | Pyramid | Line | Stacking | Unstack |\n|-----------------|---------|------|----------|---------|\n| InstructFlow             | 90%     | 100% | 90%      | 90%     |\n| InstructFlow w/ YCB-example | 60%     | 80%  | 60%      | 50%     |\n\n---\n\n### Question 2:\n\n> *“Constraint Generator assumes **clean, structured failure traces** (e.g. precise collision logs), but real‐world feedback is often noisy or incomplete, how does InstructFlow handle **ambiguous or incorrect feedback data**?”*\n> \n\n**Response**: \n\nTo directly address this concern, we conducted two sets of **robustness experiments**:\n1. **Incorrect Feedback**： the structured trace includes deliberately *incorrect object references* (e.g., reporting a collision with object B when object A was responsible); \n2. **Incomplete Feedback**： *key information* such as object IDs or failure causes is *removed*, simulating partial or vague real-world signals.\n\n|  Table4 (Reviewer-fD4v)  |Star | Arrow | Letters | Enclosed | Pyramid | Line | Packing | Unstack | YCB-Packing | YCB-Stacking | Average Drop    |\n| -------------------------------- | ---- |:-----:| ------- | -------- | ------- | ---- | ------- | ------- | ------- | -------- | --- |\n| InstructFlow                     | 100% |  80%  | 100%    | 100%     | 90%     | 100% | 90%     | 90%     | 60%     | 70%      |  -   |\n| InstructFlow w/incorrect feedback  | 100% |  80%  | 100%    | 100%     | 60%     | 80%  | 70%     | 60%     | 30%     | 40%      |   16%  |\n| InstructFlow w/incomplete feedback | 100% |  80%  | 100%    | 100%     | 70%     | 90%  | 70%     | 70%     | 40%     | 50%      |   11%  |\n\nResults (Table 4) show minimal degradation, with InstructFlow maintaining near-baseline performance. In addition, InstructFlow already handles unstructured semantic feedback via a VLM-based goal checker (`Lines 544-566, Appendix B.1.2`), enabling symbolic constraint induction from unstructed natural language feedback, making the system robust to real-world ambiguity.\n\n---\n\n### Writing issues:\n\n\nThank you for carefully pointing out the writing issues. We appreciate your attention to detail and will carefully proofread the full manuscript to ensure clarity and correctness in the final version.\n\n---\n\nWe sincerely appreciate the thoughtful and concrete suggestions you raised. Your comments have prompted us to examine several important aspects of the method more carefully, and we believe they have meaningfully strengthened the revised manuscript.\n\n---"}, {"review_id": "7jIRn4MF42", "reviewer": "Reviewer_iwsg", "summary": "InstructFlow is a symbolic, feedback-driven framework for robotic code generation that decomposes tasks using a hierarchical instruction graph. When execution failures occur, a constraint generator analyzes the feedback to infer symbolic constraints that identify the root cause of the problem. These constraints then guide targeted, efficient code repairs without regenerating the entire plan from scratch. This method significantly enhances robustness and task success rates in complex, long-horizon manipulation scenarios.", "questions": "1. Regarding the limitations of InstructFlow, what analysis has been conducted on its failure cases or the scenarios in which its performance is suboptimal? Furthermore, which component within the framework (say, the Symbolic Constraint Generator) is most frequently the point of failure?\n2. Have metrics such as wall-clock computation time and the average number of feedback queries been benchmarked for InstructFlow? If so, what are the quantitative improvements in planning efficiency and query complexity relative to the PROC3S baseline?", "limitations": "- The framework's efficacy is reduced in scenarios requiring precise manipulation of objects with irregular geometries as evidenced by the suboptimal success rates in the Arrange-YCB domain, which were 60% for packing and 70% for stacking, respectively.\n- Dependency on closed LLM models accessed via a cloud API. There is a lack of analysis of the framework's viability with open-source or locally hosted models, which is critical for applications in autonomous systems where network-independent, real-time processing is often a requirement.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n\n- Extends the PRoC3S [3] robot planning framework with symbolic reasoning features and constraint repair functionality.\n- More efficient and robust than PROC3S's reactive  method of starting over, which lacks mechanisms to reason about the task-level causes of failures. The system uses a graph as an internal plan representation that can be adjusted mid-stream.\n- This focus on generalization is a step towards more intelligent and less brittle robotic systems. Instead of solving just the immediate problem, the system learns a more fundamental rule which can help LLM planners to correctly refine the plan.\n\n## Weaknesses:\n\n- The symbolic inference phase of the constraint generator relies on unverified outputs. If these are incorrectly identified, the plan repair would most likely be incorrect.\n- Lack of real-world experiments evaluating the sim-to-real gap.\n\n### Typos:\n\n- L105 a LLMis\n- L627: offse_x", "was_revised": true, "final_justification": "I believe the approach provides several useful contributions that enhance prior LLM-based robotic planning frameworks (viz. PRoC3S) with better symbollic constraints and informed re-planning.", "rebuttal": "Thank you for your thoughtful review and encouraging assessment. We deeply appreciate your support and careful reading of our work. Below, we respond to your comments with detailed clarifications and highlight the revisions incorporated based on your suggestions.\n\n### Weakness 1:\n\n> “The symbolic inference phase of the **constraint generator relies on unverified outputs**. If these are incorrectly identified, the plan repair would most likely be incorrect.”\n\n\n**Response**:\n\nWe appreciate the reviewer’s concern. While symbolic inference can in principle introduce errors, InstructFlow mitigates this via both structured reasoning and semantic validation:\n- **Grounded diagnostic pipeline:** As detailed in **`Section 3.2`** , symbolic constraints are extracted through a four-step diagnostic workflow: from *failure entity retrieval* to *code-level and diagnostic analysis*, rather than shallow pattern matching.  This structured pipeline mitigate hallucination risk. \n- **Empirical Support:** `Ablation result` from **`Table 2(Lines 283-284)`** shows a 30-40% drop without Constraint Generator; case studies (`Appendix B.2.3`) and a illustration of constraints discovered in **`Table 4`**`(Lines 611-612)` confirm the alignment of induced constraints with task semantics.\n\n---\n\n### Weakness 2:\n\n> “*Lack of **real-world experiments evaluating the sim-to-real** gap.*”\n> \n\n**Response**:\n\nWe agree that real-robot validation is valuable. While our study is simulation-based, we explicitly model two key real-world uncertainties: **sensor noise** and **feedback ambiguity**, to evaluate robustness:\n\n1. **Perceptual noise:** We inject Gaussian perturbations into object poses, $\\sigma\\in\\{0.005,0.01,0.02\\}$. Note that object size is 0.04, the max noise is up to 50% object size. As shown in Table 1, InstructFlow maintains >70% success across tasks even under perceptual noise.\n\n\n|   Table 1 (Reviewer-iwsg)     |  Pyramid | Line | Packing | Unstack | YCB-Packing | YCB-Stacking | Averge Drop    |\n| --------------------- | ------- |:----:| ------- | ------- | ------- | -------- | --- |\n| InstructFlow          | 90%     | 100% | 90%     | 90%     | 60%     | 70%      |   -  |\n| InstructFlow $\\sigma=$ 0.005  | 90%     | 100% | 90%     | 90%     | 60&     | 70%      |   0%  |\n| InstructFlow $\\sigma=$ 0.01   | 80%     | 100% | 80%     | 80%     | 50%     | 60%      |  8.33%   |\n| InstructFlow $\\sigma=$ 0.02   | 70%     | 90%  | 70%     | 70%     | 40%     | 50%      |  18.33%   |\n\n2. **Feedback noise:** We simulate **incorrect** and **incomplete** failure feedbacks.  (1) incorrect feedbacks with wrong object references, and (2) incomplete traces with missing object IDs or causes. Table 2 shows InstructFlow remains robust, with only 11–16% average drop despite noisy or missing feedback.\n\n\n| Table 2 (Reviewer-iwsg)  |Star | Arrow | Letters | Enclosed | Pyramid | Line | Packing | Unstack | YCB-Packing | YCB-Stacking | Average Drop    |\n| -------------------------------- | ---- |:-----:| ------- | -------- | ------- | ---- | ------- | ------- | ------- | -------- | --- |\n| InstructFlow                     | 100% |  80%  | 100%    | 100%     | 90%     | 100% | 90%     | 90%     | 60%     | 70%      |  -   |\n| InstructFlow w/incorrect feedback  | 100% |  80%  | 100%    | 100%     | 60%     | 80%  | 70%     | 60%     | 30%     | 40%      |   16%  |\n| InstructFlow w/incomplete feedback | 100% |  80%  | 100%    | 100%     | 70%     | 90%  | 70%     | 70%     | 40%     | 50%      |   11%  |\n\nThese results show InstructFlow operates reliably without perfect state or supervision，a key requirement for real-world deployment. We view hardware validation as important future work and believe these robustness results provide strong evidence of real-world readiness.\n\n---\n\n### Question 1:\n\n> “Regarding the limitations of InstructFlow, **what analysis has been conducted on its failure cases or the scenarios in which its performance is suboptimal**? Furthermore, which component within the framework (say, the Symbolic Constraint Generator) is most frequently the point of failure?”\n> \n\n**Response**:\n\nWe thank the reviewer for raising this important point! Our analysis indicates that InstructFlow performs suboptimally in tasks with *high object diversity*, *irregular geometries*, or *tight layout constraints*, notably `Packing-YCB` and `Stacking-YCB` (`Fig 5, Lines 636-642 from Appendix B.2.3`). These scenarios challenge both constraint induction and plan refinement, due to complex spatial couplings and ambiguous success conditions.\n\nFor instance:\n1. **In Packing-YCB,** example shows how **constraints that require spatial precision** (like placement feasibility) are hard to induce when object shapes are varied. This clarifies **why performance degrades** in this task, a failure mode tied to **constraint induction under geometry noise**.\n2. **In Stacking-YCB,** example emphasizes failures from reasoning about **geometric compatibility and object stability**, again showing a case where **symbolic reasoning and plan refinement** can fail due to ambiguity in object interactions.\n\nAt the component level, the **Code Generator** emerges as the most common failure point. To support this, we conducted ablation experiments (Table 3, Reviewer-iwsg) by replacing GPT-4o with GPT-3.5-turbo across modules. The steepest performance drop (13%) occurs in the Code Generator, compared to 8% in the Constraint Generator and 4% in the Planner, highlighting the Code Generator’s sensitivity in long-horizon, constraint-heavy tasks.\n\n|  Table 3  (Reviewer-iwsg)|Star | Arrow | Letters | Enclosed | Pyramid | Line | Packing | Unstack | YCB-Packing | YCB-Stacking |  Average Drop   |\n| ---------------------------------------- | ---- |:-----:| ------- | -------- | ------- | ---- | ------- | ------- | ------- | -------- | --- |\n| InstructFlow w/**GPT-4o**                | 100% |  80%  | 100%    | 100%     | 90%     | 100% | 90%     | 90%     | 60%     | 70%      | -  |\n| InstructFlow w/**GPT-3.5-turbo**         | 90%  |  70%  | 90%     | 100%     | 60%     | 80%  | 70%     | 60%     | 40%     | 50%      |   17% $\\downarrow$   |\n| Code generator w/**GPT-3.5-turbo**       | 90%  |  70%  | 100%    | 100%     | 70%     | 80%  | 70%     | 60%     | 50%     | 60%      |    13% $\\downarrow$   |\n| Constraint generator w/**GPT-3.5-turbo** | 90% |  80%  | 100%    | 100%     | 70%     | 90% | 90%     | 70%     | 50%     | 60%      |   8% $\\downarrow$  |     |\n| Planner w/**GPT-3.5-turbo**              | 90% |  80%  | 100%    | 100%     | 80%     | 100% | 90%     | 80%     | 60%     | 60%      |   4% $\\downarrow$    |\n\n\n---\n\n### Question 2:\n\n> “Have metrics such as **wall-clock computation time** and the **average number of feedback queries** been benchmarked for InstructFlow? If so, what are the **quantitative improvements in planning efficiency** and query complexity relative to the PROC3S baseline?”\n> \n\n**Response**:\n\nWe appreciate the reviewer’s interest in benchmarking efficiency and query complexity. \nWhile InstructFlow introduces modular agents, it achieves superior **planning efficiency** and **comparable or improved latency** relative to PRoC3S, as shown below.\n\n\n1. **Fewer feedback queries:** As summarized in **Table 4**, InstructFlow reduces feedback-driven repair cycles by **~37%** on average. For example, in chellenging tasks like `Unstack` and `Pyramid`, InstructFlow completes planning in just 1–3 repair steps, compared to 2–5 full regenerations in PRoC3S. This gain stems from localized plain repair and high-level symbolic guidance.\n\n| Table 4 (Reviewer-iwsg)  | Star |Arrow | Letters | Enclosed | Pyramid | Line  |  Packing   |  Unstack   |   Packing  |  Stacking   |\n| ------------------ | ----------- | ---------- |:-----------:| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |\n| PRoC3S             | 0.50±0.20  |1.00±0.20 | 0.60±0.30 | 0.50±1.50 | 1.90±2.25 | 1.20±1.32 | 1.80±2.10  | 2.80±2.23 | 3.10±3.04 | 2.90+2.70 |\n| InstructFlow(ours) | 0.00±0.00      |     1.00±0.20        |    0.10±0.20      |      0.00±0.00        |      1.30+0.82       |      0.20+0.30       |       1.50+1.22      |       1.90+1.82      |     2.30+2.12        |      2.00+2.10       |\n\n2. **Lower wall-clock latency:** Despite its multi-agent design, InstructFlow achieves a **~4.7% reduction** in end-to-end latency (see **Table 5** below). Symbolic constraints (e.g., `Distance(?block_bottom) ∈ [0, 0.04]`) help prune infeasible code regions, reducing LLM sampling and retry overhead.\n\n| Table 5 (Reviewer-iwsg)  | Star |Arrow | Letters | Enclosed | Pyramid | Line  |  Packing   |  Unstack   |  YCB-Packing  |  YCB-Stacking   |\n| ------------------ | ----------- | ---------- |:-----------:| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |\n| PRoC3S             | 22.39±3.65  | 24.18±3.77 | 31.16±5.90  | 39.23±53.62 | 152.79±124.54 | 52.64±40.97 | 197.14±210.38  | 142.82±142.34 | 988.23±940.72 | 61.36±19.78 |\n| InstructFlow(ours) |    20.22+4.12     |       24.20+3.58     |   26.27+4.10      | 36.21+49.25            |      140.93+131.37       |      48.21+37.53       |       204.76+227.24      |     124.30+121.44        |      892.18+880.84       |      63.27+25.51       |\n\nThese improvements were obtained without latency-specific tuning, suggesting that modular planning does not incur significant computational penalties. We will report full statistics in the final version and are happy to release detailed logs upon request.\n\n---\n\n### Writing issues:\n\nThank you for spotting these typos. We will thoroughly proofread and fix all such issues in the final version to ensure clarity and polish.\n\n---\n\nOnce again, We are deeply appreciative of your generous support and constructive feedback. Your review not only affirmed our direction but also inspired valuable refinements."}]

Author rebuttal:

Meta-review: The paper introduces InstructFlow, a framework for robotic code generation that addresses failures in long-horizon planning tasks. It structures task prompts into a hierarchical instruction graph of subgoals, which are translated into executable code by a code generator, while a constraint generator diagnoses execution failures and induces symbolic constraints for targeted plan repair. Unlike prior flat or reactive methods, InstructFlow continuously integrates feedback into symbolic reasoning, enabling interpretable and adaptive code refinement without regenerating full plans. Experiments in drawing, block stacking, YCB object packing, and other tasks show that InstructFlow improves success rates by 20–40% over strong LLM-based baselines. 

One of the issues with this work is that while symbolic constraints are induced automatically, the framework still depends on a curated set of predicates, diagnostic rules, and failure categories, which may limit scalability across unseen tasks or domains. Additionally, the system assumes structured state information (object poses, relations), and does not yet incorporate raw visual perception or noisy sensor data. 

Given recent progress in reasoning for robotic task planning, I don't necessarily see this paper as significantly standing out in terms of novelty compared to methods such as ProC3S, or RoCo: Dialectic Multi-Robot Collaboration with Large Language Models, but it is a useful addition to the literature. 

Many of the concerns of reviewers were addressed during the rebuttal and despite the issues mentioned above, there seems to be consensus that this paper should be accepted, which is what I am inclined to recommend as well.
Decision: Accept (poster)