Enhancing LLM Planning for Robotics Manipulation through Hierarchical Procedural Knowledge Graphs
Authors: Jiacong Zhou
Jiaxu Miao
Xianyun Wang
Jun Yu
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Model
Robotics Manipulation
Primary area: applications
TL;DR: We propose HP-KG, which enhances LLMs' planning capabilities while reducing model scale requirements for robotic manipulation tasks.

Abstract: Large Language Models (LLMs) have shown the promising planning capabilities for robotic manipulation, which advances the development of embodied intelligence significantly. However, existing LLM-driven robotic manipulation approaches excel at simple pick-and-place tasks but are insufficient for complex manipulation tasks due to inaccurate procedural knowledge. Besides, for embodied intelligence, equipping a large scale LLM is energy-consuming and inefficient, which affects its real-world application.
To address the above problems, we propose Hierarchical Procedural Knowledge Graphs (\textbf{HP-KG}) to enhance LLMs for complex robotic planning while significantly reducing the demand for LLM scale in robotic manipulation. 
Considering that the complex real-world tasks require multiple steps, and each step is composed of robotic-understandable atomic actions, we design a hierarchical knowledge graph structure to model the relationships between tasks, steps, and actions. This design bridges the gap between human instructions and robotic manipulation actions. To construct HP-KG, we develop an automatic knowledge graph construction framework powered by LLM-based multi-agents, which eliminates costly manual efforts while maintaining high-quality graph structures. 
The resulting HP-KG encompasses over 40k activity steps across more than 6k household tasks, spanning diverse everyday scenarios. Extensive experiments demonstrate that small scale LLMs (7B) enhanced by our HP-KG significantly improve the planning capabilities, which are stronger than 72B LLMs only. Encouragingly, our approach remains effective on the most powerful GPT-4o model.

Enhancing LLM Planning for Robotics Manipulation through Hierarchical Procedural Knowledge Graphs
Jiacong Zhou Jiaxu Miao?* Xianyun Wang 2.3 Jun 2u
School of Computer Science, Hangzhou Dianzi University
The School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen SPengcheng Laboratory czhou@hdu edu. cn eejun@hit.edu.coo
LLM Plamning Task
Please cainast
Genarate sdmissible plass Par assenbling gift baskets These are four bosker on the sables...
Taak: Clean Clsir
Steps: Search Cloth,..
grahcookkee)
Extraction
Completion
LLM Manipulation Task
Refinement
Verification
"put the traii parluge"
wiki How
Web
MELM
How-to
Planinng
Data
Summury
HP-KG
"akck taada"
"lo gurbage"
"picce TREEE
Figure 1:
Overview of our work. We propose large-scale Procedural Knowledge Graph constructed from weh how-to data, which can significantly enhance the planning capabilities of LLMs in robot manipulation tasks.
Abstract
Large Language Models (LLMs) have shown the promising planning capabilities for robotic manipulation, which advances the development of embodied intelligence significantly. However, existing LLM-driven robotic manipulation approaches ex- cel at simple pick-and-place tasks are insufficient for complex manipulation tasks due to inaccurate procedural knowledge. Besides, for embodied intelligence, equipping a large scale LLM is consyycnn and inefficient, which affects its real-world application. To address the above problems,
propose Hierar- chical Procedural Knowledge Graphs (HP-KG) to enhance L.MMs for complex robotic planning while significantly reducing the demand for LLM scale in robotic manipulation. Considering that the complex real- world tasks require multiple steps, and each step composed of robotic-understandable atomic actions, we design a hierarchical knowledge graph structure to model the relationships between tasks, steps, and actions. This design bridges the gap between human instructions and robotic manipulation actions. To construct HP-KG, we develop an automatic knowledge graph construction framework powered by L.L--aased multi-agents, which eliminates costly manual efforts while maintaining high-quality graph struc- tures. The resulting HP-KG encompasses over 40k activity steps across more than 6k household tasks, spanning diverse everyday scenarios. Extensive experiments demonstrate that small scale LLMs (7B) enhanced by our HP-KG significantly improve the planning capabilities, which are stronger than 72B LLMs only. En-
"quual contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

couragingly, our approach remains effective on the most powerful GPT-4o model. Our code and data will be publicly available
Introduction
Embodied AI [ refers to Al that integrated into physical systems, such as robots, enabling them to interact with the physical world with the abilities of perception, reasoning, planning, and execution. Recent achievements of Large Language Models (I.LMs) [+ have shown the remarkable capabilities for robotic task planning [9 I0], greatly advancing the development of embodied AI. These models usually serve as high-level planners decompose human instructions into executable
sub-goals [1] 9 12 10
13,
while relying on pre-defined skills [4 for execution.
18 show that such LLM-driven planning methods usually generate unrealistic or logically inconsistent planning steps due to lacking procedural commonsense, especially in complex manipulation tasks. For instance, LLMs may ignore the agent's current physical state, thus failing to include necessary prerequisite actions (e.g., a standup action before any movement) [19], or overlook physical constraints such as the need to open a closed container before fetching items from inside 200. Moreover, robots are often limited by a finite energy supply, while LLM-driven planners typically require large-scale models (e-g., P-LM-E (21] with 562B parameters) to possess sufficient planning capabilities in complex, long-horizon scenarios. For embodied intelligence, equipping a large scale LLM is energy-consuming and inefficient, which affects its real-world application.
Recent studies 17
To address the above problems, we propose to construct an effective Procedural Knowledge Graph to enhance LLM--riven planners. Procedural knowledge refers the understanding of how to perform specific tasks, typically expressed as sequences of steps required to achieve specific goal [22] 19]. Since the Procedural Knowledge Graph provides the commonsense needed for planning. injecting the correct Procedural Knowledge into LLMs can effectively enhance their reasoning accuracy. Therefore, a small-scale I.LM equipped with the Procedural Knowledge Graph has sufficient capability to perform planning, alleviating the computational cost demands of embodied AI.
However, developing an effective representation of procedural knowledge to enhance robotic planning capabilities remains a challenge. Existing methods usually adopt coarse-grained task decomposition and simply represent procedural knowledge as reference documents [2 24 19,, without specific structure design. The main challenge in robot planning lies in translating high-level goals into feasible steps, due to the domain gap between language comprehension and robotic execution. A natural observation is that procedures of robot manipulation can be divided into a finite set of atomic actions that the robot can understand and execute. These atomic actions can be combined into a series of steps, which can further be combined into tasks that are highly abstract as human instruction in the real world. Motivated by this, we design a novel and effective Hierarchical Procedural Knowledge Graph (HP-KG) to organize procedures into three distinct layers: tasks, steps, and actions, as shown in Figure 2
Each procedure is further enriched with textual attributes (description, name, and tips).
Furthermore, to eliminate the need for manual knowledge engineering and reduce human effort, we introduce framework automatically construct the procedural knowledge graph through multi- agents calibration (25) 5 26). Specifically, we focus on household activities in this work as they are frequently performed by everyone and represent a promising area for robotic assistance to facilitate daily living (27). We systematically filter
tasks from the WikiHow corpus (88 and combine them with the BEHAVIÓR dataset [27] as knowledge source. We then prompt an LLM to extract steps from each household task and complete them by generating corresponding actions and their textual attributes. Subsequently, two LLM agents are employed to iteratively verify and refine the generated procedures based on designed rules. To enrich information while reducing redundancy, we perform semantic similarity clustering and LLM-based knowledge merging.
Finally, we propose a retrieval approach to leverage the constructed HP-KG. Given a language instruction, a refined query generated to retrieve relevant knowledge nodes based on semantic similarity. Through K- K-hop breadth-first search and re-ranking, it identifies the most pertinent nodes and converts their sub- -raphhs into textual descriptions for contextual planning. Extensive experiments on ActPlan- IK [18) and RLBench [29] demonstrate that our HP-KG enables smaller models (7B) to achieve stronger capabilities than larger models (72B) only. Encouragingly, our approach remains effective on the most powerful GPT-40 model. Overall, the contributions of our work are as follows:

---

We design a novel hierarchical procedural knowledge graph structure that effectively formalizes complex houschold tasks through a structure of tasks, steps, and actions.
We introduce a novel automated framework that leverages MLM--asee multi-agents to construct hierarchical procedural knowledge graphs, eliminating the need for manual knowledge engineering.
We present HP-KG, a large-scale procedural knowledge graph, containing 42,000+ activity steps across 6,000+ houschold tasks in daily scenarios.
Our method significantly improves the planning capability (+17.64% on 7B LLMs) and reduces the scale demand for LLMs.
2 Related Works
Knowledge Graph Construction. Traditional Knowledge Graph Construction methods typically involve multiple tasks, including entity extraction [30 31 and relation classification [32 33] which
incur substantial human effort and cost [4]. With the advancement
pre-trained language models like BERT (35), GPT-3 [36], end-to-end triplet extraction emerges as a promising paradigm [34] 377.
Contemporary graph construction approaches [38]40] leverage large language models (LLMs)
35 g
for entity extraction via prompting or fine-tuning.
LLMs with Knowledge Graph. Knowledge graphs (KGs), which provide structured factual repre- sentations, have emerged as powerful tool to enhance LLM performance 4414 Recent advances in Knowledge Graph Augmented Generation 44 46 demonstrate that KGs can serve as external knowledge bases to provide accurate factual information, effectively enhancing the factual correctness LL LLLM-generated responses (47 50) Furthermore, several studies have shown that leveraging the structural relationships inherent in KGs can enhance the reasoning capabilities of LLMs (31
Foundation Models for Robotics Manipulation. Recent achievements vision-language founda- tion models [54)56) have significantly influenced the field of robotics manipulation. These models demonstrate the potential for controlling robots to perform complex tasks. Recent studies can be broadly categorized into two paradigms: One stream adopts vision-language-ace
models [57] [8
such as RT-2 [59]. RT-X 60). and OpenVLA 61 to directly map visual inputs and language instruc- tions to robotic actions. The other paradigm leverages vision-language models (VLMs) to decompose
high-level instructions into sub-goals g 62 64), which are then solved through pre-defined
667 leverages Vision Language Models to generate subgoals, then employs dynamics-basee planning to solve the subgoal. Our method is orthogonal to these approaches. Benefiting from the in-context understanding capabilities of vision language models, our HP-KG can integrate seamlessly with various methods' subgoal generation processes.
skills [14 65 15 (66). For example, Yang et al.
Approach
In this section, first present structure design of the Hierarchical Procedural Knowledge Graph (HP-KG) for houschold activities. Second, we propose an automated procedural knowledge graph construction framework for building our HP-KG. Third, introduce retrieval method that retrieves relevant procedural knowledge as contextual input to enhance LLM-based planning.
3.1 Hierarchical Structure Design for Procedural Knowledge Graph
One main challenge of robotics planning tasks is how to bridge high-level human instructions and atomic actions that are understandable by robots. Representing and organizing procedural knowledge is central to its ability to assist robotics planning effectively. A well-structured procedural knowledge graph should be casily applicable to diverse downstream applications and guide reliable execution by both human operators and robotic systems.
To this end, our procedure structure design should satisfy the following principles. First, the nodes of this knowledge graph should include comprehensive tasks and atomic actions, as well as how to effectively connect them. Second, the knowledge graph should include safety-critical guidelines to prevent potential accidents, which is essential for ensuring correct procedure execution [68] We observe that complex procedures household activities are composed of set of atomic procedures (c.g., open container). Some atomic actions combine to form simple step. A series of steps, in turn,
3

---

combine to create complex tasks that approach high-level human instructions (e.g., clean the room), naturally forming hierarchical structure of procedural knowledge.
Based on these considerations, as Figure shows, we design simple yet effective Hierarchical Structure that encompasses multiple characteristics. To repre- sent the hierarchical nature of procedural knowledge, we categorize procedures into three levels: tasks, steps, and actions, which correspond to complex com- posite procedures, simple composite procedures, and atomic procedures, respectively. A task node links step nodes via HasStep edges, and step nodes link to action nodes through HasAction edges. The tempo- ral sequence is captured by NextStep edges between steps and NextAction edges between actions. Addi- tionally, each procedure contains two key attributes: HasDescription specifies what needs to be performed, HasTips provides tips and warnings.
Procedure
Tosk
Pet vlaxd
Step
Action
Step
Name
Nox Hhadvoothknn
Desciitioon
Actioon
Tips
(6) procedure subclasses and attribute
(b) Nerorchical struture
Figure 2: Overview of procedural graph struc- ture. (a) Procedure subelass and their attribute relationships. (b) The hierarchical structure between different procedures.
3.2 Automatic Graph Construction Framework
To construct our HP-KG, we propose an LLM-based multi-agent framework for automatic Procedural Knowledge Graph construction. As Figure 3 shows, the framework comprises four key stages: (1) data source cleaning and processing, (2) procedures generation and completion, (3) rules-guided iterative verification and refinement, and (4) hierarchical clustering and summarization.
Data Source Cleaning and Processing, In our work, we utilize two main data sources. The first is the WikiHow corpus (28), one of the largest online databases containing comprehensive how-to articles across multiple domains. Each article WikiHow typically consists of a title, main goal, and several methods to achieve this goal, where each method contains multiple steps with detailed descriptions. The second is the BEHAVIOR-IE dataset [27). which comprises approximately 1,000 most common houschold activities. Each activity is defined through BDDL (Behavior Domain Definition Language) file that specifies the activity goals and available objects in the environment.
The WikiHow corpus contains numerous articles across various domains, while only household activities are needed in this paper. Moreover, documents in the corpus may contain semantically similar activities, which could lead to redundancy in the extracted procedures. Therefore, we first filter documents by categories, retaining only household-related categories. Subsequently, we employ Deepseek-Y. V3 [4] to determine whether each document describes a household activity based on its title and descriptions (detailed prompt template is provided in Appendix E). Finally, we cluster documents based semantic similarity. We obtain approximately 5.8K unique houschold activities from the WikiHow corpus. Furthermore, we standardize input formats for BDDL-defined activities in the BEHAVIOR- IK dataset by using LLMs to generate WikiHoo-stylle content.
Procedures Generation and Completion. Based on collected documents, we prompt the LLM agent as the Procedure Generator to extract and complete procedures. During the extraction phase, the generator extracts each activity's 's tasks, steps, and corresponding textual attributes (e.g., task content, task tips) from each document. According to our hierarchical graph structure, we build the interconnection between tasks and steps through the hasStep relation. After extraction, we design an action completion prompt to generate atomic action procedures with minimal redundancy. Using this prompt. the generator produces corresponding atomic actions and their attributes for each step procedure. These actions along with their attributes constitute action procedures, which are then linked to their respective step procedures through the hasAction relation. These hicrarchical procedures interconnect through defined relations, forming preliminary procedures graph.
Rules-Guided Iterative Verification and Refinement. Due to the inherent limitations in Large Language Models (LLMs), the generated preliminary procedures inadequately adhere to procedural principles (e.g., they contain redundant information and tend to overly specific). To address these challenges, we propose an Iterative Verification and Refinement method to improve the quality of procedures. A Rules-Guided Verifier invokes multiple rule-checking models to assess the given procedure's compliance with established rules, subsequently aggregating these results to generate

---

(a) Data Source Cleaning and Processing
clustoring summary
Action Level
Web Data
Clean Data
Rules Chock
Step Level
Task Level
Extraction Templale
Completion Template
LLM
Läivnn how to sar-
< Livenn Skep Preeedere
Feadback
Takk anetaa
Notion roquionnnnas
Distinct Check
Forasa
Farsus
Summary
<Tlink sep by slege
-Tiik sep by skge.
Surrmery Agent
Procedure Generalor
Matth Eriliies
Mango Entiliee
Verileer
Prooedure Refiner
preliminary procedures graph
teralive refinement procees
procedures graph
final graph
(b) Procedures Generation and Completion
(c) Rules-Guided llerative Verificalion and Refinement
(d) Hierarchical Clustering and Summary
Figure 3: Overview of our automatic construction pipeline: (a) Data source cleaning and processing. (b) A Generator extracts steps with attributes from the data source and performs actions completion. (c) A Verifier and A Refiner iteratively validate procedures against designed rules while progressively refining them. (d) A Summary Agent hierarchically clusters and merges redundant procedures.
validation score and feedback. Additionally, we prompt LLM as a Procedure Refiner to iteratively modify the provided procedures until the score surpasses the predefined threshold.
Specifically, based on fundamennal principles of our Hierarchical Structure Design, we construct a comprehensive set of validation rules (e.g.. check if content accurately describes a procedure, check if action is atomic, etc.), denoted as R {r1, 72 ...,,'.',, )n). where n is the total number of rules. Given generated procedure Pr and its corresponding sub-procedures P.- = t.......PP' P=,,,,"),, we concatenate them into p = [pr; p ...:.?.?? p"] to serve as input to the Verifier, where sub- uub-proceurres represent nodes that are linked by p. via hasStep or hasAction relations. The agent then invokes rule-checking language model M to evaluate whether generated procedures p comply with rules in R. For each rule r; E R, the model outputs binary decision M(,, r.) E (0. 1), where indicates compliance and indicates violation of the respective rule. Therefore, we obtain rule compliance scores, denoted as:
s(p, R; M) = (M r:), M(). M ((P, r.)).
(1)
To identify redundant elements across sub pro-ceudcuuess Pe. the agent also employs an embedding mode encode both the name and description of each procedure. then groups sub-procedures whose pairwise embedding cosine similarity exceeds a threshold, resulting in list of clusters C (C1, C2..C...) where is the number of groups. Furthermore, the Verifier converts rule validation results s and aggregation results C' into textual descriptions separately. It then uses these results and procedures p to prompt a powerful LLM (e.g., GPT-4o) to generate a score and feedback.
After the verification, we use a Procedure Refinr to modify given procedures p. The Refiner utilizes feedbacks, textual result descriptions, and given procedures p as inputs to generate modified procedures p'. These refined procedures are then iteratively verified until their scores surpass predefined threshold. We provide the complete prompts and detailed framework rules in Appendix
Hierarchical Cluster and Summary. Although our iterative verification and refinement process enhances the quality of hierarchical procedures (e.g., tasks and their sub-steps, steps and their sub- actions), redundancy persists among procedures at the same level. To address this problem, we cluster and merge procedures of the same level via a Summary Agent.
We first match all procedures at a given level (c.g., task procedures) as the input of the Summary Agent. This agent uses the same embedding model to encode each procedure's 's name and content and computes the cos similarity between each pair of procedures. Procedures with similar semantics will be aggregated into group U, together forming the group list U (U1, ..)...)
For each group U, that contains multiple procedures, the Summary Agent employs an LLM to generate comprehensive summaries based on designed prompt "Given multiple semantically similar procedures,
hhtps//hwuggiggireee.ooovvaaaaat v5
5

---

analyze and synthesize them into single and actionable summary...". After the summarization process, we merge these newly generated procedures into the existing graph. We iteratively repeat the summarization process from task level to action level, ultimately deriving the final procedural graph.
3.3 Graph Retrieval-Augmennte Planning
In this section, we propose graph retrieval-augmented reasoning approach that utilizes HP-KG to enhance LLM planning and robotic manipulation. We aim to retrieve the top-K relevant nodes at specified target levels and transform their sub-graphs into textual descriptions to facilitate contextual planning. Our atrieval-augmented method consists of (1) Node Indexing, (2) Query Retrieval, (3) Entity Expansion and Re- fe-RRanking, (4) Procedural Graph Augmented Planning.
Node Indexing. For each node n with textual attributes In, we apply pre-trained embedding encoder (e.g., Zhang et al. (69) to obtain its representation E Rd:
Zn Encoder(x....
Query Retrieval. Let = E, X) denote our HP-KG, where V and denote the sets of nodes and edges respectively. Additionally, X E Rd I represents the node feature matrix, where [ denotes the number of nodes and d the feature dimension.
We begin with a general instruction and a retrieval target level Learges. First, we prompt LLMs to extract the task objectives as query Fy and apply same embdding model to encode 2.g as follows:
2g Encoder(xx,) E Rd.
We then retrieve the top-k1 most semantically similar procedure nodes Vky-
Vx, Top sim(zg: zn),
where žn denotes the embeddings of node n and sim(z,, zn) denotes the cosine similarity between 2g and a,. The TopK returns the top-k nodes with highest similarity.
Entity Expansion and Re- Re-Ranning. Since neighboring nodes often encapsulate complementary contextually relevant semantic information for the initially retrieved nodes, aim to expand our retrieved nodes to discover additional relevant procedures. Specifically, we expand the retrieved nodes Vi, by performing K-hop breadth-fine first traversal from each retrieved node follows:
VAL = U (n') dist(n', n) s K, n' E V).
5
nVV1,
where dist(-) denotes the path distance between nodes in HP-KG. Furthermore, we filter out nodes that do not match the target level Trerget from VE,, obtaining refined set:
Viarget {n V Level(n) Lnarget)
Where Level(-) denotes the level of node. Then, we re-rank the candidate nodes Viarget based on query Fg. and select the top-k2 nodes Vk, as follows:
V==
TTopK sim 2g. zn))
Procedural Graph Augmented Planning. After obtaining the retrieved nodes Vag, we aim to convert these nodes into textual context for LLM planning. To this end, we require not only the retrieved nodes but also the associated sub- uub-procedures for each procedure node. These sub-procedures detail the sequential steps of each task and the specific actions required within each step.
Therefore, we extract associated lower-level nodes based on predefined relations (e.g.. HasStep, HasAction, etc.), constructing hierarchical subgraphs where each retrieved node serves the root. Furthermore, we convert each hierarchical subgraph into a textual description (detailed templates are in Appendix E). To enable the procedural graph augmented planning, feed the converted textual descriptions into the LLM as additional context. The detailed algorithm is presented in Appendix B.
Experiments
In this section, we conduct comprehensive experiments on LLM planning and robotic manipulation benchmarks to evaluate the effectiveness and efficiency of our HP-KG. Extensive ablation studies are also conducted to show the effectiveness of our hierarchical structure and the construction framework.

---

Table I: Comparison of task success rates (%) for zero-shot robotic manipulation on RLBench. Our proposed HP-KG improves baseline success rates when retrieving the Top-K K (K=3,5) most relevant procedures (numbers in parentheses indicating absolute gains).
Method
HP-KG
Top-K
Open_vine
Take_scale
Take unbrella
Slide_bloo block
Play_jenga
Average
10%
15%
60%
50%
0%
27%
Voxposer 0
Top-3
40%
25%
60%
45%
0%
34%-(+7%
Top-5
40%
20%
65%
60%
0%
37%(++0%)
20%
0%
35%
10%
20%
17%
MA
Top-3
20%
0%
50%
30%
40%
28%(+11 %+
Top-5
10%
0%
40%
30%
35%
23% %
Table 2: We compare the effectiveness of our HP-KG against Chain of Thought (COT) prompting for
action planning. In both cases, we employ RVT as the action executor.
Method
Top-K
Open_vine
Take_lis
Take_scale
Take mike__mbrella
$lide_block
Average
RVT [57]
65%
5%
0%
30%
15%
23%
RVT cor
70%
10%
15%
35%
5%
27%((44%)
RVT w/ HP-KG
Top-3 Top-5
75%
10%
10%
55%
20%
34%(+11%%)
RVT
w/ HHPKG
85%
5%
15%
50%
10%
33%(+10%)
4.1
Experimental Settings
Benchmarks. We mainly evaluate our HP-KG on RLBench 29 and Blocks Arrange 70 tasks for roboties manipulation. Furthermore, we also evaluate our approach ActPlan- 18 for LLM planning. Detailed environment setup is provided in the Appendix A.
RLBench. This benchmark comprises 100 distinct hand-crafted manipulation tasks with varying complexity, ranging from basic target reaching and door opening to complex multi-stage operations. Each task is specified by natural language instruction that defines the manipulation goal. our experiments, we select set of 6 complex tasks, each requiring multiple actions. Blocks Arrange. This task requires robots to arrange randomly located colored boxes according to given textual instructions. In our experiments, we selected 5 complex tasks with varying difficulty levels, ranging from relatively simple one like but_blocks__tn lracckssin_crreeeee challenging ones such as stack_ blocks_ blockssintoo.. into___to___ three_ towers.
more
ActPlan-1K. This dataset contains multi-modal household activity planning instances. The task in ActPlan-1k 1 requires a MLLM to generate procedural plans P given a task description T and sequence of environment images {T,, .2.... } as input. Each instance is annotated with two reference procedural plans PT Pt, where each plan consists of multi-step action sequences.
Baselines. In RLBench, we compare different zero-shot manipulation approaches that leverage large language models (e.g., VoxPoser 99. MA [I].. We enhance their sub-task planning process by incorporating procedural knowledge and compare the results against their baseline performance. In the ActPlan- 1K benchmark, we evaluate various multi-modes large language models with different scales (c.g., GPT-4o [54], Gemini [55], Qwen2- (72) ) both with and without KG augmentation. We also leverage LLMs to generate subgoals and employ pretrained policy (57] for execution on RLBench. In this context, compare our HP-KG against Chain of Thought (COT) 73 method. Additionally, we compare with SayCan 71 on Blocks Arrange task.
Metrics. We employ Success Rate (SR) for RLBench and Blocks Arrange, while using the Longest Common Subsequence (LCS) metric for the ActPlan- benchmark. Detailed metrics are as follows:
Success Rate. Similar to VoxPoser, we conduct 20 trials per task and calculate the success rate. LCS. On the ActPlan- 1K benchmark, we use the longest common subsequence (LCS) metric to evaluate the generated plans against annotated reference plans (using the first one as default). Following 18] we encode each step with language embedding mode and consider those with similarity above 0.8 as matching steps in the LCS calculation.
Mix LCS. Given the diversity in planning approaches and individual preferences, we propose Mixed Longest Common Subsequence (MIX LCS) metric. In our LCS similarity comparison, we consider a generated plan step to be similar if matches any step in either reference plan.
Implementation details. In our Iterative Verification and Refinement process, the maximum iterations set 3. In our Retrieval process, we set k2 100 and set k2 to 3 depending on the experimental
7

---

Table 4: Comparison of multimodal large models of differ- ent scales on Actplan-1k IK.
Table 3: Comparison of Different Planning method on the Blocks Ar- range task.
Top-K
LCS T
Average
GPT-40-0224-1-20 S
Top-3
12,5063
12.2025/41445
Method
Top-K
Average SR
Top-5
12,7552
12.5084/43.99
104261
11.071
1077456
SayCan 77
26%
Gemini 2.0 Nash 55
Top
11.0210
11,5316
11. 11.266+44999
Top-5
10.9466
11,4767
31 22.2 44.33
GPT-4o cor
33%
Qwen2 VL
Top
9.5527
Top-3 Top-5
45% %
Top-5
8,40922
GPT-4o HP-KG
42%
8.5189
9.0664
8,7826
Qwen2 VL 728 Instruct X
Top
9-5527
10,0759
9.8 981431-11.745
Top
9.2344
9.7637
+8.41%
Table Comparison of Different Graph Structures on Table 6: Effectiveness analysis of each stage in automated graph construction frame- work on Actplan-
the Actplan- IK. All methods use Top-K=3.
Graph Structure
LCS
Míx LCS
Average
Qwen2-VLL In-strc-nttttutt
[66]
Procedures Generator
Iterative Verification
Summary Agent
Average L.C
8.5189
9.0464
8.7826
8,7826
Unstructured Does
8,9118
9,4345
9.1771
X
9,4809
Coarse Graph Structure
8.9113
9.2953
9.1033
9,5968
Hierarchical Graph (ours)
9.5527
10.0759
9.8143
V
9,8143
conditions, For zero-shot manipulation approaches, we utilize GPT-4o as the primary planner unless explicitly stated in experiments. Appendix A provides comprehensive implementation details.
4.2 Main Results
Results RLBench. Table [ presents comprehensive performance comparison of whether the method is enhanced with our proposed HP-KG. Compared to Voxposer and MA, our HP-KG significantly improves their success rates by up to 10% and 11%, respectively. For all baselines. we achieve improvements on more than half of the tasks, while the remaining tasks are not completed due the limitations of the baselines' manipulation capabilities. For instance, we find that MA struggles with tasks requiring precise operations due to its lack of precision in identifying target positions, which prevents it from accurately grasping the lid handle during lid-removal tasks. Nevertheless, our HP-KG still shows promising improvements in handling these challenging scenarios. We also compare our HP-KG with COT method, and results are presented Table 2
Results on Blocks Arrange. Table 3 compares the task success rates of different methods, with GPT-4o serving as the base LLM for all approaches. The re- sults demonstrate that LLMs enhanced with our HP-KG achieve higher success rates compared to both SayCan and COT methods. This performance gain stems from our approach's 's ability to inject explicit procedural knowledge into LLMs, whereas SayCan and COT methods merely leverage the implicit knowledge embedded in LLMs.
Whi WITTING HP KG (Tho K-B HIIK (lop-AAS)
22
2.54
Inferent Thee (4)
Massar Lcs
8:32
5.75
1:0
L
0.26
Results on ActPlan-1K. Table 4 presents comprehen- sive comparison of planning abilities across multi-modal large models with/without HP-KG enhancement. Our HP-KG improves the planning abilities of models across all scales. Specifically, our approach significantly boosts the average LCS of the small-scale model (Qwen2-VL-7B) VL-7B) by 17.64% compared to the 7B baseline, even surpassing the performance of the 72B model baseline. Furthermore, for large-scale models and Twee2--VL-722 our approach achieves
QwAn2VVL-7R
Damm2-1--72
Figure 4: Performance and inference time comparison across different model sizes.
with high capability, such as GPT-4o, Gemini-22.0FFlas performance improvements of up to 3.99%, 4. 4.90% 11.74% respectively. Comparing results of different retrieval number (top-3 v.s. top-5), we find that the performance of smaller models decreases as the Top-K increases, due to their limited long context comprehension ability. In contrast, for larger models like GPT-4o, the additional procedural knowledge further enhances their performance.

---

Query: slide the block target.
Voxposer
(Qwen2.5-7B)
task planning:: push block lowards green target
Query: slide the block larget.
Voxposer (Qwen2.5-7B)
+ HP-KG
task planning:
move the block towards the larget
align the block with the target
stop and confirm the block is next target
Figure 5: Qualitative Results:
Voxposer with small LLM (Qwen2.5-7B) correctly decomposes tasks and executes them successfully when enhanced with our HP-KG framework.
4.3 Ablation Studies
Ablation Study on Hierarchical Structure Design. To assess the effectiveness of our hierarchical structure, we transform retrieved nodes into various alternative structures and evaluate each variant on the Actplan-1K K.. Specifically, we introduce two variants: (1) Unstructured Docs, where we directly convert the retrieved nodes into text without sub-graphs; and (2) Coarse Graph Structure, where we extract sub-steps for each retrieved task node and only transform these tasks and their steps into textual descriptions without actions. Table demonstrates that our Hierarchical Graph Structure shows promising improvements over two variants. We find that providing procedural knowledge enhances LLMs' planning capabilities by supplementing limited procedural commonsense. Our hierarchical structure is more effective at organizing procedural knowledge than alternative variants.
Ablation Study on Graph Construction. We perform ablation experiments across the three stages in our Automatic Graph Construction Framework. We evaluate Qwen2-VL--72-Innst Instruct enhanced with HP-KGs from each stage on Actplan-IK, and report Average LCS metric. Table shows that our preliminary procedures graph provides modest improvement over the baseline. Both our Iterative Verification and Refinement process and Summary Agent further enhance the quality of the procedural graph, leading to additional improvements in overall performance.
Efficiency Analysis. We also compare the inference time across MLLMs of various scales to evaluate the efficiency of our HP-KG. We deploy Swen2--L-7B-Instruu and Qwen2-VL--7B-Instree
on the same hardware platform and evaluate them the Actplan-1K K benchmark, measuring both average inference time and Average LCS metric. Results are presented in Figure 4 As the retrieval number increases (from top-3 to top-S), inference time of LLMs increases slightly due to longer context length. Notably. Qwen2-VLL -L-BB enhanced with our HP-KG achieved performance comparable to the Qwen2-VIL-72B baseline while saving 50% of inference time, demonstrating that our method can effectively reduce reliance on large-scale L.LMs. We also compare different L.MMs within Voxposer for robotics manipulation tasks and evaluate their computational efficiency Appendix
4.4 Case Studies
prresents qualitative analysis by comparing scenarios with and without HP-KG enhancement. We utilize Qwen2.5-7B as the planner in Voxposer to decompose tasks into different subtasks and execute them sequentially. The Qwen2.5-7B model alone struggles with proper task decomposition. hindering the robotic arm's 's ability to complete assigned tasks. In contrast, with HP-KG enhancement, the model generates more logical subtasks, enabling successful execution of the overall task objective. We also conduct experiments in long-horizen robot task planning benchmark using the Kitchen- World environment [67]. Due space constraints, detailed results are presented in Appendix D.
Limitations and Conclusion
In this work, we propose Hierarchical Procedural Knowledge Graphs (HP-KG) to advance long- horizon planning capabilities of LLMs. We introduce a hierarchical graph structure that captures

---

complex real-world task relationships. We also develop an automatic knowledge graph construction framework powered by LLM-based multi-agents. Furthermore, we propose a retrieval approach to leverage the constructed HP-KG enhancing L.MS for robotic manipulation. Extensive experiments on multiple benchmarks demonstrate our improvements compared existing approaches. However, our knowledge graph is limited to household activities, constraining its applicability in general scenarios. We plan to develop general procedural graph and apply it to wider field in the future.
Acknowledgments and Disclosure of Funding
This work supported by the National Natural Science Foundation of China (NSFC) under Grant No. 62125201, U24B20174, 74, No. 62306273. The authors declare no competing interests.
10

---

References
[1] Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, al. Embodiedscan: A holistic multi-modal 3d perception suite towards embodied AL. In CVPR, 2024.
Alessandro Suglia, Claudio Greco, Katie Baker, Jose L. Part, al. Alanavim: A multimodal embodied AI foundation model egocentric video understanding. In Yaser Al- Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, EMNLP, 2024.
[3] OpenAI. Gpt-4v(ision) system card. http::/upi..smmante http::/heeeuuuttttuauuu/e////aoo 2023.
[4] DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, et al. Deepseek-v3 technical report. arXiv preprint 1943,,, 2024.
[5] DeepScek-Al, Daya Guo, Dejian Yang, et al. Deepseck-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
[6] Albert Q. Jiang. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel. Guillaume Lample, Lucile Saulnier. et al. Mistral 7b. arXiv preprint 1.0.vv:2300006888. 2023.
[7] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Wu, et al. Vicuna: An open chatbot impressing gpt-4 with 90%* chatgpt quality. 2023, httpps:/:///msst org/ httg//llg///0// 2023 00-0002333000000000 30-30--1 vicuna/
[8] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
[9] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In CoRL, 2023.
[10] Jacky Liang, Wenlong Huang. Fei Xia, Peng Xu, Karol Hausman, Brian lehter, Pete Florence, and Andy Zeng. Code policies: Language model programs for embodied control. In ICRA, 2023.
[11] Jiafei Duan, Wentao Yuan, Wilbert Pumacay, Yi Ru Wang, Kiana Ehsani, Dieter Fox, and Ranjay Krishna. Manipulate-nnything Automating real-world robots using vision-language models. In CoRI, 2024.
[12] Siyuan Huang, Haonan Chang, Yuhan Liu, Yimeng Zhu, Hao Dong, Peng Gao, Abdeslam Boular- ias, and Hongsheng Li. A3vIm: Actionable articulation aware vision language model. arXiv preprint 20l.:2496..7549
[13] Xiaoqi Li. Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang. Jiaming Liu, and Hao Dong Maniplim: Embodied multimodal large language model for object-centric robotic manipulation. In CVPR, 2024.
[14] Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipula- tion with low-cost hardware. In Robotics: Science and Systems XIX, Duegu, Republic of Korea, July 10- 14, 2023, 2023.
[15] Yecheng Jason Ma, William Liang, Guanzhi Wang, et al. Eureka: Human-level reward design via coding large language models. In ICLR, 2024.
[16] Ajay Mandlekar, Yuke Zhu, Animesh Garg, al. ROBOTURK: A crowdsourcing platform for robotic skill learning through imitation. In CoRL, 2018
[17] Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, et al. Embodied agent interface: llms for embodied decision making. In Neurips, 2024.
Benchmarking
[18] Ying Su, Zhan Ling, Haochen Shi, Cheng Jiayang, Yauwai Yim, and Yangqiu Song. Actplan-1k: 1k: Bench- marking the procedural planning ability of visual language models in household activities. In EMNLP, 2024.
[19] Valentina Anita Carriero, Antonia Azzini, Ilrria Baroni, Mario Scrocca, and Irene Celino. Human evaluation of procedural knowledge graph extraction from text with large language models, In EKAW, 2024.
[20] Haoquan Fang. Markus Grotz, Wilbert Pumacay, Yi Ru Wang, Dieter Fox, Ranjay Krishna, and Jiafei Duan. Sam2act: Integrating visual foundation model with memory architecture for robotic manipulation. arXiv preprint 25X.::5500118664 2025.
11

---

[21] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery. Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied multimodal language model. 2023.
[22] Anisa Rula, Gloria Re Calegari, Antonia Azzini, Davide Bucci, Ilaria Baroni, and Irene Celino. Eliciting and curating procedural knowledge industry: Challenges and opportunities. Qurator, 2022.
[23] Kaichen Zhao, Yaoxian Song, Haiquan Zhao, Haoyu Liu, Tiefeng Li, and Zhixu Li. Towards barse-grained visual language navigation task planning enhanced by event knowledge graph. In CIKM, 2024.
[24] Michael Beetz, Philipp Cimiano, Michaela Kümmel, Enrico Motta, Ilaria Tiddi, and Jan- Philipp Töberg. Transforming web knowledge into actionable knowledge graphs for robot manipulation tasks. ESWC. 2024.
[25] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Iehter, Fei Xia, Ed Chi, Quoc V. Le, and Deany Zhou. ffain-of-thougee prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Newral Information Processing Systems 2022, NeurlPS 2022. New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[26] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024.
[27] Chengshu Li. Ruohan Zhang, Josiah Wong, Cem Gokmen, et al. BEHIAVIRR-IK A benchmark for embodied AI with 1, 000 everyday activities and realistic simulation. In Karen Liu, Dana Kulie, and Jeffrey Ichnowski, editors, CORL, 2022.
[28] Li Zhang. Qing Lyu, and Chris Callison-Bureh. Reasoning about goals, steps, and temporal ordering with wikihow. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, EMNLP, 2020.
[29] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark& & learning environment. IEEE Robotics Autom. Lett., 2020.
[30] Pedro Henrique Martins, Zita Marinho, and André Martins. Joint learning of named entity recognition and entity linking. In ACL, 2019.
[31] Aniruddha Mahapatra, Sharmila Reddy Nangi, Aparna Garimella, and Anandhavelu Entity extraction in low resource domains with selective pre-training of large language models. In EMNLP, December 2022.
[32] Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao, Distant supervision for relation extraction via piecewise convolutional neural networks. In EMNLP, 2015.
[33] Zexuan Zhong and Danqi Chen. A frustratingly easy approach for entity and relation extraction. In ACI, 2021.
[34] Lingfeng Zhong, Jia Wu, Qian Li, Hao Peng. and Xindong Wu. A comprehensive survey on automatic knowledge graph construction. ACM Comput. Surv,, 2023.
[35] Jacob Devlin, Ming-Was Chang, Kenton Lec, and Kristina Toutanova. BERT: pre-training deep bidirectional transformers for language undorstanding. In NAACL-HLT, 2019.
[36] Tom B. Brown, Benjamin Mann, et al. Language models are few-shot learners. In NewrIPS, 2020.
[37] Lishan Qiao, Limei Zhang, Songcan Chen, and Dinggang Shen. Data-driven graph construction and graph learning: A review, Neurocompuating. 2018
[38] Zhen Bi, Jing Chen, Yinuo Jiang, Feiyu Xiong, Wei Guo, Huajun Chen, and Ningyu Zhang Codekge: Code language model for generative knowledge graph construction. ACM Trans. Asian Low Resour. Lang. Inf. Process., 2024.
[39] Yansong Ning and Hao Liu. Urbankgent: A unified large language model agent framework for urban knowledge graph construction. In NewrIPS, 2024.
[40] Bowen Zhang and Harold Soh. Extract, define, canonicalize: An Ilmeb framework for knowledge graph construction. In EMNLP, 2024.
[41] Qinggang Zhang. Shengyuan Chen, Yuanchen Bei, Zheng Yuan, Huachi Zhou, Zijin Hong. Junnan Dong. Hao Chen, Yi Chang, and Xiao Huang. A survey of graph retrieval- augmented generation for customized large language models. arXiv preprint 11X5525501.33958 2025.
12

---

[42] Xiaoxin He, Yijun Tian. Yifei Sun, Nitesh V. Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. G-retriever: retrieval-augmentee generation for textual graph understanding and question answering, In NeurlPS, 2024.
[43] Jiho Kim, Yeonsu Kwon, Yohan Jo, and Edward Choi. KG-GPT: A general framework for reasoning on knowledge graphs using large language models. In EMNIP, 2023.
[44] Bohan Chen and Andrea L. Bertozzi. Autokg: Efficient automated knowledge graph generation for language models. In 2023 IEEE International Conference on Big Data (BigData), 2023.
[45] Xiangrong Zhu, Yuexiang Xie, Yi Liu, Yaliang Li, and Wei Hu. Knowledge graph-guided retrieval augmented generation. In NAACL, 2025.
[46] Meng-Chieh Iee, Qi Zhu, Costas Mavromatis, Zhen Han, Soji Adeshina, Vassilis N. Ioannidis, Huzefa Rangwala, and Christos Faloutsos. Hybgrag: Hybrid retrieval-augmennte generation on textual and relational knowledge bases. arXiv preprint 4rXiv:2412.16311, 2024.
[47] Xuejiao Zhao, Siyan Liu, Su- Yin Yang, and Chunyan Miao. Medrag: Enhancing retrieval- augmented generation with knowledge graph-elicited reasoning for healtheare copilot. In WWW, 2025.
[48] Lei Liang, Mengshu Sun, Zhengke Gui, et al. KAG: boosting Ilms in professional domains via knowledge augmented generation. arXiv preprint 2rXiv:2409.1777 2024.
[49] Junjie Wang, Mingyang Chea, Binbin Hu, et al. Learning to plan for retrieval-augmentte large language models from knowledge graphs. In EMNLP, 2024.
[50] Mufei Li, Siqi Miao, and Pan Li. Simple is effective: The roles graphs and large language models in knowledge-graph-basee retrieval-augmented generation. In ICI.R, 2025.
[51] Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Heung- Yeung Shum, and Jian Guo. Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph. In ICLR, 2024.
[52] Shengjic Ma, Chengjin Xu, Xuhui Jiang, Muzhi Li, Huaren Qu, Cehao Yang, Jiaxin Mao, and Jian Guo. Think-on-graph 2.0: Deep and faithful large language model reasoning with knowlcdge-guides retricval augmented generation. arXiv preprint arXiv:2407. 108005, 2024.
[53] Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Reasoning on graphs: Faithful and interpretable large language model reasoning. In ICLR. 2024.
[54] OpenAL, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, et al. Gpt-4o system card. arXiv preprint axXiv:2410.21276, 2024.
[55] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05533 2024.
[56] Peng Wang, Shuai Bai, Sinan Tan, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12/91, 2024.
[57] Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, and Dieter Fox. RVT: robotic view transformer for object manipulation. In CoRL., 2023.
[58] Zhutian Yang, Caelan Reed Garrett, Tomás Lozano-Péree Leslie Pack Kaelbling, and Dieter Fox. Sequence- based plan feasibility prediction for efficient task and motion planning. In Robotics: Science and Systems XIX, 2023.
[59] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, al. RT-2: vision-languagg-action models transfer web knowledge to robotic control. In CoRL, 2023.
[60] Abby O' Neilil,, Abdul Rehman, Abhiram Maddukuri, et al. Open x-embodiment: Robotic learning datasets and RT-X models: Open x-embodiment collaboration. ICRA, 2024.
[61] Moo Jin Kim, Karl Pertsch, Siddharth Karameheti, Ted Xiao, al. Openvla: An open-source vision- aanuuage-action model. arXiv preprint arXiv:2406.0940 09246, 2024.
[62] Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. In CORL, 2024.
[63] Yingdong Hu, Fangi Lin, Tong Zhang, Li Yi, and Yang Gao. Look before you leap: Unveiling the power of GPT-4V in robotic bision-langure planning, arXiv preprint arXiv:2311.178. 2023.
13

---

[64] Kevin Lin. Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: from natural language instructions feasible plans. Auton. Roboty, 2023.
[65] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In CoRL, 2022.
[66] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Trans, Mach. Learn. RioffRiiFR3a)
Res., 2024. URL http::///pppeeeeeiiaii
[67] Zhutian Yang. Caelan Reed Garrett, Dieter Fox, et al. Guiding long- ong-horizon task and motion planning with vision language models. arXiv preprint K1v:2440.002193, 2024.
[68] Arda Uzunoglu, Gözde Gül Sahin, and Ahdulfattah Safa. PARADISE: evaluating implicit planning skills of language models with procedural warnings and tips dataset. ACL, 2024.
[69] Dun Zhang, Jiacheng Li. Ziyang Zeng, and Fulong Wang. Jasper and stella: distillation of sota embedding models. arXiv preprint arXiv:2412.199 19044998,, 2025.
[70] Yongchao Chen, Yilun Hao, Yang Zhang, and Chuchu Fan. Cade-as--ymbolicpplaeeer Foundation model-based robot planning via symbolic code generation. arXiv preprint arXiv:2503.01700, 2025.
[71] Brian Ichter, Anthony Brohan, Yevgen Chebotar, et al. Do as can, as I say: Grounding language in robotic affordances. In CoRL, 2022.
[72] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, et Qwen2 technical report. arXiv preprint arXiv:2407.1067/ 2024.
[73] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thoughh prompting elicits reasoning in large language models. NeurlPS, 2022.
[74] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, et al. Expanding performance boundaries of open- source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 2024.
[75] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[76] Qwen, :, An Yang, Baosong Yang, et al. Qwen2.5 technical report. arXiv preprint 15Xi: 2025.
[77] Shiduo Zhang, Zhe Xu, Peiju Liu, Xiaopeng Yu, Yuan Li, Qinghui Gao, Zhaoye Fei, Zhangyue Yin, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. Vlabench: A large-scale benchmark for language-conditioned roboties manipulation with long-horizon reasoning tasks. In ICCV, 2025.
[78] Jinguo Zhu, Weiyun Wang. Zhe Chen, Zhaoyang Liu, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint 2xXiv2554.10477 2025.
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: yes.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: discuss the limitations in our paper.
15

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
Justification:
we do not provide theoretical conclusion
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
we provide via detailed instructions for how to replicate the results in our paper.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification; we promise make code public and help the community reproduce our results when our work is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/public/eee.c www.ddddeeeooeuuummmcecom ssi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: we provide via detailed instructions for how to replicate the results our paper and promise to make code public and help the community reproduce our results when our work accepted.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: No need calculate error bars.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [TODO]
Justification: provide detailed instructions about computer resources in paper.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://aaarra.ccccoooo/oc/ccccca
Answer: [Yes]
Justification: our research conforms, in every respect, with the NeurlPS Code of Ethies.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: In our paper, we propose mainly an effective method to enhance the planning capacity of LLM, which is not subject to high risk of abuse.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
functioning correctly, harms that could arise when the
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [No]
19

---

Justification: In our paper, propose mainly an effective method to enhance the planning capacity of LLM, which is not subject to high risk of abuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: we have credited all asset creators and respected the associated licenses and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, papprswi wwwode.tthcc....m com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: we promise to make code public and help the community reproduce our results when our work is accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their
submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
20

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification: we did not use crowdsourcing or conducted research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [No]
Justification:
we did not use crowdsourcing or conducted research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification: No.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
21

---

A Overall Experimental Details
Software and Hardware Configurations, We conduct our experiments servers equipped with NVIDIA A6000 GPUs (48GB VRAM), with NVIDIA CUDA Toolkit version 11.8, For inference time comparison, we deploy different-sized models (7B and 72B) using the vLLM inference framework and use the AWQ quantized version for the 72B model to fit within the available GPU memory.
Experimental Parameter Settings. In our Iterative Verification and Refinement process, the maximum iterations is set to 3. For all clustering operations, we employ cosine similarity with threshold of 0.85. In our Procedural Graph Retrieval process, we set Y 100 and set l/2 to 3 or 5 depending on the experimental conditions. For zero-shot manipulation approaches, utilize GPT-4o the primary planner unless explicitly stated in experiments and set the target retrieval level Taaggtt to the step procedure. For experiments on Actplan- we set the target retrieval level Ltrrget to the Task procedure due its more complex task objectives.
Details of the Constructed Graph. Our HP-KG encompasses a rich variety of actions and objects, enabling comprehensive coverage of diverse tasks in common daily scenarios. Additionally, we present the task distribution of our HP-KG, demonstrating its potential for assisting robots in complex task comple- tion. The specific details are shown below:
Table 7: Statistics of nodes and their corre- sponding average number of child nodes.
Node Type
Node Count
Average Sub-Nodes
Task Nodes 6380
6.58
Step Nodes
40659
4.11
Action Nodes
136943
Graph Statistics. Our HP-KG comprises 6K Task Nodes, 40K Step Nodes, and 136K Action Nodes in total, with comprehensive statistics detailed in Table 7 Furthermore, HP-KG incor- porates over 1,300 distinct verbs and approximately 26,275 objects, whose distributions are illustrated through word cloud visualizations in Figure 6
Task Length Distribution. We present the distribution of total task lengths in HP-KG in Figure Our tasks span from simple procedures (approximately 3-7 actions) to complex operations (over 30 actions). This diversity highlights HP-KG's capability to support robotic systems in executing both straightforward and complex task sequences.
switch tralipp Splppaann Lant prevent
positico
beat
flatten
pick
test
fold
RACE
TM
Leavens 1ILL sDrE deent HHgiSh
Lean
Jirthneeee WEOT BN
spread
Refold arrange
wa S h chine ggg
detergent Beest dish
vingga
Inspec
stain
the
neasur ing cup
ma
screws
Chi
Bcr
Cabinet net door
1
tables
potatoes
pot
AteAk kip
bowl
Aittee
rices
paints
Heoll occeie ahorre
brew geltur
doo
aiga
Veg
manntain
Tamp rubwig y
ing ingre ngredd
i
ennts
processile
Loth
ttpp
replace
close
dry
cab transter
inter pret
ilid St
fagie
e
Cheat
soda
SCRUD
need
pai
rood S ponge
Ol
rag
E grefrig ingggrrtt
oven
NIR OPS
SOLEr ranse
solution
selethes
butttr
chicken
ben
aa
small
meslies
amount
Lacaee hundll
con
C
a
nel
reduEs
pars
moisten fastenshut i gnteeneleet
spray
fabric
Figure 6: Statistical analysis of verbs and objects in HP-KG actions. (Left) Word cloud visualization of verbs. (Right) Word cloud visualization of objects.
B Procedural Graph Retrieval-Augmented Planning Algorithm
The complete formulation of the retrieval algorithm is detailed in Algorithm T
C Additional Experiments Results
In this section, we present additional experimental results on both ActPlan- 1K and RLBench bench-
marks.
Additional comparison on ActPlan-1K, Table X
presents a broader comparison of diverse large
dision-langual models (c.g., Intemn 774,, 3.2.-....--9----iiii-
(75) on the
22

---

300
154
200
Figure 7: Distribution of total task lengths in HP-KG.
Algorithm 1 Procedural Knowledge Graph Retrieval and Planning.
Input:
General instruction, hierarchical procedural knowledge graph g (V, E, X), target level
Itarget: parameters k1, k2, hop limit K, encoder model
Output: Retrieved procedural knowledge in textual format
1: Query Retrieval:
2: Extract task objectives as query 2i using LLM
3: Encode query: 2g = Encoder(r,)) E Rdd
4: Retrieve initial candidates: VA, TopKnev sim(zz, Top-k1 semantically similar nodes
5: Entity Expansion and Re-Ranking
6: Expand retrieved nodes: ViS = UneVi {n') dist(n', S K,n' V}
K-hop expansion
7: Filter by target level: Viarget = {n E Vi, Level(n) = Inarget)
8: Re-rank candidates: Ve, = Topp né Viargnt sim(2g, 2n)
Final top-k2 nodes
9: Procedural Text Formatting:
10: Text < I
> Initialize formatted text collection
11: for each node n E Vk,
12: Retrieve attributes and child nodes of
13: Format node information into structured text segment in
14: Text T )
15: end for
Concatenate all text segments in Text into final textual context T
17: Return:
Formatted procedural text
ActPlan-1e 1K 18 benchmark. Since some models cannot fully adhere to instructions and produce complete planning outputs, record the count of successfully generated planning solutions for each model. Our HP-KG approach demonstrates consistent effectiveness across models of varying scales. We present detailed ablation results of our construction framework in Table
Additional comparison on RLBench. In addition to GPT-4o, we integrate several open-souree models 10-en22.57BBInstrueee Qwen2.5-72B-Ine Instruct (76)) into Voxposer and conduct efficiency analysis. The success rates across different tasks and inference time comparisons are presented in Tob and Figure & respectively. Compared to these models without HP-KG, approach improves their success rates by up to 10.83% and 5.83%, respectively. Furthermore, with HP-KG enhancement, Qwen2.5-7B ----trrutt achieves performance comparable Qwen2.5-72B-Imm Instrrutt while requiring less than 20% of its inference time.
Detailed Inference Time Analysis. We conduct experiments to evaluate the efficiency of our retrieval method. Specifically, we measure the average Instruction Encode Time (Encode Remei,ee Time, K-hop Time, Reranking Time, Graph Conversion Time and LLM Inference Time during
23

---

Table 8:
We evaluated additional large multimodal language models on the ActPlan IK benchmark. In our analysis, the Success metric indicates the number of instances where models successfully generated planning solutions in the required format.
Model
HP-KG
Top-K
Success
LCS f
Mix LCS
Average 1
237/237
8,4430
8.6920
8.5675
IntermVL-2.5-220
741
Top-3
237/237
8.5147
8.9915
8.7531(+2.16%)
Top-5
237/237
8,4599
8.9957
8.7278(+1.875)
197/237
10.3756
10,9086
10.6421
1lama-3.2-90B: 3.2-990----iiionn S-8--Visoo--nstuut
7S
Top-3 Top-5
204/237
11.6715
12.4166
12.0440(+13.1755
10.8563
11.1602
11.0082((3.44%%
Table 9: Detailed results of each stage in automated graph construction framework on Actplan
Procedures Generator
Ieerative Verification and Refinement
Summary Agent
LCS
Mix LCS
Average LCS
8.5189
9.0464
8.7826
9,2320
9,7299
9,4809
9,2953
9.9578
9.6265
9.5527
10.0759
9.8143
the Top-3 and Top-5 search processes on the Actplan- IK [8]]- All settings are consistent with the Experimental Details Appendix [A] [TT presents the detailed timing breakdown. Our retrieval pipeline is highly efficient, with the combined overhead (encoding, retrieval, k-hop, reranking, and graph conversion) being significantly smaller than the MILM inference time.
Additional experiments on complex manipulation benchmark. We conduct experiments on a more complex benchmark, VLABench(Z2], which con- tains multiple complex tasks involving common sense knowledge, physical rules, and reasoning capabilities. VLABench designed evaluate not only VLAs but also MILMs. In the MLLM evaluation setting, models must output both the skills be invoked and their corresponding parameters.
30
Seeee The
54,00
75.8%%
1.75
2.8
798%
We select four task categories: CommonSense, Com- plex, M&T, and PhysicsLaw, with each category con- taining 5-8 tasks. The ConmonSense, M&T, and PhysicsLaw categories evaluate the model's 's under- standing of world procedural knowledge, while the Complex category requires not only world knowledge but also long-horizon planning capabilities.
1.70
Voposer (2w 70)
Vagpost (Que125
Figure 8: Performance and inference time comparison of different model sizes in the Voxposer framework.
We compare the accuracy of MILMs enhanced with HP-KG against those without such enhancement, with each task evaluated over 50 trials. As shown in Table [2 our HP-KG consistently improves planning performance even on complex manipulation benchmark. This indicates that our approach effectively enhances MLLMs' planning capabilities by providing structured prior knowledge that guides decision-making in challenging scenarios.
Table 10: Additional comparison task success rates (%) for zero-shot robotic manipulation on RLBench using Voxposer We integrated various open-source large language models into the Voxposer framework.
Mudels
HP-KG
Top-K
Open_vine
Take_
Take_acole
Twww..nebrai
S1-de,hi bllock
Play-jeens
Average
(Qwnn25-7BB Instrot
Mingoner
Top-3
Top-5
25 25/0055
30.0055
15.00%
0.00
0005
0UD0S
25,0075
15,005
5.00%
20,00%
10.00%
5.00
45000%
35,00%
15,00%
0.00%
0.00%
0100%%
(5009((77588)
(-.33) %)
7.50%
(Qwe2255 fnstruct
Vesposer
Top
Top
35.00%
30,00%
3.00%
0.00%
0008
0.00%
25.00%
35 005
000
55,00%
65.00%
63.00%%
20.00%
35.00%
20.00%
0.00%
0.00%
0,00%
25.35%(=3.333)
83.
20.00%

---

Table 11: Detailed Inference Time Analysis with Qwan2-VL-72B-Issttuuc on ActPlan.
Top-k
Encode Time
Retrieval Time
K-hop Time
Reranking Time
Graph Conv.
LLM Inference
Time
Time
Top-3 Top-5
0.038s 0.039s
0.045s 0.046s
0.001s 0.001s
0.025s 0.027s
0.001s 0.001s
1.28s
1.49s
Table 12: Performance Comparison with Different Top-k Settings VLABench
Models
HP-KG
Top-k
CommonSense
M&T
PhysicsLaw
Complex
Average
26.15
28.19
10.21
19.01
20.89
Intern3-VI-880
28.77
29.21
18.95
21.47
24.58
29.55
28.70
18.22
21.16
24.41
D Long-Horizon Planning Experiments
We further investigate the capability of our HP-KG to support embodied robots in long-horizon planning tasks by conducting evaluations in the more sophisticated simulated environment Kitchen World [67]
Kitchen World Settings. The environment comprises either single-arm dual-arm robot and generated kitchen scenarios containing multiple movable objects and articulated objects or surfaces. The robot required complete tasks based on given natural language instructions. Following Yang et al. (67]. we consider a task where a single-arm robot is required to make chicken soup in this kitchen environment. It is worth noting that this represents highly complex long-horizon task, requiring approximately a dozen sequential actions for successful completion. Specifically, we consider a complex scenario where all cabinet doors are initially closed and the pot is covered with its lid.
We adopt GPT-4o [54
base model for generating sub-goals and apply the default TAMP
approach from Yang et al. [67] to solve the generated sub-goals. Similar to Yang et al. [67], we use Task Completion Percentage all subgoals.
metric, defined the proportion of solved subproblems among
presents performance comparison between models enhanced with proposed HP-KG and those without such enhancement. The results demon- strate that base models enhanced with HP- KG achieve higher Task Completion Per- centage, indicating that our HP-KG can ef- fectively facilitate robot execution of com- plex long-horizon tasks. Additionally, illustrates the subgoals generated by different methods during the experiments. GPT-4o enhanced with our HP-KG can generate more reasonable subgoals, while the baseline model without HP-KG may omit certain essential operations (e.g.. opening cabinets and picking salt and pepper).
Results. Table 13
Table 13: Comparison of Task Completion Percentage for Kitchen Task Execution.
Method
Top-K
Task Completion Percentage
GP [
77.4%
GPT-4o w/PP-KG
Top-3
(+9.5%
GPT-4o w/HP-KG
Top-5
83.5%((6..19) 1%)
present qualitative comparison in Figure which
E Implementation Details of the Automated Graph Construction Framework
Details Verification Rules In our Iterative Verification and Refinement process, we construct comprehensive set of validation rules to evaluate whether the generated procedures adhere to procedural principles. We categorize these rules based on their application types (c.g., tasks, steps, actions) and present several examples in Table
Instruction Template in Construction Framework. We provide the prompt templates used at each stage of our Framework. Specifically, Table[15]and and (6] the prompts for procedure generation and completion. Table 17 shows the prompt that guides the model to aggregate rule verification results and generate corresponding feedback and scores, while the prompt in Table T8 instructs the Refiner to modify procedures based on this feedback. Finally, the Summary Agent employs the prompt in Table 19 to merge redundant procedures. In each stage of our framework,
25

---

Goal: Using chicken and water, make chicken soup.
Environment:
The pot lid is on the pot body.
The chicken leg on the fridge shelf.
Both salt and pepper shakers are the cabinet.
All doors closed.
All knobs and handles turned off.
Generated Planning Resalts
Generated Planning Resaltts
1. Open the fridge docc.
1. Open the tridge door.
12. Place chicken leg inso the por body.
chicken from fridge.
Pick chicken log from fridge.
Sprinklee salt the body.
fridge
Close fiigge
Spriakle pepper
pot
por
GPT-4o
Open cabinet door.
bady.
GPT-4o
Turm the fiucer handle.
chickes ley into the pot body.
Pick popper shaleer.
salli shikerr
17, Turnn on stove knob.
the pot body wore..
Fill the por body with water.
HP-KG
Closn cobinct cloar
Turn off the faucet handle.
Remove the por Lid from body.
Place back the body.
Tur faucet handle.
the
Tum
Figure 9: Qualitative Results in the Kitchen World Environment. GPT-4o enhanced with our HP-KG generates more reasonable subgoals, while the baseline model without HP-KG may omit certain essential operations.
Table 14: Validation rules based on established procedural principles
Procedure Type
Rules
Check if task name contains specific and unambiguous descriptors. Check if task name avoids vague terms. Check if task name clearly indicates the purpose.
Task Procedure
Check step name avoids ambiguous terms descriptors.
Check if description explains purpose and necessity.
Step Procedure
Check if description avoids redundant information.
Check if steps follows logical sequence.
Check if action follows basic structure. Check if action is atomic and independently executable.
Check if description explains specific goal.
Action Procedure
Check if actions follows logical sequence. Check actions follows logical sequence. Check if actions avoids redundancy with other actions.
employ GPT-4o our primary LLM to generate outputs based on the prompts. Notably, for verifying procedures' compliance with rules, we utilize Deepseek-V3. V3,, which offers comparable performance GPT-4o but lower operational cost.
Example of Procedural Graph
We present example of our HP-KG, as illustrated in Figure 10 This details the procedural steps and associated actions for completing the task 'clean broom and remove stains", Each procedure includes three components: name, detailed description, and practical tips.
26

---

Table 15: This designed prompt is used to extract tasks and their corresponding step-by-step instruc- tions from WikiHow documents.
Analyze multiple similar Wikihow examples to extract canonical procedural knowledge.
Input:
[axamples]
Thinking Process:
1. Compare all examples to identify:
Common goal across all tasks Essential steps
in examples
Conflicting instructions (resolve via majority vote)
2. For each step:
the most precise action verb used Note critical tips from all variants
Output following JSON format:
"task": "(specific purposelgoal] [main how-ta]",
"description": "Contextual summary containing: [Operational significance and scope boundaries, Key differentiators from alternative approaches, Optimal application scenarios]",
"lips": "Comprehensive list potential issues, important tips, and warnings", "steps": [[
"step": "[PracisioVVerb] [TargetObject] (specific purpose]".
"description": "Detailed step description including: What actions to take, Why these actions are necessary.
Expected outcomes, Important considerations precautions")
"tips: "Step-specife considerations, timing requirements, measurements, safety precautions"
Please ensure your analysis:
Includes safaty considerations and edge
Provides clear, actionable steps
relevant lips and warnings
step
Provides generalized solutions that can be adapted to various scenarios
Considers different environmental conditions and constraints that might affect the task execution
Naming and Description Requirements:
1. Task Names:
structure: "(specific purpose/goal) [main purpose]"
Use clear, specific, and unambiguous descriptions
Clearly indicate the purpose
2. Task Descriptions:
Explain the purpose importance
Include task-specific tips Highlight key warnings or special requirements
3. Step Names:
Follow logical sequence
Avoid ambiguous terms
why or
Ensure logical flow belween steps
4. Step Descriptions:
Detail specific actions and their purposes
Explain why action necessary
Describe expected outcomes
Provide step-specitic tips precautions
27

---

Table This action completion prompt facilitates the generation of specific actions.
will provide complete task breakdown with the following components:
Task Objective: (task)
Task Description: (description)
Important Tips and Considerations: (tips)
Detailed Steps: (steps]
Please Generate Action For All Steps.
Requirements for Action Generation:
1. Action Requirements
Action sentences structure: [Action] [Object] [Spatial: Object) [Orientation] [Direction[[State]
Required Information
Action:
Description: Detailed explanation including specific goal of this atomic action, Why it's necessary this step, Required preconditions. Expected outcome/sucoess criteria.
Tips: List specific considerations: Salety precautions, Quality control points, Common mistakes avoid, Environmental considerations.
3. Anti- Retu-neddnyacy Guidelines:
Consolidate Actions When: Identical action verbs on similar object types, Same execution methoditoolpatttem, Order foobbleinterchangebblee Common success metrics apply. Single verification works for all items.
Keep Separate When: 1. Difierent tools/methods per item. Strict sequence dependencies exist, 3. Unique
salety protocols per item, 4. Individual verification required, Different quality standards apply.
Please, think through step:
1. What is the goal of this step?
2. What the key constraints and requirements?
3. What safety factors need consideration?
4. What tools or resources are needed?
Then, break down the step into atomic actions, finally provide your response in this JSON format:
"action": "(ection:verbnet verb] (Objectworddet noun]
"tips": "Kay safety and execution tips" "description": "Why this action and what to expect"
Table 17: This prompt is used for rule-based validation, compiling different rules results and generated procedures, then prompting the language model to produce comprehensive feedback and score.
Given validation results from rule checks, please analyze the findings and generate improvement recommendations.
Input:
Original Prompt: [prompt]
Generated Response: (response)
Rule Check Results: (rules checking resuils]
Analysis Tasks:
Review each rule validation result
2. Identify patterns in rule violations
Determine
4. Provide corrections feedback
Please provide your evaluation the following JSON format:
"reason":
"Detailed analysis explaining the scoring rationale key observations", "score": number
"feedback":
"Clear and actionable improvement suggestions", "Each point focusing a specific aspect enhance",
28

---

Table 18: This prompt utilized to optimize generated procedures based on verification results.
Given original response and multiple feedback points, along with their corresponding refined versions, please synthesize a final refined response.
Original Response:/ (ssponse)
Feedback Points: feedss//eed
Rule Cheks:
[rules checking results)
Please analyze all refinements and combine their improvements generate an optimized final response that addresses all feedback points cohesively.
Please return only the final refined response.
Please provide your response refinements in the following JSON format:
"reason": "Detailed analysis of why these retinements are suggested" "refined": (refine result)
Table 19: We employ this prompt synthesize multiple candidate procedures into consolidated summary.
Given multiple semantically similar items, analyze and synthesize them into single. comprehensive and actionable summary.
Input: (candidate lisi)
Follow these steps to analyze and summarize the items:
1. Semantic Analysis:
Iddettfy actions and objectives each item Find common elements and key differences
Extract essential meaning and purpose
2. Content Synthesis:
Use specific verbs and nouns, avoid abstract terms.
Preserve all crilical operational sleps and important details.
Distill redundant information while keeping context specific requirements. Synthesize similar concepts into unilled expressions.
3. Structured Output Requirements:
Title: Use concrete verbe that reflect the main task, include both the action and the target object. keep brief but specitic, make immediately understandable.
purpose and scope, detail specific steps mathods, include key conditions requirements, explain the expected outcome, use precise, actionable language.
Descrrition: Start with the
List specific, executable actions, include crucial warnings requirements, focus practical guidance, avoid vague suggestions, synthesize similar tips into stronger, unitied points.
-Tips:
4. Quality Verification:
Ensure caplure all essential information from original items.
Ensure all instructions specific and actionable.
Ensure all content logically organized.
Ensure language clear and concrete.
29

---

Action Proccdure
mecilption cim tite
Action Prodscure
the
Tee: maue stratt.
Nac
Tip man ve raro of's
oe Tee
clate frem
mnants side
(tais....
Action Przzadure
Flom uree wungg were
brow
Noe
Action Procedum
tha
Seee
YEAR
Procodure
Hoorkter mnterr on
San regesate Lieh
Ties sundaa mesie metrrse
Action Procedure
Ture desengen terttee
Action Procedur
Proccdure
Action Procedure
Bilig reneve
No
Tiass Bling cay wiill
socuaiee Ioporiean...
Peeiigg.
Hap roved or oen
Figure 10: An example of cleaning and removing stains from brooms in our procedural knowledge graph.
30


Reviews:
[{"review_id": "iWTpU1zzyx", "reviewer": "Reviewer_A3V5", "summary": "This paper aims to address the limitations of LLMs in complex robotic manipulation tasks. It proposes a novel approach that utilizes a Hierarchical Procedural Knowledge Graph (HP-KG) with web \"how-to\" data (e.g., wikiHow). The HP-KG captures hierarchical relationships between high-level tasks, subtasks, and fine-grained actions, while also providing graph information retrieval methods. This framework is designed to provide LLMs with more accurate procedural knowledge and reduce the energy consumption associated with deploying large-scale LLMs only. Experimental results demonstrate that HP-KG significantly enhances LLM planning capabilities, and show how this scales as the model size increases.", "questions": "1. Currently, structured information is provided to LLMs only at the textual level. Is it possible to integrate graphical representations and directly input them to the LLM for better performance?\n\n2. How does the system handle tasks that require knowledge outside the graph, or tasks that necessitate dynamic replanning in response to unforeseen environmental events?\n\n3. Can the HP-KG framework be extended to other modalities (e.g., visual observations, tactile feedback) to further enhance planning robustness in real-world robotic environments?", "limitations": "1. The study is limited to household activities and specific robotic arms, constraining its applicability in general scenarios.\n\n2. Although the paper proposes a method for HP-KG construction, this process might become a bottleneck when generating knowledge graphs covering an extremely wide range of complex manipulation tasks.\n\n3. Reliance on wikiHow data might introduce biases or limitations due to the specificity of information. Its generalizability to tasks not well-covered by such web data might be limited.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The design utilizes web \"how-to\" data (e.g., wikiHow) to construct and use a Hierarchical Procedural Knowledge Graph (HP-KG), making LLM planning more efficient and reliable.\n2. By providing external knowledge, HP-KG alleviates the high energy consumption associated with deploying large-scale LLMs for knowledge acquisition.\n3. The paper provides reliable experimental evidence on robot manipulation tasks, validating the consistent effectiveness of the HP-KG method compared to baseline methods.\n\n## Weaknesses \n1. The quality of data automatically constructed by the LLM Agents has not been verified, nor have any examples been shown.\n2. Despite the experiments conducted, it is now unclear how robust the generated plans are to real-world settings (uncertainties, noise, or unexpected environmental changes), which are common in robot manipulation. The planning might be highly dependent on how well the environment representation matches the HP-KG's knowledge.", "was_revised": true, "final_justification": "The paper makes a contribution to the research community, with good results and explainability. The authors have also provided timely and thorough responses to the reviewers’ comments.", "rebuttal": "- **W1: LLM agent-generated data lacks quality verification and examples.**\n\nThank for your valuable comment. we conduct experiments to verify the quality of our generated data. Specifically, we sample a subset(100task and 864 steps) of our HP-KG. We use three data quality metrics, including truly atomic actions precision (AAP), step ordering precision (SOP), and actions ordering precision (AOP).\n\nTo measure the atomic actions precision, we extracted all verbs from the HP-KG and manually created an atomic verb dictionary containing only atomic verbs (e.g., pick, turn on) while excluding non-atomic ones (e.g., arrange). We then calculated the percentage of atomic verbs in the generated actions based on this dictionary. For Steps Precision and Action Precision, we annotated the correct steps/actions ording and calculate the average percentage of correctly ordered steps and actions, respectively. Detailed results are presented in following Table. Experiments show that our LLM Agents can generate reasonable atomic actions and acceptable steps, helping to bridge high-level human instructions with atomic actions that robots can understand.\n\n||AAP |SOP|AOP| \n|-|-|-|-|\n|HP-KG|82.39%|86.92%|85.78%|\n\nTo better illustrate our HP-KG approach, we include a graph example in Figure 5 and detailed graph statistics in the Appendix of the Supplementary Material. We hope these supplementary materials may offer helpful context for understanding our HP-KG.\n\n- **W2: it is now unclear how robust the generated plans are to real-world settings(uncertainties, or unexpected environmental changes).**\n\nThank you for your comments. We conduct experiments to simulate real-world environmental variations and uncertainties, demonstrating the robustness of our HP-KG. \n\nSpecifically, we utilize the Counterfactual Planning split from Actplan-1K[1], which assesses the model's ability to adapt reasoning and planning when task scenarios involve modified conditions or constraints that frequently arise in real-world applications. We present results in the following Table, which demonstrate that our approach successfully leverages procedural commonsense to enable robust LLM planning capabilities even in uncertainty scenarios.\n\n|Model|Top-k|Average LCS↑| \n|-|-|-|\n|Qwen2-VL-7B|-| 8.51|\n|Qwen2-VL-7B + HP-KG|top-3|9.12|\n|Qwen2-VL-7B + HP-KG|top-5|8.86|\n|Qwen2-VL-72B|-| 9.16|\n|Qwen2-VL-72B + HP-KG|top-3|9.34|\n|Qwen2-VL-72B + HP-KG|top-5|9.76|\n\nFurthermore, we will simulate more real-world settings (noise, unexpected environments) in future experiments to demonstrate the robustness of our approach.\n\n- **Q1: Is it possible to integrate graphical representations and directly input them to the LLM.**\n\nYes, it is possible to directly integrate graphical representations into LLM. Howerver integrating graphical representations and directly inputting them to LLMs requires training the models to adapt to graphical inputs, which incurs high computational costs and may lead to catastrophic forgetting[3] of the LLMs' existing knowledge. Instead, we convert retrieved structured procedural information into textual context for LLM input. This method enables seamless integration without architectural changes, leverages existing reasoning capabilities of LLMs for procedural knowledge application, and maintains interpretability across diverse procedural formats[2]. Due to time constraints, we are unable to fine-tune an LLM for direct graph integration and will compare such methods in future revisions.\n\n\n- **Q2: How does the system handle tasks that require knowledge outside the graph, or tasks that necessitate dynamic replanning.**\n\n\nIn our experiments, the HP-KG is constructed from how-to data that is distinct from the benchmark datasets (Actplan-1K[1], RLBench[4]) used for evaluation, which demonstrates the generalization capability of our approach. The HP-KG contains a rich collection of reusable atomic actions that serve as fundamental building blocks. Within the household domain, these action units can be flexibly combined and reconfigured to create diverse procedural sequences, enabling effective generalization across new environments, unseen tasks, and varying contextual conditions. However, it may fail in scenarios that are completely outside the graph.\n\n\n- **Q3: Can the HP-KG framework be extended to other modalities.**\n\nYes, our HP-KG has the potential to be extended to other modalities. MLLMs have demonstrated strong visual understanding capabilities [6], and research has shown that providing additional visual cues can help MLLMs better comprehend the world and enhance their reasoning abilities [7]. We envision incorporating visual imagery and manipulation videos into our HP-KG, which would allow for instruction-based retrieval of relevant task operation diagrams or videos. Providing such visual inputs to MLLMs might improve their understanding of sequential operations and procedural tasks. We plan to explore the integration of visual modalities into our HP-KG in future research.\n\n\n- **L1: The study is limited to household activities**\n\nThank you for your insightful feedback. We have also acknowledged and discussed this limitation in the Limitation Section.\n\n- **L2: HP-KG construction might become a bottleneck when generating knowledge graphs covering an extremely wide range of complex manipulation tasks**\n\nThank you for your insight. Our construction framework may not be able to cover extremely complex manipulation tasks.\n\n- **L3: Reliance on wikiHow data might introduce biases or limitations due to the specificity of information. Its generalizability to tasks not well-covered by such web data might be limited.**\n\nThank you for your valuable comment. Web data may not be able to ensure good generalizability across diverse tasks. We plan to improve the generalization of our knowledge graph in future work.\n\n\n\n[1] ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities. EMNLP 2024.\n\n[2] G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering. Neurips 2024.\n\n[3] Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal. ACL 2024.\n\n[4] RLBench: The Robot Learning Benchmark & Learning Environment. IEEE Robotics Autom. Lett.\n\n[5] Exploring Large Language Models for Knowledge Graph Completion. ICASSP 2025.\n\n[6] Qwen2.5 Technical Report. Arxiv 2412.15115.\n\n[7] Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers. Arxiv 2506.23918."}, {"review_id": "WToMY2fLNh", "reviewer": "Reviewer_uR6N", "summary": "The paper tackles an important task of making the robot manipulation both flexible thanks to LLM model and following the given procedure by utilizing knowledge graphs. The introduced hierarchical knowledge graphs when integrated with LLM outperform compared former models as well as CoT on most of the selected household tasks in RLbench and Actplan. It significantly advances performance especially of the smaller LLM models.", "questions": "1) p.5 l193 feedback, textual results descriptions - what these mean, where they are collected and how are these utilized?\n2) how do you make sure not to loose some important rule or information by summary agent? two similar procedures might differ only in one small, but important parameter\n\n3) why in Table 1/2 top 5 sometimes performs worse and sometimes better than top 3? why CoT drops performance in Table 2 for slide block compared to RVT? why top 3/5 sometimes performs even worse than RVT? are there some typical problems?\n\n4) table 1/2 shows mainly pick and place tasks, what about more complex tasks like preparing table or tasks like screwing, wiping, and longer tasks like tidying up a table? could you show results on these or discuss limitations of your approach?", "limitations": "Limitations are very sparse, mentioning only extension outside household tasks.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper is very nicely written, clearly structured and provides several meaningful evaluations including inference time comparison. It also tackles an important issue how to make open vocabulary models more efficient by utilizing contextual knowledge. The proposed hierarchical knowledge graphs are clearly described and meaningfully designed.\n\nWeaknesses: \nthe paper seem to focus evaluation on pick and place tasks. it is not clear how more complex tasks with different parameters (trajectory to follow, pressure, etc) and dexterous manipulations could be represented and how well it performs.\n\nIt should be clearer described in which cases it decreases performance compared to model without it or compared to CoT and why.\n\nthe evaluation is comparing different methods in different cases, making it difficult to compare.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable insights and comprehensive review. Your thoughtful comments will greatly enhance our manuscript.\n\n- **W1: Validation on complex tasks and performance remain unclear.**\n\nwe conduct comprehensive experiments on VLABench[1], a more complex benchmark comprising eight distinct manipulation skills (e.g., Insert, Hang, Twist) that extend beyond basic pick-and-place operations. These tasks require comprehensive commonsense knowledge, understanding of physical principles, and sophisticated reasoning capabilities to solve effectively. VLABench can evaluate not only VLAs but also MLLMs. In the MLLM evaluation setting, MLLMs are required to output the skills to be invoked and the corresponding parameters for each skill. \n\nWe select four task categories (CommonSense, Complex, M&T, PhysicsLaw), with each category containing 4-8 tasks. CommonSense, M&T, and PhysicsLaw categories evaluate the model's understanding of world procedural knowledge, while Complex categoriy, beyond world knowledge, demand long-horizon planning. We compare the accuracy of MLLMs with and without our HP-KG enhancement, and we run each task for 50 trials. Results are presented in the following table.\n\n|Models|Top-k|CommonSense|M&T|PhysicsLaw|Complex|\n|-|-|-|-|-|-|\n|Intern3-VL-8B|-|26.15 |28.19 |10.21|19.01|\n|Intern3-VL-8B + HP-KG |3|28.70|29.21|18.95|21.47|\n|Intern3-VL-8B + HP-KG |5|29.55|28.7|18.22|21.16|\n\nResults demonstrate that our HP-KG can enhance MLLMs' planning capabilities by injecting procedural commonsense into the LLM context, thereby providing structured prior knowledge that facilitates robust planning in complex tasks.\n\nAdditionally, our HP-KG framework is primarily designed for providing procedural commonsense to LLM context and facilitating more reasonable planning, rather than addressing dexterous manipulation benchmarks such as [3,4]. These benchmarks primarily evaluate models' ability to perform precise finger movements, coordinated multi-finger grasping, and object manipulation. Therefore, our method is not suitable for dexterous manipulations task.\n\n- **W2: It should be clearer described in which cases it decreases performance compared to model without it or compared to CoT and why.**\n\n1. On the Slide Blocks task, our method with top-5 retrieval yields lower performance compared to the approach without HP-KG. The task's ambiguous instructions result in more irrelevant information being retrieved, thereby degrading planning quality.\n\n2. Our method shows comparable performance to COT on Take_lid and Take_scale tasks. This may be due to the inherent limitations of the RVT model on these two tasks, where neither our method nor COT achieves satisfactory performance.\n\n\n- **W3: the evaluation is comparing different methods in different cases, making it difficult to compare.**\n\nTable 1 and Table 2 compare different types of methods. In Table 1, we present a comparison of zero-shot robotic manipulation methods on RLBench. In Table 2, we validate whether our method can enhance pretrained action models. Given that zero-shot methods and pretrained action models appear to be better suited for different types of tasks, we selected slightly different test cases for each comparison. Since our main comparison is between methods with and without our HP-KG enhancement, our experimental setup is fair.\n\n- **Q1: p.5 l193 textual results descriptions,feedback - what these mean, where they are collected and how are these utilized?**\n\n**what these mean:** 'textual results descriptions' refers to the consolidated textual description that combines all rules with their respective verification outcomes. 'feedback' represents the LLM's assessment of the given procedure $p$, identifying potential issues.\n\n**where they are collected:** 'textual results descriptions' are collected by converting each rule verification result along with its original rule text into a structured textual format (e.g., \"rule:{rule_check_result}\"), and then concatenating all these formatted descriptions to form the final comprehensive textual results. 'feedback' is obtained by feeding these textual results descriptions along with procedure $p$ to an LLM, which then generates the final score and corresponding feedback.\n\n**how are these utilized:** The 'textual results descriptions' are fed into an LLM along with procedure $p$ to generate the final score and corresponding feedback. 'feedback' is subsequently fed to the Refiner, which modifies procedure $p$ based on the feedback.\n\n\n- **Q2: how do you make sure not to loose some important rule or information by summary agent?**\n\nIn Table 6, the results show that has been processed by the Summary Agent further enhances the planning capacity of the Base Model, suggesting that our summary agent largely retains important information. Additionally, we selectively summarize procedures with high similarity, which is a common strategy for eliminating redundancy and addressing distributional imbalance, thus reducing the knowledge imbalance[2].\n\n\n- **Q3: why in Table 1/2 top 5 sometimes performs worse and sometimes better than top 3? why CoT drops performance in Table 2 for slide block compared to RVT? why top 3/5 sometimes performs even worse than RVT? are there some typical problems?**\n\n1. In our experiments, we also observe this phenomenon. Under the models shown in Tables 1 and 2, retrieving more information (Top-5) may introduce irrelevant information during the model's planning process. This suggests that current models may have limited capacity to effectively filter and utilize extensive retrieved information, thereby degrading planning quality and leading to performance decline with Top-5 retrieval. Our experiment in Table 4 demonstrates that more advanced models can effectively handle Top-5 retrieval without performance degradation.\n\n2. Due to the ambiguous instructions(e.g., 'slide the block to target') in Slide Blocks task, COT may generate suboptimal planning, leading to performance degradation.\n\n3. In Slide blocks task, our method with Top-3 retrieval achieves superior performance compared to the RVT. However, Top-5 retrieval yields lower performance, likely due to irrelevant information introduced by retrieving more content, thereby degrading planning quality.\n\n4. As shown in Table 2, our method with Top-3 retrieval outperforms the baseline RVT model in all cases. Our method with Top-5 retrieval shows decreased performance in only one case, where ambiguous instructions cause the retrieval process to introduce irrelevant information, thereby degrading planning quality.\n\n\n- **Q4: what about more complex tasks? could you show results on these or discuss limitations of your approach?.**\n\nWe conduct experiments on more complex tasks using VLABench. Detailed experimental results can be found in **W1**.\n\n- **L1: Limitations are very sparse, mentioning only extension outside household tasks.**\n\nThank you for your insightful feedback. Our work may also face limitations in constructing reasonable task planning when dealing with an extremely complex manipulation tasks, due to the limited planning and reasoning capabilities of MLLMs. Furthermore, our HP-KG, which is derived from how-to data, may not generalize well to all complex manipulation scenarios.\n\n\n[1] VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks. ICCV 2025.\n\n[2] Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI. IEEE Trans. Knowl. Data Eng. 36(11)\n\n[3] DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects. CVPR 2023.\n\n[4] DexH2R: A Benchmark for Dynamic Dexterous Grasping in Human-to-Robot Handover. ICCV 2025."}, {"review_id": "v99b1PP5Yi", "reviewer": "Reviewer_LNsU", "summary": "This paper introduces Hierarchical Procedural Knowledge Graphs (HP-KG) as a means to enhance the planning capabilities of Large Language Models (LLMs) for complex robotic manipulation tasks, while also aiming to reduce reliance on large-scale models. The authors identify two key limitations in current LLM-driven manipulation systems: (1) a lack of accurate procedural knowledge for complex tasks and (2) the inefficiency and high energy cost of deploying large LLMs in real-world robotic systems.\n\nTo address these issues, the proposed HP-KG models the hierarchical structure of tasks — breaking them down into steps and atomic robotic actions — thereby aligning human instructions with machine-executable procedures. The knowledge graphs are automatically constructed using an LLM-based multi-agent system, reducing the need for manual annotation while maintaining quality. The resulting dataset includes over 40,000 activity steps covering more than 6,000 household tasks.\n\nEmpirical results suggest that small-scale LLMs (e.g., 7B parameters) augmented with HP-KG outperform significantly larger models (e.g., 72B) in planning tasks, and the benefits also transfer to state-of-the-art models like GPT-4o.", "questions": "N/A", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Summary of Strengths\n1.\tThis paper is well written and clearly organized, making it easy to follow.\n\n2.\tThe idea of using procedural knowledge graphs to enhance the planning of LLMs is straightforward and experimental results well supported the results.\n\n3.\tI like the idea of constructing the knowledge graph from the wikihow dataset, the authors promised to release the knowledge graph, so it would be helpful for the community to build more powerful LLM planning modules for embodied AI.\n\n## Summary of Weaknesses\n1.\tThe anonymous link didn’t provide any code or data for us to review.\n\n2.\tThe lack of experiments or demos in real-world robots, which could decrease the contribution of the world. Real-world scenarios are more complicate and it would be nice to test the robustness of HP-KG there.", "was_revised": true, "final_justification": "Overall, I think it's a good paper. My concerns about the anonymous link and the real robot deployment are not resolved, but I think it's acceptable.", "rebuttal": "- **W1: The anonymous link didn’t provide any code or data for us to review.**\n\nThank you for your valuable feedback. Due to NeurIPS's policy this year, we cannot update our repository or submit any links during the rebuttal period. We commit to updating our repository and releasing the data immediately after the rebuttal period ends to further facilitate Procedural Knowledge development and research.\n\nTo better illustrate our HP-KG approach, we include a graph example in Figure 5 and detailed graph statistics in the Appendix of the Supplementary Material. We hope these supplementary materials may offer helpful context for understanding our HP-KG.\n\n- **W2: The lack of experiments or demos in real-world robots, which could decrease the contribution of the world.**\n\nThank you for your sincere comment. Due to time and cost constraints, we have not yet deployed our approach to real-world robots. We will further extend our HP-KG to real-world applications in future work."}, {"review_id": "93wAGyZAEb", "reviewer": "Reviewer_VoF5", "summary": "The paper proposes HP-KG, a large-scale hierarchical procedural knowledge graph designed to enhance LLM-based planning for complex manipulation tasks while allowing the use of smaller LLMs.", "questions": "- Could you provide automatic or human-annotated precision/coverage statistics (e.g., % of actions truly atomic, % of step ordering correct) for HP-KG?\n\n- Do you foresee the same structure and pipeline scaling to industrial assembly or navigation tasks? Any preliminary results or challenges?\n\n- Also, I would encourage the trials on more challenging manipulation benchmarks like RoboCerebra.\n\n- Please report average retrieval + sub-graph conversion time and memory footprint for top-3/top-5 on ActPlan-1K, and discuss impact on total inference.\n\n- To mitigate potential bias from GPT-4o judging, could you include a small human study validating that HP-KG plans are indeed more feasible/executable?\n\n- The KG Retrieval reminds me the the in-context prompts used in CaP or Instruct2Act, can the authors provide more discussions. \n\n- Overall, due to the limited task complexity in the chosen RLBench, it did not fully demonstrate the effectiveness of the promised KG.", "limitations": "yes", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The overall method is well-motivated where hierarchical KG that directly aligns language instructions with executable robot actions, filling the abstraction gap.\n- Fully automated construction via LLM agents with rule feedback reduces manual effort and produces a sizable, publicly promised dataset.\n- Model-agnostic enhancement – works with small (7 B) and large (GPT-4o) planners, yielding both quality gains and inference-time savings.\n\n## Weakness\n\n -  focuses on table-top tasks; transfer to industrial or other domains untested.\n\n-  although clustering/verification exists, no quantitative quality metrics (precision/consistency) for the final KG are reported. \n\n-  author mentioned \"safety-critica\"  in Line 125, but no further information provided. \n\n- embedding + K-hop search latency on large graphs is not profiled", "was_revised": true, "final_justification": "Overall, I think this is a good paper in the view of agent-style VLA to enhance its reasoning or planning ability. My main concern still holds, I thought it would greatly enhance the paper's strength when the authors demonstrated the effectiveness where the proposed KG would further boost the performance of the current SOTA VLA like Pi-0 on the large-scale and more challenging benchmarks, like LIBERO-10 and others. \n\nIn view of this, I am currently holding my current score, but would like to lower my confidence.", "rebuttal": "- **W1: Focuses on table-top tasks; transfer to industrial or other domains untested.**\n\nIt is possible that our pipeline could be adapted to other domains such as industrial assembly. Our core methodology may be applicable provided that suitable large-scale datasets are available for these domains. However, due to time constraints during the rebuttal period, we do not have sufficient time to construct new large-scale knowledge graphs, as this process requires substantial data collection and curation efforts. As mentioned in our limitations, the study is limited to household activities. Different domains would likely require domain-specific adaptations in entity types, relationship categories, etc.\n\n\n- **W2: No quantitative quality metrics (precision/consistency) for the final KG are reported.**\n\nWe conduct experiments to verify the quality of our generated data. Specifically, we sample a subset(100task and 864 steps) of our HP-KG. We use three data quality metrics, including truly atomic actions precision (AAP), step ordering precision (SOP), and actions ordering precision (AOP).\n\nTo measure the atomic actions precision, we extracted all verbs from the HP-KG and manually created an atomic verb dictionary containing only atomic verbs (e.g., pick, turn on) while excluding non-atomic ones (e.g., arrange). We then calculated the percentage of atomic verbs in the generated actions based on this dictionary. For Steps Precision and Action Precision, we annotated the correct steps/actions ording and calculate the average percentage of correctly ordered steps and actions, respectively. Detailed results are presented in following Table. Experiments show that our LLM Agents can generate reasonable atomic actions and acceptable steps, helping to bridge high-level human instructions with atomic actions that robots can understand.\n\n||AAP |SOP|AOP| \n|-|-|-|-|\n|HP-KG|82.39%|86.92%|85.78%|\n\nTo better illustrate our HP-KG approach, we include a graph example in Figure 5 and detailed graph statistics in the Appendix of the Supplementary Material. We hope these supplementary materials may offer helpful context for understanding our HP-KG.\n\n\n- **W3: author mentioned \"safety-critica\" in Line 125, but no further information provided.**\n\nThe term 'safety-critical' refers to the 'safety-critical guidelines' mentioned in Line 125. These are essential safety warnings and precautionary instructions that our procedure should incorporate to alert robot to potential accidents during procedure execution. Following this principle, we incorporated a Tips attribute when defining the properties of the Procedure. An example of a Tip is: \"Handle the notebook carefully to avoid bending or tearing pages.\n\n- **W4: embedding + K-hop search latency on large graphs is not profiled.**\n\nWe conduct experiments to evaluate the efficiency of our retrieval method. Specifically, we measure the average Instruction Encode Time (Encode Time),Retrieval Time, K-hop Time, Reranking Time, Graph Conversion Time and LLM  Inference Time during the Top-3 and Top-5 search processes on the Actplan-1K[1]. \n\nThe results are presented in the following table. \n\n|Top-k|Encode Time |Retrieval Time|K-hop time|Reranking Time|Graph Conversion Time|LLM Inference Time |\n|-|-|-|-|-|-|-|\n|Top-3|0.038s|0.045s|0.001s|0.025s|0.001s|1.28s|\n|Top-5|0.039s|0.046s|0.001s|0.027s|0.001s|1.49s|\n\nResults show that our retrieval process is highly efficient, with only minimal additional time overhead that is significantly less than MLLM inference time.\n\n\n- **Q1: Could you provide automatic or human-annotated precision/coverage statistics (e.g., % of actions truly atomic, % of step ordering correct) for HP-KG?.**\n\nThank you for your valuable comment. We have conducted experiments to evaluate the precision and coverage statistics of HP-KG. Detailed results can be found in **W2**\n\n\n- **Q2: Do you foresee the same structure and pipeline scaling to industrial assembly or navigation tasks? Any preliminary results or challenges?**\n\nIt is possible that our pipeline could be adapted to other domains such as industrial assembly or navigation tasks. The core methodology may be applicable provided that suitable large-scale datasets are available for these domains. However, due to time constraints during the rebuttal period, we do not have sufficient time to construct new large-scale knowledge graphs, as this process requires substantial data collection and curation efforts.\n\nAs mentioned in our limitations, the study is limited to household activities. Different domains would likely require domain-specific adaptations in entity types, relationship categories, etc.\n\n\n- **Q3: Also, I would encourage the trials on more challenging manipulation benchmarks like RoboCerebra.**\n\nThank you for your insight. We further conducted experiments on a more complex benchmark. Since the RoboCerebra benchmark is not open source, we conducted our experiments on VLABench[2]. \n\nVLABench, which contains multiple complex tasks that involve common sense knowledge, physical rules, and reasoning capabilities. VLABench can evaluate not only VLAs but also MLLMs. In the MLLM evaluation setting, MLLMs are required to output the skills to be invoked and the corresponding parameters for each skill. \n\nWe select four task categories (CommonSense, Complex, M&T, PhysicsLaw), with each category containing 5-8 tasks. CommonSense, M&T, and PhysicsLaw categories evaluate the model's understanding of world procedural knowledge, while Complex categoriy, beyond world knowledge, demand long-horizon planning. We compare the accuracy of MLLMs enhanced with our HP-KG against those without such enhancement, with each task run for 50 trials. Results are presentsed on following Table.\n\n|Models|Top-k|CommonSense|M&T|PhysicsLaw|Complex|\n|-|-|-|-|-|-|\n|Intern3-VL-8B|-|26.15 |28.19 |10.21||\n|Intern3-VL-8B + HP-KG |3|28.70|29.21|18.95|\n|Intern3-VL-8B + HP-KG |5|29.55|28.7|18.22|\n\nResults demonstrate that our HP-KG can enhance MLLMs' planning capabilities by injecting procedural commonsense into the LLM context, thereby providing structured prior knowledge that facilitates robust planning in complex tasks.\n\n\n- **Q4: Please report average retrieval + sub-graph conversion time and memory footprint for top-3/top-5 on ActPlan-1K, and discuss impact on total inference.**\n\nThank you for your valuable comment. We have conducted experiments to evaluate the time performance of the Top-k retrieval process. Detailed results can be found in **W4**. The results demonstrate that our retrieval process is highly efficient, incurring only minimal additional time overhead that is significantly less than MLLM inference time\n\n\n- **Q5: To mitigate potential bias from GPT-4o judging, could you include a small human study validating that HP-KG plans are indeed more feasible/executable?.**\n\nWe conduct experiments to verify the feasibility of our plans of HP-KG. Specifically, we sample a subset (100 tasks) from our HP-KG and manually annotate whether each action is feasible or executable for the robot. We then calculate the average percentage of feasible actions, denoted as the feasible action rate (FAR).\n\nDetailed results are presented in following Table. Experiments show that our LLM Agents can generate feasible atomic actions, helping to bridge high-level human instructions with atomic actions that robots can understand.\n\n|| FAR|\n|-|-|\n|HP-KG| 71.54%|\n\n\n\n- **Q6: The KG Retrieval reminds me the the in-context prompts used in CaP or Instruct2Act, can the authors provide more discussions**\n\nInstruct2Act presents an agent-based robotic manipulation framework that generates Python code based on task instructions, leverages this code to invoke vision tools (such as SAM) for environmental perception, and subsequently generates plausible robotic actions for low-level controllers to execute.\n\nOur approach is orthogonal to these agent-based robotic manipulation frameworks and provides complementary capabilities. The constructed HP-KG can be seamlessly integrated into various planning generation systems, offering procedural commonsense knowledge to facilitate more rational action planning.\n\n\n- **Q7: Overall, due to the limited task complexity in the chosen RLBench, it did not fully demonstrate the effectiveness of the promised KG.**\n\nWe conduct experiments on more complex tasks using VLABench. Detailed experimental results can be found in **Q3**.\n\n\n[1] ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities. EMNLP 2024.\n\n[2] VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks. ICCV 2025."}]

Author rebuttal:

Meta-review: This paper introduces Hierarchical Procedural Knowledge Graphs (HP-KG) to enhance LLM planning for robotic manipulation by bridging high-level instructions to atomic robot actions through structured web "how-to" data. Four reviewers provided ratings of 4, 5, 5, and 3, with substantive concerns about data quality, real-world robustness, and practical effectiveness that the authors largely addressed through comprehensive rebuttals including quality metrics experiments, counterfactual planning validation, and additional experiments demonstrating benefits beyond simple pick-and-place tasks. While Reviewer VoF5's core concern about integration with current SOTA VLA models on established benchmarks remains unresolved, and the work is limited to household manipulation tasks with no real robot deployment, the technical contribution is sound with thorough experimental validation within its defined scope. The automatic knowledge graph construction framework represents a novel solution to procedural knowledge limitations in LLM-based planning, with clear benefits demonstrated for smaller models (7B) outperforming larger ones (72B), justifying acceptance based on technical merit and potential to establish foundations for future research despite acknowledged domain limitations.
Decision: Accept (poster)