NOD-TAMP: Generalizable Long-Horizon Planning with Neural Object Descriptors
Authors: Shuo Cheng
Caelan Reed Garrett
Ajay Mandlekar
Danfei Xu
Venue: CoRL 2024 (2024)
Keywords: Robot Learning
Robot Planning
Manipulation
TL;DR: We introduce NOD-TAMP, a TAMP-based framework that can solve broad long-horizon manipulation tasks by adapting and composing short manipulation trajectories from a handful of human demonstration.

Abstract: Solving complex manipulation tasks in household and factory settings remains challenging due to long-horizon reasoning, fine-grained interactions, and broad object and scene diversity. Learning skills from demonstrations can be an effective strategy, but such methods often have limited generalizability beyond training data and struggle to solve long-horizon tasks. To overcome this, we propose to synergistically combine two paradigms: Neural Object Descriptors (NODs) that produce generalizable object-centric features and Task and Motion Planning (TAMP) frameworks that chain short-horizon skills to solve multi-step tasks. We introduce NOD-TAMP, a TAMP-based framework that extracts short manipulation trajectories from a handful of human demonstrations, adapts these trajectories using NOD features, and composes them to solve broad long-horizon, contact-rich tasks. NOD-TAMP solves existing manipulation benchmarks with a handful of demonstrations and significantly outperforms prior NOD-based approaches on new tabletop manipulation tasks that require diverse generalization. Finally, we deploy NOD-TAMP on a number of real-world tasks, including tool-use and high-precision insertion. For more details, please visit https://nodtamp.github.io/.

NOD-TAMP: Generalizable Long-Horizon Planning with Neural Object Descriptors
Shuo Cheng , Caelan Garrett" 2, Ajay Mandlekar" Danfei Xu'
GGoorgia Institute of Technology "NVIDIA Corporation
Abstract: Solving complex manipulation tasks in household and factory settings remains challenging due to long-horizon reasoning, fine-grained interactions, and broad object and scene diversity. Learning skills from demonstrations can be an effective strategy, but such methods often have limited generalizability be- yond training data and struggle solve long-horizon tasks. To overcome this, we propose synergistically combine two paradigms: Neural Object Descrip- tors (NODs) that produce generalizable object-centric features and Task and Mo- tion Planning (TAMP) frameworks that chain short-horizon skills to solve multi- step tasks. We introduce NOD-TAMP, a TAMP-based framework that extracts short manipulation trajectories from a handful of human demonstrations, adapts these trajectories using NOD features, and composes them solve broad long- horizon, contact-rich tasks. NOD-TAMP solves existing manipulation bench- marks with handful of demonstrations andsignificantly outperforms prior NOD- based approaches new tabletop manipulation tasks that require diverse gen- eralization. Finally, we deploy NOD-TAMP on a number of real-world tasks, including tool-use and high-precision insertion. For more details, please visit https: //oo://noddaagg aerodaamp.iitttee github........ .i/.
Keywords: Robot Learning, Robot Planning, Manipulation
Introduction
From children playing with Lego blocks adults rearranging a room, our remarkable ability to plan long sequences of actions to achieve our goals is still beyond the capabilities current robots. Consider the challenges involved in daily tabletop tasks shown in Fig. 1. First, these tasks are often long-horizon and full of sequential dependencies. Here, the robot must reason about the best pose to grasp mug in order to stow it in cabinet along with other steps to organize the entire table. Second, steps such as placing the mug in a tight cabin or stowing the screwdriver on the tool rack require intentional contact, which can render most motion planners that focus on avoiding collisions ineffective [1]. Finally, to effective across broad environments, robot must handle a wide variation of object shapes and scene layouts.
Task and Motion Planning (TAMP) [2. 3] is an effective approach for such problems because it can effectively resolve sequential dependencies through hybrid munbolic-contnnouu reasoning. However, TAMP systems typically require accurate, special-purpose perception systems and hand- engineered manipulation skills. Thus, it is difficult to apply them to unseen objects and tasks that require complex motion trajectories. Recent works have proposed learn manipulation skills from demonstration [4, 5] partially relax these constraints. However, their generalization ability re- mains bounded by the training data, which is costly to collect scale [6].
By contrast, neural representation models have shown remarkable potential in enabling generalizable manipulation systems [7, 8, 9, 10]. In particular, Neural Object Descriptors (NODs) [8. 11, 12] are a powerful tool to extract dense, part-level features that generalize across object instances. Neural Descriptor Fields (NDFs) [8], a type of NOD that encodes SE(3) poses relative to given object, can
"Equal Contribution.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

One Demo for
Pre-tratning
Perform Broad Ramge of Tasks
NOD
Long-horkeon goals
Fine grained mations
tree)
NOD-TAMP
configurations
geometries
Figure I: Overview. NOD-TAMP is TAMP-based framework that adapts demonstration trajecto- ries to new situations to accomplish long-horizon, fine-grained tasks.
adapt key-frame poses (e.g. grasps) for one object instance to others in the same object category (e.g. mugs), thereby achieving category-level generalization. However, existing NOD-based methods [8, 13, 14] are limited to adapting individual key-frame poses and thus struggle tasks involving complex motion and multi-step reasoning.
In this paper, we propose combine these complementary paradigms and introduce NOD-TAMP, TAMP-based framework that extracts adaptable skills from handful of human demonstrations ing NOD features and composes them solve long-horizon tasks. Central to NOD-TAMP is skill reasoning module that composes short-horizon skills to solve novel long-horizon goals that were never demonstrated, thereby achieving compositional generalization. To synthesize fine-grained manipulation trajectories for new objects, we propose a NOD-based trajectory adaptation module that can consistently adapt recorded skill trajectory according the observed objects. Finally, NOD-TAMP flexibly integrates the adaptation of recorded trajectories with traditional motion plan- ning generalize across drastically different scene layouts.
We empirically evaluate NOD-TAMP on many simulated multi-step manipulation tasks that test different factors of generalization across long-horizon tasks, including object shapes, number of objects, scene layout, task length, and task objectives. We find that NOD-TAM can solve existing manipulation benchmarks [15], with a fraction (4 vs. 500 demos) the data required by behavioral cloning methods. On new task suite that stress-test generalization capabilities, NOD-TAMP also outperforms other existing methods [8, 16]. some of which share subset of its traits, highlighting the value of building cohesive manipulation planning system. Finally, successfully demonstrate NOD-TAMP on 6 real-world manipulation tasks.
Related Work
TAMP. Task and Motion Planning (TAMP) is a powerful paradigm for addressing long-horizon manipulation challenges by decomposing complex planning problem into series of simpler sub- problems [2, 17, 18. 19, 3]. Nonetheless, TAMP techniques presuppose knowledge of the object models and the underlying system dynamics. Such presuppositions can be limiting, particularly for domains with diverse objects and complex physical processes such as contact-rich manipulation.
Learning for TAMP. Recent works have set to address such limitations by replacing hand-crafted components in TAMP system with learned ones. Examples include environment models [20, 21, 22, 23, 24], object relationships [25, 26, 27]. skill operator models [28, 4] skill samplers [29, 30), and learned policies [31, 32, 33). However, these learned components are often limited to the tasks and environments that they are trained on. Two notable exceptions are MOM [34] and GenTP [35], but both methods plan with predefined manipulation skills. In contrast, our work directly tackles the generalization challenge at the level of motion generation. Closely related to our work are methods that learn manipulation skills for TAMP systems [4, 36, 37]. However, the resulting systems remain bottlenecked by the generalizability of the skills, which are trained using conventional Reinforcement Learning [36] Behavior Cloning [4, 37]. Instead, our work develops TAMP-come TAMP-compatible skills with object category-leve generalization.

---

Learning from Human Demonstrations. Moder deep imitation learning techniques have shown remarkable performance in solving real-world manipulation tasks [38, 39, 40, 6, 41, 42]. However, the prominent data-centric view of imitation learning [43, 6, 42], i.e. scaling up robot learning via brute-force data collection, remains limited by the sample efficiency of the existing learning algo- rithms and the challenges in collecting demonstrations for long-horizon tasks in diverse settings. Other recent works have proposed to replay small set of human demos in new situations facil- itate sample-efficient generalization [16, 44, 45, 46, 47, 48, 49, 50], but replay without adaptation can fail for novel object instances. Some other works leverage pretrained object representations to dramatically improve the generalization of policies given a handful of demonstrations [10, 8, 14]. However, these methods are limited to adapting a short skill [10] or a single manipulation action [8]. Our work develops long-horizon planning framework that seamlessly integrates skills augmented with latent object representations into classical TAMP framework.
3 Problem Setup and Background
The central question we aim to answer is: given set of demonstration trajectories, can adapt and recompose segments of them to solve new lusks? Our solution adopts the TAMP framework, where a high-level planner orchestrates a set of short-horizon motion generators (skills) to produce coherent long-horizon plans. The framework allows us to divide the problem into three technical sub- sub-proble probllrrbes..... (1) How to represent demonstration trajectory snippets as TAMP skills? In particular, how should we represent their precondition and effect constraints?
(2) How to adapt skills instanti- ated with recorded trajectories to new scenes and objects? (3) Given new task goal, how chain these skills together generate trajectory plan? Our insight is that NOD features will enable us to adapt both motion trajectories and skill constraints to scene layouts and object shapes. Our goal is to develop cohesive TAMP framework that addresses these sub-problems by building its core components on NOD representations.
3.1 Problem Setup
We consider the problem of object rearrangement, where robot must manipulate objects achieve desired scene configuration. The robot observes the scene in RGB-D frames and uses off-the-shelf segmentation models [51] to extract instance point cloud P. E R ggxx x3 for each manipulable object o. Accordingly, we represent the environment state as a set of object point clouds and the robot end- effector pose = ({ ( ) where Th E SE(3) is the end-effector pose in the world frame. The goal is specified as a set of task-relevant object point clouds g = {Po). for example, a mug inside a cabinet. The robot must generate a sequence of actions [ai...., arl that manipulate the objects to reach final configuration that closely matches the goal g, each action is an end-effector pose in the world frame T E SE(3). We measure task success by checking whether the desired scene configuration is reached. Our framework assumes access set of demonstration trajectories {Ti), each of which is a sequence of actions π; [a aa :..., ( and the object point clouds capturing the initial state of the recorded scene. The objective is adapt and compose the trajectories to generate action plans for solving a new task given new scene layout with unseen objects.
Neural Descriptor Fields (NDF). Our approach leverages Neural Descriptor Fields (NDFs) [8] to compactly represent object poses and features. An NDF is learned function diNDE that maps an ob- ject point cloud P E N-N3 x3 and a query pose T e SE(3) in the same frame to a feature descriptor Rd: 2 + N P) E IRd (1). We focus on two key properties of NDFs: Intra-category consistency: For objects of the same category (e.g., mugs), trained v'NDF maps geometrically sim- ilar query points (c.g., mug rims) similar feature descriptors Pose invariance: The descriptors are invariant to the object's global pose To enabling generalization to new layouts.
We use NDF to solve an inverse problem: given a query pose TV and its feature 2 derived from object point cloud P., recover the pose Ty relative to a new object cloud Pa. This optimization problem can be solved with gradient descent: NDF-OPTIMIZE(P, 2) argmin viNDF Po (2). T2
3.2 Skill Representation
We employ NDFs in skills to represent not only their control trajectories but also their start and end states. Accordingly, we represent each skill as tuple: = (name, iname,param, pre, eff, traj).
3

---

Task Specification
Planning
Skill Adaptation
Perception
Constraints: fx2 EditTe Paa...
Skeleton:
Gradient- based
Pick (mag_)))PPaac Placee
(mug.e Dl-m
constroint Feature matching
dem oredrated
Optimization
piherved
object point cloud
All Demo Skills
Selected Skills
Robot Execution
Pick (mug)
Pick (screwdriver)
(mug)
(mug)
Figure 2: NOD-TAMP Pipeline. Given goal specification. task planner plans sequence of skill types. Then, a skill reasoner searches for the combination of skill demonstrations that maximizes compatibility. Using learned neural object descriptors (e.g., NDFs), each selected skill demonstra- tion is adapted to the current scene. Finally, the adapted skills are executed in sequence.
We show a concrete example of skill representation in supplementary material Fig. 16. Here, name denotes the skill type (e.g., PICK, INSERT). param - {o: Pa E NRNx E Oz) are skill parameters, which include the skill-relevant objects O, and their observed point clouds. Preconditions pre specifies set of constraints that must hold before skill execution. The eff is change in conditions, i.e. adding removing constraints, as result of successfully executing the skill. A constraint is represented by the relative configuration of two point clouds, which is encoded through NDF features (e.g., Apr E Rdd). For example, an INSERT skill may require the robot to hold object in a specific way. Executing the skill results in a new constraint between the object and a receptacle. Finally, let Iraj = тı be set of end-effector poses for this skill. core objective of our method compose coherent multi-step plans by selecting and adapting suitable trajectory within each constituent skill. We include details of all skills in our experiments in the supplementary material.
4 NOD-TAMP
We present NOD-TAMP, a method for adapting and recomposing a set of skill demonstrations to solve new tasks. First, we show how single skill can be adapted to a new environment using NDFs (Sec. 4.1). Then, we propose planning algorithm that identifies skill segments from multi- ple demonstrations maximize compatibility (Sec. 4.2). Finally, we use motion planning connect each skill in order to efficiently and robustly generalize to environments (Sec. 4.3). The work- flow for NOD-TAMP is illustrated in Fig.
4.1 Skill Adaptation
We seek to adapt a skill to newly observed scene, which may be populated with new objects and layouts. To do so, we leverage key invariance: the skill trajectory still needs satisfy the recorded constraints (i.e., relative configurations between pairs of objects). Our skill adaptation module (1) transforms the skill trajectories to constraint-centric NDF feature trajectories Z, [21, 22,
using Equation and (2) adapts the trajectory to the observed scene via sequential optimization using Equation 2. Specifically, to encode a recorded skill trajectory and adapt it to the test scenarios, we consider common rearrangement skills that can be divided into two categories: hand-object interaction, such as grasping and manipulating constrained mechanisms (doors, etc.), and object- object interaction, where a robot uses the object in hand to interact with another object, such as placing and insertion.
Hand-object interaction. In this case, the constraint is between the manipulated object and robot end-effector at the end of trajectory. Thus we use the recorded object point cloud as the conditioning input to the NDF to encode the demonstrated robot end-effector trajectory as an NDF feature trajectory Z, [Z1, :2..... where z, = (NDr(TT.[] P.).
Object-object interaction. Here, the constraint is between pair of objects o'), where of is the in-han ha- source object, and is target object, e.g., a receptacle. Assuming rigid transform

---

between the end-effector and o' (i.e. secure grasp), we can place query pose T on o' and create an object-centric demonstration trajectory. The encoded trajectory is thus 2, = [2 =2,... where 2; = VNND (TH [+] Po). Note that the robot may also manipulate an unseen in-hand object o' during deployment. For example, the robot may be asked to stow a larger mug in bin when the demonstration is with a small mug, Thus we must also featurize the query pose with respect to o' in order to satisfy the precondition constraint between of and the end-effector. To do so, encode the constraint between the query frame and object o' as žg T'NDr (7 Po.). This way, with feature bg and Zr, we can fully characterize the constraints between object and of across time while considering their shapes.
Trajectory adaptation. Given the feature trajectory Zr, we will use the NDF fune- tion conditioned on the observed point cloud recover transformed skill trajectory, i.e., DUD--OPTIMIZE(Paaeeeee observed: 2), for each E 2,. We employ sequential optimization procedure to speed up the convergence, where each optimized pose serves as the initialization to warm-start the optimization of the next pose. In the case of object-hand interaction, the optimization output is an end-effector trajectory that can be directly used as a sequence of robot control setpoints. In the case of object-object interaction, the output is an object-centric trajectory (the constraint between query frame and the receptacle object across time), which we need to convert to robot controls. To do this, we first adapt the constraint between the query frame and the in-hand object in test scenarios using the recorded feature 2g. resulting a rigid transform between the query frame and the in-hand object. Then with the in-hand pose, we can derive the end-effector poses for control. We describe the skill adaptation with extended notation and pseudocode algorithm in the supplementary material.
4.2 Skill Planning
Given a set of skills, the goal of the skill planner to select a sequence of skills and their motion trajectories that can be chained together reach a task goal. The trajectories are then adapted using the procedure described in Sec. 4.1.
While our system can generate task plans independently or be embedded in an outer TAMP algo- rithm thanks to our skills functioning as PDDL operators, we assume the task plan is given. Our focus is on the bi-level planning problem chaining and adapting skill trajectories to create coher- ent Long-horizon motion plans. For given task, we assume an H -step task plan skeleton [#1.... #H] that defines sequence of selected skills, e.g., (mick,mug), PI.ACE(mug, bin), ...]. Recall that each skill can contain multiple candidate demonstration trajectories. The start and end of each trajectory represent its precondition and effect constraint, respectively. The essential step in skill planning is, for each skill in the plan, choose a candidate trajectory that is most compatible with the constraints of its adjacent skills. We calculate compatibility based on the distance between pairs of constraints in the NDF space. For simplicity, for the i-th skill in the plan, we denote the NDF-encoded precon- dition of a candidate skill trajectory as =/rre and the effect as =en" The compatibility is calculated as || all || Finally, we parse the goal configuration into a set of pair-wise
c= =1 pre cff object constraints and encode them as a set of NDF features Zg. We then compute the plan cost as the distance between Z, and the final accumulated constraints of the entire plan sequence.
We use Depth-firs search (DFS) to traverse over possible skill plans. After we obtain all costs for all skill trajectory combinations, the plan with the lowest plan-wide NDF feature distance is returned. For simplicity, we present this Cartesian product over relevant skills, but this can be done more efficiently by performing Uniform Cost Search in plan space, where the NDF feature distance serves as a the cost function. Algorithm 2 in the supplementary material displays the pseudocode for the NOD-TAMP planner.
4.3 Transit & Transfer Motion
Adapting demonstrated skills is particularly effective at generating behavior that involves contact. However, demonstrations typically contain long segments without contact (outside of holding an object). Because these components do modify the world, it is often not productive to replicate them. Thus, temporally trim skill demonstrations to focus on the data points that involve contact. In our implementation, we simply select the 20 steps before contact.
5

---

After trimming, and in often before trimming, two adjacent skills might be far away in task space. While linear interpolation is an option, this is not generally safe because the straight-line path may cause the robot to unexpectedly collide. To address this, we use motion planning to optimize for safe and efficient motion that reaches the start of next the skill. Motion planning generally requires some characterization of the collision volume of the obstacles avoid. Because we do not assume access to object models, we use the segmented point clouds the collision representation. For each pose yielded by the skill, we use Operational Space Controller (OSC) [52] to track them.
5 Experiments
We validate NOD-TAMP and how its components contribute to solve long-horizon tasks, perform fine-grained manipulation, and generalize to new object shapes. We select three evaluation set- tings: (1) LIBERO [15), a standard manipulation benchmark that feature diverse objects and long- horizon tasks, (2) a set of custom tabletop tasks that stress test spatial generalization and skill rea- soning&reuse, and (3) six real-world tasks with noisy perception and multitudes of challenges com- bined. We highlight key conclusions here and leave additional details in the supplementary material.
5.1 Experimental Setup
LIBERO Benchmark: LIBERO [15] is an existing multi-task manipulation benchmark. Our evaluation covers the "LIBERO-SPe" itatiall i (10 tasks), "LIBERO-Ob (10 tasks), and three of the "LIBERO-Long" tasks (Task 1, 5, and 8). For "LIBERO-Spatial tasks, we provide our system with just one demo of manipulating a bowl instance and test each system's ability to generalize over different initial bowl poses and goal configurations. For "LIBERO 1e1000O-jjee Object" tasks, provide our system with four demos of manipulating cheese box, milk box, ketchup bottle, and soup can and then test our system's s ability to generalize over similar objects shapes (e.g.. salad dressing bottle, pudding box) and poses.
To push the limit of the system, we design a suite of rearrangement tasks that have large variation in task horizon, object instances, scene layouts, goal configurations, Pick up mugs with varying shapes and initial poses; "MugInsertion innssrrtttn" Insert mugs of varying shape into tight cabinet. Both the mug and the cabinet are randomly placed on the table; "Tabl eCl Place two mugs into two bins, which aims to test the ability to achieve long-term goals by reusing the skills; "TableClearHal Stow one mug into a cabinet with side opening and place another mug into a bin. The robot must reason about proper grasp strategy to achieve the goal; "ToolHang" [39] Insert the frame into stand with tight tolerance, and then hang the tool object on the inserted frame, which tests the cabability of handling fine-grained motions.
Customized Tabletop Tasks:
and precision tolerances (See Fig 3). SougPickingg
We provide only two demos, which manipu- late one mug instance in two different ways by grasping either the handle or the rim, and test the methods on other nine different mug shapes. For the "Too l Hang" task, we provide one demo of how insert the frame and hang the tool on the frame after it is assembled.
Mug Insertion
Table Clear
Table Clear Hard
Tool Hang
Figure 3: Customized tasks. Examples of initial state and goal state (in green bounding box).
5.2 Baselines
NDF+ + [8] We augment the original NDF method, which has only shown single-pos, optimization, with task skeleton and the skill planning module. This baseline also uses a motion planner to transi- tion between key- key-frame poses; MimicGen [16] MimicGen directly transforms the demonstrated poses to the relevant object frame and then sent to the controller without further adaptation. For fair comparison, we augment MimicGen with a motion planner for collision avoidance; BC - The best- performing BC baseline (ViT-T) from LIBERO benchmark [15]. We list the reported performance in the multi-task learning setting as is an upper bound for lifelong imitation learning.
We also compare full system with different variants: Ours/SR This ablation removes the skill planning module. For each skill, we randomly choose reference trajectory from the collected

---

demonstrations belonging to this skill. This baseline validates the importance of skill reasoning for generalizing across tasks; Ours/MP- This ablation removes the motion planning component and uses linear trajectory interpolation to achieve transitions between the adapted skill trajectories. This baseline validates the benefit of leveraging motion planning, capability present in TAMP systems; NSC (naive skill chaining) This baseline ablates both the skill reasoning and the motion planning component, it randomly selects reference trajectory for each skill, adapts the skill with NDF, and uses linear trajectory interpolation for transitions between the selected trajectories.
5.3 Evaluation on the LIBERO Benchmark
This experiment compares the best behavior cloning (BC) performance provided by the LIBERO benchmark [15] with methods that combine generalizable neural representations and model-based planners, such as NOD-TAMP and several baselines (See Tab. 1). For the "LIBERO-Spatial tasks, the BC method is trained with 500 demos, and achieves 78% suc- cess rate. In contrast, our system only requires namely demo for "LIBERO-Spatial" and demos for "L the success rates are 84% and 94% respectively.
and "LIBERO--O
Table 1: Success rates on LIBERO tasks. MimicGen+ Ours/MP, and Ours/SR are abbrevi-
We hypothesize that the performance gap is caused by different state/action representations and the structural information leveraged to build the system. The BC methods directly
ated as M O/MP, and O/SR.
Tasks
BC
NDF
MG
NSC
0/M
OSR
Ours
Spatial
078
0.72
0.82
0.74
0.72
0.84
learn a mapping from the scene observation
Object
0.78
0.80
0.88
0.80
0.76
0.90
0.94
Long1
0.80
0.50
0:40
0.10
0.10
0.70
0.70
to the actions and thus require a huge broad
Long5
0.52
0.70
0.60
0.20
0.20
0.60
0.70
data cover diverse situations. Our method,
Longs
0.00
0.30
0.80
0.20
0.10
0.80
0.90
along with the cetioontransferring baselines (e.g., NDF, MimicGen), utilize object-centric representations to adapt the spatial correspondences from demo scenes to test scenes. MimicGen assumes identical correspondences between demon- stration objects and test objects, directly replaying trajectories the local frames of test objects. In contrast, our approach leverages learned object representations to infer spatial correspondences, enabling the transferred actions to be more robust to variations in object geometry. Compared to NDF+, NOD-TAMP transfers a sequence of actions that represent each dynamic manipulation skill instead of just single target state, in this case, the last end-effector pose in the demonstration tra- jectory. This improves NOD-TAMP's performance in fine-grained manipulation tasks. As LIBERO tasks are less constrained on grasping strategies (e.g., grasping the bowl to place it on a plate), we found the O/SR baseline performs similarly to our full system.
5.4 Evaluation on Customized Tabletop Tasks
In the tabletop tasks, NOD- TOD- consistently achieves a high success rate (80-90%) across all tasks and outperforms the other baselines and ablations (see Tab. 2). Below, we highlight specific comparisons and underscore the importance of each component NOD-TAME Additional analysis in supplementary material.
Table 2: Success rates on customized tabletop tasks. MimicGent, Ours/MP, and Ours/SR abbreviated as M O/MP, and O/SR.
NOD-TAMP exhibits strong performance across long-horizon tasks and is able to reuse skills in new contexts. The "TableClear" task requires re-using the existing two pick- and-place human demonstrations, which only consisted of single mug and bin interactions, stow mugs into two bins. NOD-TAMP achieves strong performance and outperforms MimicGen+ by 15% and NDF by 25% this task, showcasing a superior ability on re- re-purposing short-horizon skills for long-horizon manipulation.
Tasks
NDF
MG*
NSC
0/MP
OSR
Ours
MugPicking
0.80
0.70
0.85
0.0
0.85
0.85
MugInsertion
0.75
0.55
0.0
085
0.80
0.90
TableClear
0.80
Tablec learHand
0.40
0.55
0.5
0.50
0.10
0.50
Toollaang
0.75
0.75
NOD-TAMP exhibits strong generalization capability across goals, objects, and scenes in long- horizon tasks. The "Tables cellellees rdar task requires intelligent selection and application of demonstration trajectories to achieve diverse mug placements. We see the clear benefit of the skill planning component to achieve the different goals in this task NOD-TAMP outperforms Ours/SR by 70% and NSC by 65%. The omission of the skill planning module results in an incompatible
7

---

Clear Mugs (Same Goal)
Clear Mugs
Insert Screwdriver
Make Coffee
Sort Tableware
(Multi Goal)
Use Tool
Figure 4: Real-world tasks. Examples of initial and intermediate goal states.
composition of skills. For example, the robot may grip the rim of mug and attempt to insert it into the cabinet, leading to collisions between the cabinet and the gripper.
ROD-TAMP is able to solve low-tolerance manipulation tasks. The ToolHang task requires fine- grained manipulation skills that adapt to various object poses. NOD-TAMP achieves 75% success rate, outperforming NDF* and MimicGen* that cannot adapt their trajectories based on environ- ment changes. Since this task does require obstacle avoidance constrained grasping strategies, ablated baselines (Ours/MP, Ours/SR, and NSC) perform similarly full method.
5.5 Real-world Evaluation
We deploy NOD-TAMP on a real Franka Panda robot solve six challenging manipulation tasks (Fig. 4): "SortTableware"
Insert dish into narrow slot the rack and stack two bowls top of it; "Ma kekkoeoffeeee Place mug under the coffee machine, insert a coffee pod into tight holder, close the lid and then press the button; "InsertScrewdrl iver" Insert screwdriver into a tight slot on the storage rack; "UseTool" (inspired by [19, 53]) Use the L-shape tool to poke a box out from a narrow tunnel, and hook another box that out reach, and stack them; "ClearMugs (SameCoal)" Hang two mugs on the mug tree; "Clea MMlggs (MultiGoal] Hang one mug on the tree, and insert another mug into the cabinet. We provide single demon- stration for each (skill, object category) pair and test skill reasoning and reuse across tasks, object instances (c.g., round vs. square plates), and scene configurations. We include more details, includ- ing the list of object instances and reset range in the supplementary material.
We use front-mounted Microsoft Azure Kinect camera to capture RGB-D images and SAM [51] segment object point cloud. NOD-TAMP plans directly based on the partial-view object point cloud and executes the plans with impedance control. NOD-TAMP achieves a 60% success rate on "MakeCoffee", 70% success rate "Insert:Screwzz handling fine- fine-grained motions (e.g., inserting the coffee pod screwdriver with tight tolerance). achieves 90% success rate on
showing its capability
and "UseTool", and 80% success rate on "Cl "CleerMMun ruggs (SameGoa) 1 and "Cl "ClearMugs (MultiGoal) ", suggesting its capability on skill reusing and reasoning based on long-horizon goals, e.g., how to grasp the mug to store into bin vs. hang on mug tree and grasping different part of the tool poke and hook.
Conclusions and Limitations
We introduced NOD-TAMP, planning algorithm for long-horizon and fine fine-grained manipulation that can generalize across object shapes. NOD-TAMP directly leverages human demonstrations to implement manipulation skills. To ensure that these skills generalize to new settings, NOD-TAMP uses NDFs to adapt demonstrated object-centric motion to new, unseen objects. These skills are chained together using feature matching to ensure plan feasibility. Finally, they are executed using traditional motion planning and control to generalize across environments. We evaluated NOD- simulated task suites and six real-world tasks.
TAMP and competitive baselines
A limitation of NOD-TAMP is the computational efficiency in NOD-based trajectory adaptation. This bottleneck might be addressed with lightweight neural networks or more efficient optimization techniques. For skill planning. we use DFS to traverse over possible skill plans, this part can be further accelerated through optimization the implementation (e-g., parallel DFS or building K- trees in the NDF constraint space) accommodate large-scale demos. Besides, extending the system to handle deformable objects would necessitate adapting the current NOD to account for time- varying features, which represents challenging but interesting area for future work.

---

7 Acknowledgment
This work was partially supported by NSF Award #2409016 and NSF Award #2101250.

---

References
[1] B. Aceituno-Cabeaa and A. Rodriguez. A global quasi-dynamic model for contact-trajectory optimization in manipulation. 2020.
[2] L. Kaelbling and T. Lozano-Pérez. Hierarchical task and motion planning in the now. In ICRA, 2011.
[3] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling. and T. Lozano- Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:265-293, 2021.
[4] T. Silver, R. Chitnis, J. Tenenbaum, L. P. Kaelbling, and T. Lozano-Pérez Learning sym- bolic operators for task and motion planning. In 2021 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 3182-3188 IEEE, 2021.
[5] Cheng and D. Xu. League: Guided skill learning and abstraction for long-horizon manipu- lation. IEEE Robotics and Automation Letters, 2023.
[6] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. Ju- lian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, Parada, Peralta, E. Perez, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-
Mordatch, O. Nachum,
Pertsch, Quiambao, K. Rao,
1: Robotics transformer for real-world control at scale. In arXiv preprint ar 17Xiv:2212.06817,
2022.
[7] W. Huang, Wang, Zhang, Y. Li, Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973. 2023.
[8] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitz- mann. Neural descriptor fields: Se(3)-equivariant object representations for manipulation. 2022.
[9] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Conference Robot Learning, pages 894-906. PMLR, 2022.
[10] B. Wen, W, Lian, K. Bekris, and S. Schaal. You only demonstrate once: Category-Ievel ma- nipulation from single visual demonstration. arXiv preprint 1rXiv:2201.12716, 2022.
[11] P. R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. arXiv preprint arXiv: 2018.
[12] Sundaresan, J. Grannen, Thananjeyan, A. Balakrishna, M. Laskey, Stone, J. Gon- zalez, and K. Goldberg. Learning rope manipulation policies using dense object descriptors trained synthetic depth data. In 2020 IEEE International Conference Robotics and Au- lomation (ICRA), pages 941 9411-9418. IEEE, 2020.
[13] A. Simeonov, Y. Du, Y.-C. Lin, A. R. Garcia, L. P. Kaelbling, T. Jozzno-Pérez,, and P. Agrawal. Se (3)-equivariant relational rearrangement with neural descriptor fields. In Conference on Robot Learning, pages 835-846. PMLR, 2023.
[14] E. Chun, Y. Du, A. Simeonov, T. Lozano-Perez, and L. Kaclbling. Local neural descrip- tor fields: Locally conditioned object representations for manipulation. arXiv preprint arXiv:2302.03573, 2023.
[15] B. Liu, Y. Zhu, C. Gao. Y. Feng, Q. Liu, Y. Zhu, and P. Stone. Libero: Benchmarking knowl- edge transfer for lifelong robot learning. arXiv preprint arXiv:2306.0333 033100 2023.
10

---

[16] A. Mandlekar, S. Nasiriany* B. Wen", I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. In Conference on Robot Learning (CoRL), 2023.
[17] L. P. Kaelbling and T. Lozano-Pérez. Pre-image backchaining in belief space for mobile ma- nipulation. In Robotics Research, pages 383-400. Springer, 2017.
[18] C. R. Garrett, Lozano-Pérns and P. Kaelbling. Pddistream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 30, pages 440 0400 2020.
[19] M. A. Toussaint, K. R. Allen, K. A. Smith, and J. B. Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. 2018.
[20] G. Konidaris, P. Kaelbling, and T. Lozano-Perez. From skills to symbols: Learning symbolic representations for abstract high-level planning. Journal of Artificial Intelligence Research, 61: 215-289, 2018.
[21] Z. Wang, C. R. Garrett, L. P. Kaelbling, and T. Lozano-Pérez. Learning compositional models of robot skills for task and motion planning. The International Journal of Robotics Research, 40(6-7):866-89) 2021.
[22] J. Liang, M. Sharma, A. LaGrassa, S. Vats, S. Saxena, and O. Kroemer. Search-based task planning with learned skill effect models for lifelong robotic manipulation. In 2022 Interna- tional Conference on Robotics and Automation (ICRA), pages 6351-6357. IEEE, 2022.
[23] A. Simeonov, Y. Du, B. Kim, Hogan, Tenenbaum, P. Agrawal, and A. Rodriguez. A long horizon planning framework for manipulating rigid pointcloud objects. Conference on Robot Learning, pages 1582-1601. PMLR, 2021.
[24] Y. Huang, A. Conkey, and T. Hermans. Planning for multi-object manipulation with graph neural network relational classifiers. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1822-1829. IEEE, 2023.
[25] W. Yuan, C. Paxton, K. Desingh, and D. Fox. Somet: Spatial object-centric representations for sequential manipulation. In Conference Robot Learning, pages 148-157. PMLR, 2022.
[26] T. Migimatsu and Bohg. Grounding predicates through actions. In 2022 International Conference on Robotics and Automation (ICRA), pages 3498-3504. IEEE, 2022.
[27] K. Kase, C. Paxton, H. Mazhar, T. Ogata, and D. Fox. Transferable task execution from pixels through deep planning domain learning. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 10459-10465. IEEE, 2020.
[28] H. M. Pasula, L. Zettlemoyer, and L. Kaelbling. Learning symbolic models of stochastic domains. Journal of Artificial Intelligence Research, 29:309-352, 2007.
[29] R. Chitnis, D. Hadfield-Mennn A. Gupta, S. Srivastava, E. Groshev, C. Lin, and P. Abbeel. Guided search for task and motion plans using learned heuristics. In ICRA. IEFF, 2016.
[30] B. Kim, L. Shimanuki, L. Kaelbling, and T. Lozano-Pérez. Representation, learning, and planning algorithms for geometric task and motion planning. I/RR, 41),), 2022.
[31] M. J. McDonald and D. HadfieldMennll. Guided imitation of task and motion planning. In Conference on Robot Learning, pages 630-640. PMLR, 2022.
[32] J. Gu, Xiang. X. Li. Z. Ling, X. Liu, T. Mu, Y. Tang. S. Tao, X. Wei, Y. Yao, et al. Maniskil12: A unified benchmark for generalizable manipulation skills. arXiv preprint arXh::2302.04653 2023.
11

---

[33] M. Dalal, A. Mandlekar, C. Garrett, A. Handa, R. Salakhutdinov, and D. Fox. Imitating task and motion planning with visuomotor transformers. arXiv preprint arXiv:2305. 16309, 2023.
[34] A. Curtis, X. Fang, L. Kaelbling, T. Lozano-Pérez, and R. Garrett. Long-horizon manip- ulation of unknown objects via task and motion planning with estimated affordances. In 2022 International Conference on Robotics and Automation (ICRA), pages 1940-1946. IEEE, 2022.
[35] C. Wang, D. Xu, and L. Fei-Fei. Generalizable task planning through representation pretrain- ing. IEEE Robotics and Automation Letters, 7(3):8299-8855 8306, 2022.
[36] S. Cheng and D. Xu. Guided skill learning and abstraction for long-horizon manipulation. arXiv preprint arXiv:2210.12631, 2022.
[37] A. Mandlekar, C. R. Garrett, D. Xu, and D. Fox. Human- comman-in-theer in-the-loop task and motion planning for imitation learning. In 7th Annual Conference on Robot Learning, 2023.
[38] T. Zhang, Z. McCarthy, O. Jow, D. Lee, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. arXiv preprint arXiv:1710.04615. 2017.
[39] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L., Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL), 2021.
[40] A. Mandlekar, D. Xu, R. Martin-Martín, S. Savarese, and L. Fei-Fei. Learning to general- ize across long-horizon tasks from human demonstrations. arXiv preprint arXiv:2003.0600 06005, 2020.
[41] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learn- ing, pages 991-1002. PMLR, 2022.
[42] M. Ahn, Brohan, N. Brown, Chebotar, O. Cortes, B. David, Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0169/ 2022.
[43] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In Conference on robot learning, pages 1113-1132. PMLR, 2020.
[44] N. Di Palo and Johns. Learning multi-stage tasks with one demonstration via self-replay. In Conference Robot Learning, pages 1180-1189. PMLR, 2022.
[45] E. Johns. Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstra- tion. ICRA, 2021.
[46] V. Vosylius and E. Johns. Where to start? transferring simple skills complex environments. arXiv preprint arXiv:2212.06111,,00 2022.
[47] A. Chenu, O. Serris, O. Sigaud, and N. Perrin-Gilbert. Leveraging sequentiality in reinforce- ment learning from a single demonstration. arXiv preprint arXiv:2211.04770 2022.
[48] E. Valassakis, G. Papagiannis, N. Di Palo, and E. Johns. Demonstrate once, imitate imme- diately (dome): Learning visual servoing for one-shot imitation learning. In 2022 JEFF/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 8614-8621. IEEE, 2022.
[49] J. Liang, B. Wen, K. Bekris, and Boularias. Learning sensorimotor primitives of sequential manipulation tasks from visual demonstrations. In 2022 International Conference on Robotics and Automation (ICRA), pages 8591-8597. IEEE, 2022.
12

---

[50] F. Di Felice, S. D' D'Avela, A. Remus, P. Tripicchio, and C. A. Avizzano. One-shot imitation learning with graph neural networks for pick-and-place manipulation tasks. IEFE Robotics and Automation Letters, 2023.
[51] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick. Segment anything. arXiv:2304.02643, 2023.
[52] O, Khatib. A unified approach for motion and force control of robot manipulators: The opera- tional space formulation. IEEE Journal on Robotics and Aulomation, 3(1):43-53, 1987.
[53] D. Xu, A. Mandlekar, R. Martin, Y. Zhu, S. Savarese, and L. Fei-Fei. Deep affor- dance foresight: Planning through what can be done the future. In 202/ IEEE International Conference Robotics and Automation (ICRA), pages 6206-6213. IEEE, 2021.
[54] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages $748-8763. PMLR, 2021.
[55] J. J. Kuffner and S. M. VaValle. Rrt-connect: An efficient approach to single-query path planning, In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 2, pages 995-1001. IEEE, 2000.
[56] N. Hogan. Impedance control: An approach to manipulation: Part ii im-imllementttionn 1985.
[57] M. Tölgyessy, M. Dekan, L. Chovanec, and Hubinský. Evaluation of the azure kinect and its comparison kinect v1 and kinect v2. Sensors, 21(2):413, 2021.
[58] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
[59] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martin What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2108. 032998 2021.
[60] A. X. Chang, T. Funkhouser, I.. Guibas, P. Hanrahan, Q. Huang. 7. Li, S. Savarese, M. Savva, S. Song. H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNet: An Information-Rice 3D Model Reposi- tory. Technical Report W.Xiv:1512.0330 [es. [cs. Stanford University Princeton University
Toyota Technological Institute at Chicago, 2015.
[61] D. P. Kingma and Ba. Adam: method for stochastic optimization. arXiv preprint 2014.
13

---

A Table of Contents
The supplementary material has the following contents:
Real-World Experiments (Sec. B): Elaborate on our real-world experimental setting and analyze the results
Skill Reasoning Visualization (Sec. C): Visualize the feature matching results for real- world trials to analyze the skill reasoning process
Simulation Experiments (Sec. D): Elaborate on our customized tabletop task setting and analyze the results
Robustness to Perception Noise (Sec. E): Perform experiments that analyze the effect of sensor noise on NOD-TAMP success rates
Computation Efficiency (Sec. F): Break down the runtime of the different components of NOD-TAMP
Demonstration Extraction and Skill Representation (Sec. G): Elaborate on the demo extraction process and skill representation
Demo Quality Analysis (Sec. H): How demonstration quality affects performance
NDF Training (Sec. I): Describe how we trained our NDFs
LIBERO Qualitative Results and Failure Modes (Sec. J): Review failure modes on the
LIBERO tasks
Pseudocode (Sec. K): Present algorithms for trajectory adaptation and skill planning
14

---

B Real-world Experiments
B.1 System Setup
We demonstrate deploying our method on a real Franka Emika Panda robot in Fig. 5. The system perceives the scene with a Microsoft Azure Kinect camera and uses the Segment Anything Model (SAM) [51] to generate instance segmentation masks. To identify the target objects for each task, we extract visual features for each mask region using a CLIIP model [54], and retrieve the target masks through the text descriptions target objects. We project the pixels belonging each target object into the robot base frame generate point clouds. For pixel with coordinate (u, v) and depth d. the corresponding 3D location can be recovered by
p p= R 1K-..++
where / = (ud, vd, d), [M[ denotes the camera pose obtained through calibration, and K denotes the camera intrinsic matrix.
The motion planning component is built on [55]. We execute trajectories using open-loop control and track them with joint impedance controller [56] operating at a frequency of 20 Hz.
Camera
Robot
Figure 5: Hardware Setup. An illustration of the hardware setup.
B.2 Task Details
The objects used in each task, an example of start/goal state, reset range, and skills recorded are illustrated in Fig. 6 and Fig. 7. The skills extracted from single demonstration of the full task. Below we describe each real world task setup and the skill demonstrated.
"InsertScrewdxivee"" A fine-grained manipulation task. Pick up a screwdriver by the handle and insert into a tight slot (approx. 5mm) on the storage rack.
Tableware"" multi-step manipulation where the robot must place dishes onto dish rack. Grasp and insert dish into narrow slot on the dish rack and stack two bowls next to it. Dishes and bowls vary in shapes and size in each evaluation trial.
"MakeCo Operate Keurig machine to make coffee a multi-step task with fine- grained manipulation steps. Pick up mug by the handle and place it the coffee machine, insert a coffee pod into the tight holder, close the lid and then press the button.
"ClearMugs ( (SameGoal) ": Grasp and hang two mugs on the mug tree. The robot must reason about how to pick up the mug (by the rim, not the handle), in order to hang the mugs.
"ClearMuge (Mul t (iGoal)" " Grasp and hang one mug on the mug tree, and grasp and stwo another mug into the cabinet. The robot must reason about how to pick up the mug: to hang the mug, pick up by the rim. To stow mug, pick up by the handle.
15

---

Objects unseen)
Range
Demonstratec Skills
Inital State
Target State
Insert Screwdriver
(sorewddiver)
Insert esciiiiet
Sort Tableware
Pick Dowl)
Place (bowl
Pisk (Haae)
Insert (plaae)
Make Coffee
Pick (mug)
Place Imug)
Diise (K))
Push (buloo)
(poi)
Shart (puc))
Clear Mugs (Same Goal)
Pick (mug)
Place (mrgg)
Clear Mugs (Muls Goal)
(No additional demo need to be recorded. All required skills are reused from other tasks)
Figure 6: Real-world Task Setup (part 1). Visualization of the real world tasks. For each row, we show the objects used for the task. The objects used in the demonstration are visualized using bounding boxes with green dotted lines. We then show an example start and goal state, the reset range by overlaying the initial frames of each trial, and the skills recorded for the task.
16

---

Dop esn
Bool
(tan])
(plcok)
Hook (hinck)
Pick (block)
Place (bloc))
Place Blons
Figure 7: Real-world Task Setup (part 2).
"UseTool": A classical physical problem-solving task that requires multi-step reason- ing [19, The robot must reason about how pick up the tool in order interact with objects in the scene. Grasp the junction of an L-shape tool and use it to poke box out from a narrow tunnel, and then regrasp the long handle of this tool to hook another box that out of reach, and finally stack the two boxes.
the tool to
Skill reuse. For the "Clea EMags (SameGoal) task, we only record a demonstration of how to grasp mug and hang it on the mug tree, as the skills of grasping mug and stow it into cab- inet can be re-used from the "MakeCof fee task; For "Clea cclearMugs (MultiGoal G. we do not record any new demonstration as all the required skills for this task can be re-used from "ClearMugs (SameGoal) task.
Evaluation setup. We conduct 10 evaluations per task. Select tasks involve different object in- stances for each evaluation. Objects are placed randomly within their respective initialization range.
B.3 Performance Analysis
The quantitative results are shown in Tab. 3.
Table 3: Success rates of our system on real world tasks.
Tasks
SortTableware
MakeCoffec
InsertScrewdriver
Success Rate
9/10
6/10
7/10
Tasks
UseToool
ClearMugs (SameGoal)
ClearMugs (MultiGoal)
Success Rate
9/10
8/10
8/10
The task execution process is visualized Fig. and Fig. 9. NOD-TAMP can handle fine-graine grained motions (c.g., inserting the coffee pod or screwdriver with tight tolerance), and demonstrate its capability re-use skills and reasoning over them to achieve long-horizon goals (e.g., grasping different parts of the tool achieve poking and hooking behaviors). We also notice the major failures are caused by that the robot fails to grasp the handle of the mug, pod with the holder of the machine, where the errors can be attributed to noisy depth perception, or incomplete object point clouds due to partial view observation. We conduct further analysis on perception noise in Sec. E.
precisely align the
17

---

Insert Screwdriver
f
Sort Tableware
Sort Tableware
Sort Tableware
Make Coffee
Figure 8: Real-world Results. Key frames of real world task execution processes, the planning results are shown below each frame.
18

---

Clear Mugs (Multi Goal)
Clear Mugs (Multi Goal)
Clear Mugs (Same Goal)
Clear Mugs (Same Goal)
Use Tool
4
Figure 9: Real-world Results (Continued).
19

---

C Skill Reasoning Visualization
To analyze the skill planning process, we use real world trials of "ClearMugs (SameGoal) " "Clea "ClearMugs (MultiGoal) ) ", and "UseTool" for visualizing the feature distance of different skill combinations (See Fig. 10). For the left part the figure, each row represent different strategies of picking a mug (i.e., grasp the rim or grasp the handle), each column represent different strategies of placing the holding mug (i.e., insert the mug into cabinet, hang on mug tree). For the right part of the figure, each row represents different ways of pick up the tool (i.e., grasp either the junction or the long handle), and each column show different ways of using the tool (i.e., poke object out of a tunnel or hook object that out of reach). Lower feature distance means better compatibility of the skill combination. The results show that by leveraging the learned object descriptor features that characterizing geometric configurations, our skill planning module is able correctly evaluate the skill compatibility.
0.0007
0.8
0.0006
0.5
0005
0.0004
0.000238
2 15
0,000425
0.582
0.577
0.538
D 0003
0.00002
0.2
0,0508
0.0001
0.1
Figure 10: Feature Matching. The NOD feature distance of different skill combinations for real world trials. Lower score indicates more compatible skills (pre-post condition matching).
20

---

D Simulation Experiments
We visualize all reference demos used in the customized tabletop tasks in Fig. 11. We record just one demo for each task and post process the recorded data into skills. We conduct 20 evaluation trials for each task, and we change the object shapes and poses for each trial test the generalization of the system, we visualize the task reset ranges by overlying the first image frame of each trial in Fig. 12.
Fig. 13 show the generated trajectories of our framework and the execution process for our proposed tabletop tasks in simulation, highlighting the capability of our system on handling diverse shapes, configurations, and task goals.
Pick (mug)
Place (mug)
Pick (mug)
Place (mug)
Pick (frame)
Insert (frame)
Pick (tool)
Hang (tool)
Figure 11: Simulation Demos. visualization of the reference skill demos used for each tomized tabletop task. Here, the trajectories for each skill are projected into the camera coordinate frame and drawn top of the initial RGB image.
Mug Insertion
Table Clear
Table Clear Hard
Tool Hang
Figure 12: Customized Tabletop Task Reset Ranges. The task reset ranges.
21

---

Figure 13: Simulation Results. Key frames of task execution and planning.
22

---

E Robustness to Perception Noise
To understand the performance of our system under different levels of perception noise, such as levels present real-world sensors, we perform an experiment where we inject noise the point cloud observation simulation. We peform evaluation on the first stage of the "Tool Hang" task, high-precision task with tolerance of approximately 5mm. The robot needs to pick up the frame object and insert it into stand. A study of the depth accuracy of the Microsoft Azure Kinect [57] showed that, within a distance of 0.8 meters, the noise standard deviation is 5.546 10-1 meters.
To simulate this and settings with increased noise, we inject Gaussian noise with standard devia- tions of 0.05, 0.1, 0.15, and 0.2 centimeters. The results of the experiment are shown in Fig. NOD-TAMP only experiences a 5% reduction in success rate for real-world levels of noise. Our ex- periments show that NOD-TAMP robustly complete precise tasks even in the presence of typical sensor noise.
Success Rate v.s. Perception Noise
1.0
Success Rate
0.4
0.2
08.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Noise Level
1e-3
Figure 14: Robustness to Perception Noise. We evaluate the performance of NOD-TAMP under different levels of perception noise on the first stage of the simulated "Tool "ToolHang" task. The vertical dotted line represents the Gaussian noise standard deviation of Microsoft Azure Kinect. The success rate of NOD-TAMP only slightly decreases for real-world levels of noise, indicating that NOD-TAMP is robust to sensor noise.
23

---

Computation Efficiency
We provide a planning runtime analysis of our system in Fig. 15. We evaluate NOD-TAME on two-stage task that involves skill chaining and reasoning. report the runtime of the trajectory adaptation, constraint transfer skill reasoning, and trajectory tracking components. Since most daily tasks be achieved through sparsely represented trajectories with around 10-20 poses, al- together, the planning time is typically 1-2 minutes, where the gradient-based NDF optimization occupies most of the runtime.
Time Cost Distribution
TA
7.3
CT SR
19.0
300
250
(8)
5.3
19.2
319.3
150
100
4.6
161.1
3.8
68.3
10
100
Trajectory Length
Figure 15: Time v.s. Trajectory Lengths. We show the runtime of our full system for two- stage task. TA is short for trajectory adaptation, CT SR short for constraint transfer and skill reasoning. Tr is short for trajectory tracking. We see that trajectory adaption is the most computationally expensive operation in NOD-TAMP.
We also observe that the computational bottleneck is trajectory adaptation, which involves NDF optimization of individual poses to align with the reference trajectory feature. The runtime this component can be improved by utilizing lightweight neural networks for feature encoding and lever- aging more efficient optimization techniques. This is left for future work.
24

---

G Demonstration Extraction and Skill Representation
Here, we provide additional detail on segmenting and representing skills from recorded demos, to supplement Sec. III.B and IV.B-C the main text. To segment skill-level demonstrations from a longer task demonstration, we identify kinematic switches, which can be detected from gripper open & close actions and contact. Specifically, detect object contacts and pinpoint the time step which these changes oceur to establish the boundaries of each skill, similar to prior works that uses signals such gripper-object contact [58]. Some data sources, such as LIBERO, contain noisy actions such as repeated grasps and accidental contacts. To correct for this, we manually inspect and filter out low-quality skill demonstrations. To better leverage transit and transfer motion planning, we trim the skill segments to just the actions before changes in contact. In our implementation, we simply consider the 50 steps before contact. Further discussion is included in Sec. IV.C.
to
Skill name: Place (mug, bin)
Categories: mug, bin
Trajectory feature
Required constraints: IZo=NDF(eef. pose mug))
Add constraints: (Z:=NDF(bin pose mmug)
Delete constraints: IZo=NDF(eef. posel | mmg))
Figure 16: Skill Representation. How we represent pick skill in NOD-TAMP: the "required constraints" represent preconditions and the "add & delete constraints" represent effects.
Fig. illustrates how skill is represented the skill planning step (Sec. III.B). During skill planning, candidate skill is currently executable only if the currently active set of constraints, which are updated after each skill is added to the current partial plan during the search, include the required constraints of the candidate skill. Additionally, we use compatibility score in the form of the feature distance between two matched constraints to rank plan viability.
25

---

H Demo Quality Analysis
To analyze how demo quality affects the performance of our system, we use the "Can" task from Robomimic benchmark [59] to test our system, which paired with hybrid human demos. Accord- ing to Robomimic, the demos are categorized
three groups with quality "better", "okay", and "worse". We randomly sampled demos from each group, and source demo with randomly initialized object placements, the results presented in Fig. 17.
10 evaluation trials for each
Performance Demo Quality
0.8
Task Success
0.6
0.4
0.0 better
okay Demo Quality
worse
Figure 17: Task success rate v.s. Demo Quality for the "Can" task in Robomimic [59]. Demon- strations of different qualities are extracted from the accompanying dataset.
The results show that system's performance is affected by the quality of the reference demonstra- tions, similar to other learning from demonstration methods. However, the performance degradation is minimal. We also notice that some of the failure cases come from insecure grasps. This is proba- bly due to sub-optimal grasp poses. Incorporating failure detection and re- re-planning capability could further mitigate this issue.
26

---

I NDF Training
We train per-category NDF models using 3D mesh models extracted from ShapeNet [60]. We adopt same model architecture and learning hyperparameters as Simeonov et al. [8], namely learning rate of 0.0001 and batch size of 16. 'he model is optimized using the Adam optimizer [61] and trained for 80k epochs. We employ 3D occupancy prediction descriptor features, and randomly rotate and scale the object model to make the learned model more robust to shape variation. We use the same NDF models across all experiments in simulation. For cellwwrldd experiments, we further augment the training data by synthesizing partial point cloud to reflect the real-world perception input.
pre-training task acquire object
27

---

LIBERO Qualitative Results and Failure Modes
Fig. 18 visualizes the execution of several LIBERO tasks. Typical failure modes of our approach include gripper collisions due to a tight cabinet drawers and object slippage due to sub-optimal grasp poses. Our system's performance affected by the quality of the reference demonstrations, similar other frapiing-roommmmmosstttee providing higher-quality demos. Additionally, incorporating the ability to replan would make the system more robust to skill execution failures.
methods. Thus, some of the failures be improved through
Figure 18: LIBERO Results. Key frames of three task execution processes for LIBERO benchmark.
28

---

K
Pseudocode
Algorithm shows the trajectory adaption process. Let o and o' be the source and target objects in test scene for the adapted skill. Let Cace denote the acquired constraints during the execution of prior skills (e.g., the grasp pose after executing PICK). Finally, let To and Tu be poses for object and of respectively.
Algorithm Trajectory adaptation
Declare: Source object in test scene target object in test scene o', rohot end-effector Declare: Global accumulated constraints Coce.
Declare: Planned skills т. [d),... dn.
I: procedure ADAPT-TRAJ(o, d',d)
2: Eq: Z7 t- d
3: P. PERCEPTION(O)
PERCEPTION(O)) 5: T + ADF-OPTIMMIEEPP,
6:
b Adapt query pose to test scene based on target object
7:
T3 + (Tg') - T
8:
if dd.mode==bj-okt then
9:
Te + Coca (o', e)]
Extract constraints from Cace
10:
for Z E 2, do
11:
T + NDF-OPTIMIZH(P,,,2
12:
D Adapt motion to test scene based on source object
13:
if dd..oee=objooa then
14:
Ts + (17) -1 (1r2) )-1
15:
else
16:
T T
17:
yield T
Yield target to controller
18:
if d.mooe==ooj-ee then
19:
delete (o',) e6]
Remove constraint from Case
20:
else
TO + Ty (TH)-1.
Acquire the constraint
22:
Cocal o')] T3
Append constraint to Case
Note that the object poses used in equations just intermediate variables that help bridge the desired transformations, therefore these object poses do not need to carry any actual meaning. Here, we explain how we use NDFs to estimate novel object Pnew's 's point cloud transform Tnew w.r.t. given reference object Pref with pose Tref- To do so, we simply define the rotation of as identity, and define translation as the mean of Pref:
2yef <- VNDD(TTre Pres)
Tnew + NDF-OPTIMIZE(PPem 2ref).
The overall NOD-TAMP planning algorithm is shown in Algorithm
29

---

Algorithm NOD-TAMP planner
Declare: Plan skeleton [#1, f.,, #H1
Declare: Task goal specification Zg
I: procedure LELAN--ND-SKILLS((I))) $2.... a.). (w]:2p) Zg)
2: D
List of demos per skill
for: E [l.... H) do
4:
D+D= [T]].. where {r} is the trajectory set of skill ft
T. None
Optimal trajectory plan b Lowest cost
6:
C. x0
7:
for PRODUCT(D) do
All valid traj. sequences Feature cost
8:
Ze f- 0
All accumulated constraints
10:
for
E [l....,
H 1] do
11:
2/v PARSE(T(()
Parse pre. and eff. constraints
12:
21+
2t+ PARSE(T(i +
aet
13:
C+ c+ || of
14:
Compute feature distance among skills
15:
Zeo Zeco U (2
16:
Update acquired constraints Enumerate goal constraints
17:
for (k, 2x) Zy do
18:
2 Zwo [K]
19:
t c + 12 zz
20:
D Compute feature distance of the goal configuration b. Update best plan
21:
if < 6, then
22:
T. t-
23:
return w.
30


Reviews:
[{"review_id": "RRyNIWMUbv", "reviewer": "Reviewer_3zeS", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this work, the authors consider the task of long-horizon object rearrangement manipulation tasks that may involve multiple sequential rearrangments. The authors seek to combine two prominent frameworks: Neural Object Descriptors (which have been shown to promote intra-class and configuration generalization) and Task and Motion Planning (which is very useful for dealing with long-horizon planning). They devise an algorithm that learns NDF trajectories from demonstrations, and at test time searches over candidate trajectory options to find a plan to execute. They show promising results on several benchmarks.", "summary_of_recommendation": "In short, the execution and evaluation of the method are strong, but the method itself requires a large number of somewhat-unreasonable assumptions, and has limited utility without further exploration. There is no clear path forward to generalizability in new scenes which are not simple rearrangements (with minor in-class geometric variations) of previous demonstrations. Some of the fundamental limitations ought to be addressed - otherwise I see this contribtuition as incremental.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "* How would this method scale to larger numbers of demonstrations?\n* How can the task plans themselves be generated?\n* Has this method been applied to truly long-horizon tasks? AKA moving 10+ objects around the scene.\n* How can multimodality be handled?\n* How can situations where the trajectory matters (aka “thread the needle” tasks) over the course of the entire trajectory be handled?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZePG4iLYWa", "reviewer": "Reviewer_s2BX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work leverages neural descriptor fields (NDFs) to generalize a skill demonstration to novel task settings (e.g., new object types, poses). A skill-reasoning module chains a sequence of skills together to solve a long-horizon task by checking the preconditions and effects of the skills in the latent representation provided by the NDF. Experiments done on a benchmark suite and a real-world robot show promising results.", "summary_of_recommendation": "This paper shows a novel way of learning from demonstrations using neural descriptor fields. The method uses NDFs to combine trajectories to solve long-horizon tasks. Although there are some vague points regarding the method, I think the set of experiments shows some promising results, which might ignite some future ideas.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "- Why does O/SR perform so well?\n- What are the limitations of the method in addition to the mentioned computational bottleneck?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sHuFeiqFZy", "reviewer": "Reviewer_Stau", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This method uses an object centric TAMP approach to efficiently learn several tasks from very few demonstrations. They use neural descriptor feilds to encode joint object pointclouds with end effector poses. Then then use these encodings to generate trajectories and select which skills to chain together.", "summary_of_recommendation": "I am leaning reject because on the main contributions (generating trajectories from learned descriptors) is not sufficiently motivated.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. Can you justify the use of NDF for generating trajectories?\n2. Can you discuss how the current formulation of the method requires very specific skills which limits the generalization of the method.\n3. Have you encountered any ambiguities in optimizing the order of skills or are there any limitations to this method of chaining skills?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers raised concerns regarding the fact that a plan skeleton is provided, so the task planning is essentially given, lack of implementation details, unstated assumptions. The rebuttal submitted by the authors addressed these concerns to a large extent, and two out of three reviewers recommend acceptance.
Decision: Accept