NOD-TAMP: Generalizable Long-Horizon Planning with Neural Object Descriptors

Authors: shuo  cheng
caelan  garrett
ajay  mandlekar
danfei  xu
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 04:34:49

Abstract: Solving complex manipulation tasks in household and factory settings remains challenging due to long-horizon reasoning, fine-grained interactions, and broad object and scene diversity. Learning skills from demonstrations can be an effective strategy, but such methods often have limited generalizability beyond training data and struggle to solve long-horizon tasks. To overcome this, we propose to synergistically combine two paradigms: Neural Object Descriptors (NODs) that produce generalizable object-centric features and Task and Motion Planning (TAMP) frameworks that chain short-horizon skills to solve multi-step tasks. We introduce NOD-TAMP, a TAMP-based framework that extracts short manipulation trajectories from a handful of human demonstrations, adapts these trajectories using NOD features, and composes them to solve broad long-horizon, contact-rich tasks. NOD-TAMP solves existing manipulation benchmarks with a handful of demonstrations and significantly outperforms prior NOD-based approaches on new tabletop manipulation tasks that require diverse generalization. Finally, we deploy NOD-TAMP on a number of real-world tasks, including tool-use and high-precision insertion. For more details, please visit https://nodtamp.github.io/.

Main paper: 
 introduction: 
 From children playing with Lego blocks to adults rearranging a room, our remarkable ability to plan long sequences of actions to achieve our goals is still beyond the capabilities of current robots. Consider the challenges involved in daily tabletop tasks shown in Fig. 1 . First, these tasks are often long-horizon and full of sequential dependencies. Here, the robot must reason about the best pose to grasp a mug in order to stow it in a cabinet along with other steps to organize the entire table. Second, steps such as placing the mug in a tight cabin or stowing the screwdriver on the tool rack require intentional contact, which can render most motion planners that focus on avoiding collisions ineffective [1] . Finally, to be effective across broad environments, the robot must handle a wide variation of object shapes and scene layouts.Task and Motion Planning (TAMP) [2, 3] is an effective approach for such problems because it can effectively resolve sequential dependencies through hybrid symbolic-continuous reasoning. However, TAMP systems typically require accurate, special-purpose perception systems and handengineered manipulation skills. Thus, it is difficult to apply them to unseen objects and tasks that require complex motion trajectories. Recent works have proposed to learn manipulation skills from demonstration [4, 5] to partially relax these constraints. However, their generalization ability remains bounded by the training data, which is costly to collect at scale [6] .By contrast, neural representation models have shown remarkable potential in enabling generalizable manipulation systems [7, 8, 9, 10] . In particular, Neural Object Descriptors (NODs) [8, 11, 12] are a powerful tool to extract dense, part-level features that generalize across object instances. Neural Descriptor Fields (NDFs) [8] , a type of NOD that encodes SE(3) poses relative to a given object, can Learning from Human Demonstrations. Modern deep imitation learning techniques have shown remarkable performance in solving real-world manipulation tasks [38, 39, 40, 6, 41, 42] . However, the prominent data-centric view of imitation learning [43, 6, 42] , i.e. scaling up robot learning via brute-force data collection, remains limited by the sample efficiency of the existing learning algorithms and the challenges in collecting demonstrations for long-horizon tasks in diverse settings. Other recent works have proposed to replay a small set of human demos in new situations to facilitate sample-efficient generalization [16, 44, 45, 46, 47, 48, 49, 50] , but replay without adaptation can fail for novel object instances. Some other works leverage pretrained object representations to dramatically improve the generalization of policies given a handful of demonstrations [10, 8, 14] . However, these methods are limited to adapting a short skill [10] or a single manipulation action [8] . Our work develops a long-horizon planning framework that seamlessly integrates skills augmented with latent object representations into a classical TAMP framework. 
problem setup and background: 
 The central question we aim to answer is: given a set of demonstration trajectories, can we adapt and recompose segments of them to solve new tasks? Our solution adopts the TAMP framework, where a high-level planner orchestrates a set of short-horizon motion generators (skills) to produce coherent long-horizon plans. The framework allows us to divide the problem into three technical sub-problems. (1) How to represent demonstration trajectory snippets as TAMP skills? In particular, how should we represent their precondition and effect constraints? (2) How to adapt skills instantiated with recorded trajectories to new scenes and objects? (3) Given a new task goal, how to chain these skills together to generate a trajectory plan? Our insight is that NOD features will enable us to adapt both motion trajectories and skill constraints to new scene layouts and object shapes. Our goal is to develop a cohesive TAMP framework that addresses these sub-problems by building its core components on NOD representations.We consider the problem of object rearrangement, where a robot must manipulate objects to achieve a desired scene configuration. The robot observes the scene in RGB-D frames and uses off-the-shelf segmentation models [51] to extract instance point cloud P o ∈ R N ×3 for each manipulable object o. Accordingly, we represent the environment state as a set of object point clouds and the robot endeffector pose s = ⟨{P o }, T e w ⟩, where T e w ∈ SE(3) is the end-effector pose in the world frame. The goal is specified as a set of task-relevant object point clouds g = {P o }, for example, a mug inside a cabinet. The robot must generate a sequence of actions [a 1 , ..., a T ] that manipulate the objects to reach a final configuration that closely matches the goal g, each action is an end-effector pose in the world frame T e w ∈ SE(3). We measure task success by checking whether the desired scene configuration is reached. Our framework assumes access to a set of demonstration trajectories {τ i }, each of which is a sequence of actionsEQUATIONT ], and the object point clouds capturing the initial state of the recorded scene. The objective is to adapt and compose the trajectories to generate action plans for solving a new task given a new scene layout with unseen objects.Neural Descriptor Fields (NDF). Our approach leverages Neural Descriptor Fields (NDFs) [8] to compactly represent object poses and features. An NDF is a learned function ψ NDF that maps an object point cloud P ∈ R N ×3 and a query pose T q ∈ SE(3) in the same frame to a feature descriptorz ∈ R d : z ← ψ NDF (T q | P ) ∈ R d (1). We focus on two key properties of NDFs: Intra-category consistency: For objects of the same category (e.g., mugs), a trained ψ NDF maps geometrically similar query points (e.g., mug rims) to similar feature descriptors z. Pose invariance: The descriptors are invariant to the object's global pose T o w , enabling generalization to new layouts. We use NDF to solve an inverse problem: given a query pose T q w and its feature z derived from object point cloud P o , recover the pose T q ′ w relative to a new object cloud P o ′ . This optimization problem can be solved with gradient descent: NDF-OPTIMIZE(P o ′ , z) ≡ argminT q ′ w ||z -ψ NDF (T q ′ w | P o ′ )|| (2).We employ NDFs in our skills to represent not only their control trajectories but also their start and end states. Accordingly, we represent each skill π as a tuple: π = ⟨name, param, pre, eff, traj⟩.Constraints:𝑍 ! = {𝜓(𝑇 " |𝑃 " ), … } Plan Skeleton: Pick (mug_0)→Place (mug_0, cabinet_0)→ ⋯ Perception Skill Planning … 𝜋! … 𝜋" … 𝑍# 𝑐 Skill Adaptation demonstrated observedAll Demo Skills Finally, let traj = τ i be a set of end-effector poses for this skill. A core objective of our method is to compose coherent multi-step plans by selecting and adapting a suitable trajectory within each constituent skill. We include details of all skills in our experiments in the supplementary material. 
nod-tamp: 
 We present NOD-TAMP, a method for adapting and recomposing a set of skill demonstrations to solve new tasks. First, we show how a single skill can be adapted to a new environment using NDFs (Sec. 4.1). Then, we propose a planning algorithm that identifies skill segments from multiple demonstrations to maximize compatibility (Sec. 4.2). Finally, we use motion planning to connect each skill in order to efficiently and robustly generalize to new environments (Sec. 4.3). The workflow for NOD-TAMP is illustrated in Fig. 2 .We seek to adapt a skill to a newly observed scene, which may be populated with new objects and layouts. To do so, we leverage a key invariance: the skill trajectory still needs to satisfy the recorded constraints (i.e., relative configurations between pairs of objects). Our skill adaptation module (1) transforms the skill trajectories τ to constraint-centric NDF feature trajectories Z τ = [z 1 , z 2 , ..., ] using Equation 1 and (2) adapts the trajectory to the observed scene via sequential optimization using Equation 2. Specifically, to encode a recorded skill trajectory and adapt it to the test scenarios, we consider common rearrangement skills that can be divided into two categories: hand-object interaction, such as grasping and manipulating constrained mechanisms (doors, etc.), and objectobject interaction, where a robot uses the object in hand to interact with another object, such as placing and insertion.Hand-object interaction. In this case, the constraint is between the manipulated object o and robot end-effector e at the end of a trajectory. Thus we use the recorded object point cloud as the conditioning input to the NDF to encode the demonstrated robot end-effector trajectory as an NDF feature trajectory Z τ = [z 1 , z 2 , ..., ] where z i = ψ NDF (T e w [i] | P o ). Object-object interaction. Here, the constraint is between a pair of objects (o, o ′ ), where o ′ is the in-hand source object, and o is a target object, e.g., a receptacle. Assuming a rigid transform between the end-effector and o ′ (i.e. a secure grasp), we can place a query pose T q ′ w on o ′ and create an object-centric demonstration trajectory. The encoded trajectory is thusZ τ = [z 1 , z 2 , ..., ],wherez i = ψ NDF (T q ′ w [i] | P o ).Note that the robot may also manipulate an unseen in-hand object o ′ during deployment. For example, the robot may be asked to stow a larger mug in a bin when the demonstration is with a small mug. Thus we must also featurize the query pose with respect to o ′ in order to satisfy the precondition constraint between o ′ and the end-effector. To do so, we encode the constraint between the query frame and object o ′ as z q = ψ NDF (T q ′ w | P o ′ ). This way, with feature z q and Z τ , we can fully characterize the constraints between object o and o ′ across time while considering their shapes.Given the feature trajectory Z τ , we will use the NDF function conditioned on the observed point cloud to recover a transformed skill trajectory, i.e., NDF-OPTIMIZE(P observed , z), for each z ∈ Z τ . We employ a sequential optimization procedure to speed up the convergence, where each optimized pose serves as the initialization to warm-start the optimization of the next pose. In the case of object-hand interaction, the optimization output is an end-effector trajectory that can be directly used as a sequence of robot control setpoints. In the case of object-object interaction, the output is an object-centric trajectory (the constraint between query frame and the receptacle object across time), which we need to convert to robot controls. To do this, we first adapt the constraint between the query frame and the in-hand object in test scenarios using the recorded feature z q , resulting a rigid transform between the query frame and the in-hand object. Then with the in-hand pose, we can derive the end-effector poses for control. We describe the skill adaptation with extended notation and pseudocode algorithm in the supplementary material.Given a set of skills, the goal of the skill planner is to select a sequence of skills and their motion trajectories that can be chained together to reach a task goal. The trajectories are then adapted using the procedure described in Sec. 4.1.While our system can generate task plans independently or be embedded in an outer TAMP algorithm thanks to our skills functioning as PDDL operators, we assume the task plan is given. Our focus is on the bi-level planning problem of chaining and adapting skill trajectories to create coherent long-horizon motion plans. For a given task, we assume an H-step task plan skeleton [π 1 , ..., πH ] that defines a sequence of selected skills, e.g., [PICK(mug), PLACE(mug, bin), ...]. Recall that each skill can contain multiple candidate demonstration trajectories. The start and end of each trajectory represent its precondition and effect constraint, respectively. The essential step in skill planning is, for each skill in the plan, choose a candidate trajectory that is most compatible with the constraints of its adjacent skills. We calculate compatibility based on the distance between pairs of constraints in the NDF space. For simplicity, for the i-th skill in the plan, we denote the NDF-encoded precondition of a candidate skill trajectory as z i pre , and the effect as z i eff . The compatibility is calculated asc = ||z i pre -z i-1 eff || + ||z i eff -z i+1 pre ||.Finally, we parse the goal configuration g into a set of pair-wise object constraints and encode them as a set of NDF features Z g . We then compute the plan cost as the distance between Z g and the final accumulated constraints of the entire plan sequence.We use Depth-first search (DFS) to traverse over possible skill plans. After we obtain all costs for all skill trajectory combinations, the plan with the lowest plan-wide NDF feature distance is returned. For simplicity, we present this as a Cartesian product over relevant skills, but this can be done more efficiently by performing a Uniform Cost Search in plan space, where the NDF feature distance serves as a the cost function. Algorithm 2 in the supplementary material displays the pseudocode for the NOD-TAMP planner.Adapting demonstrated skills is particularly effective at generating behavior that involves contact. However, demonstrations typically contain long segments without contact (outside of holding an object). Because these components do not modify the world, it is often not productive to replicate them. Thus, we temporally trim skill demonstrations to focus on the data points that involve contact. In our implementation, we simply select the 20 steps before contact.After trimming, and in often before trimming, two adjacent skills might be far away in task space. While linear interpolation is an option, this is not generally safe because the straight-line path may cause the robot to unexpectedly collide. To address this, we use motion planning to optimize for safe and efficient motion that reaches the start of next the skill. Motion planning generally requires some characterization of the collision volume of the obstacles to avoid. Because we do not assume access to object models, we use the segmented point clouds as the collision representation. For each pose yielded by the skill, we use Operational Space Controller (OSC) [52] to track them. 
experiments: 
 We validate NOD-TAMP and how its components contribute to solve long-horizon tasks, perform fine-grained manipulation, and generalize to new object shapes. We select three evaluation settings: (1) LIBERO [15] , a standard manipulation benchmark that feature diverse objects and longhorizon tasks, (2) a set of custom tabletop tasks that stress test spatial generalization and skill rea-soning&reuse, and (3) six real-world tasks with noisy perception and multitudes of challenges combined. We highlight key conclusions here and leave additional details in the supplementary material.LIBERO Benchmark: LIBERO [15] is an existing multi-task manipulation benchmark. Our evaluation covers the "LIBERO-Spatial" (10 tasks), "LIBERO-Object" (10 tasks), and three of the "LIBERO-Long" tasks (Task 1, 5, and 8). For "LIBERO-Spatial" tasks, we provide our system with just one demo of manipulating a bowl instance and test each system's ability to generalize over different initial bowl poses and goal configurations. For "LIBERO-Object" tasks, we provide our system with four demos of manipulating a cheese box, milk box, ketchup bottle, and soup can and then test our system's ability to generalize over similar objects shapes (e.g., salad dressing bottle, pudding box) and poses.Customized Tabletop Tasks: To push the limit of the system, we design a suite of rearrangement tasks that have large variation in task horizon, object instances, scene layouts, goal configurations, and precision tolerances (See Fig 3 )."MugPicking" -Pick up mugs with varying shapes and initial poses; "MugInsertion" -Insert mugs of varying shape into a tight cabinet. Both the mug and the cabinet are randomly placed on the table; "TableClear" -Place two mugs into two bins, which aims to test the ability to achieve long-term goals by reusing the skills; "TableClearHard" -Stow one mug into a cabinet with side opening and place another mug into a bin. The robot must reason about proper grasp strategy to achieve the goal; "ToolHang" [39] -Insert the frame into a stand with tight tolerance, and then hang the tool object on the inserted frame, which tests the cabability of handling fine-grained motions. We provide only two demos, which manipulate one mug instance in two different ways by grasping either the handle or the rim, and test the methods on other nine different mug shapes. For the "ToolHang" task, we provide one demo of how to insert the frame and hang the tool on the frame after it is assembled.NDF + [8] -We augment the original NDF method, which has only shown single-pose optimization, with task skeleton and the skill planning module. This baseline also uses a motion planner to transition between key-frame poses; MimicGen + [16] -MimicGen directly transforms the demonstrated poses to the relevant object frame and then sent to the controller without further adaptation. For fair comparison, we augment MimicGen with a motion planner for collision avoidance; BC -The bestperforming BC baseline (ViT-T) from LIBERO benchmark [15] . We list the reported performance in the multi-task learning setting as it is an upper bound for lifelong imitation learning.We also compare our full system with different variants: Ours/SR -This ablation removes the skill planning module. For each skill, we randomly choose a reference trajectory from the collected demonstrations belonging to this skill. This baseline validates the importance of skill reasoning for generalizing across tasks; Ours/MP -This ablation removes the motion planning component and uses linear trajectory interpolation to achieve transitions between the adapted skill trajectories. This baseline validates the benefit of leveraging motion planning, a capability present in TAMP systems; NSC (naïve skill chaining) -This baseline ablates both the skill reasoning and the motion planning component, it randomly selects a reference trajectory for each skill, adapts the skill with NDF, and uses linear trajectory interpolation for transitions between the selected trajectories.This experiment compares the best behavior cloning (BC) performance provided by the LIBERO benchmark [15] with methods that combine generalizable neural representations and model-based planners, such as NOD-TAMP and several baselines (See Tab. 1). For the "LIBERO-Spatial" and "LIBERO-Object" tasks, the BC method is trained with 500 demos, and achieves 78% success rate. In contrast, our system only requires namely 1 demo for "LIBERO-Spatial" and 4 demos for "LIBERO-Object", the success rates are 84% and 94% respectively. We hypothesize that the performance gap is caused by different state/action representations and the structural information leveraged to build the system. The BC methods directly learn a mapping from the scene observation to the actions and thus require a huge broad data to cover diverse situations. Our method, along with the action-transferring baselines (e.g., NDF, MimicGen), utilize object-centric representations to adapt the spatial correspondences from demo scenes to test scenes. MimicGen + assumes identical correspondences between demonstration objects and test objects, directly replaying trajectories in the local frames of test objects. In contrast, our approach leverages learned object representations to infer spatial correspondences, enabling the transferred actions to be more robust to variations in object geometry. Compared to NDF + , NOD-TAMP transfers a sequence of actions that represent each dynamic manipulation skill instead of just a single target state, in this case, the last end-effector pose in the demonstration trajectory. This improves NOD-TAMP's performance in fine-grained manipulation tasks. As LIBERO tasks are less constrained on grasping strategies (e.g., grasping the bowl to place it on a plate), we found the O/SR baseline performs similarly to our full system.In the tabletop tasks, NOD-TAMP consistently achieves a high success rate (80-90%) across all tasks and outperforms the other baselines and ablations (see Tab. 2). Below, we highlight specific comparisons and underscore the importance of each component in NOD-TAMP. Additional analysis is in supplementary material. composition of skills. For example, the robot may grip the rim of a mug and attempt to insert it into the cabinet, leading to collisions between the cabinet and the gripper.NOD-TAMP is able to solve low-tolerance manipulation tasks. The ToolHang task requires finegrained manipulation skills that adapt to various object poses. NOD-TAMP achieves 75% success rate, outperforming NDF + and MimicGen + that cannot adapt their trajectories based on environment changes. Since this task does not require obstacle avoidance or constrained grasping strategies, our ablated baselines (Ours/MP, Ours/SR, and NSC) perform similarly to our full method.We deploy NOD-TAMP on a real Franka Panda robot to solve six challenging manipulation tasks (Fig. 4 ): "SortTableware" -Insert a dish into a narrow slot on the rack and stack two bowls on top of it; "MakeCoffee" -Place a mug under the coffee machine, insert a coffee pod into a tight holder, close the lid and then press the button; "InsertScrewdriver" -Insert a screwdriver into a tight slot on the storage rack; "UseTool" (inspired by [19, 53] ) -Use the L-shape tool to poke a box out from a narrow tunnel, and hook another box that out of reach, and stack them; "ClearMugs(SameGoal)" -Hang two mugs on the mug tree; "ClearMugs(MultiGoal)" -Hang one mug on the tree, and insert another mug into the cabinet. We provide a single demonstration for each ⟨skill, object category⟩ pair and test skill reasoning and reuse across tasks, object instances (e.g., round vs. square plates), and scene configurations. We include more details, including the list of object instances and reset range in the supplementary material.We use a front-mounted Microsoft Azure Kinect camera to capture RGB-D images and SAM [51] to segment object point cloud. NOD-TAMP plans directly based on the partial-view object point cloud and executes the plans with impedance control. NOD-TAMP achieves a 60% success rate on "MakeCoffee", 70% success rate on "InsertScrewdriver", showing its capability on handling fine-grained motions (e.g., inserting the coffee pod or screwdriver with tight tolerance).It achieves 90% success rate on "SortTableware" and "UseTool", and 80% success rate on "ClearMugs(SameGoal)" and "ClearMugs(MultiGoal)", suggesting its capability on skill reusing and reasoning based on long-horizon goals, e.g., how to grasp the mug to store into bin vs. hang on mug tree and grasping different part of the tool to poke and hook. 
conclusions and limitations: 
 We introduced NOD-TAMP, a planning algorithm for long-horizon and fine-grained manipulation that can generalize across object shapes. NOD-TAMP directly leverages human demonstrations to implement manipulation skills. To ensure that these skills generalize to new settings, NOD-TAMP uses NDFs to adapt demonstrated object-centric motion to new, unseen objects. These skills are chained together using feature matching to ensure plan feasibility. Finally, they are executed using traditional motion planning and control to generalize across environments. We evaluated NOD-TAMP and competitive baselines on two simulated task suites and six real-world tasks.A limitation of NOD-TAMP is the computational efficiency in NOD-based trajectory adaptation. This bottleneck might be addressed with lightweight neural networks or more efficient optimization techniques. For skill planning, we use DFS to traverse over possible skill plans, this part can be further accelerated through optimization in the implementation (e.g., parallel DFS or building K-D trees in the NDF constraint space) to accommodate large-scale demos. Besides, extending the system to handle deformable objects would necessitate adapting the current NOD to account for time-varying features, which represents a challenging but interesting area for future work. 
 
 Appendix: We demonstrate deploying our method on a real Franka Emika Panda robot in Fig. 5 . The system perceives the scene with a Microsoft Azure Kinect camera and uses the Segment Anything Model (SAM) [51] to generate instance segmentation masks. To identify the target objects for each task, we extract visual features for each mask region using a CLIP model [54] , and retrieve the target masks through the text descriptions of target objects. We project the pixels belonging to each target object into the robot base frame to generate point clouds. For a pixel with coordinate (u, v) and depth d, the corresponding 3D location can be recovered by p = R • K -1 • I + t, where I = (ud, vd, d), [R|t] denotes the camera pose obtained through calibration, and K denotes the camera intrinsic matrix.The motion planning component is built on [55] . We execute trajectories using open-loop control and track them with a joint impedance controller [56] operating at a frequency of 20 Hz. The objects used in each task, an example of start/goal state, reset range, and skills recorded are illustrated in Fig. 6 and Fig. 7 . The skills are extracted from a single demonstration of the full task. Below we describe each real world task setup and the skill demonstrated.• "InsertScrewdriver": A fine-grained manipulation task. Pick up a screwdriver by the handle and insert it into a tight slot (approx. 5mm) on the storage rack.• "SortTableware": A multi-step manipulation where the robot must place dishes onto a dish rack. Grasp and insert a dish into a narrow slot on the dish rack and stack two bowls next to it. Dishes and bowls vary in shapes and size in each evaluation trial.• "MakeCoffee": Operate a Keurig machine to make coffee -a multi-step task with finegrained manipulation steps. Pick up a mug by the handle and place it the coffee machine, insert a coffee pod into the tight holder, close the lid and then press the button.• "ClearMugs(SameGoal)": Grasp and hang two mugs on the mug tree. The robot must reason about how to pick up the mug (by the rim, not the handle), in order to hang the mugs.• "ClearMugs(MultiGoal)": Grasp and hang one mug on the mug tree, and grasp and stwo another mug into the cabinet. The robot must reason about how to pick up the mug: to hang the mug, pick up by the rim. To stow a mug, pick up by the handle. • "UseTool": A classical physical problem-solving task that requires multi-step reasoning [19, 53] . The robot must reason about how to pick up the tool in order to use the tool to interact with objects in the scene. Grasp the junction of an L-shape tool and use it to poke a box out from a narrow tunnel, and then regrasp the long handle of this tool to hook another box that out of reach, and finally stack the two boxes.Skill reuse. For the "ClearMugs(SameGoal)" task, we only record a demonstration of how to grasp a mug and hang it on the mug tree, as the skills of grasping a mug and stow it into cabinet can be re-used from the "MakeCoffee" task; For "ClearMugs(MultiGoal)", we do not record any new demonstration as all the required skills for this task can be re-used from the "ClearMugs(SameGoal)" task.Evaluation setup. We conduct 10 evaluations per task. Select tasks involve different object instances for each evaluation. Objects are placed randomly within their respective initialization range.The quantitative results are shown in Tab. 3. The task execution process is visualized in Fig. 8 and Fig. 9 . NOD-TAMP can handle fine-grained motions (e.g., inserting the coffee pod or screwdriver with tight tolerance), and demonstrate its capability to re-use skills and reasoning over them to achieve long-horizon goals (e.g., grasping different parts of the tool to achieve poking and hooking behaviors). We also notice the major failures are caused by that the robot fails to grasp the handle of the mug, or not precisely align the pod with the holder of the machine, where the errors can be attributed to noisy depth perception, or incomplete object point clouds due to partial view observation. We conduct further analysis on perception noise in Sec. E. Sort Tableware Sort Tableware Sort Tableware Make CoffeeTo analyze the skill planning process, we use real world trials of "ClearMugs(SameGoal)", "ClearMugs(MultiGoal)", and "UseTool" for visualizing the feature distance of different skill combinations (See Fig. 10 ). For the left part of the figure, each row represent different strategies of picking a mug (i.e., grasp the rim or grasp the handle), each column represent different strategies of placing the holding mug (i.e., insert the mug into cabinet, or hang it on mug tree). For the right part of the figure, each row represents different ways of pick up the tool (i.e., grasp either the junction or the long handle), and each column show different ways of using the tool (i.e., poke object out of a tunnel or hook object that out of reach). Lower feature distance means better compatibility of the skill combination. The results show that by leveraging the learned object descriptor features that characterizing geometric configurations, our skill planning module is able to correctly evaluate the skill compatibility. We visualize all reference demos used in the customized tabletop tasks in Fig. 11 . We record just one demo for each task and post process the recorded data into skills. We conduct 20 evaluation trials for each task, and we change the object shapes and poses for each trial to test the generalization of the system, we visualize the task reset ranges by overlying the first image frame of each trial in Fig. 12 .Fig. 13 show the generated trajectories of our framework and the execution process for our proposed tabletop tasks in simulation, highlighting the capability of our system on handling diverse shapes, configurations, and task goals. To understand the performance of our system under different levels of perception noise, such as levels present in real-world sensors, we perform an experiment where we inject noise in the point cloud observation in simulation. We peform evaluation on the first stage of the "ToolHang" task, a high-precision task with tolerance of approximately 5mm. The robot needs to pick up the frame object and insert it into a stand. A study of the depth accuracy of the Microsoft Azure Kinect [57] showed that, within a distance of 0.8 meters, the noise standard deviation is 5.546 × 10 -4 meters.To simulate this and settings with increased noise, we inject Gaussian noise with standard deviations of 0.05, 0.1, 0.15, and 0.2 centimeters. The results of the experiment are shown in Fig. 14 . NOD-TAMP only experiences a 5% reduction in success rate for real-world levels of noise. Our experiments show that NOD-TAMP can robustly complete precise tasks even in the presence of typical sensor noise. We provide a planning runtime analysis of our system in Fig. 15 . We evaluate NOD-TAMP on a two-stage task that involves skill chaining and reasoning. We report the runtime of the trajectory adaptation, constraint transfer & skill reasoning, and trajectory tracking components. Since most daily tasks can be achieved through sparsely represented trajectories with around 10-20 poses, altogether, the planning time is typically 1-2 minutes, where the gradient-based NDF optimization occupies most of the runtime. We show the runtime of our full system for a twostage task. TA is short for trajectory adaptation, CT & SR is short for constraint transfer and skill reasoning. Tr is short for trajectory tracking. We see that trajectory adaption is the most computationally expensive operation in NOD-TAMP.We also observe that the computational bottleneck is trajectory adaptation, which involves NDF optimization of individual poses to align with the reference trajectory feature. The runtime of this component can be improved by utilizing lightweight neural networks for feature encoding and leveraging more efficient optimization techniques. This is left for future work.Here, we provide additional detail on segmenting and representing skills from recorded demos, to supplement Sec. III.B and IV.B-C in the main text. To segment skill-level demonstrations from a longer task demonstration, we identify kinematic switches, which can be detected from gripper open & close actions and contact. Specifically, we detect object contacts and pinpoint the time step at which these changes occur to establish the boundaries of each skill, similar to prior works that uses signals such as gripper-object contact [58] . Some data sources, such as LIBERO, contain noisy actions such as repeated grasps and accidental contacts. To correct for this, we manually inspect and filter out low-quality skill demonstrations. To better leverage transit and transfer motion planning, we trim the skill segments to be just the actions before changes in contact. In our implementation, we simply consider the 50 steps before contact. Further discussion is included in Sec. IV.C. Fig. 16 illustrates how a skill is represented in the skill planning step (Sec. III.B). During skill planning, a candidate skill is currently executable only if the currently active set of constraints, which are updated after each skill is added to the current partial plan during the search, include the required constraints of the candidate skill. Additionally, we use a compatibility score in the form of the feature distance between two matched constraints to rank plan viability.To analyze how demo quality affects the performance of our system, we use the "Can" task from Robomimic benchmark [59] to test our system, which paired with hybrid human demos. According to Robomimic, the demos are categorized into three groups with quality "better", "okay", and "worse". We randomly sampled 4 demos from each group, and we run 10 evaluation trials for each source demo with randomly initialized object placements, the results are presented in Fig. 17 The results show that our system's performance is affected by the quality of the reference demonstrations, similar to other learning from demonstration methods. However, the performance degradation is minimal. We also notice that some of the failure cases come from insecure grasps. This is probably due to sub-optimal grasp poses. Incorporating failure detection and re-planning capability could further mitigate this issue.We train per-category NDF models using 3D mesh models extracted from ShapeNet [60] . We adopt the same model architecture and learning hyperparameters as Simeonov et al. [8] , namely a learning rate of 0.0001 and batch size of 16. The model is optimized using the Adam optimizer [61] and trained for 80k epochs. We employ 3D occupancy prediction as a pre-training task to acquire object descriptor features, and we randomly rotate and scale the object model to make the learned model more robust to shape variation. We use the same NDF models across all experiments in simulation. For real-world experiments, we further augment the training data by synthesizing partial point cloud to reflect the real-world perception input.Algorithm 1 shows the trajectory adaption process. Let o and o ′ be the source and target objects in the test scene for the adapted skill. Let C acc denote all the acquired constraints during the execution of prior skills (e.g., the grasp pose after executing PICK). Finally, let T o w and T o ′ w be poses for object o and o ′ respectively. Po ← PERCEPTION(o) 4:P o ′ ← PERCEPTION(o ′ ) 5:T q w ← NDF-OPTIMIZE(P o ′ , zq) 6:▷ Adapt query pose to test scene based on target object 7:T q o ′ ← (T o ′ w ) -1 • T q w 8: if d.mode=obj-obj then 9:T o ′ e ← Cacc[⟨o ′ , e⟩] ▷ Extract constraints from Cacc 10:for z ∈ Zτ do 11:T q w ← NDF-OPTIMIZE(Po, z) 12:▷ Adapt motion to test scene based on source object 13:if d.mode=obj-obj then 14:T e w ← T q w • (T q o ′ ) -1  


Decision: The reviewers raised concerns regarding the fact that a plan skeleton is provided, so the task planning is essentially given, lack of implementation details, unstated assumptions. The rebuttal submitted by the authors addressed these concerns to a large extent, and two out of three reviewers recommend acceptance.

Reviews: review: title: title: Review
paper_summary: paper_summary: In this work, the authors consider the task of long-horizon object rearrangement manipulation tasks that may involve multiple sequential rearrangments. The authors seek to combine two prominent frameworks: Neural Object Descriptors (which have been shown to promote intra-class and configuration generalization) and Task and Motion Planning (which is very useful for dealing with long-horizon planning). They devise an algorithm that learns NDF trajectories from demonstrations, and at test time searches over candidate trajectory options to find a plan to execute. They show promising results on several benchmarks.
main_review: main_review: # Quality

The paper is of reasonably high quality. The motivation is clear, and the proposed method is sound (despite its limitations). The experiments are diverse and thorough, comprising both  simulated and real-world tasks. Ablations are thorough as well. The major quality issue is with the method design itself - see weakness section for more details. Additionally, I’d argue that their proposed “TAMP” section barely qualifies as TAMP, as the rough plan + constraints/effects have already been constructed - it’s mostly the “MP” part of TAMP, and jointly optimizing these two things is usually the hardest part.

# Clarity

This work is reasonably clear, although some of the trajectory adaptation math / intuition is unclear and I had to refer to the appendix algorithms to understand. It might make sense to move one or both of the algorithms to the main text.

# Originality

The papers is not particularly novel, as it adapts a well-explored framework (NOD / NDF) for efficiently representing geometric relationships to a rather restricted long-horizon setting using simple escape hatches to the challenging problems (multimodality+free-space motion -> motion planning; exponential search space -> tiny # of demonstrations).

# Significance

Because it only offers limited potential to generalize, and requires many prerequisites (rough plan, prespecified effects/preconditions), the paper is not particularly significant.

# Strengths

* The paper is well-written, and evaluates their method thoroughly on many different tasks.
* The sample-efficiency of the method is attractive, given the known-good properties of the NDF submodules.
* Structural generalization over SE(3) is a good property to have / exploit.
* The method outperforms the baselines.

# Weaknesses

* The major weakness of this paper is that it makes extremely significant assumptions about problem structure:
    * Task planning is already complete - a high-level plan is already provided as input, as are various constraints/effects for each transition. This is an extremely strong assumption - there is no real path given for generating these plans, which is highly nontrivial. In my opinion, this method doesn’t really qualify as TAMP because the task planning is provided.
    * The problems are not multimodal
    * The actual intermediate trajectories between keyframes are inconsequential
    * Each demonstration can be carved up into clear quasistatic keyframes
    * All objects are rigid, as are the attachments to the gripper
* The strong sample complexity seems like a red herring - as the algorithm itself fundamentally does not scale to larger sample complexity of varying quality / diversity. This is evident from the inclusion of a cartesian product in the algorithm for searching over trajectories, which is naive and exponential (and in fact, is usually the hardest part of long-horizon planning problems). 
* There is no real discussion of multimodality, or generalizability to different quantities of objects (i.e. train on 0-2 mugs in the scene, test on a scene w/ 3 mugs)
* Composition of estimations (mug-object and object-object relationships) can incur various errors which may compromise the success rates over time
questions: questions: * How would this method scale to larger numbers of demonstrations?
* How can the task plans themselves be generated?
* Has this method been applied to truly long-horizon tasks? AKA moving 10+ objects around the scene.
* How can multimodality be handled?
* How can situations where the trajectory matters (aka “thread the needle” tasks) over the course of the entire trajectory be handled?
limitations: limitations: 2
review_summary: review_summary: In short, the execution and evaluation of the method are strong, but the method itself requires a large number of somewhat-unreasonable assumptions, and has limited utility without further exploration. There is no clear path forward to generalizability in new scenes which are not simple rearrangements (with minor in-class geometric variations) of previous demonstrations. Some of the fundamental limitations ought to be addressed - otherwise I see this contribtuition as incremental.
title: title: Use neural descriptor fields to represent skill demonstrations
paper_summary: paper_summary: This work leverages neural descriptor fields (NDFs) to generalize a skill demonstration to novel task settings (e.g., new object types, poses). A skill-reasoning module chains a sequence of skills together to solve a long-horizon task by checking the preconditions and effects of the skills in the latent representation provided by the NDF. Experiments done on a benchmark suite and a real-world robot show promising results.
main_review: main_review: The overall structure of the paper is good, however, the details of the method are not very clear; I understood the general idea behind the paper by making some educated guesses.

I think using NDFs to transform the demonstrations into a latent representation is a promising direction to learn skills from various data sources, and/or generalize to novel settings. I appreciate that there are a lot of experiments, but it is not very obvious in which cases the method shines and fails. For instance, the performance on MugPicking (0.85) is less than MugInsertion (0.90), which I would expect the reverse, and which is actually the case for other baselines except O/MP. And this is the general theme I felt while going over the experiments; I got that the method works, but could not imagine where and why the method would work or fail, or how difficult the task is to understand the increment over the baselines. For instance, does grasping a mug generalize to grasping a tumbler mug without a handle, or a teapot? This would help future readers to assess its current limitations and improve upon it.

Below are my other comments:
- In Sec. 3.2, it’s a bit vague whether ‘pre’ and ‘eff’ are normal point clouds or latent representations through NDFs. Is ‘param’ a latent representation as well? It would be nice to write this part with the notation consistent with Sec 3.1.
- Figure 2 could be improved. For instance, it was not obvious to me that those boxes in ‘skill planning’ are skills; I thought they were some sort of features.
- It’s hard to follow Sec 4.1. “Our skill adaptation module (1) transforms the skill trajectories to constraint-centric NDF feature trajectories and (2) adapts the trajectory to the observed scene via sequential optimization.” It’s not clear what is meant by the constraint-centric NDF feature trajectories. In line with my previous comment, it would be much easier to understand it with a proper notation.
- In Table 1, why does O/SR perform so well? Isn’t it basically picking random reference trajectories? Does this mean that the tasks do not require reasoning? I didn’t get why a random trajectory selection performs so well.
- In Table 2, while it’s mentioned that NOD-TAMP achieves strong performance, its ablated version NSC (without motion planning and skill reasoning) performs quite well, too!
- I am not sure if this can be thought of as long-horizon planning, but I agree that using neural descriptors might be good for learning behaviors from demonstrations that can generalize to other objects etc.
questions: questions: - Why does O/SR perform so well?
- What are the limitations of the method in addition to the mentioned computational bottleneck?
limitations: limitations: 1
review_summary: review_summary: This paper shows a novel way of learning from demonstrations using neural descriptor fields. The method uses NDFs to combine trajectories to solve long-horizon tasks. Although there are some vague points regarding the method, I think the set of experiments shows some promising results, which might ignite some future ideas.
title: title: Interesting work but many open questions
paper_summary: paper_summary: This method uses an object centric TAMP approach to efficiently learn several tasks from very few demonstrations. They use neural descriptor feilds to encode joint object pointclouds with end effector poses. Then then use these encodings to generate trajectories and select which skills to chain together.
main_review: main_review: The paper is well written and clear and the resulting robot experiments are impressive. However, I have some major concerns about this paper which need to be addressed. 

1. I am not convinced by the usefulness of using neural descriptor fields to generate trajectories. You already have the start and goal end-effector poses, why not use classical trajectory generation with collision checking which is well established and shown to be fast and robust? This would also make it easier to implement your proposed future work to apply constraints to the trajectory such as keep the mug upright. I don't see any benefit from using learned trajectories for these simple pick and place tasks.

2. It is impressive that learned skills can be deployed on a real robot with as few as one demonstration. However, the specificity of of each skill makes this a much simpler problem. Separate skills for picking a bowl vs plate vs screw driver is in fact quite a limitation which I think should be addressed in the paper. 

3. In the skill planning module I intuitively feel there may become ambiguities in selecting the right skill based solely on pointcloud and end-effector pose. Have the authors encountered any examples of this? What are the failure modes of the skill planning?

**Edit: Thank you to the authors for the additional explanations. On the basis of their responses I change my recommendation to Weak Accept.**
questions: questions: 1. Can you justify the use of NDF for generating trajectories?
2. Can you discuss how the current formulation of the method requires very specific skills which limits the generalization of the method.
3. Have you encountered any ambiguities in optimizing the order of skills or are there any limitations to this method of chaining skills?
limitations: limitations: 2
review_summary: review_summary: I am leaning reject because on the main contributions (generating trajectories from learned descriptors) is not sufficiently motivated.