SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields
Authors: Anthony Simeonov
Yilun Du
Yen-Chen Lin
Alberto Rodriguez Garcia
Leslie Pack Kaelbling
Tomás Lozano-Pérez
Pulkit Agrawal
Venue: CoRL 2022 Poster (2022)
Keywords: Neural Fields
Relations
Manipulation
Object Rearrangement

Abstract: We present a framework for specifying tasks involving spatial relations between objects using only 5-10 demonstrations and then executing such tasks given point cloud observations of a novel pair of objects in arbitrary initial poses. Our approach structures these rearrangement tasks by assigning a consistent local coordinate frame to the task-relevant object parts, localizing the corresponding coordinate frame on unseen object instances, and executing an action that brings these frames into alignment. We propose an optimization method that uses multiple Neural Descriptor Fields (NDFs) and a single annotated 3D keypoint to assign a set of consistent coordinate frames to the task-relevant object parts. We also propose an energy-based learning scheme to model the joint configuration of the objects that satisfies a desired relational task. We validate our pipeline on three multi-object rearrangement tasks in simulation and on a real robot. Results show that our method can infer relative transformations that satisfy the desired relation between novel objects in unseen initial poses using just a few demonstrations.

SE())-Equivarian Relational Rearrangement with Neural Descriptor Fields
Anthony Simeonov", .1.2, Yilun Du".,, Lin Yen-Chen'
Alberto Rodriguez², Leslie Pack Kaelbling'. Tomás Lozano-Pérez' Pulkit Agrawal
Massachusetts Institute of Technology
"CSAIL," 2 Improbable Al Lab, DDepartment of Mechanical Engineering, "Equal Contribution
We present a framework for specifying tasks involving spatial relations between objects using only ~5510 demonstrations and then executing such tasks given point cloud observations of a novel pair objects in arbitrary initial poses. Our approach structures these rearrangement tasks by assigning consistent local coordinate frame to the task-relevant object parts, localizing the corresponding coordinate frame unseen object instances, and executing action that brings these frames into alignment. We propose optimization method that uses multiple Neural Descriptor Fields (NDFs) and single annotated 3D keypoint to assign a set of consistent coordinate frames to the task-relevant object parts. We also propose an energy-based learning scheme to model the joint contiguration of the objects that satisfies a desired relational task. We validate our pipeline on three multi-object rearrangement tasks in simulation and on a real robot. Results show that our method infer relative transformations that satisfy the desired relation between novel objects in unseen initial poses using just few demonstrations.
Abstract:
Keywords: Object Relations, Rearrangement, Manipulation, Neural Fields
Introduction
Many tasks we want robots to perform e.g., stacking bowls and plates to declutter table, putting objects together to build an assembly, and hanging mugs on a rack with hooks - involve rearranging objects relative to one another. Such tasks can be described in terms of spatial relations between part features of set of objects, where a local coordinate frame is attached the task-relevant part the object, and the relation achieved by transforming the objects to bring these coordinate frames into specified alignment. For example, hanging a mug rack a relation between the mug's handle and the rack's 's hook, while stacking bowl on mug involves aligning the bottom of the bowl with the top of the mug (see Fig. 1).
Specifying and solving tasks in this way requires the ability to (i) assign a consistent local coordinate frame to the task-relevant object parts, and (ii) detect the corresponding coordinate frames on new object instances. Prior works have demonstrated these capabilities using techniques such as supervised keypoint detection [1, 2], but the use of large task-specific datasets labeled by humans limits easy deployment for wide diversity of tasks. Neural Descriptor Fields (NDFs) [3] have also been used to perform these abilities, with the added benefit of requiring just small set (~ 5-10) of task demonstrations. They achieve this by combining task-agnostic self-supervised pretraining and a few labeled examples of objects with consistent coordinate frames attached to their relevatt parts, effectively performing few-shot learning for task-relevant part localization on new instances.
Despite these benefits, it be tedious label the relevant part of each demonstration object with consistent pose (e.g., label the "handle" of each mug with consistent orientation). Prior work [3] reduced this burden by permitting the user to label single coordinate frame near the task-relevant part of known secondary object, and use the demonstrations to associate the frame with the task-relevant part of each unknown object (e.g., label a frame on the "hook" a known rack once, and associate
Project website: https: attps://anthoon //t.t/ottssimmoooooooooo wwwpiiimeoooo..iiuuobeeeeeeeeeee io/rrndf/
6th Conference Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

Initial configuration (unseen objects random poses)
Infer and exocuie relation "moug upright on table"
Infer and execute relation "bowl upright on magg"
Final configuration, relation "bovol uprightt on nans" satisfied
Figure I: Given point cloud of pair of unseen objects in aritrary initial configurations (top left), Relational Neural Descriptor Fields (R-NDFs) obtain relative transformations that satisfy relational task objective, such as "placing the mug upright on the table" (middle) and "stacking the bowl upright on top of the mug" (right). Our framework obtains these transformations by inferring the 6D pose of local coordinate frames at the task-relevant parts of the objects using just a small handful (~5-100 of demonstrations of each relational task.
this frame with each mug's 's "handle" based on the final mug placements). This labeled frame also specifies the alignment target for test-time objects once their relevant parts are localized. While this enables generalization to unseen objects in diverse poses [3], assuming a known secondary object is limiting for the hanging example, the system generalizes to scenarios with unseen mugs, but fails with both an unseen mug and rack with new shape. In this paper, we address this limitation. In particular, we present Relational Neural Descriptor Fields (R-NDFs), a framework, using ~ 5-10 demonstrations, that takes as input 3D point clouds of a pair of unseen objects in arbitrary initial poses and outputs a relative transformation between them that satisfies a relational task objective.
The central difficulty in applying NDFs to scenarios with changing pairs of objects is to assign set of consistent local coordinate frames to the task-relevant parts of the objects in the demonstrations, which may be both unaligned and differently shaped. We propose an optimization method that uses two NDFs (one per object) and a single 3D keypoint label in just one of the demonstrations, to assign a set of local coordinate frames that are consistently posed relative to the task-relevant parts of the objects. We then apply NDFs localize the corresponding coordinate frames for unseen pairs of objects presented in arbitrary initial poses, and solve for the relative transformation between them that satisfies the desired relation. However, errors can accumulate when inferring relative transformation based on pair of coordinate frames that have been independently localized. To mitigate this effect, we also propose a learning approach that directly models the joint configuration of the pair of objects and helps refine the transformation for satisfying the relation.
We validate R-NDFs on three relational rearrangement tasks in both simulation and the real world. Our simulation results show that R-NDF' outperform a set of baseline approaches, and our proposed real
optimization and learning-based refinement schemes benefit overall task success. Finally, world results exhibit the effectiveness of R-NDFs on pairs of diverse real world objects in tabletop pick-and-place, and highlight the potential for applying our approach to multi-step tasks.
2 Background: Neural Descriptor Fields
A Neural Descriptor Field (NDF) [3] represents an object using function ff that maps 3D coordinate x E R and an object point cloud P E R3xN to spatial descriptor in Rd:
f(x P): F x R 3x + Rd.
(1) such that
The function f is parameterized as a neural network constructed to be an object is subject to a rigid body transform T E SE(3) its spatial descriptors transform accordingly":
f(x [TT
(2)
This enables NDFs to behave consistently for the same object, regardless of the underlying pose. NDFs are also trained to learn correspondence over objects in the same category, so that points near similar geometric features of different instances (e.g., a point near the handle of two different mugs) are mapped to similar descriptor values. The equivariance property is obtained by using SO(3)-equivariant neural network layers [4] and mean-centered point clouds, while the category-level correspondence is obtained by training f category-level 3D reconstruction task [3, 5].
We use homogeneous coordinates for ease of notation, i.e., Tx denotes Rx + t where T = (R, t) E SE(3).

---

NDFs can also be redefined to model field over full SE(3) poses, rather than individual points. This achieved by concatenating the descriptors of the individual points in a rigid set of query points X' E R3x No, i.e., a set of three or more non-collinear points X;, i i= I...Ng. g. that are constrained to transform together rigidly. This construction allows NDFs to represent an SB(3) pose T via its action on X, i.c., the coordinates of the transformed query point cloud TA':
2 = F(T(P) = ① [(Tx:
(3)
X.EA
Thus, F maps point cloud P and an SE(3) pose T to category-level pose descriptor 2 E RRdxN, where F inherits the same [E(3))-qquvarianee from f.
3 General Problem Setup and Preliminaries
Our high level goal is to enable a user to specify task involving geometric relationship between pair of rigid objects, and enable a robot to perform this task on unseen object instances presented in arbitrary initial poses. Examples of relations we consider include "mug hanging on a rack", "bowl stacked upright on a mug", and "bottle placed upright on a tray".
Concretely, our goal is to build system that takes as input (nearly complete) 3D point clouds PA and PB (each segmented out from the overall scene) of objects O and Op, and outputs an SE(3) transformation T for transforming On n into configuration that satisfies a desired relation between O and O O.. We represent the relation as an alignment between pair of local coordinate frames attached to task-relevant geometric features the objects, and break down the problem of obtaining Ta n into (i) assigning a set of consistent coordinate frames to the task-relevant local object parts and (ii) localizing these coordinate frames on the relevant parts of the new objects.
Furthermore, we assume a user specifies the relational task by providing small handful of K task demonstrations { such that it's intuitive and efficient to specify a wide diversity of tasks with minimal engineering effort. A demonstration 2> consists of point clouds PA and P (of objects Ó and Ó p) and relation-satisfying transformation Tp.
NDFs for Encoding Single Unknown Object Relations. Prior work on NDFs may be applied to simplified version of this task, where the geometry and state of O is known. Given that O is known, we can initialize set of query points XA near the task-relevant part of O and use the query points to encode the relative pose To via Equation (3). Thus, a demonstration D is mapped to target pose descriptor = F(T B PPB) representing the (inverse of the) final pose of ÓB relative to A- In practice pose descriptors from multiple demonstrations { are averaged to obtain an overall descriptor 2 1 Σ 2, for the whole set, which has important implications in the version of the task with two unknown objects (see Section 4.1 for further discussion).
K
Given a novel object instance represented by point cloud P B, we can compute transformation Tp such that transforming Og by Ta satisfies the demonstrated relation between O and Og. This achieved by minimizing the L1 distance to the target pose descriptor 2:
T argmin|| [T 21..
(4)
T
Intuitively, Equation (4) performs well across different objects due to the fact that NDFs are pretrained to enable reconstruction across large dataset of 3D shapes. As a result, shared descriptors are discovered across different instances in shape category. In contrast, training a model directly on the few demonstrations (e.g., for regressing pose Tz)) would be more susceptible overfitting.
4 Method
We deseribe how we apply NDFs to infer relations between pairs of unknown objects. In Section 4.1, we propose iterative optimization method for assigning consistent task-relevant coordinate frames to multiple objects. In Section 4.2, we discuss how we train a neural network on top of NDF features to model the joint object configuration and refine an inferred transformation. The system inputs consist of pretrained NDFs ʃA and fB for each object category, demonstra- tions (D ((PA, PA, 1 and a single labeled 3D coordinate XAB for one of the demonstrations, indicating approximately where the respective demonstration objects interact.
3

---

Demonstration:
(B)
Inference (Step 1): Object A part localization
(C)
Inference (Step Object part localization
Execution: Trunsformed Object B
Pose descriptor encoding
PB
T
TB
21 FaCTx, PA)
2,l T orgmin| Fa En1
2n = FR Fr, Pa)
Figure 2: Method Overview. (A) demonstration (P Pa. is TB) of relation encoded into pair of pose descriptors by randomly sampling set of query points XA the origin and transforming it by Txa be near the task-relevan interaction point XAB. NDFs fA and fs are then used to obtain descriptors ΣA and 2B representing coordinate frames near the task-relevant local parts on the objects. (B) Given point cloud PA of a novel object, NDF fA, and pose descriptor A, pose Xx, of the corresponding coordinate frame on P is found. (C). This procedure is then repeated with P B. fs, and 2g to find pose T the relevant parts of P B relative to pose Tx, found in the first inference step. (D) Transforming Pa B by Ts satisfies the desired relation.
4.1 Multiple NDFs for Inferring Pairs of Task-Relevant Local Coordinate Frames
Consider a scenario where O and O, have unknown underlying shapes and configurations. We now show how NDFs can be used for inferring a pair of task- local coordinate frames on both objects and recovering a transformation Tg that satisfies the relation. The key idea of our approach is to formulate this problem as a bi-level optimization (illustrated in Figure 2), where we first optimize to find task-relevant portion of O A, and subsequently optimize a relative transform of a local part of Og B with respect the local region of O
We begin with two pretrained NDFs, fA and SB. and query points X' in a canonical pose at the world frame origin. We obtain N's by sampling Ng points from a zero-mean Gaussian and scaling such that X'1 has scale similar to the salient object parts. We then use the key- point XAB to transform X'A near the task-relevant features in the demonstration associated with XAB- Denote this transformation as Txx- Finally, we encode frand-frame pose Tx, into a descriptor conditioned on PA. as ZA FA PA), and relative pose T as ZB = Fn Fn (Pr). conditioned on PB. At test-time, we optimize both the world-frame pose of the query points Tx, and the (inverse of) pose TB relative to the initial pose found in the first step:
Tx, = argmin|/Fa((TPP.)
(5) TEL argmin|| CaaminnlFFaaaa Fa PB) Enl
(6)
T
Figure 2 shows an example of this pipeline, where the resulting Ta is applied to the point cloud P of O to satisfy the "hanging" relation.
Minimizing Descriptor Variance. In practice, solv- ing Equations (5) and (6) works better if pose descrip- tors {2 from multiple demonstrations are averaged together to obtain overall target descriptor 2 K >m 2, (see Sec. 6.1 and [3]). The reason is that a single demonstration underspecifies which object parts are relevant for the task, allowing 2 to be sensitive to ob- ject features which are not relevant to the desired relation. Instead, set of demonstrations using slightly different objects (e.g., with different scales) reveals regions lo- cal interactions that are shared across the demonstrations, which helps disambiguate between parts that are critical vs. irrelevant for the specified relation.
Demo
1
Demo
Query points not aligned vs.
Query points digged
Figure 3: Demo alignment. We align the query points by minimizing the variance across the descriptor set before averaging.
However, to avoid the pitfalls of averaging across a potentially multimodal or disjoint set, we want descriptors in the set { { to be sensitive to nearby local geometry in way that is consistent (i.e., unimodal) across the demos. This only occurs if the query points used obtain the descriptors themselves consistently aligned relative to each respective object (see Figure 3). Therefore, need

---

to find a transformation TxA. for each demonstration D; that transforms the canonical query points X' into configuration that leads the descriptors {2, F each other. We address this by finding the set transformations (Tx,,))
(Tx x PAs)) to be consistent with
K that minimizes the
variance across the descriptor set {2, FA (T x PA.))
min 11:1
(ar( subject to 2. = (T fori=1, K
(7)
where Var(.) denotes the sum of the per-element variance across set of vectors. We perform this minimization by applying NDFs in an alternating optimization procedure. Starting with initial reference pose (constructed using XAB) placing X'A near the task-relevant object parts in one of the demonstrations, we iteratively apply Equation (5) to obtain a descriptor for each demonstration that matches the reference. At the outer level, we refit the reference descriptor using the mean of the most recently obtained individual descriptors, and repeat. More details can be found in the Appendix.
4.2 Capturing Joint Descriptor Alignment through Learned Energy Functions
The method in Section 4.1 proposes to infer a desired relation by sequentially localizing independent coordinate frames for each object. While this approach is generally effective, small errors can accumulate and cause slight misalignments that lead to failure the execution. We thus propose to learn neural network which directly captures the joint configuration of O and Og that satisfies the desired relation, and use this model to refine predictions made by the method in Section 4.1.
Pairwise Energy Functions. We train an Energy- -asee Model (EBM) E-(-) [6] to parameterize a learned energy landscape over NDF encodings of relative poses between O and Og 8 (i.e.. Eg acts learned analogue for the L1 distance in Section 4.1). The energy function Ea(-) is trained so that the ground truth transform of Оg with respect to O is recovered given NDFs fA and fø (note that f corresponds to descriptor evaluation at single coordinate x while F is defined over sets of coordinates). Explicitly, our energy function is trained so that:
T B argmin [Eo(fn(-ITE B), (A(-
(8)
T
Since each NDF is continuous field, difficult to input them directly into our energy function E-).). We represent the energy function as the of the point-wise evaluation of each NDF set of different query points X' sampled from transformed pointcloud TPp B-
Σ
xE-E
At test-time, we use Equation (8) to refine the transformation obtained using Equations (5) and (6).
4.3 Learning
NDF training. We represent NDFs fA and fB as two neural networks with identical architecture and separate weights. Following [3], the architecture consists a PointNet [7] point cloud encoder with SO(3) equivariant Vector Neuron [4] layers, and a multi-layer perceptron (MIP) decoder. The NDF represented as a function mapping 3D coordinate and a point cloud to the vector of concatenated activations of the MLP. The models are trained end-to-end to reconstruct 3D shapes given object point clouds. We use dataset of ground truth 3D shapes and generate corresponding set of 3D point clouds in simulation. More architecture and training data details can be found in the Appendix. Energy-Based Model Training. We supervise the EBM Eg so that optimization over the learned energy landscape recovers the relative transform between O and Og- In particular, we follow the training objective in [8] and train arg min [E5 (-) to match target pose using the following procedure. We first apply small delta perturbation TA TRP (i.e., the point cloud of O in its final configuration) to obtain PBA Ta TEP.. We then train Eg to iteratively refine an initial random pose To with translation to and rotation Ry to undo the perturbation pose TA. We run n steps of optimization on to and Rg. where an individual step is given by t; 1 XV. Fo a), f.. B.A)) and R = R
(- JB (TTP.)))))))
We may train the energy function so that T,, corresponds to the inverse of the perturbation pose TA using Luans ||t, tx and Lyo |RR, R However, with symmetric objects, there are multiple different rotations R,, which may satisfy the desired relation (c.g., bowl is still "on"
5

---

mug, regardless of the angle about its radial axis). To account for these symmetries, we implicitly enforce consistency between an optimized transform T,, and A by enforcing that its application on P B, leads similar point cloud to TEPE g. We achieve this by minimizing the Chamfer loss [9] between the optimized transformed point cloud T,P.B. B.a and the demonstration point cloud T T BBPB
5 Application to Tabletop Manipulation
Robot and Environment Setup. We apply the method in Section to the problem of tabletop object rearrangement using a Franka Panda robotic arm with a Robotiq 2F140 parallel jaw gripper. The arm is used to collect the demonstrations and to execute the inferred transformation test-time. Our environment consists of the arm on table with four calibrated depth cameras.
Providing and Encoding Demonstrations. When collecting demonstration, initial object point clouds PA PA and B of objects Ó and Og are obtained by fusing set of back projected depth images. The demonstrator moves the gripper
pose T grasp- grasps Ô and finally moves the gripper to a pose I "pncee that satisfies the desired relation between Ô A and Ö F obtained as T ppacce guus In one of the demonstrations, a 3D keypoint XAB is labeled near the parts of the objects that interact with each other by moving the gripper to this region and recording its position.
Test-time Task Setup and Inference. At test time, are given point clouds PA and Pa of new objects O and Og. Equations (5), (6), and (8) are applied in sequence to obtain Tg. TB is applied to D by transforming an initial grasp pose grasp (obtained using a separate grasp generation pipeline) by TB to obtain a placing pose place = TB graup and off the shelf inverse kinematics and motion planning used to reach T grapp and T place.
Experiments and Results
Our experiments are designed to evaluate R-NDFs in executing relational rearrangement tasks with unseen objects using only few demonstrations. We seek to answer three questions: (1) How well do R-NDFs predict transformations that satisfy a relational task? (2) How important is each component in R-NDFs? (3) Can R-NDFs be used to perform multi-object pick-and-plase tasks in the real world? Baselines. As existing rearrangement methods are not directly applicable with few demonstrations, we compare with two constructed baselines. The first is to train an MLP to directly regress the relative transformation between objects ("Pose Regression"). The MLP takes as input the point cloud encodings obtained from the same PointNet [7] encoder with Vector Neuron [4] layers used in NDFs, and is trained directly on the demonstrations. The second method is based on 3D point cloud registration ("Patch Match"). We use state of the art registration method [10] to align the test-time shapes to the demonstration shapes and then compute the resulting relative transformation.
Task Setup and Evaluation Metrics. We consider three relational rearrangement tasks for evaluation: (I) Hanging a mug on the hook of a rack, (2) Stacking a bowl upright on top of a mug, and (3) Placing a bottle upright inside of box-shaped container. We provide 10 demonstrations of each task and evaluate if each method, using the demonstrations, can infer transformation that satisfies the desired relation for unseen pairs of object instances with randomly sampled poses. Experiments conducted in both the real-world and in simulation using PyBullet [11]. In simulation, the transformation obtained by each method is directly applied by resetting the simulator to the transformed object states. To quantify performance, we report the success rate over 100 trials, where we use the ground truth simulator state to compute success (objects must be in contact, have the correct relative orientation, and not interpenetrate).
6.1 Simulation Results
We begin by evaluating how well R-NDFs can infer the desired transformations in simulation. We consider two settings of varying difficulty. First, the pair of unseen objects are positioned randomly on the table with a randomly sampled "upright" orientation (similar to those used in the demonstrations). Second, the orientation of O is randomly sampled from the full space of 3D rotations.
Results in Table 4a compare the performance of our approach to the baselines. We find that training model to directly regress the relative pose using the point cloud embeddings leads to substantial

---

Nacilagrake Dric
Bowl on Mug Arbitrary
Mug on Rack
Bottle in Container
Method
Upright
Upright Arbitrary
Upright
Arbitrary
Pose Regression
35,0
6.0
13.0
10.0
37.9
12.0
Patch Match
34.0
32,0
56.0
44.0
44.0
42.0
Builass Concimes
R-NDF
74.0
70.0
84.0
75.0
80.0
75.0
(a)
(b)
Figure 4: (a) Relation inference success rates in simulation. R-NDF performs better than the baseline
approaches. (b) Example predictions. Representative predictions made by each method in simulation
Multiple Demonstration
Query Point Alignment
EBM Refinement
Upright Pose
Arbitrary Pose
1,00
vs, Estimatin Noise
No
No
No
39.3
43.6
0,50
Yes
No
No
60.0
Yes
Yes
No
78.0
72.0
0.25
Yes
Yes
Yes
84.0
75.0
0.00
1.2 Noise Standand (Normalized Object Sios)
(a)
(b)
Figure 5: (a) Ablations. R-NDF performance with different components ablated. Success rate highest when using multiple demonstrations, query point alignment, and EBM refinement. (b) Success vs. Keypoint Noise. Success rate magnitude of noise (normalized by object size) added to the single labeled 3D keypoint xAB.
overfitting and much lower success rates. On the other hand, the registration registrationbbaedd method can sometimes find transformations that correctly align the unseen shapes to the demonstration objects, and thus achieves higher success rates than pose regression. However, 3D registration is susceptible to locally optimal results that align the task irrelevant parts the objects. Common failure modes of using 3D registration the tasks we consider include aligning the body of the mug but ignoring the handle, or aligning the racks be upside down. Figure 4b illustrates the final simulator state after applying some of the representative predictions of each method.
In contrast, R-NDFs more accurately localize the task-relevant object parts and assign coordinate frames to these parts that are consistent with the demonstrations, leading to the highest success rates. Consistent with [3], the performance gap between the "upright" and "arbitrary" pose settings is small, which can be attributed the built-in equivariance of the features used in R-NDF.
6.2 Ablations
Next, we analyze the importance the individual components of R-NDFs. We investigate ablations on the simulated "mug on rack" task, again considering both "upright" and "arbitrary" pose settings.
The top of Table 5a illustrates that R-NDF performs there are multiple possible explanations for the alignment between two objects when given one example of the desired relation, pose descriptors obtained from single demonstration are more sensitive to task irrelevant object features. The second row of Table 5a investigates the effect of averaging descriptors across the set of demonstrations without first aligning the query points relative to the objects each demo, We modified the demonstrations to provide keypoints {xAB.)) KK near the relevant region in each demonstration, and then transform the query points this region withour aligning their orientations. Removing the query point alignment reduces the performance. The third row of Table 5a shows that removing the EBM refinement also decreases the success rate.
with single demonstration. Since
We further examine the importance of accurately specifying the 3D keypoint X near the task- relevant region on one of the demonstrations. We run the trials multiple times with Gaussian distributed noise added to the labeled point. Figure 5b shows plot of the success rate vs. the noise magnitude normalized by the approximate size of the object. The plot indicates that with limited noise perturbation, the success rate does not suffer significantly, though observe steep decline with more substantial perturbations. These larger perturbations shift the query points to regions near geometric features that are less relevant to the desired relation.
7

---

Bowl on Mug
Mug on Rack
Bottle in Container
Figure 6: Real Execution Results. Example executions of relational tasks unseen mugs. bowls, bottles, racks, and containers in the world. Our framework enables inferring the relative transformation between pairs of unseen objects arbitrary initial poses from small handful unaligned demonstrations of cach task.
6.3 Real Results
Finally, we validate that R-NDFs can used to perform the real world. Figure 6 shows the execution on our three tasks. Our method successfully infers tranformation between the objects that satisfies the relations, despite the objects being presented in challenging array of initial configurations. Figure shows multi-step rearrangement application of R- NDFs for the "bowl mug" task. First, relation between the mug and the table is specified and inferred for placing the mug upright. Then, the system executes the "stacking" relation between the bowl and the upright mug. This highlights how R-NDFs can enable executing sequential chains of relations to satisfy task objectives involving more than two objects. Please see our attached supplemental video for additional real world results.
on pairs of unseen objects in
Related Work
Novel Object Rearrangement. Several methods exist for novel object rearrangement [1, 12-29], many of which don't 't consider multiple varying objects that interact. CatBC [30] uses dense correspon- dence models to achieve impressive pick-and-place policy generalization from a single demonstration, but assumes known receptacle for placing. Neural shape mating [31], OmniHang [32], and kPAM 2.0 [2] generalize to pairs of unseen objects, but these approaches train on large task-specific datasets. TransporterNets [33, 34] enables rearrangement with varying pick and place locations from a few demonstrations, but focuses top-down manipulation and struggles with out-of-plane reorientation. In contrast, we focus executing relations involving large 3D reorientations.
Neural Fields in Robotics. Neural fields use neural networks parameterize functions over continuous spatial temporal coordinates [35]. They have been applied model various signals and scene properties, such images [36], geometry [5, 37, 38), appearance [39, 40], tactile imprints [41], and sound [42], with high fidelity and memory efficiency. Neural fields have been applied to represent objects for manipulation [3, 43-46] and environment states for dynamics and policy learning [47- 49]. They have also been used for pose estimation [50, 51], SLAM [52, 53], and representing object geometry without depth cameras [54, 55].
8 Limitations and Conclusion
Limitations. R-NDFs require pretrained NDF for each category used in the task, which can be nontrivial to obtain for novel object categories without existing 3D model datasets. Our approach also requires an annotated keypoint to localize task-relevant object parts. Future work could explore automated discovery of task-relevant regions directly from a set of demonstrations. Our system uses depth cameras, which often struggle with noise and objects with thin and transparent features. An RGB-only approach offering a similar level of generalization would be interesting to investigate. Finally, we require segmented object point clouds. While object instance segmentation quite mature, pretrained segmentation models regularly struggle when objects are in diverse orientations.
Conclusion. This work presents an approach for learning from a limited number of demonstrations to rearrange novel objects into configurations satisfying a relational task objective. We develop methods that build upon prior applications of neural fields for representing objects and increase the scope of tasks they can achieve. Our results illustrate the general applicability of our framework across diverse range of relational tasks involving pairs of novel objects in arbitrary initial poses.

---

Acknowledgments
This work is supported by the NSF Institute for Al and Fundamental Interactions, DARPA Machine Common Sense, NSF grant 2214177, AFOSR grant FA9550-22-1-08 ONR grant N00014-22 1- 2740, MIT-IBM Watson Lab, MIT Quest for Intelligence and Sony. Anthony Simeonov and Yilun Du are supported in part by NSF Graduate Research Fellowships. We thank the members of the Improbable AI Lab and the Learning and Intelligent Systems Lab for the helpful discussions and feedback on the paper.
Author Contributions
Anthony Simeonov developed the idea of minimizing descriptor variance for aligning multiple demonstrations, set up the simulation and real robot experiments, played primary role in paper writing, and led the project.
Yilun Du came up with and implemented the energy- baage modeling framework for relative pose inference, helped develop the overall framework of using NDFs for relational rearrangement tasks, simulated experiments, helped with writing the paper, and co-led the project.
Yen-Chen Lin participated research discussions about different ways to approach 6-DoF pick- and aaddplacc/rearrngemment tasks, helped suggest improvements to the NDF training and optimization procedure, and helped with editing the paper.
Alberto Rodriguez helped with early brainstorming how multiple NDF models could be used for multi-object rearrangement tasks and gave feedback on the tasks and real robot results.
Leslie Kaelbling helped develop the idea of chaining multiple pairwise relations together to perform multi-step tasks, provided suggestions on interesting rearrangement tasks solve, and helped write and edit the paper.
Tomás Lozano- Peréz also helped suggest the application to multi-step tasks via sequencing relations, reinforced the investigation of representations grounded in local interactions between object parts, and provided valuable feedback on the paper.
Pulkit Agrawal was involved in early technical discussions about how use multiple NDF models for rearrangement tasks, helped clarify key technical insights regarding query point labeling in the demonstrations, advised the overall project, and helped with paper writing and editing.
References
[1] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam:
Keypoint affordances for category- level robotic manipulation. In The International Symposium of Robotics Research, pages 132-157. Springer, 2019. 1,
[2] W. Gao and R. Tedrake. kpam 2.0: Feedback control for category-level robotic manipulation. IEEE Roboties and Automation Letters, 6(2):2962 222299999999,,,,,- 2021. 1,
[3] A. Simconov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, Agrawal, and V. Sitz- mann. Neural descriptor fields: -quevurrar object representations for manipulation. In 2022 International Conference Robotics and Automation (ICRA), pages 6394-6400. IEEE,
2022. 2, 4,
[4] C. Deng, O. Litany, Y. Duan, A. Poulenard, Tagliasacchi, and Guibas. Vector neurons:
A general framework for (eq)---quvaa networks. In ICCV, 2021. URL https://arxiv.
org arg/abs/2104. 122/29991..2.......2/2//2/1 12229. 2, 5.
[5] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4460-4470, 2019. 2, 8
[6] Y. Du and I. Mordatch. Implicit generation and modeling with energy based models. In NeurIPS.
2019. 5

---

[7] C. QI, H. Su, K. Mo, and L. J. Guibas, Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017. 5, 6
[8] Y. Du, S. Li, Y. Sharma, J. B. Tenenbaum, and I. Mordatch. Unsupervised learning of composi- tional energy concepts. NeurIPS, 2021.
[9] H. G. Barrow, J. M. Tenenbaum, R. C. Bolles, and H. C. Wolf. Parametric correspondence and chamfer matching: Two new techniques for image matching. Technical report, SRI International Menlo Park CA Artificial Intelligence Center, 1977.
[10] W. Gao and R. Tedrake. Filterreg: Robust and efficient probabilistic point-set registration using gaussian filter and twist parametcrization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10095-11104, 2019.
[11] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. GitHuu repository, 2016.
[12] S. Cheng, K. Mo, and L. Shao. Learning to regrasp by learning to place. In 5th Annual Confer- ence on Robot Learning, 2021. URL hattps://opperee www.:eewwwweeeettoddddeeedd-deee wwwdidiiddddddnnnaaadaaaaaa
8
[13] R. Li, C. Esteves, A. Makadia, and P. Agrawal. Stable object reorientation using contact plane registration. In 2022 International Conference on Robotics and Automation (ICRA), pages 6379-6385. IEEE, 2022.
[14] S. Thompson, L. P. Kaclbling, and T. Lozano-Perez. Shape-based transfer of generic skills. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 5996-60000
IEEE, 2021.
[15] D. Batra, A. X. Chang, S. Chernova, A. J. Davison, Deng, V. Koltun, S. Levine, J. Malik, I. Mordatch, R. Mottaghi, et al. Rearrangement: A challenge for embodied ai. arXiv preprint ar Nttv:2011.019975. 2020.
[16] A. Simeonov, Y. Du, B. Kim, F. R. Hogan, J. Tenenbaum, P. Agrawal, and A. Rodriguez. long horizon planning framework for manipulating rigid pointcloud objects. In Confer- ence Robot Learning (CoRL), 2020. URL https: / /tt///atttnysmmmmmmaeaae github. rpo-planning framework/.
[17] S. Lu, R. Wang, Y. Miao, C. Mitash, and K. Bekris. Online object model reconstruction and reuse for lifelong improvement of robot manipulation. In 2022 International Conference on Robotics and Automation (ICRA), pages 1540-1546. IEEE, 2022.
[18] M. Gualtieri and R. Platt. Robotic pick-and-place with uncertain object instance segmentation and shape completion. IEEE robotics and automation letters, 6(2):1753-1166) 2021.
[19] P. Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 2019.
[20] A. Curtis, X. Fang, L. P. Kaelbling, T. Lozano-Pérez. and C. R. Garrett. Long-horizon manipu- lation of unknown objects via task and motion planning with estimated affordances. In 2022 International Conference on Robotics and Automation (ICRA), pages 1940-1946. IEEE, 2022.
[21] C. Paxton, C. Xie, T. Hermans, and D. Fox. Predicting stable configurations for semantic placement of novel objects. In Conference on Robot Learning, pages 806-815. PMLR, 2022.
[22] W. Yuan, C. Paxton, K. Desingh, and D. Fox. Sornet: Spatial object-centric representations for sequential manipulation. In 5th Annual Conference on Robot Learning, pages 148-157 PMLR, 2021.
[23] A. Goyal, A. Mousavian, C. Paxton, Y.W. Chao, B. Okorn, J. Deng, and D. Fox. Ifor: Iterative flow minimization for robotic object rearrangement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14787-14797, 2022.
[24] A. H. Qureshi, A. Mousavian, C. Paxton, M. Yip, and D. Fox. NeRP: Neural Rearrangement Planning for Unknown Objects. In Proceedings of Robotics: Science and Systems, Virtual, July
2021. doi: 1ww11/55..SS.... 202 XVIL0072.
[25] D. Driess, J.-S. Ha, and M. Toussaint. Learning to solve sequential physical reasoning problems 1466,
from a scene image. The International Journal of Robotics Research,
10

---

2021.
[26] D. Driess, J.-S. Ha, and M. Toussaint. Deep visual reasoning: Learning to predict action sequences for task and motion planning from an initial scene image. In Robotics: Science and Systems 2020 (RSS 2020). RSS Foundation, 2020.
[27] W. Liu, C. Paxton, T. Hermans, and D. Fox. Structformer: Learning spatial structure for Linguag-guidded semantic rearrangement of novel objects. In 2022 International Conference on Robotics and Automation (ICRA), pages 6322-6329. IEFF, 2022.
[28] W. Goodwin, S. Vaze, I. Havoutis, and I. Posner. Semantically grounded object matching for robust robotic scene rearrangement. In 2022 International Conference on Robolics and Automation (ICRA), pages 1 11138-11144, 144. IEEE, 2022.
[29] M. Danielczuk, A. Mousavian, C. Eppner, and D. Fox. Object rearrangement using learned colliioooo collision functions. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 6010-6017. IEEE, 2021. 8
[30] B. Wen, W. Lian, K. Bekris, and S. Schaal. You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration. In Proceedings of Robotics: Science and
Systems, New York City, NY, USA, June 2022. doi:10.15607/RSS. http:/RSSS...2..... XVIII.044. 8
[31] Y.-C. Chen, H. Li, D. Turpin, A. Jacobson, and A. Garg. Neural shape mating: Self-supervised object assembly with adversarial shape priors. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pages 12724-12733, 2022. 8
[32] Y. You, L. Shao, T. Migimatsu, and J. Bohg. Omnihang: Learning to hang arbitrary objects using contact point correspondences and neural collision estimation. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 5921-5927. IEEE, 2021. 8
[33] A. Zeng. P. Florence, J. Tompson, S. Welker, Chien, M. Attarian, T. Armstrong, Krasin, D. Duong, V. Sindhwani, and Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference Robot Learning (CoRL), 2020. 8
[34] H. Huang, D. Wang. R. Walters, and R. Platt. Equivariant Transporter Network. In Proceedings of Robotics: Science and Systems, New York City, NY, USA, June 2022. doi: 10.156//SSS.
2022.XVII.007. 8
[35] Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, Tombari, J. Tompkin, V. Sitzmann, and S. Sridhar. Neural fields in visual computing and beyond. Computer Graphics Forum, 2022.
ISSN 1467-8659. doi: 10. 1/egf. 14505. 8
[36] T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila. Alias-free
generative adversarial networks. NeurIPS, 34, 2021. 8
[37] Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proc. CVPR, 2019. 8
[38] Z. Chen and H. Zhang. Learning implicit fields for generative shape modeling. In Proc. CVPR,
pages 9939-5948, 2019. 8
[39] B. Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proc. ECCV, 2020. 8 [40] V. Sitzmamn, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous ad-structtue-awaree neural scene representations. In NewrlPS, 2019. 8
[41] R. Gao, Y.-Y. Chang. S. Mall, L. Fei-Fei, and J. Wu. Objectfolder: A dataset of objects with
implicit visual, auditory, and tactile representations. In CoRL, 2021. 8
[42] A. Luo, Y. Du, M. Tarr, J. B. Tenenbaum, A. Torralba, and C. Gan. Learning neural acoustic
fields. arXiv preprini arXiv:2204.0066) 2022. 8
[43] J.-S. Ha, D. Driess, and M. Toussaint. Deep visual constraints: Neural implicit models for manipulation planning from visual input. IEEE Robotics and Automation Letters, 7(4):10857- 10864, 2022. 8
[44] Y. Wi, P. Florence, A. Zeng, and N. Fazeli. Virdo: Visio-tactile implicit representations of deformable objects. arXiv preprint arXiv:2202.00 200020002000.... 2022.
[45] Z. Jiang, Zhu, M. Svetlik, K. Fang, and Y. Zhu. Synergies between affordance and geometry:
6-dof grasp detection via implicit representations. Robotics: science and systems, 2021.
11

---

[46] Z. Jiang, C.-C. Hsu, and Y. Zhu. Ditto: Building digital twins of articulated objects from interaction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5616-5626, 2022. 8
[47] Y. Li, S. Li, V. Sitzmann. P. Agrawal, and A. Torralba. 3d neural scene representations for visuomotor control. In CoRI,, 2021.
[48] D. Driess, Z. Huang, Y. Li, R. Tedrake, and M. Toussaint. Learning multi-object dynamics with compositional neural radiance fields. arXiv preprint arXiv:2202.//85. 2022.
[49] D. Driess, I. Schubert, P. Florence, Y. Li, and M. Toussaint. Reinforcement learning with neural radiance fields. arXiv preprint arXiv:2206.0166
2022.
[50] L. Yen-Chen, P. Florence, J. T. Barron, A. Rodriguez, P. Isola, and T.-Y. Lin. iNeRF: Inverting
neural radiance fields for pose estimation. In IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS), 2021.
[51] M. Adamkiewicz, T. Chen, A. Caccavale, R. Gardner, P. Culbertson, J. Bohg, and M. Schwager.
Vision-only robot navigation in a neural radiance world. In RA-L, 2022. 8
[52] A. Moreau, N. Piasco, D. Tsishkou, B. Stanciulescu, and A. de La Fortelle. Lens: Localization
enhanced by nerf synthesis. In Conference on Robot Learning, 2021. 8
[53] E. Sucar, S. Liu, J. Ortiz, and A. Davison. iMAP: Implicit mapping and positioning in real-time.
In ICCV, 2021. 8
[54] I.. Yen-Chen, P. Florence, J. T. Barron, T.-Y. Lin, A. Rodriguez, and P. Isola. NeRF-Supervision:
Learning dense object descriptors from neural radiance fields. In ICRA, 2022.
[55] J. Ichnowski*, Y. Avigal*, J. Kerr, and K. Goldberg. Dex-NeRF: Using neural radiance field
to grasp transparent objects. In Conference on Robot Learning (CoRL), 2021. 8
12


Reviews:
[{"review_id": "oMsFzOJ86sC", "reviewer": "Reviewer_haL1", "summary": "This paper presents R-NDF, a framework for pair-wise object rearrangement that is robust to pose variations in SE(3). The framework builds on prior work in Neural Descriptor Fields (NDFs) – an approach for learning spatial descriptors of objects that are robust to 6-DoF transformations. NDFs are limited to a single object, whereas R-NDF can learn geometric relationships between pairs of rigid-bodies. Experimental evaluations include simulated experiments and baseline comparisons to simple pose regression and patch matching. The results also include some ablations that investigate individual components of R-NDF. Qualitative results include real-robot demonstrations, which showcase R-NDF’s ability to precisely place mugs on wooden racks with just a handful number of demonstrations.   \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n+ On the whole, the idea of using SE(3) equivariant frameworks for robot perception is an interesting and practical direction for vision-based manipulation. Naive image-based perception systems struggle to generalize to unseen object poses and are not robust to scene changes. Equivariance is a property that has been crucial to data efficiency and robustness in 2D vision (like with UNets), so it’s great to see equivariance being adapted to 6-DoF manipulation in 3D settings, more specifically in multi-object scenarios. While data-augmentation is one possible alternative, it’s hard to guarantee robustness to pose-variations like with equivariant frameworks.  \n\n+ R-NDF is evaluated both in simulated and real-world setups. Particularly, the real-world results are compelling. The framework can imitate high-precision tasks with just 5-10 demonstrations, and do this with any given pose-variation. Also, this robustness to pose-variations is evaluated with simulated experiments, which might be easily reproducible if the authors open-source their code-base. \n \n+ The figures in the paper are generally helpful. Figure 1 does a good job of providing an overview of the problem. Figure 2 provides a good summary of the approach and the high-level concepts. The experimental setup in Section 6 is mostly clear, and also includes a good set of ablations investigating individual aspects of the framework like energy-model based refinement and aligning query points. \n\n**Weaknesses**\n\n- One major concern is that R-NDF makes a lot of assumptions specific to the two-object setup, which could be better highlighted in the paper. Specifically, R-NDF assumes access to: (1) object segmentations, good object point-clouds, and approximate object localization (at least a centroid), (2) no distractor objects with only two-objects in the scene, (3) collision-free motion plans, and (4) pre-trained NDFs for each object. \n\n(1) Object segmentation and localization might be obtainable for large rigid-body objects, but it’s unclear how robust R-NDF would be in cluttered scenes with heavy occlusions and inaccurate segmentations. \n\n(2) The EBM formulation in Section 4 is also highly-engineered towards a setup with two-rigid objects. It’s unclear how easily extendible or generally applicable R-NDF would be to N-object rearrangement or anything that involves more than a single-step geometric transformation.\n\n(3) Section 5 briefly mentions that a motion-planner is used to reach grasp and place poses, but how is the collision avoidance handled? Particularly for the “Mug on Rack” example, how does the motion-planner know not to collide with the wooden stem before hanging the handle on it? \n\n(4) How long does it take to pre-train the NDFs? The paper claims only 5-10 demonstrations are needed to learn rearrangement transformations, but there is also the cost of pre-training NDFs on object instances, right?\n\nThese limitations don’t necessarily diminish the contributions of the paper, but it might be worth highlighting these requirements for future practitioners trying to adapt the framework to real-world manipulation challenges. \n\n- Another issue is that the relationship between R-NDF and k-PAM (Gao et al, Manuelli et al) is not sufficiently discussed. The related work section states “.. kPAM 2.0 generalize to pairs of unseen objects, but these approaches are trained on large task-specific datasets” but the methods section for R-NDF states that “NDFs are pretrained to enable reconstruction across a large dataset of 3D shapes”. So does kPAM use more shapes than R-NDF or not? Also, is there anything preventing the experiments from using kPAM (Gao et al) as a baseline for the experiments? \n\n- Minor: Section 4 could be re-written for better readability. There is a lot of linear algebra interweaved with the text, which makes the sentences hard to read at times. The text could be made more accessible by providing high-level intuitions before delving into mathematical definitions. \n\nReferences: \\\nGao et al: [https://groups.csail.mit.edu/robotics-center/public_papers/Gao20.pdf](https://groups.csail.mit.edu/robotics-center/public_papers/Gao20.pdf) \\\nManueli et al: [https://arxiv.org/abs/1903.06684](https://arxiv.org/abs/1903.06684) \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes a novel SE(3) equivariant framework for object-rearrangement and achieves compelling real-world results with limited data. There are some issues with the presentation of methods and results, but the framework itself could be of interest to the manipulation community.\n\n**Post Rebuttal**  \nThe authors spent a lot of time and effort updating the paper and adding new sections, and I really appreciate that! These updates address most of my concerns.\n\nMy only reservation is still the strong assumption about \"large rigid objects\". As pointed out, there has been a lot of progress in instance segmentation and 3D encoders, but it's still unclear how R-NDFs would work with objects that, to begin with, are hard to represent with segmentations or pointclouds in the real-world like particles, clothes, and liquids. But nonetheless, \"larger rigid objects\" still cover a broad category of manipulation problems, and overall, this paper is quite relevant to the CoRL audience. So I am keeping my score of Weak Accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See issues mentioned in the Weaknesses section.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "cO5NlxZJ-rFS", "reviewer": "Reviewer_gkTn", "summary": "The paper proposes a learning from demonstration approach based on pretrained category-level dense 3D descriptors (NDFs). The paper formulates single-step object arrangement tasks as a SE(3) transformation between two object instances. The proposed method first encodes SE(3) transformations as a set of point descriptors and then optimizes for the transformation between objects by minimizing the distance in the descriptor space. The method requires a few (10) demonstrations for each task and works for unseen object instances of known categories.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Clever formulation of learning from a few demonstrations.\n- Multiple improvements that address the limitation of prior work (NDF).\n- Nice real world demo in the supplement.\n\nWeaknesses\n- Reliance on object segmentation and complete point cloud.\n- Only works for known categories. Implicitly relies on labeled 3D object databases.\n- More qualitative examples would improve the presentation, such as showing the variety of objects.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper's contribution is borderline for me, but I would recommend acceptance based on the novelty and good real-robot demonstration, if not objected by other reviewers.\n\nThe paper cleverly leverages category-level descriptors that establishes correspondence between object parts to learn tasks from a few demonstrations and almost no annotation. However, a big underlying assumption is that these category level descriptors can be learned well. I have doubts about these descriptors, as they assume a full correspondence can be established between two object instances from the same category. I don't see how they will work for categories like chairs. I think a descriptor that can deal with partial-to-partial correspondences is crucial to have.\n\nThe method also assumes that object rearrangements can be simplified into SE(3) transformations, where in reality many more constraints exist, such as robot kinematics or collisions between object parts. Relying on motion planner is fine for simple scenarios, but eventually the pick and place poses need to consider these factors, especially if objects have more complex geometry and the environment is more clutterred.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Line 71 says that translation equivariance is obtained by mean-centering point clouds. How does that work? Does this assumes the point clouds are complete? A bit more details will be helpful.\n- Line 128 mentioned scaling $X_A$ to have scale similar to the salient object parts. How is this scale specified? From what I understand, the objects have various different scales, right?\n- How are \"task-relevant part of $O_A$\" (line 98) specified in the original NDF?\n- Why are the EBM model (line 188-202) that learns metrics on the descriptor space not susceptible to overfitting? It seems to be trained on only 10 examples."}}, {"review_id": "OkD5KRSZX3g", "reviewer": "Reviewer_krtE", "summary": "The paper builds on Neural Descriptor Fields and extends it to be used on two arbitrary objects (as opposed to one unknown and one fixed). \nThis is a more difficult problem as the query pointcloud location has to be defined from demonstrations. \nThe paper proposes an optimization method to consistently arrive at initial pointcloud location with a help of annotating its (initial) center.\nAfter the pose of the query is defined on object A using NDF of A, its instance on object B is also defined using NDF of B, yielding pose descriptors Z_A and Z_B.\nThese pose descriptors are averaged over demonstrations.\nDuring inference first the query pointcloud pose is defined and then the relative object pose is computed via optimization.\nTo improve accuracy further the paper also proposes the minimization of an Energy Based Model that fine-tunes the final pose.\nThe method was tested in simulations and real experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths\n\nThe paper addresses a relevant topic in robot learning and manipulation.\nNDF is a promising and novel method to encode relative object poses from pointcloud recordings and it shows good potential for generalization to similar objects.\nThis paper further improves its applicability to two-object-class scenarios, where the two object instances might have been unseen during training.\nThe paper presents a reasonable and technically mostly sound approach to achieve this goal and shows evaluations to support the desgin choices.\nThe clarity of the paper is also good, it is easy to follow.\nThe figures illustrate the approach well.\n\nWeaknesses\n\nI found the story around the Energy Based Model minimization vague. \nWhile it seems to mitigate the problem of inaccurate poses (after solving (4) and (5)), for me it is not entirely clear what is the source of pose error.\nThe paper discusses the accumulation of errors, but why do these error appear in the first place.\nDoes it relate to inaccurate demonstrations, limited expressiveness of NDFs, quality of the pointcloud and its segmentation?\n\nThe overall solution consists of multiple steps starting from camera calibration, training NDFs and pose descriptors till inference in real time, point cloud segmentation, optimizations to derive relative pose and grasp pose, etc.\nWhile a few failure points (e.g. quality of depth) have been mentioned in the limitation section, more discussion on the limitation of the whole algorithmic pipeline would be welcome.\nIt seems to take a substantial effort to implement the algorithm, what are the key components to pay attention to, what are prone to fail?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper provides a well motivated improvement to NDFs, the technical quality and clarity are good, but could use more improvements (see Strengths & Weaknesses). The simulation and real world experiment support the claim of the paper. More insights into further limitations and implementation details could improve my rating.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "Please address the limitations of the full training and inference pipeline.\n\nPlease comment on scaling to more objects, e.g., what happens with 3 objects? For example, place bottle left to the plate and right to the box.\n\nCan NDFs be used for tasks other than pick-and-place? \n"}}, {"review_id": "McxC8G5Ued", "reviewer": "Reviewer_1zr8", "summary": "The paper proposes a way to infer the SE(3) transform to move an object (represented by a point cloud) to satisfy some required relation with another object (also represented by a point cloud) e.g. to hang a mug on a rack. Both objects should belong to some a priori known classes, for each of which a dataset of point cloud instances should exist. These datasets are used to pre-train neural networks that allow subsequent inference of a transform from a small number (around 10) demonstrations. The proposed approach is evaluated on a set of experiments, including a real robot setting, and compared with two baselines showing improvements.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper builds heavily on the previous work on Neural Descriptor Fields. This work enabled inferring an SE(3) transformation for moving one object to satisfy a certain relation with the other when this other object is stationary and fixed. On the one hand, this paper goes in the right direction by removing this superfluous assumption. On the other hand, the path to this is rather incremental: two Neural Descriptor Fields are used instead of one. Additional refinement is added by using an additional multilayer perceptron resulting in a slight improvement in performance. Examples, including one with a real robot, that show improvements over some baselines are present. However, no code is provided in the supplementary and no promise of publishing the code upon acceptance is made which, in my opinion, is a big drawback. \n\n**Minor Comments**\n\n1. Equation (2). The property seems to be invariance rather than equivariance: usually equivariance of F:X->Y means that F(Tx)=TF(x) where T is an element of a group that acts on both X and Y, while invariance is F(Tx)=F(x). Equation (2) is similar to the latter.\n2. Footnote on page 2. “T = (R, T)” probably should be “T = (R, t)”.\n3. Line 93. “it’s” -> “it is”.\n4. Line 106 and other places. I suggest using the usual “l^1” instead of “L1”.\n5. Line 117. Maybe change “3D coordinate” to “3D point”. In my humble opinion a coordinate means each of the three numbers in a tuple representing a 3D point, not the whole tuple.\n6. Equation (7). As far as I understand this is not a constrained optimization problem. Phrasing it as a constrained one seems misleading to me.\n7. Lines 166-167. You write “small errors can accumulate”. What is the iterative procedure meant here? Is this the two-step procedure defined by Equations (5) and (6). If so, the word “accumulate” reads a bit weird for a two-step procedure.\n8. Figure 5 (b). At the bottom left the “0.00” and “0.0” interpenetrate.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although being somewhat incremental, the paper makes some progress in a relevant problem. It does not feature a (promise of) an open source implementation of the method, which is a big drawback.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Please comment or act on the following.\n\n- Line 228. Do you train the MLP on 10 examples only? If so, how do you manage to get something reasonable from it?\n- Figure 6. The container lying on a side with a bottle about to drop when the manipulator lets go of it is a genuine situation from an experiment or an improperly rotated picture?\n- In your bibliography, you have a number of arXiv references for papers that were published. For some entries you have page numbers and for some you do not, same for urls. Please make the bibliography consistent and cite the published versions of papers.\n"}}]

Author rebuttal:

Decision: Accept (poster)