One-shot Imitation Learning via Interaction Warping
Authors: Ondrej Biza
Skye Thompson
Kishore Reddy Pagidi
Abhinav Kumar
Elise van der Pol
Robin Walters
Thomas Kipf
Jan-Willem van de Meent
Lawson L.S. Wong
Robert Platt
Venue: CoRL 2023 Poster (2023)
Keywords: 3D manipulation
imitation learning
shape warping

Abstract: Learning robot policies from few demonstrations is crucial in open-ended applications. We propose a new method, Interaction Warping, for one-shot learning SE(3) robotic manipulation policies. We infer the 3D mesh of each object in the environment using shape warping, a technique for aligning point clouds across object instances. Then, we represent manipulation actions as keypoints on objects, which can be warped with the shape of the object. We show successful one-shot imitation learning on three simulated and real-world object re-arrangement tasks. We also demonstrate the ability of our method to predict object meshes and robot grasps in the wild. Webpage: https://shapewarping.github.io.

One-shot Imitation Learning via Interaction Warping
Ondrej Biza', Skye Thompson', Kishore Reddy Pagidi', Abhinay Kumar',
Elise van der Pol3.", Robin Walters Thhoms Kipf
Jan- Willem van de Meent Lawson L.S. Wong"-1, Robert Platt Northeastern University, 2 Brown University, Microsoft Research, Google DeepMind, University of Amsterdam Equal contribution. t Equal advising. biza. ofnortheaseern. edu
Learning robot policies from few demonstrations is crucial in open- ended upplications. We propose new method, Interaction Warping, for one-shot learning SE(3) robotic manipulation policies. We infer the 3D mesh of each ob- ject in the environment using shape warping, a technique for aligning point clouds across object instances. Then, we represent manipulation actions as keypoints on objects, which can be warped with the shape the object. We show suc- cessful one-shot imitation learning on three simulated and real-world object re- arrangement tasks. We also demonstrate the ability of our method predict object meshes and robot grasps in the wild. Webpage: https://shapewarping github.io.
Abstract:
Keywords: 3D manipulation, imitation learning, shape warping
Introduction
In one-shot imitation learning, are given sin- gle demonstration of a desired manipulation behav- ior and we must find policy that can reproduce the behavior in different situations. A classic example the Mug Tree task, where robot must grasp a mug and hang on a tree by its handle. Given a single demonstration of grasping a mug and hanging on tree (top row of Figure 1), we want to ob- tain policy that successfully generalize across objects and poses, e.g. differently-shaped mugs and trees (bottom row of Figure 1). This presents two key challenges: First, the demonstration must gener- alize to novel object instances, e.g. different mugs. Second, the policy must reason in SE(3), rather than in (2) where the problem is much easier [1].
Figure 1: The Mug Tree task.
To be successful in SE(3) manipulation, it is generally necessary to bias the model significantly toward the object manipulation domains in question. One popular approach is to establish a corre- spondence between points on the surface of the objects in the demonstration(s) with the same points on the objects seen at test time. This approach generally implemented using keypoints, point de- scriptors that encode the semantic location of the point on the surface of an object and transfer well between different novel object instances [2, 4]. E.g., points handles from different mugs should be assigned similar descriptors, thereby helping to correspond handles different mug instances. A key challenge therefore becomes how to learn semantically meaningful keypoint descriptors. Early work used hand-coded feature labels [4]. More recent methods learn a category-level object descrip- tor models during a pre-training step using implicit object models [5] or point models [2].
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

This paper proposes different approach to the point correspondence problem based on Coherent Point Drift (CPD) [6]. a point-cloud warping algorithm. We call this method Interaction Warping- Using CPD, we train a shape-completion model to register a novel in-category object instance to a canonical object model in which the task has been defined via single demonstration. The canonical task can then be projected into scenes with novel in-category objects by registering the new objects to the canonical models. Our method has several advantages over the previous work mentioned above [2, 3, 4]. First, it performs better in terms of its ability to successfully perform novel instance of demonstrated task, both in simulation and on robotic hardware. Second, it requires order-of- magnitude fewer object instances to train each new object category tens of object instances rather than hundreds. Third, our method agnostic to the use of neural networks - the approach presented based on CPD and PCA models, though using neural networks is possible.
2 Related Work
We draw prior works in shape warping [7. 8] and imitation learning via keypoints [4]. Shape warping uses non-rigid point cloud registration [9], a set of methods for aligning point clouds or meshes of objects, to transfer robot skills across objects of different shape. Our paper is the first to use shape warping to perform relational object re-arrangement and to handle objects in arbitrary poses. Second, keypoints are state abstraction method that reduces objects states to the poses of set of task-specific keypoints. We use keypoints to transfer robot actions. The novelty in our work is that interaction points are found automatically and warped together with object shape.
Few-shot Learning of Manipulation Policies: Keypoint based methods have been used in few- shot learning of object re-arrangement (4, 10, 11J. These methods rely on human-annotated object keypoints. Follow-up works proposed learned keypoints for learning tool affordances [12, 13, 14] and for model-based RL [15]. related idea the learning of 2D [16] and 3D [5, 17, 18, 19] descriptor fields, which provide semantic embeddings for arbitrary points. A keypoint can then be matched across object instances using its emhedding. We specifically compare to Simeonov et al. [5, 17] and show that our method requires fewer demonstrations. In separate lines of works, Pan et al. [2] (also included in our comparison) tackled object re-arrangement using cross-attention [20] between point clouds and Wen et al. [21] used pose estimation to solve precise object insertion.
Shape Warping and Manipulation: We use a learned model of in-class shape warping originally proposed by Rodriguez al. [22]. This model was previously used transfer object grasps [7, 23, 24] and parameters for skills such pouring liquids [25, 8]. Our method jointly infers the shape and pose of an object; prior work assumed object pose to be either given [8] or detected using neural pose detector [24]. Gradient descent on both the pose and the shape was previously used by Rodriguez et al. [7], Rodriguez and Behnke [23], but only to correct for minor deviations in pose. second line of work transfers grasps by warping contact points [26, 27, 28, 29, 30, 22, 31, 32]. Finally, point-cloud warping has been used to manipulate deformable objects [33, 34].
3 Background
Initial
Final
Coherent Point Drift (CPD) Given two point clouds, ( E Rax3 and x E gmx3 Coher-
Source over target
ent Point Drift (CPD) finds a displacement Wi+s
Rax3 of the points (() that brings them as close as possible (in L2 sense) to the points X [6]. CPD a non-rigid point-cloud registration method each
point in ( can be translated independently. CPD
minimizes the following cost function,
Figure 2: Coherent Point Drift warping.
J(W,,))
Σ log
exp
X + (W (W.,)) - -x )
(WWW,))), (1)
20
T=1

---

using expectation maximization over point correspondences and distances (see [6] for details). This can be viewed as fitting Gaussian Mixture Model n components to the data X Here, (W)) prior on the point displacements that regularizes nearby points in X to move coherently, preventing the assignment of arbitrary correspondences between points in X and X
Generative Object Modeling Using CPD: CPD be used part of generative model for in-category object shapes as follows [7]. Assume that we are given set of point clouds, {x(") ..... ( }, point clouds describing different mug instances. Each of these point clouds must be full point cloud in the sense that it covers the entire object. Select a "canonical" object X(O), C {1, 2, .... K} and define a of displacement matrices Wc-: = CPD(X(C), x (()))) E (1.2..., K}. The choice of C' is arbitrary, but heuristically choose the C that is the most representative (Appendix A.2). Now, calculate low rank approximation of the space of object-shape deformations using PCA. For each matrix W E g let We-i E [R denote the flattened version. We form the 3n K data matrix We = [Wc.1..... Wc-k] and calculate the d-dimensional PCA projection matrix W E R3nxd_ This allows us to approximate novel in-category objects using a low-dimensional latent vector novel E Rd, which can be used to compute point cloud
that describe K object instances that all belong to single category, e.g. set of
Y ( + Reshape Vnovell))
(2)
where the Reshape operator casts back to an n x 3 matrix.
Shape Completion From Partial Point Clouds: In practice, want to be able to approximate complete point clouds for objects for which we only have partial view [8]. This be accom- plished using the generative model by solving for
((Y) D(Y, ((Dxrttaaaal)))
using gradient descent on Essentially, struction closest to the observed points. To account for the partial view, Thompson et al. [8] use the one-sided Chamfer distance [35],
solving for the latent vector that gives recon-
D (x()) (x1)xx0)=
Σ min
x
x(,)
(4)
iE (l....)
Note that (+) E Rgx3 and x() E m x3 do not need to have the same number of points (n * m).
Interaction Warping
This section describes Interaction Warping (IW), our proposed imitation method (Figure 3). We assume that we have first trained a set of category-level generative object models of the form de- scribed in Section Then, given single demonstration desired manipulation activity, we
warping model
inferred scene
grasp demo
placement demo
object shape and pose inference
V. v.. L, R.
grasp prediction
placement prediction
V,, tyr R
JyMe
observed point cloud
Figure 3: Interaction Warping pipeline for predicting grasp and placement poses from point clouds.

---

detect the objects in the demonstration using off off-the-shelf models. For each object in the demon- stration that matches a previously trained generative model, we fit the model to the object in order to get the pose and completed shape of the object (Section 4. and 4.2). Next, we identify interaction points on pairs of objects that interact and corresponding those points with the matching points in the canonical object models. Finally, we reproduce the demonstration in a new scene with novel in-category object instances by projecting the demonstrated interaction points onto the completed object instances in the new scene (Section 4.3).
4.1 Joint Shape and Pose Inference
In order to manipulate objects in SE(3), we want to jointly infer the pose and shape of an object represented by a point cloud (partiall) To do so, we warp and transform point cloud Y E R to minimize a loss function,
((Y) D(Y, X (paatiil)) 3 maxx||X
k
which is akin to Equation 3 with the addition of the second term, a regularizer on the size of the decoded object. Our implementation regularizes the object to fit into the smallest possible ball. The main reason for the regularizer is to prevent large predicted meshes in real-world experiments, which might make impossible to find collision-free motion plans.
We parameterize Y warped, scaled, rotated and translated canonical point cloud,
(x (C) + Reshape(Wvr) Os) R² + t.
(6)
Equation 2
Here, X(C) is a canonical point cloud and v E Rd parameterizes a warped shape (as described rotation matrix and E R represents
in Section 3), E R3 represents scale, E SO(3) translation. We treat and i row vectors in this equation.
We directly optimize L with respect to u, and t using the Adam optimizer [36]. We parameterize R using R E R223 an arbitrary matrix, and perform Gram-Schmidt orthogonalization (Algorithm 5) to compute a valid rotation matrix R. This paramcterization has been shown to enable stable learning of rotation matrices [37, 38). We run the optimization with many initial random restarts, please see Appendix A.4 for further details. The inferred v, 8 represent the shape of the object captured by X(partial) and R, & represent its pose.
4.2 From Point Clouds to Meshes
We infer the shape and pose of objects by warping point clouds. But, we need object meshes to perform collision checking for finding contacts between objects and motion planning (Section 4.3). We propose a simple approach for recovering the mesh of a warped object based on the vertices and faces of the canonical object.
First, we warp the vertices of the canonical object. To do so, the vertices need to be part of (()) because our model only knows how to warp points in X(C) (Section 3). However, these vertices (extracted from meshes made by people) are usually very biased (e.g. 90% of the vertices might be in the handle of a mug), which results in learned warps that ignore some parts of the object. Second, we add points to (C) that are randomly sampled on the surface of the canonical mesh. X(C) is then composed of approximately the same number of mesh vertices and random surface samples, leading better learned warping. We construet (C) such that the first V points are the vertices; note that the ordering of points in (C) does not change it is warped.
Given a warped, rotated and translated point cloud Y (Equation 6), the first V points are the warped mesh vertices. We combine them with the faces of the canonical object to create a warped mesh M. Faces are represented as triples of vertices and these stay the same across object warps.
4.3 Transferring Robot Actions via Interaction Points

---

Consider the example of a point cloud of a mug Y that is warped using Equation We can se- lect any point Y; and track it as the mug changes its shape and pose. For example, lies on the handle of the mug, we can use it to align handles of mugs of different shapes and sizes. That can, in turn, facilitate the transfer of manipulation policies across mugs. The key question is how to find the points Yú relevant to a particular task. We call these interaction points.
point
Figure 4: (a) Contacts between gripper and bowl extracted from a demonstration. (b) Nearby points between a mug and tree extracted from a demonstration of hanging the mug on the tree. (c) A virtual point (red) representing the branch of the tree intersecting the handle of the mug. The red point is anchored to the mug using k nearest neighbors on the mug (four are shown green).
Grasp Interaction Points: We define the grasp interaction points as the pairs of contact points between the gripper and the object at the point of grasp. Let (A) and (A)) be the point cloud and mesh respectively for the grasped object inferred by our method (Section 4.1, 4.2). Let M(G) be a mesh our gripper and To the pose of the grasp. We use pybullet collision checking to find
P pairs of contact points (pj .Pj ),= where Pj (AA is on the surface of M and Pj is on the (1) (G)
surface of M(C) in pose Ta (Figure 4a). We want to warp points P3 (A) onto different shape, but our model only knows how to warp points in ((A), Therefore, we find set of indices IG = (i.....ir)) where (A) is the nearest neighbor of AA)))
Transferring Grasps: a new scene, we infer the point cloud of the new object y(A)) (Eq. 6). We solve for the new grasp as the optimal transformation TG that aligns the pairs of points (Y,(4), Pj (()), E (),...,P,, Ia- Here, y-(4") are the contact points from the demonstration warped to new object instance. Note that there is correspondence between the points in ( and y(A'); shape warping does not change their order. We predict the grasp To (Figure 5a) that minimizes the pairwise distances analytically using an algorithm from Horn et al. [39].
Placement Interaction Points:
For placement actions, look at objects being placed in relation to each other, such as mug being placed on mug-tree. Here, define interaction points as pairs of nearby points between the two object, generalization of contact points. We use nearby points so that the two objects do not have to make contact in the demonstration; e.g., the mug might not be touching the tree before it is released from the gripper. Similarly, the demonstration of an object being dropped into a container might not include contacts.
Let (4) and (F) be the inferred point clouds of the two objects. We capture the original point clouds from demonstration right before the robot opens its gripper. We find pairs of nearby points
with L2 distance below ó, (p(AA E (4)) pp(B) E y(B)) -|pp4)) - p( < 5}. Since there might
be tens of thousands of these pairs, we find representative sample using farthest point sampling [40]. We record the indices of points p) in yBB), as Tp = (11, (2,...,.p)) ir}.
We further add pj (B) as virtual points into ( this idea is illustrated in Figure 4 (b) and (c). For example, we wish to solve for pose that places a mug on a tree, such that the branch of the tree intersects the mug's 's handle. But, there is no point in the middle of the mug't handle that
use. (4) Hence, we add the nearby points (B) (e.g. points on the branch of the tree) as virtual points dj to (A), We anchor dfa) using Lnniaoost so warps together with (A), Specifically, for each point /B8) we find L nearest neighbors (N,,,,..., n,,L) in (A) and anchor ofaa) as follows,
(A) ngk
+ (p)
(B)
( ng
= (B)
To transfer the placement, save the neighbor indices ni and the neighbor displacements A
5

---

(a)
(b)
Figure 5: Predicting grasps using interaction point warping. (a) the predicted grasp for a bowl/plate changes based on the curvature of the object. (b) the placement of mug on mug tree changes as the mug grows larger so that the branch of the tree is in the middle of the handle.
Figure 6: Example of an episode of putting a mug on tree starting from a tilted mug pose.
Transferring Placements: We infer the point clouds the pair of new objects y and ( We calculate the positions of the virtual points with respect to the warped nearest neighbors,
(4))
(A) ng.k
+jkk-
(8)
We then construct pairs of points (9) (4)) formation of the first object Tp that minimizes the distance between the point pairs. Since we know how we picked up the first object, we can transform Tp into the coordinate frame of the robot hand and execute action of placing object A' onto object (Figure 5b).
(
).j f1.....,):: E Ip and find the optimal trans-
Experiments
We evaluate both the perception and imitation learning capabilities of Interaction Warping. Sec- tion 5.1, we perform three object re-arrangement tasks with previously unseen objects both sim- ulation and physical robot. In Section 5.2, we show system is capable of proposing grasps in cluttered kitchen setting from single RGB-D image.
We use ShapeNet [41] for per-category (mug, bowl, bottle and box) object pre-training (required by our method and all baselines). We use synthetic mug-tree meshes provided by [17]. Our method (IW) uses 10 training example per class, whereas all baselines use 200 examples. The training meshes are all aligned in canonical pose.
5.1 Object Re-arrangement
Setup: We use open-source simulated environment with three tasks: mug mug-tree, bowl on mug and a bottle in a container [17]. Given a segmented point cloud of the initial scene, the goal is to predict the pose of the child object relative to the parent object (e.g. the mug relative to the mug-tree). A successful action places the object on a rack / in a container so that does not fall down, but also does not clip within the rack container. The simulation does not test grasp prediction. The three tasks are demonstrated with objects unseen during pre-training. We described how our method (IW) uses single demonstration in Section 4.3; to use multiple demonstration. IW uses training prediction enror to select the most informative one (Appendix A.5).

---

#
# Train. Meshes
Mug Tree Arbitrary
Bowl on Mug
Bottle in Container
Method
Demo
Upright
Upright
Arbitrary
Upright
Arbitrary
R-NDF [17]
1
200
60.0
51.0
69.0
68.0
19.0
8.0
TAX-Pose [2]
1
200
61.0
41.0
16.0
9.0
4.0
1.0
IW (Ours)
86.0
83.0
82.0
84.0
62.0
60.0
R-NDF [17]
5
200
88.0
89.0
53.0
46.0
78.0
47.0
TAX-Pose [2]
200
82.0
51.0
29.0
14.0
6.0
2.0
IW (Ours)
5
10
90.0
87.0
75.0
77.0
79.0
79.0
R-NDF [17]
10
200
71.0
70.0
69.0
60.0
$1.0
59.0
TAX- [2]
10
200
82.0
52.0
20.0
20.0
2.0
1.0
IW (Ours)
10
10
88.0
88.0
83.0
86.0
70.0
83.0
Table 1: Success rates of predicted target poses of objects in simulation. Upright and Arbitrary refer to the starting pose of the manipulated object. Measured over 100 trials with unseen object pairs.
Mug on Tree Pick&Place
Bowl on Mug Pick&Place
Bottle in Container Pick&Place
Mean
Method
Pick
Pick
Pick
Pick
Pick&Place
NDF [5]
93.3
26.7
75.0
33.3
20.0
6.7
62.8
22.2
R-NDF [17]
64.0
12.0
37.5
37.5
26.7
20.0
42.7
23.2
IW (Ours) 96.0
92.0
87.5
83.3
86.7
83.3
90.1
86.2
Table 2: Success rates of real-world pick-and experiments with single demonstration. The manipulated object (e.g. mug) starts in arbitrary pose (we use stand to get a range of poses) and the target object (e.g. a mug-tree) starts in an arbitrary upright pose. 'The target object (e.g. the mug tree) is in fixed pose for this experiment, NDF does not handle target object variation. Each entry is measured over 25 30 trials with unseen object pairs.
In our real-world experiment, we perform both grasps and placements based on a single demonstra- tion. We capture a fused point cloud using three RGB-D cameras. We use point-cloud clustering and heuristies to detect objects in the real-world scenes (details in Appendix B. and perform motion planning with collision checking based on the meshes predicted by our method. We evaluate the ability of each method to pick and place unseen objects with varying shape and pose (Figure 8). We provide single demonstration for each task by teleoperating the physical robot. We do not have access to the CAD models of objects used in the real-world experiment.
Result: We find that our method (IW) generally outperforms R-NDF [5] and TAX-Pose [2] on the simulated relational-placement prediction tasks (Table 1) with 20 times fewer training objects. We chose these two baselines recent state-of-the-art SEE(3) few-shot learning methods. IW can usually predict with above 80% success rate even with 1 demo, whereas R-NDF and TAX-PoSt can only occasionally do with 5+ demos, and often fail to reach 80% success rate all. We use an open-source implementation of R-NDF provided by the authors [42], which differs in performance from the results reported in [17]. TAX-Pose struggles with precise object placements in the bowl on mug and bottle in box tasks; it often places the pair of objects inside one another. Occasionally, adding more demonstrations decreases the success rate because some demonstrations are of low quality (e.g. using decorative mugs with strange shapes).
In real-world pick and place experiments, we demonstrate the ability of IW to solve the three object re-arrangement tasks mug on tree, bowl on mug and bottle in box with unseen objects (Figure 8) and variation in the starting pose objects (Table 2). We find that NDF and R-NDF [5, 17] struggle with the partial and noisy real-world point clouds. This often results in both the pick and place actions being too imprecise to successfully solve the task. Pre-training (R-)NDF on roal-world point clouds could help, but note that IW was also pre-trained on simulated point clouds. We find that the warping of canonical objects is more robust to noisy and occluded point clouds. We show an example episode of placing mug tree in Figure 6.
7

---

(a)
(b)
(c)
(d)
(c)
Figure 7: Grasp prediction in the wild: (a) an RGB-D (depth not shown) image. (b) open- apea-locabularyy object detection and segmentation using Detic [43] and Segment Anything [44], (c) object meshes predicted by our method based on segmented point clouds (we filter distant and small objects), (d) meshes projected into the original image, (e) grasps predicted by Interaction Warping projected into the original image. Figure has additional examples.
We use the meshes predicted by IW to perform collision checking during motion planning. We do not perform collision checking (other than to avoid contact with the table) when using (R-)NDF as these methods do not predict object meshes, but failures due to collision between the robot and one of the object were infrequent in real-world (R-)NDF trials.
5.2 Grasp Prediction in the Wild
Setup: In this experiment, we show that we can combine our method with a state-of-the-an object detection and segmentation pipeline predict object meshes and robot grasps from single RGB-D image. We use open- open- vocabulary object detector Detic [43] to predict bounding boxes for common household objects and Segment Anything [44] to predict segmentation masks within these bounding boxes. We turn the predicted RGB-D images into point clouds and use our shape warping model to predict a mesh for each object. Finally, we use interaction warping to predict a robot grasp based on a single demonstration per each object class (details in Appendix B.2).
Result: We show the results for two example scenes in Figure 7 and 9, Our perception pipeline can successfully detect objects in images with cluttered backgrounds. Our warping algorithm accounts for the variation in the shape and size of objects and our interaction warping algorithm can generalize the demonstrated grasps to the novel objects.
6 Limitations and Conclusion
We introduced Interaction Warping, a method for one-shot learning of SE(3) robotic manipulation policies. We demonstrated that warping of shapes and interaction points leads to successful one- shot learning of object re-arrangement policies. We also showed that we can use open-vocabulas detection and segmentation models detect objects in the wild and predict their meshes and grasps.
I Limitations: Our method requires segmented point clouds of objects. We demonstrated real-world object detection pipeline in Section 5.2, but it can be difficult to capture clean point clouds aligned with image-based segmentations. The joint inference of shape and pose of an object takes around 25 seconds per object on an NVIDIA RTX 2080 Ti GPU, Future work could train an additional neural network to amortize the inference, or to predict favorable initialization. We use a PCA model of shape warps for simplicity; this model cannot capture the details of objects, such as the detailed shape of the head of a bottle. A model with higher capacity should be used for tasks that require high precision. Finally, predicted policy is fully determined by the shape warping model and single demonstration; our method does not learn from its failures, but it is fully differentiable.

---

Acknowledgments
This work was supported in part by NSF 1724191, NSF 1750649, NSF 1763878, NSF 1901 117, NSF 2107256, NSF 2134178, NASA 8ONSSCI9K 1474 and NSF GRFP awarded Skye Thompson. We would like the thank the CoRL reviewers and area chair for their feedback.
References
[1] D. Wang, R. Walters, and R. Platt. So(2)-equivariant reinforcement learning. In The Tenth International Conference Learning Representations. ICLR 2022, Virtual Event, April 25- 29, 2022. OpenReview.net, 2022.
[2] C. Pan, Okor,, H. Zhang, B. Eisner, and D. Held. TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulation. In 6th Annual Conference on Robot Learning, Nov. 2022.
[3] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5):1-12, 2019.
[4] L. Manuelli, W. Gao, P. R. Florence, and R. Tedrake. KPAM: KeyPoint Affordances for Leve Robotic Manipulation. In T. Asfour, E. Yoshida, J. Park, H. Christensen, and O. Khatib, editors, Robolics Research The 19th International Symposium ISRR 2019, Hanoi, Vietnam, October 6-10, 2019, volume 20 of Springer Proceedings in Advanced Robotics, pages 132-157. Springer, 2019.
[5] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitz- mann. Neural Descriptor Fields: (Elii--quiva Object Representations for Manipulation. In 2022 International Conference on Robotics and Automation, ICRA 2022, Philadelphia, PA, USA, May 23-27, 2022, pages 6394-6400. IEEE, 2022.
[6] A. Myronenko and X. Song. Point-Set Registration: Coherent Point Drift. IEEE Trans. Pattern Anal. Mach. Intell., 32(12):2262 22772225,2 Dec. 2010. ISSN 0162-8828.
[7] D. Rodriguez, Cogswell, Koo, and Behnke. Transferring Grasping Skills to Novel Instances by Latent Space Non-Rigid Registration, Sept. 2018.
[8] S. Thompson, L. P. Kaelbling, and T. Lozano-Perez. Shape-Base Transfer of Generic Skills. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 5996- 6002, May 2021.
[9] X. Huang, G. Mei, J. Zhang, and R. Abbas. A comprehensive survey on point cloud registra- tion. CoRR, abs/2103.02663 022,0290, 2021.
[10] W. Gao and R. Tedrake. kPAM 2.0: Feedback Control for Category-Level Robotic Manipula- tion. IEEE Robotics and Automation Letters, 6(2):2962-290 22229929299,,, Apr. 2021. ISSN 2377-3766.
[11] W. Gao and R. Tedrake. kPAM-SC: Generalizable Manipulation Planning using KeyPoint Affordance and Shape Completion. In 202/ IEEE International Conference on Roboties and Automation (ICRA), pages 6527-6533, May 2021.
[12] Z. Qin, K. Fang, Y. Zhu, L. Fei-Fei, and S. Savarese. KETO: learning keypoint representations for tool manipulation. In 2020 IEEE International Conference on Robotics and Automation, ICRA 2020, Paris, France, May August 31, 2020, pages 7278-7285. IEEE, 2020.
[13] M. Vecerik, J.-B. Regli, O. Sushkov, D. Barker, R. Pevceviciute, T. Rothörl, C. Schuster, R. Hadsell, L. Agapito, and J. Scholz. S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via View Consistency, Oct. 2020.
[14] D. Turpin, L. Wang, S. Tsogkas, S. Dickinson, and A. Garg. GIFT: Generalizable Interaction- aware Functional Tool Affordances without Labels, June 2021.

---

[15] L. Manuelli, Y. Li, P. Florence, and R. Tedrake. Keypoints into the Future: Self-Supervised Correspondence Model-Based Reinforcement Learning, Sept. 2020.
[16] R. Florence, L. Manuelli, and R. Tedrake. Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation. In 2nd Anmual Conference Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, volume 87 of Proceedings of Machine Learning Research, pages 373-385, PMLR, 2018.
[17] A. Simeonov, Y. Du, Y.-C. Lin, A. R. Garcia, L. P. Kaelbling, T. Lozano-Pérez, and P. Agrawal. SE(3)- Equiivaraa Relational Rearrangement with Neural Descriptor Fields. In 6th Annual Conference Robot Learning, Nov. 2022.
[18] H. Ryu, Lee, H. Lee, and Choi. Equivariant descriptor fields: Se(3)-qquivarian energy- based models for end-to-end visual robotic manipulation learning. CoRR, abs/2206.08321, 2022.
[19] E. Chun, Y. Du, A. Simeonov, T. Lozano-Perez, and L. Kaelbling. Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulation, Mar. 2023.
[20] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, Jones, A.N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[21] B. Wen, W. Lian, K. Bekris, and S. Schaal. You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration. In Robotics: Science and Systems XVIII. Science and Systems Foundation, June 2022. ISBN 978-0-99233477-855
Robotics:
[22] D. Rodriguez, A. Di Guardo, A. Frisoli, and S. Behnke. Learning Postural Synergies for Cat- egorical Grasping Through Shape Space Registration. In 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids), pages 270-276, Nov. 2018.
[23] D. Rodriguez and Behnke. Transferring Eategory-Based Functional Grasping Skills by Latent Space Non-Rigid Registration. IEEE Robotics and Automation Letters, 3(3):2662- 2669, July 2018. ISSN 2377-3766.
[24] T. Klamt, D. Rodriguez, M. Schwarz, C. Lenz, D. Pavlichenko, D. Droeschel, and Behnke. Supervised Autonomous Locomotion and Manipulation for Disaster Response with a Centaur- Like Robot. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1-8, Oct. 2018.
[25] S. Brandi, O. Kroemer, and Peters. Generalizing pouring actions between objects using warped parameters. In 2014 JEEE-RAS International Conference on Humanoid Robots, pages 616-621, Nov. 2014.
[26] Y. Li, J. L. Fu, and N. S. Pollard. Data-Driven Grasp Synthesis Using Shape Matching and Task- Baskd Pruning. IEEE Trans. Visual. Comput. Graphics, 13(4):732-747, July 2007. ISSN 1077-2626.
[27] H. Ben Amor, O. Kroemer, U. Hillenbrand, G. Neumann, and J. Peters. Generalization of human grasping for multi-fine fingered robot hands. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2043-2050, Oct. 2012.
[28] U. Hillenbrand and M. A. Roa. Transferring functional grasps through contact warping and lo- cal replanning. In 2012 IEEE/RS International Conference on Intelligent Robots and Systems, pages
Oct. 2012.
[29] R. Jäkel, S. R. Schmidt-Rohr, S. W. Rühl, A. Kasper, Z. Xue, and R. Dillmann. Learning of Planning Models for Dexterous Manipulation Based on Human Demonstrations. Int J of Soc Robotics, 4(4):437-448, Nov. 2012. ISSN 1875-4805.
10

---

[30] T. Stouraitis, U. Hillenbrand, and M. A. Roa. Functional power grasps transferred through warping and replanning. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 4933-4940, May 2015.
[31] D. Pavlichenko, D. Rodriguez, C. Lenz, M. Schwarz, and S. Behnke. Autonomous Bimanual Functional Regrasping of Novel Object Class Instances. In 2019 JEEE-RAS 19th International Conference on Humanoid Robols (Humanoids), pages 35 351-358, Oct. 2019.
[32] H. Tian, C. Wang, D. Manocha, and X. Zhang. Transferring Grasp Configurations using Active Learning and Local Replanning. In 2019 International Conference on Robotics and Automa- tion (ICRA), pages 1622-1628, May 2019.
[33] A. X. Lee, A. Gupta, H. Lu, S. Levine, and P. Abbeel. Learning from multiple demonstrations using trajectory-awert non-rigid registration with applications to deformable object manipula- tion. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5265-5272, Hamburg, Germany, Sept. 2015. IEEE ISBN 978-1-4799-9994--
[34] J. Schulman, J. Ho, C. Lee, and P. Abbeel. Learning from Demonstrations Through the Use of Non-rigid Registration. In M. Inaha and P. Corke, editors, Robotics Research, volume 1 14, pages 339-354, Springer International Publishing, Cham, 2016. ISBN 978-3-319-28870-3 978-3-319 28-27-7--
[35] H. G. Barrow, J. M. Tenenbaum, R. C. Bolles, and H. C. Wolf. Parametric correspondence and chamfer matching: Two new techniques for image matching. In R. Reddy, editor, Proceed- ings of the 5th International Joint Conference on Artificial Intelligence Cambridge, MA, USA, August 22-25, 1977, pages 659-663. William Kaufmann, 1977.
[36] D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization, Jan. 2017.
[37] I.. Falorsi, P. de Haan, T. R. Davidson, N. D. Cao, M. Weiler, P. Forré, and T. S. Cohen. Explorations in homeomorphic variational auto-encoding. CoRR, htbs/1807.04689 2018.
[38] Park, O. Biza, I.. Zhao, J. van de Meent, and Walters. Learning symmetric embeddings for equivariant world models. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 17372-17389. PMLR, 2022.
[39] B. K. P. Horn, H. M. Hilden, and S. Negahdaripour. Closed-form solution of absolute orienta- tion using orthonormal matrices. J. Opt. Soc. Am. A, 577) Jul 1988.
[40] Y. Eldar, M. Lindenbaum, M. Porat, and Y. Y. Zeevi. The farthest point strategy for progressive image sampling. In 12th IAPR International Conference on Pattern Recognition, Conference C: Signal Processing / Conference D: Parallel Computing, ICPR 1994, Jerusalem, Israel, 9-13 October, 1994, Volume 3, pages 93-97 IEEE, 1994.
[41] A. X. Chang, Funkhouser, I. Guibas, P. Hanrahan, Q. Huang, 7. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNet: An Information-Rich 3D Model Reposi- tory, Dec. 2015.
[42] A. Simeonov, and Agrawal. -qquivaaan relational rearrangement with neural descriptor fields. 2022.
Y. Du, L. Yen-Chen, A. Rodriguez, L. P. Kaelbling, T. L. Perez,
[43] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra. Detecting wrenty-thousand classes using image-level supervision. In S. Avidan, G. J. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, editors, Computer Vision ECCV 2022- 17th European Conference, Tel Aviv, Is- rael, October 23-27, 2022, Proceedings, Part IX, volume 13669 of Lecture Notes in Computer Science, pages 550-368. Springer, 2022.
11

---

[44] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W. Lo, P. Dollár, and R. B. Girshick. Segment anything. CoRR, abs/2304.02643, 2023.
[45] K. He, G. Gkioxari, P. Dollár, and R. B. Girshick. Mask R-CNN. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2980- 2988. IEEE Computer Society, 2017.
[46] B. Cheng, I. Mísra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention Mask Transformer for Universal Image Segmentation, June 2022.
[47] T.-Y. Lin, M. Maire, Belongie, Bourdev, R. Girshick, Hays, Perona, D. Ramanan, C. Zitnick, and P. Dollár. Microsoft COCO: Common Objects in Context, Feb. 2015.
[48] B. Zhou, H. Zhao, X. Puig. S. Fidler, A. Barriuso, and A. Torralba. Scene Parsing through ADE20K Dataset. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5122-5130. IEEE, July 2017.
[49] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn A. Dosovitskiy, A. Ma- hendran, A. Arnab, M. Dehghami, Z. Shen, X. Wang. X. Zhai, T. Kipf, and N. Houlsby, Simple Open- Vocabulayy Object Detection with Vision Transformers, July 2022.
12

---

A
Method Details
We included the code for both our simulated and real-world experiments for reference. Please find it in the supplementary material under iw_code. Algorithms and 2 describe our warp learning and inference.
Algorithm Warp Learning
Input: Meshes of K example object instances (objı jbbg.,,bjj....... objx].
Output: Canonical point cloud, vertices and faces and a latent space of warps. Parameters:
Smoothness of CPD warping and number of PCA components T.
Sample small point cloud per object (Appendix A.1). > Select canonical object with index C (Appendix A.2). 3: canon = Concat(objc.verttin SabpleL(objc)).
1: PCD (SampleS(obi,)
2: C' Select Cellectaanoniic((((D))
b Use both vertices and surface samples.
4: for i E 21, .... K) + C do
5:
Wc = (cnnonn
> Coherent Point Drift warping (Section 3).
6: end for
7: Dw
([
b Dataset of displacements of canon.
8: PCA = FitPCA (Dw. components = L). Learn a latent space of canonical object warps.
9: return Canon(points canon, vertices obje. vertices, faces objc.faces), PCA.
Algorithm Warp Inference and Mesh Reconstruction
Input: Observed point cloud ped, canonical object canon and latent space PCA.
Output: Predicted latent shape v and pose T. Parameters: Number of random starts S, number of gradient descent steps T, learning rate 7) B.
and object size regularization
1: to ped Σ pcd|
ped,
21=1
2: ped ped tg.
Center the point cloud.
3: for = 1 to do
4: Riit Random initial 3D rotation matrix.
9
5: Initialize v = (0 0 s= (1 1), ,4 = (0 0) ft
6: Initialize Adam [36] with parameters v, 8, l,, and learning rate yj.
7:
for j = to do
8:
Reshape(Wey
9:
X = canon.points +
Warped canonical point cloud.
10:
R (F)
X (X @ 8) R RT + tj.
Scaled, rotated and translated point cloud. > One-sided Chamfer distance.
12:
C =
[ned]
1 Σ pcd
min
XX|
|pcd, Xi112-
C C 3
Object size regularization.
14:
Take a gradient descent step to minimize L using Adam.
end for
16: end for
Find parameters v* ,t Ripit R² with the lowest final loss across E
18: X = canon. points + Reshape(Wa W"r"
= (X O (R (RR)) + t + ty.
Complete point cloud in workspace coordinates.
20: vertices = (X,, X2, X caanon venoonvertioes)
> First /canon.vertices/ points of X are vertices. canon. facce)...ce).
verticos
21: return Mesh( vertices = vertices, faces
Warped mesh.
13

---

A.1 Point Cloud Sampling
We trimesh' to sample the surface of object meshes. The function trimesh. sample. sample surface even samples specified number of points and then rejects points that are too close together. We sample 2k points for small point clouds (SampleS) and 10k point for large point clouds (SampleL).
A.2 Canonical Object Selection
Among the K example objects, we would like to find the one that is the easiest to warp to the other objects. For example. if we have ten examples mugs, but only one mug has square handle, should not choose it it might be difficult warp it conform to the round handles of the other nine mugs. We Algorithm 3, which computes K K - warps and picks the object that warps to the other K 1 objects with the lowest Chamfer distance. We also note an alternative and computationally cheaper algorithm from Thompson et al. [8], Algorithm 4. This algorithm simply finds the object that the most similar to the other K objects without any warping.
Algorithm Exhaustive Canonical Object Selection
Input: Point clouds of K training objects (X(1), X(2) ..... X(x)).
Output: Index of the canonical object.
1: for to K do
2: for 1 to K, do
3:
W,+j CPD(X(1) x
D. Warp point cloud i to point cloud j-
4: Ci. = x 1 min
X
x
(X0) W
5: end for
6: end for
7: for = 1 to K do
8:
C
م ش- 1ight Ci.i
> Cumulative cost of point cloud warps.
9: end for
10: return arg min C.
K
Pick point cloud that is the easiest warp.
Algorithm 4 Approximate Canonical Object Selection [8]
Input: Point clouds of K training objects (X(1), X(2) x(K)). Output: Index of the canonical object.
for 1 to do
2: for 1 to K, # do
3:
Ci. XCJ
Σ
x k=
min
1=1
X
(
x.00
4: end for
5: end for
6: for = 1 to K do
7: C, C = 1z=1 Cia
8: end for
9: return arg mink K C,
A.3 Gram-Schmidt Orthogonalization
We compute a rotation matrix from two 3D vectors using Algorithm 5 [38].
bbnnss//gthubbbom/mmiiktthttttts
14

---

Algorithm 5 Gram-Schmidt Orthogonalization
Input: 3D vectors u and v.
Output: Rotation matrix.
u/ u/ //||u||
(w' 2: o' = [
3: w' ul o
4: return Stack(u', v', w')
A.4 Shape and Pose Inference Details
The point clouds Y R starts in canonical form with latent shape e equal zero. We set the initial scale one, translation zero and rotation R identity,
s s= 1), t= (0 0), R 0
- (9)
0
R is then transformed into R E SO(3) using Algorithm 5. We minimize L with respect to v, and R using the Adam optimizer [36] with learning rate 10-2 for 100 steps. We set / 10-2 We found the optimization process is prone to getting stuck in local minima; e.g., instead of aligning the handle of the decoded mug with the observed point cloud, the optimizer might change the shape of the decoded mug to hide its handle. Hence, we restart the process with many different random initial rotations and pick the solution with the lowest loss function. Further, we randomly subsample Y to 1k points each gradient descent step this allows us run 12 random starting orientations NVIDIA RTX 2080Ti GPU.
A.5 Using Multiple Demonstrations
Our method transfers grasps and placements from a single demonstration, but our simulated ex- periment, have access to multiple demonstrations, We implement simple heuristic for choosing the demonstration that fits our method the best: we make prediction of the relational object place- ment from the initial state of each demonstration and select the demonstration where prediction is closest to the demonstrated placement. The intuition is that we are choosing the demonstration where our method was able to warp the objects with the highest accuracy (leading to the best place- ment prediction). This is especially useful in filtering out demonstrations with strangely shaped objects.
B Experiment Details
B.1 Object re-arrangement on a physical robot
We use a URS robotic arm with a Robotiq gripper. We capture the point cloud using three RealSense D455 camera with extrinsics calibrated to the robot. For motion planning, we use Movelt with ROSI. To segment the objects, we use DBSCAN cluster the point clouds and simple heuristics (e.g. height, width) detect the object class.
B.2 Grasp prediction in the wild
We use a single RealSense D435 RGB-D camera. Our goal is to be able to demonstrate any task in the real world without having to re-train our perception pipeline. Therefore, we chose an open- vocabulary object detection model Detic [43], which able to detect object based on natural lan- guage descriptions. We used the following classes: "cup", "bowl", "mug", "bottle", "cardboard", "box", "Tripod", "Baseball bat", "Lamp", "Mug Rack", "Plate", "Toaster" and "Spoon". We use
15

---

(a)
Figure 8: Objects used for the real- tasks: (a) mug on tree, (b) bowl (or plate) on mug and (c) bottle in box. We use single pair of objects to generate demonstrations and test on novel objects.
the predicted bounding boxes from Detic to condition a Segment Anything model [44] to get ac- curate class-agnostic segmentation masks. Both Detic² and Segment Anything' come with several pre-trained models and we used the largest available. Finally, we select the pixels within each seg- mentation mask and use the depth information from depth camera create per-object point cloud. We use DBSCAN to clouster the point cloud and filter out outlier points. Then, we perform mesh warping and interaction warping to predict object meshes and grasps.
Previously, we experimented with Mask R-CNN [45] and Mask2Former [46] trained on standard segmentation datasets, such as COCO [47] and ADE20k [48]. We found that these dataset lack the wide range of object classes would see in household environment and that the trained models struggle with out-of-distrriution viewing angles, such as looking from a steep top-down angle. We also experimented with an open- open-vocabulary object detection model OWL-ViT [49] and found it to be sensitive to scene clutter and the viewing angle.
C Additional Results
Training and inference times: We measure the training and inference times of TAX-Pose, R-NDF and IW (Table 3). Both R-NDF and IW take tens of seconds either perceive the environment or to predict an action. This is because both of these methods use gradient descent with many random restarts for inference. On the other hand, TAX-Pose performs inference in fraction of second but requires around 16 hours of training for each task. Neither R-NDF nor IW require task tasc-spe training. We do not include the time it takes to perform pre-training for each class of objects, which required by all three methods, because we used checkpoints provided by the authors of TAX-Poss -oos and R-NDF.
Additional real-world grasp predictions: We include additional examples of real-world object segmentation, mesh prediction and grasp prediction in Figure 9.
D Limitations
Limitations of shape warping: Shape warping works well when we can smoothly warp shapes between object instances, but it would struggle with changing number of object parts. For example, if we had a set of mug trees that have between one and six branches, shape warping would pick one of these trees the canonical object and it would not be able to change the number of branches in the canonical tree.
Further, many object-oriented point cloud based methods (like IW and NDF) are limited by the receptive field of the point cloud they model. For example, if we wanted to perform cooking task, both of these methods would not be able model the entire kitchen aisle the entire stove. We
16

---

Method
Training
Perception
Grasp prediction
Placement prediction
TAX-Pose [2] R-NDF [17]
16.5 + 1.3h
0.02 + 0.01
0.02 = 0.01
21.4 = 0.5
42.5 1.8 s
IW (Ours)
29.6 + 0.2 s
0.01 + 0.01
0.003 + 0.004
Table 3: Approximate training and inference times for our method and baselines measured over five trials. R-NDF and IW do not have an explicit training phase, as they use demonstrations nonpara- metrically during inference. Only IW has perception step that is separate from the action prediction step. We do not include the time it takes to capture a point cloud or to move the robot. Training and inference times were measured system with single NVIDIA RTX 2080Ti GPU and an Intel 17-9700K CPU.
Figure 9: Additional examples, please see Figure 7.
Figure 10: Example of mug on tree episode.
17

---

Figure 11: Example of bowl/plate on mug episode.
Figure 12: Example of bottle in box episode.
would have to manually crop the point cloud only to the top of the stove or the particular burner we want to place a pan onto.
Limitations of joint shape and pose inference: Joint shape and pose inference is prone to getting stuck in local minima. For example, instead of rotating mug to match its handle to the observed point cloud, our inference method might change the shape of the mug to make the handle very small. We address this problem by using many random starting orientations the full inference process takes 25 seconds per object on an NVIDIA RTX 2080 GPU.
Pose inference might also fail when do not the bottom of the object. We subtract the table from the point cloud, and at the bottom. Then, the inference process might not be able to tell if the mug is right side up or upside down.
observed point cloud of a mug might have opening both at the top
18


Reviews:
[{"review_id": "spq3qNeL8XR", "reviewer": "Reviewer_M4iE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The work proposes the use of Coherent Point Drift as opposed to direct point correspondence, in order to find the semantically corresponding SO(3) manipulation action to a single demonstration. The method outperforms all baselines in several tasks, and its application is successfully demonstrated with a real robot. ", "summary_of_recommendation": "My recommendation is weak acceptance since the work is interesting, novel and improving the state of the art, however, there are some more questions that need to be addressed. I accept because I am willing to argue for this paper’s acceptance. However, I’m also willing to argue the contrary based on any specifics that other reviews point out and can ultimately support either decision.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "-  Was at any point a real-world demonstration used, or was real world data only used for inference tests?\n\n- L125-L128 are a bit unclear to me, why would it be large without the regularizer?\n\n- Is there any explanation for the big drop in performance on upright mugs for 10 demonstrations (Table 1)? Typically more demonstrations are expected to produce better results. Similarly unexpected drops are observed in some of the other cells as well. This might indicate that the number of trials was not sufficient?\n\n- Did you consider transferring grasps to novel grippers. Would it be possible with this method", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths:\n\nThe evaluation is good – a good choice of baseline methods is chosen, and the experimental design is appropriate. The presentation of the work is clear and well-structured, with well-designed figures. The method is original and clearly better than the baselines. \n\nWeaknesses:  \n- If my understanding of the work is correct, the biggest weakness of this approach is that parts of the method appear heuristic/arbitrarily focused on the mug on rack task  - such as the choice of how virtual points are determined; or choosing to use the mug points before release from the gripper. This limits the applicability of the method to other use-cases. \n\n- (L93) A full point cloud of the objects is necessary for the demonstrations, which may be difficult to obtain.\n\n- The method is limited to inferring on objects that are similar to the ones in the demonstration set, and it would likely fail when dealing with objects that have additional parts, or missing parts - even if these are not relevant to the task\n\n- The limitations section contains three limitations – the difficulty of using real RGB-D cameras along with segmentation, the 25 second inference time, and PCA being too simplistic for capturing of details. I expected a much more in-depth analysis of failure modes, use-cases in which the model would not be applicable, etc. \n\n## Minor Comments for paper improvements that could be considered:\n\n- Phi(.) from CPD can be described in more details.\n\n- L147 typo “that are ignore” \n\n- Figure 4. typo: “k nearest neighbours”\n\n- L211 type “three RGB-D [cameras/sensors/images].”\n\n- 4.2 could use a figure to illustrate the problem\n\n- Tekden et al.,  “Affordance Transfer based on Self-Aligning Implicit Representations of Local Surfaces” appears to be doing something similar and might provide inspiration for a potential continuation of this method toward using object parts instead of full objects for this purpose.\n\n- L255: single desktop GPU is less informative than a specific model GPU.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "aek69Ihq43x", "reviewer": "Reviewer_XXj1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a new method, interaction warping, to tackle one-shot imitation learning. The overall pipeline uses keypoints on the object point cloud to predict grasps and other robot actions, and such keypoints are obtained through collision checking and other heuristics.\n\nThe main novelty of this work lies in how keypoints detected in the training object can be effectively transferred to unseen objects in the same category. Unlike previous works with descriptor fields, this paper uses shape warping, a way to align a new object to a canonical object through individual point displacements. This approach is shown to achieve higher performance when compared to previous methods.", "summary_of_recommendation": "The paper is novel and interesting, though some flaws need to be addressed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "For Con-1:\n- More details about the data used for training the generative model/PCA should be provided.\n- Could NDF baselines leverage these data? Is the comparison fair?\n\nFor Con-2\n- More discussions about which object categories and tasks will be suitable for this approach.\n- What are some example categories that warping methods may struggle? Does NDF approaches have similar issues? What are some object categories that both warping and NDF would struggle?\n- This would give readers who are less familiar with this line of work better intuition.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*quality*\n\nPros\n- Using shape warping to align point cloud is novel for one-shot imitation learning, and overall a neat idea.\n- Empirical analysis includes two baseline and 3 sim/real tasks. 100 trials are conducted for each task*method combination, making the results significant and convincing.\n\nCons\n- It is not clear if comparing the proposed method with Neural Descriptor Field (NDF) baselines is a fair comparison. The proposed method requires access to \"a set of category-level generative object models\" (line 112, page 3), which I assume requires additional training data. While NDF baselines may not be able to make use of this additional information.\n- It is not clear how general the proposed method is. Similar to previous works, the method is tested only on cup/bowl-like objects. The procedure to extract \"nearby points\" (line 175, page 6) appears to be quite specific to the Mug Tree task.\n\n*clarity*\n- The presentation is clear and detailed.\n\n*originality*\n- The proposed method is original and novel.\n\n*Significance*\n- The paper is interesting and significant.\n\n\n\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "FJ3lFATnwc3", "reviewer": "Reviewer_odM7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose Interaction Warping, a method for learning policies from a single demonstration for pick-and-place tasks involving two objects. The proposed method assumes prior knowledge and access to several CAD models from each object category. The authors use CPD with PCA to create a generative model for point cloud completion and warping. Given a single demonstration on one of the CAD models comprising the dataset used to fit the generative model, Interaction Warping first detects the contact points between the gripper and the object to be picked, and between the grasped object and the object it should be placed/hung on. At test time, the partial point cloud of both test objects is compleated and warped to the point clouds used during the demonstration using the learned generative model. The output warping is used to map the contact points retrieved from the demonstration to the novel object instances, which are then realised to complete the task.", "summary_of_recommendation": "Although the proposed method is interesting and appealing, the novelty appears to be incremental compared to prior work, introducing a method for transferring placement/hanging contact points that is analogous to the method used for transferring grasping contact points. Moreover, from my understanding, a major limitation of the method is that all CAD models from a single object category have to be pre-aligned, and that the demonstration must be provided in simulation on one of the models used to fit the generative model. Moreover, the paper could be better organised and its clarity improved. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "- Do all CAD models used to fit the generative model need to be pre-aligned?\n- Is the demonstration provided in simulation (as it must be provided on one of the CAD models used to fit the generative model)?\n- Is there a way to provide the demonstration in the real world without having access to the object CAD model?\n- How does the proposed method change when using more than one demonstration, as done in the experiments section?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Quality**: The idea of detecting contact points from a demonstration, warping them to novel object instances, and analytically solving for end-effector and object poses that realise these contact points is very appealing and intuitive. The majority of the paper is also technically sound. However, there are some claims that are not well supported. For example, the authors claim that one of the advantages of the proposed method over prior work is that their \"method is agnostic to the use of neural networks\" without providing a justification for why this is an advantage. Moreover, in the limitations section, the authors say that their generative model \"cannot capture the details of objects\" while many neural-network-based models can, making it seem as if the lack of neural networks in the proposed method is a disadvantage.\n\n**Originality**: Transferring grasp poses using the CDP + PCA model has been done in [1], while gradient descent on both the pose and the shape was previously done in [2]. The authors acknowledge both of these papers in their related work section and extend the previous line of work to placing/hanging.\n\n**Clarity**: The clarity of the paper could be improved, as currently, the paper raises some questions, including:\n- Do all CAD models used to fit the generative model need to be pre-aligned?\n- Is the demonstration provided in simulation (as it must be provided on one of the CAD models used to fit the generative model)?\n\n**Strengths**:\n- The method requires only a single demonstration.\n- Learned policies generalise to novel object instances.\n\n**Weaknesses**: \n- The method requires prior object knowledge.\n- The demonstration must be given in simulation.\n\n**References**:\n\n[1] D. Rodriguez, A. Di Guardo, A. Frisoli and S. Behnke, \"Learning Postural Synergies for Categorical Grasping Through Shape Space Registration,\" 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids), Beijing, China, 2018, pp. 270-276, doi: 10.1109/HUMANOIDS.2018.8625032.\n\n[2] D. Rodriguez, C. Cogswell, S. Koo and S. Behnke, \"Transferring Grasping Skills to Novel Instances by Latent Space Non-Rigid Registration,\" 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia, 2018, pp. 4229-4236, doi: 10.1109/ICRA.2018.8461169.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "EUKdaJ_1mjk", "reviewer": "Reviewer_mU8N", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces a method for one-shot learning of SE(3) robotic manipulation policies called Interaction warping it works such that for an object class it has one demonstration. The shape and contact points are warped to a different shape in the same class, therefore, with only one method you can theoretically, if all works well, manipulate all objects of that class. Results show improvement over other methods from the literature", "summary_of_recommendation": "Nice conference paper with potentially a very impactful and useful method.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "the paper is interesting and the results look promising. As the authors point out themsleves, they use PCA model of shape warps, when something with higher capacity might work better. Also, the objects are somewhat simple, would it work on momre complex object (i.e. chair, even though a chair is somewhat bigger and more difficult to manipulate)?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is interesting and quite clear, with supplementary material to clear up additional questions. The originality and significance of the paper are considerable.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new method for one-shot imitation learning, which proposes the use of Coherent Point Drift to align different object instances from the same object category, based on point cloud observations. Heuristics used to detect keypoints from a demonstration then allow the demonstration to be transferred to novel objects of the same category. Simulation and real-world experiments show that this method outperforms some recent well-known baselines.

Reviewer scores before the rebuttal were 1 x “weak reject”, 2 x “weak accept”, and 1 x “strong accept”. Authors provided a rebuttal, and an updated paper which included some minor clarifications to the text. Following the rebuttal, reviewer scores were 3 x “weak accept”, and 1 x “strong accept”. The “weak reject” reviewer upgraded to “weak accept” following satisfactory answers to their questions. The AC then initiated a discussion amongst reviewers, although there was already a positive consensus amongst the reviewers and so the discussion was short.

Overall, the AC and reviewers find this to be an interesting and sensible method, with compelling results. Some general criticisms still remain, such as (1) the need for aligned CAD models for each object category, (2) the use of a heuristic for the virtual points which may not be appropriate for other tasks, and (3) the limited range of tasks evaluated on. However, there is still a consensus that this paper is useful for the robot learning community and that it should therefore be accepted at CoRL. Please be sure that for the final paper, the clarifications added to the updated paper remain (the updated paper is currently over 8 pages), and that the three limitations above are adequately addressed.
Decision: Accept (Poster)