TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulation
Authors: Chuer Pan
Brian Okorn
Harry Zhang
Ben Eisner
David Held
Venue: CoRL 2022 Poster (2022)
Keywords: Learning from Demonstration
Manipulation
3D Learning

Abstract: How do we imbue robots with the ability to efficiently manipulate unseen objects and transfer relevant skills based on demonstrations? End-to-end learning methods often fail to generalize to novel objects or unseen configurations. Instead, we focus on the task-specific pose relationship between relevant parts of interacting objects. We conjecture that this relationship is a generalizable notion of a manipulation task that can transfer to new objects in the same category; examples include the relationship between the pose of a pan relative to an oven or the pose of a mug relative to a mug rack. We call this task-specific pose relationship “cross-pose” and provide a mathematical definition of this concept. We propose a vision-based system that learns to estimate the cross-pose between two objects for a given manipulation task using learned cross-object correspondences. The estimated cross-pose is then used to guide a downstream motion planner to manipulate the objects into the desired pose relationship (placing a pan into the oven or the mug onto the mug rack). We demonstrate our method’s capability to generalize to unseen objects, in some cases after training on only 10 demonstrations in the real world. Results show that our system achieves state-of-the-art performance in both simulated and real-world experiments across a number of tasks. Supplementary information and videos can be found at https://sites.google.com/view/tax-pose/home.

TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulation
Chuer Pan, Brian Okorn, Harry Zhang; Ben Eisner; David Held Robotics Institute, School of Computer Science Carnegie Mellon University, United States (chuerp, bokorn, haolunz, baeisner, dheld] cmu, edu
Abstract: How do we imbue robots with the ability to efficiently manipulate seen objects and transfer relevant skills based on demonstrations? End-to-end learning methods often fail generalize novel objects unseen configura- tions. Instead, we focus on the task-specific pose relationship between relevant parts interacting objects. We conjecture that this relationship notion of a manipulation task that can transfer to new objects in the same category; examples include the relationship between the pose of a pan relative an oven or the pose of a mug relative to mug rack. We call this task-specific pose rela- tionship "cross-pose" and provide a mathematical definition of this concept. We propose a vision-basee system that learns to estimate the cross-pose between two objects for given manipulation task using learned cross-object correspondences. The estimated pooss-pose is then used to guide downstream motion planner to manipulate the objects into the desired pose relationship (placing pan into the oven or the mug onto the mug rack). We demonstrate our method's 's capability to generalize to unseen objects, some cases after training on only 10 demonstra- tions in the real world. Results show that our system achieves state-of-the-an per- formance in both simulated and real-world experiments across a number tasks. Supplementary information and videos can be found on project website.
generalizable
Keywords: Learning from Demonstration, Manipulation, 3D Learning
TAB
Corrected Correspondences Estimated Cross-Pose and Importance Weights Figure 1: To solve relative placement task, TAX-Pose cross-object attention estimate dense object correspondences and importance weights for each object point. This dense estimate mapped to single "cross-por pesss"poee" which the robot uses to accomplish the given task.
Observation
Cross-Objett Attention
Robot Execution
Introduction
Many manipulation tasks require a robot to move an object location relative to another object. For example, a cooking robot may need to place lasagna in an oven, place pot on stove, place plate in microwave, place mug onto mug rack, place cup onto shelf. Understanding and placing objects in task-specific locations a key skill for robots operating in human environments. Further, this skill should generalize novel objects within the training categories, such placing new trays into the oven or new mugs onto mug rack. common approach robot learning is to train a policy "end-to-end," mapping from pixel observations to low-level robot actions. However, end-to-end trained policies cannot easily reason about complex pose relationships such as the ones described above, and they have difficulty generalizing unseen object configurations.
"Equal Contribution. See Appendix H for a detailed list of each author's contributions.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

In contrast, we propose a method that learns to reason about the 3D geometric relationship between a pair of objects. For the type of tasks defined above, the robot needs reason about the relationship between key parts on one object with respect to key parts on another object. For example, to place a mug on a mug rack, the robot must reason about the relationship between the pose of the mug handle and the pose of the mug rack; if the mug rack changes its pose, then the pose of the mug must change accordingly in order to still be placed on the rack (see Figure 3). We name this task- specific notion of the pose relationship between pair of objects as "cross-pose" and we formally define it mathematically. Further, we propose a vision system that can efficiently estimate the cross- pose from small number of demonstrations of given task, generalizing to novel objects within the training categories. To complete the manipulation task, we use the estimated cross- as the target of motion planning algorithm, which will move the objects into the desired configuration (e.g. placing the mug onto the rack, placing the lasagna into the oven, etc).
In this paper, we present TAX-Pose (TAsk-specific Cross- Pose), a deep 3D vision-based method that learns to pre- dict task-specific pose relationship between a pair of objects from a set of demonstrations. Our cross-pose esti- mation system is provably translation equivariant and can generalize from a small number of real-world demonstra- tions (in some cases as few 10) to new objects in unseen poses.
Observations
Robot Execution
The contributions of this paper include:
1. A precise definition of "cross-pose," which defines task-specific pose relationship between two objects.
A novel method that estimates soft-correspondences between two objects, from which the cross-pose be- tween the objects can be estimated (see Figure I); this method is provably translation equivariant and can learn from a small number of real-world demonstra- tions.
Figure 2: We study relative placement tasks, which one object needs to be placed in a position relative to another object. Here are two of the tasks that we demonstrate our method on: Top: PortNet-Mobilty Place- ment Task requires one object (e.g. block) placed relative to another object (e.g. a drawer) by a semantic goal position (e.g. in- side); Bottom: Mug Hanging Task requires placing the mug's s handle the mug rack.
A robot system to manipulate objects into the desired cross-pose to achieve given manipulation task.
We present simulated and real-world experiments test the performance of our system in achieving a variety of relative placement manipulation tasks. We demonstrate our method on a semantic placement task, in which the robot must place an object in, on, or around a novel ob- ject (Figure 2, top). We also demonstrate our method on precise placement tasks, such as hanging a mug on rack (Figure 2, bottom) or placing a bottle or bowl on shelf; in both cases our method generalizes to new object configurations and new objects within the training categories.
2 Related Work
Object Pose Estimation: Pose estimation is the task of detecting and inferring the 6DoF pose of an object, which includes its position and orientation, with respect some previously defined object reference frame [1, 2, 3, 4,5, 5, 6]. Recent work [7, 8, 9, 10] proposed to use 3D semantic keypoints as an alternative form of object representation. While keypoint-based methods can generalize within an object class, they require a significant amount of hand annotated data or access to a simulated version of the task to learn estimate the keypoint locations. In contrast, our method is able to learn from just 10 real-world demonstrations Another approach is to use dense embeddings, such as Dense Object Nets (DON) [11] and Neural Descriptor Fields (NDF) [12], which achieve generalization across classes by predicting dense embeddings in the observation and matching them to embeddings of the demonstration objects. However, DON [11] and NDF [12] assume that the target object is moved relative to static reference object in a "known canonical configuration" (e.g. the pose of the mug rack in NDF [12] is assumed to be known and fixed). In contrast, our method reasons about the geometric relationship between pair of objects and hence does not need to assume

---

a static environment. Thus, for example, our method is able to perform the mug hanging task while varying the pose of the mug rack (see our project website), whereas the baselines (DON [11], NDF (12]) cannot. Further, we show that our method significantly outperforms both the DON [11] and NDF [12] baselines, especially when given a very small number of demonstrations.
Point Cloud Registration: Our method for estimating the cross-pose between two objects builds upon previous work in point cloud registration. The typical objective in point cloud registration is to find the optimal rigid alignment between two point clouds, to minimize the sum of squared distances between two sets of points. Traditionally, Iterative Closest Point (ICP) [13] and its vari- ants [14, 15, 16, 17, 18, 19] have been used to compute the optimal rigid alignment between two point clouds. Deep Closest Point (DCP) [20] avoids local minima common for ICP by seeking to approximate correspondence in high- -iggehinnniionl learned feature space. Our method builds upon the architecture of DCP for cross-pose estimation; however, contrast to point cloud registration, in which the objective is to minimize the sum of squared distances between two sets of points on the same object in two different poses, our objective is to estimate a task-specific pose relationship between two different objects. Extending the framework from DCP, we learn residual to the soft correspondences, allowing for points to match outside the convex hull of each object. This com- ponent is necessary when computing soft correspondences between objects of drastically different morphologies (such as a mug and a mug rack).
3 Problem Statement
Relative placement tasks: In this paper, we are specif- ically interested in "relative placement tasks." Given two objects, A and B, "relative placement task" is the task of placing object A a pose relative to object B. For ex- ample, consider the task of placing lasagna in an oven, placing mug on a rack, or placing a robot gripper on the rim of mug. All of these tasks involve placing one object (which we call the "action" " object A) at a semantically meaningful location relative to another object (which we call the "anchor" object B
Goal Configuration
Also Goal Configuration
T T-TA
TA
T B
T T
Figure 3: If we transform both the action object (mug) and the anchor object (rack) by the same transform, then the relative pose between these objects is unchanged (the mug is still "on" the rack) the mug is still in the goal configuration.
Specifically, suppose that TA and T3 are SE(3) poses for objects A and B respectively (in a shared world refer- ence frame") for which a desired task is considered com- plete (lasagna is the oven; mug is on the rack, etc). Then for a relative placement task, if objects A and B are in poses T TA and T T3 (respectively) for any trans- form T, then the task will also be considered to be complete, as seen in Figure 3. In other words, T5 represents the pose of the rack and T represents the pose of the mug on the rack (at task completion); then if we transform the both the mug and rack poses by T, then the mug will still be located on the rack. Formally, this property can be defined with the following Boolean function,
RelPlace TB) = SUCCESS iff 3T E SE(3) s.t. TA = T T and Ts = T T (I)
For many real semantic placement tasks, there are actually sets of valid solutions which solve each task (i.e., there are many potential locations to place object on table to achieve semantic "object-on-table" relationship). However, for this work, we consider precise placement tasks under the simplifying assumption that, for a given pose of object B, there is single, unambiguous pose of object A needed to achieve the task.
Definition of Cross-Poser Given the above definition of a relative placement task, our goal will be to determine how to move object A so that it will be in the "goal pose," which, as described above, defined relative to the pose of object B. To achieve this, one option is to estimate the poses of objects A and B separately and then compute the transformation needed to move object A into the goal pose. However, the pose estimate of each object will have errors, and these errors will accumulate when the poses are combined into the single relative pose needed to reach the goal configuration.
Note that the definition of action and anchor is symmetric; either object can treated as the action object and the other as the anchor.
All SE)3 transformations in this work are defined in a fixed, arbitrary world frame.
3

---

Instead of estimating the pose of each object independently, we aim to learn function f(P A.P.P.P Ps), which takes as input the point clouds PA and Ps E for both objects and B where PA R and PB E TR are 3D point clouds of sizes NA and Ng, respectively. This function outputs an SE(3) rigid transformation, A: Ps) = TAS. where we refer to TAS as the "cross-pose" between object A and object B. For notational convenience, we occasional write as a function of the poses TA, TB of point clouds P.A and Ps respectively (with respect to global reference frame) such that f(T TB) f(P PB). This notational change is to make the transformation math more intuitive; in practice, this function only ever receives point clouds as input.
We will define the cross-poss TAB (below) such that, if we transform object A by TAB. then object will be in the goal pose relative to object B for the relative placement task. For example, suppose that T and Tg are poses for objects A and B, respectively, for which a desired relative placement task considered complete. be J(T": Tg) = I where I is the identity, as object A does not need to be moved to complete the task. Further, based on the definition of relative placement task given above, if both objects are transformed by the same transform T, then the objects will still be in the desired relative pose,
this configuration, the cross-pose of these objects would
F(T- T T TB) S(T TB) =I
(2)
for any transform T SE(3). Now, let us assume that objects A and B are the goal configu- ration and have pose TA Ta T and Ts -Tg Tg. respectively, for arbitrary transforms T, and Ts E SE(3). We then define the "cross-pose" of objects and B as:
A(TA, TB) f(T. Tú T3 TB) TAS := T3 T
Note that this definition equívalent Equation for the special case of T -T, This definition of cross-pose allows us to move object A into the goal configuration, relative to object B:
TAB T (T TT (T. Tÿ) TT Tx
satisfying the relative placement condition defined in Equation 1 with T Tg. Alternatively, we
could have instead transformed object B by the inverse of the cross-pose to achieve the task.
4 Method
Overview: We frame the task of cross- posse estimation a soft correspondence prediction task be- tween pair of point clouds, followed by an analytical least-squares optimization to find the optimal cross-pose for the predicted correspondences. As described in Appendix B, this correspondence- based approach allows our method to be translation-equivrriant translating either object (AA or B) will lead to translated cross-pose prediction. This allows our method to automatically adapt to novel positions of both the anchor and action objects, unlike previous work which assumes a static anchor [12]. Our method for task-specific cross-pose estimation, known as TAX-Pose consists of the following steps, as shown in Figure 4:
1. Soft Correspondence Prediction: For pair of objects A, B, a neural network learns to predict per-point embedding to establish (soft) correspondence between A and B, which are called
Cross- Cross-Correspondence Estimator
Poan Bonthanettgg
Creediget ATTARM
Camiid Cknopecknns
stort Afrittoo Weggt
Varta sett
Comood
Cabosking dicCee)
Cres Ixtever
A Cr
Cren sljen (rrenaermn
Wrighted Avriage
V
ex(+)
Po
VILF
Chew Oliped $R. O
Tepaciane
(huring Sytwwn
Bo(-)
Figure 4: TAX-Pose Training Overview: Given a specific task, our method takes input two point clouds and outputs the cross-pose between them needed to achieve the task. TAX-Pose first learns point clouds features using two DGCNN [21] networks and two Transformers [22]. Then the learned features are each input to a point residual network to predict per-point soft correspondences and weights across the two objects. The desired cross pose can be inferred analytically from these correspondences using singular value decomposition.

---

"virtual soft correspondences. The corresponding points are constrained to be within the convex hulls of B and A respectively.
2. Adjustment via Correspondence Residuals: For most estimation tasks, some points in object A may not be within the convex hull of object B; for instance, when mug is placed on a mug rack, most points the mug will be outside of the convex hull of the mug rack. To accommodate these cases, we apply pointwise residual vector to displace each of the predicted soft correspondences. These "corrected virtual correspondences" allow points in A to correspond to locations in free space near B.
Find the Optimal Transform: Because the cross-pose defined as a rigid transformation of object A. we use differentiable weighted SVD to find the transformation that minimizes the weighted least squares difference to the corrected virtual correspondences.
Because each step above is differentiable, the whole model can be optimized end-to-end, despite having an interpretable internal structure which we describe below. Our method is heavily inspired by Deep Closest Point (DCP) [20]. The key difference between pose alignment model and DCP that we predict the cross- pose between two different objects for a given task instead of registering two point clouds of an identical object. Additionally, TAX-Pose predict relationships where these clouds may not have any points of contact or overlap.
We now describe our cross-pose estimation algorithm in detail. To recap the problem statement, given objects and B with point cloud observations PA E R3xNa, P8 E R33NS respectively, our objective is to estimate the task-specific cross-pose TAB f(P,, PB) SE(3). Note that the cross-pose between object A and B is defined with respect to a given task (e.g. putting a lasagna in the oven, putting mug the rack, etc).
4.1 Cross-Pose Estimation via Soft Correspondence Prediction
The first step of the method is to compute two sets of correspon- dences between A and B, one which maps from points in A to B, and one which maps from points in B to A. These need not be a bijection, and can asymmetric. As we want each step to be differ- entiable, we follow DCP's conventions and estimate soft correspondence. This assigns a virtual soft corresponding point v V to every point p E PA by computing a convex combination of points in PB, and vice versa. Formally:
Soft Correspondence Prediction:
No
v P PB S.E.
Σ
w =1 (5a) v PAW s.t.
w
Σ w 8 =1 (5b)
with normalized weight vectors wA W A-B8 and w - E W WB-A. Importantly, these virtual corresponding points are not constrained to the surfaces of or B; instead, they are constrained to the convex hulls of Ps and PA, respectively.
To compute the weights wAA+B w - in Equations 5a and 5b, we first encode each point cloud PA and Pg into a latent space using neural network encoder, DGCNN [21]. This encoder head comprised of two distinct encoders 9A and gIs each of which receives point cloud PA and Ps. respectively, zero-centers them, and outputs dense, point- wise embedding for each object (see Figure 4): A = (P) E Rxxx WE us(Pa) E RRd where v//" E Mx is the d- dimensional embedding of the i-th point object K., and Px: is the zero-centered point cloud for object K. Because we want the cross-corrspoondence to incorporate information about both point clouds, we then employ a cross-object attention module between the two dense feature sets to obtain cross-object point embeddings, & E N and 2g N defined as:
A V A 9TA A, L.) 4B V + 9Ts L A
where 9T 9Ts are Transformers [22].
Finally, recall that our goal was to compute set of normalized weight vectors W WA-BB. WE-AA- To compute the virtual corresponding point vz assigned any point p.d PAA, we can extract the desired normalized weight vector w from intermediate attention features of the cross-object attention module as:
KKq*) Vd
KAq vd
w. A softmax
w B-A softmax
5

---

where qf E Qr. and Qx. Ke E RNxx are the query and key values (respectively) for object I. associated with cross-object attention Transformer module 9Tk (see Appendix C for details). These weights are then used to compute the virtual soft correspondences V., Vi using Equation 5.
Adjustment via Correspondence Residuals: As previously stated, the virtual soft correspondences VA, VB given by Equations 5a and 5b are constrained to be within the convex hull of each object. However, many relative placement tasks cannot be solved perfectly with this constraint. For instance, we might want a point on the handle of a teapot to correspond to some point above a stovetop (which lies outside the convex hull of the points on the stovetop). To allow for such off-object correspondences, we further learn residual vector, 5,4 Aл for each point i that corrects each virtual corresponding point v²A. This allows us to displace each virtual corresponding point to any arbitrary location that might be suitable for the task. To compute these residual vectors, we use point-wise neural network A IRH to map each point's embedding into 3D residual vector:
8z =
E ERR ttth yRo
E RR
Applying these residual offsets to the virtual points, we get a set of corrected virtual correspon- dences, vid E VA and v." E VB, defined as
vz 5/ -
These corrected virtual correspondences v.A define the estimated goal location relative to object B for each point P; E PA of object A, and likewise for each point in object B (see visualization in Appendix A.
Least-Squares Cross-Pose Optimization with Weighted SVD: Given the sets of dense correspon- dences, A VA) and (Pss VRS. we would like to compute a single rigid transformation for ob- ject A. To do so, we solve for the transformation TAs (the cross-poss pose) that minimizes the weighted distance between each point and its corrected virtual correspondence. Formally, this leads to the following weighted least squares optimization:
NA
NB of T VAIB + Σ 1=1
J(TAB) Σ
15 AB
where the weights of a, a E as signify the importance of each correspondence and predicted by a point-wise MIP as shown in Figure 4. These weights are learned end-to-end as parameters of our network; they visualized in Appendix A.2, which shows that the network has learned to assign more weight to the parts of the object that are most important for the task, such as the region around the mug handle (on the mug) and the region around the peg (on the rack), Equation 9 is the well-known weighted Procrustes problem, for which there exists an analytical solution. To maintain the differentiablity of the system, we use weighted differentiable SVD operation [23] to compute the cross-pose TAB that minimizes this objective (see Appendix D for details). This allows us train the system end-to-end described below.
4.2 TAX-Pose Training Pipeline
To train our model, we use a segmented set of demonstration point clouds of pair of objects in the goal configuration. For each demonstration point cloud, we generate multiple training examples by transforming each object's point cloud, P.A and P& by random SE(3) transformations T, and T3, respectively. The predicted cross-pose, TA8, is then compared with the ground truth cross- pose, T := T using an average distance loss [24] with dense regularization (see more details on training losses in Appendix 1.
5 Experiments
To evaluate TAX-Pose, we conduct wide range of simulated and real-world experiments on two classes of relative placement tasks: NDF [12] Tasks and PartNettMobility placement Tasks. All tasks involve placing an "action" object at specific location relative to an anchor object, which the relative pose is specified by set of demonstrations. Our method then generalizes to perform this task on novel objects in unseen configurations. We refer the reader to our project website for additional results and videos.

---

Simulaned Demo
Ubservation
Execution
Train Mugs
Test Mugs
Execution
Figure 5: Real-world experiments summary. Left: In object placement task, we train using simulated demon- strations and test real-world objects. Right: Mug Hanging real-war well-world experiments. We train from just 10 demonstrations from 10 training mugs in the real world and test 10 unseen test mugs.
5.1 NDF Tasks
We evaluate our method on all three NDF [12] tasks (mug hanging, bottle placement, and bowl placement); see Appendix F.1.3 for results on bottle and bowl placement. Results on mug hanging are described in more detail below.
Simulation Experiments:
For our simulation experiments, we perform the task of hanging a mug on rack as two sequential cross pose estimation steps: grasping the mug (estimating the cross- pose between the gripper and the mug) and hanging the mug on the rack (estimating the cross- pose between the mug and the rack). In Pybullet [25], we simulate Franka Panda above table with 4 depth cameras placed on the corners of the table. The model is trained on 10 simulated demonstrations of mug hanging. We evaluate task execution success on unseen mug instances in randomly generated initial configurations. We measure task success rates of 1) Grasping, where success is achieved when the object is grasped stably; 2) Placing, where success is achieved when the mug is placed stably on the rack; 3) Overall, when the predicted transforms enable both grasp and place success in sequence. We compare our method to Neural Descriptor Field (NDF) [12] and Dense Object Nets (DON) [11]. Details of these methods can be found in prior work [12].
We evaluate our method in simulation in 100 trials consisting of unseen mug instances in random initial and goal configurations for both Upright and Arbitrary poses. As shown in Table 1, our method significantly outperforms the baselines for simulated mug hanging. We report additional results for simulated bottle and bowl placement tasks in Table 8 in Appendix F.1.3.
Simulation Results:
Ablation Analysis: Effects of Number of Demonstrations. To study how the number of demonstra- tions observed affects our method's performance, we train our model on {10, 1} demonstrations of upright pose mug hanging. Results are found in Table 2. Our method outperforms the baselines for all number of demonstrations; TAX-Pose can perform well even with as few demonstrations.
Cross-Pose Estimation Design Choices. We analyze the effects of design choices made in our Cross- Pose estimation algorithm for the upright pose mug hanging task. Specifically, we analyze the effects of 1) computing residual correspondence; 2) the use of weighted SVD over non-weighted in computing cross-pose; 3) using transformer the cross-object attention, as opposed simpler model such as a 3-layer MLP. Table 3 shows that each major component of our system is important for task success. See more ablation experiments in Appendix F.1.I.
Grasp
Place
Overall
Grasp
Place
Overall
Model
Demos
Ablation
Grasp
Place
Upright Pose
Arbitrary Pose
3
10
No Res.
0.97
0.96
0.93
DON [1]]
091
0.50
0.45
0.35
0.15
0.17
DON
0.32
0.36
0.5
Unw. SVD
0.92
0.9
0.83
NDF [12]
0.%6
0.92
0.88
0.78
0.75
0.58
NDF
0.46
0.70
0.88
No Alln.
0.90
0.82
0.76
Toxe
0.99
0.97
0.96
0.75
0.84
0,63
TAX-Pase
0.77
0.90
0.96
TAX-Pase
Table 1: Mug on rack simulation success rate (1)
Table 2:
Demos vs.
Table
Mug hanging abla-
Overall success rate (f)
tions success rate (t)
Real- World Experiments: We explore the hanging component of the mug on a rack task in a real world environment, which requires estimating the cross-pose between the mug and the rack. We train TAX-Pose using real demonstrations of 10 different mugs hung on rack (1 demonstration each, for total of only 10 real- real-world demonstrations for training). A motion primitive is used to grasp each mug, after which the robot plans a trajectory to apply the predicted cross-pose to the grasped mug. We evaluate the model on the 10 training mugs in novel poses, as well as on 10 unseen mugs (see Figure 5). For each of the 20 mugs, we conduct 5 trials, varying the mug's 's and rack's 's starting poses in each trial. Success is recorded if a peg penetrates the mug handle at the end of the trial. Our method achieves success rate of 62% on training mugs in novel poses and 54%
7

---

on unseen mugs. A visualization of the results can be seen in Figure 5 (right) and on the project website. Note that our method is able to perform the mug hanging task while varying the pose of the mug rack (see our project website), whereas the baselines (NDF [12], DON [11]) cannot because they assume a fixed, known rack position (see NDF [12] for baseline details).
5.2 PartNet-Mobility Placement Tasks
Task Description: We also define PartNet- Placement task as placing a given action object relative to an anchor object based on semantic goal position. We select set of household furniture objects from the PartNet-Mobility dataset [26] as the anchor objects, and a set of small rigid objects released with the Ravens simulation environment [27] as the action objects. For each anchor object, we define a set of semantic goal positions (i.e. 'top'. 'left', 'right', 'in'), where action objects should be placed relative to each anchor. Each semantic goal position detines a unique task in our pross-pose prediction framework. Given a synthetic point cloud observation of both objects, the task is to predict cross-pose that places the object at the specific semantic goal. We evaluate both condiiondt variant (TAX-Pose GC), which is trained across all goals, and task-specific variant (TAX-Pose) of our model, which trains a separate model per goal type (see Appendix F.2.2 for details). In both cases we train only 1 model across all action and anchor objects. All models are trained entirely on simulated data and transfer directly to real-world with no finetuning. Further task details can be found in the Appendix G.2.
Baselines: We compare our method to a variety of end-to-end imiration-learinn-baass methods trained from motion planner expert in simulation (see Appendix G.2.4 for details). Note that in the PartNet-Mobility Placement experiments, the pose of the anchor object poses are randomly varied. As such, we omit a comparison to methods that assume static anchor, such the Neural Descriptor Field (NDF) [12] and Dense Object Nets (DON) [11] baselines used in the mug hanging task (Section 5.1), as both methods assume that the anchor objects in fixed, known position.
Results: We report rotation (ER) and translation (E+) error between our predicted transform and the ground truth as geodesic rotational dis- tance [28, 29] and 1,2 distance, respectively. periments (Table 4 Top) and our real-world experiments (Table 4 Bot- tom), we find that TAX-Pose outperforms the baseline end-to-end im- itation learning methods, with the goal-conditioned variant, -ose GC, performing the best. In real-world experiments, our method gener- alizes to novel distributions of starting poses better than the Goal Flow baseline, placing action objects into the goal regions with a 92% suc- cess rate. See Figure 5 (left) and the website for results; see Appendix F.2 for more detailed tables and Appendix G.2.4 for baseline details.
Average
En
EIE BC
42.26
0.75
both our simulated ex-
DAgger
37.96
Ita.. Flow
35.93
0.67
Flow
26.64
0.17
TAX- Pose
6.64
0.16
Poxe GC
4.34
0.16
Average SR
Gral Flow
0.31
TAXePose
Table 4: Top: Simulation Rotational (*) and Transla-
tional (m)) Errors (L). Bot-
tom: Real-world goal place- ment success rate (f).
6 Conclusion and Limitations
In this paper, we show that dense soft correspondence can be used learn task specific object re- lationships that generalize to novel object instances. Correspondence residuals allow our method to estimate correspondences to virtual points, outside of the objects convex hull, drastically increasing the number of tasks this method can complete. We further show that this "cross can be learned for a task, using a small number of demonstrations. Finally, we show that our method far outper- forms the baselines on two challenging tasks in both real and simulated experiments. While our method is able to predict relative pose relationships with high precision, it has several limitations:
Requires segmentation: Our method requires accurate segmentation of two objects in order to predict their relative goal pose.
Performance degrades under occlusion: Our method performs best when complete point clouds are provided, captured via multiple cameras or by repeatedly reorienting the objects.
Poorly defined for multimodal relationships: Because our method extracts single global es- timate of relative pose from a fixed set of correspondences, performance on objects with multi- ple valid goals is not well-defined. Our method might be augmented with consensus-based or sampling-based approach to capture the multimodality of the solution space in these cases. We leave this for future work.
8

---

Acknowledgements
This material based upon work supported by the National Science Foundation under Grant No. UIS-184914, This work was also supported by LG Electronics. We are grateful to Daniel Seita and Jenny Wang for their helpful feedback and discussion on the paper.
References
[1] D. G. Lowe. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pages 1150-1157. leee, 1999.
[2] Rothganger, S. Lazebnik, C. Schmid, and J. Ponce. 3d object modeling and recognition using local affine-invarian image descriptors and multi-view spatial constraints. International journal of computer vision, 66(3):231 2006.
[3] Y. Xiang, Schmidt, V. Narayanan, and D. Fox. Posecnn: convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint arXiv:1711.00199, 2017.
[4] Y. He, W. Sun, H. Huang, J. Liu, H. Fan, and J. Sun. Pvn3d: A deep point-wise 3d keypoints voting network for 6dof pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11632-11641 2020.
[5] Y. He, H. Huang, H. Fan, Q Chen, and J. Sun. Ffb6d: full flow bidirectional fusion network for 6d pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3003-3013, 2021.
[6] D. Turpin, L. Wang, S. Tsogkas, S. Dickinson, and A. Garg. Gift: Generalizable interaction-
aware functional tool affordances without labels. Robotics: Science and Systems (RSS), 2021.
[7] L. level robotic manipulation. International Symposium on Robotics Research (ISRR) 2019, 2019.
Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category-
[8] Z. Qin, K. Fang. Y. Zhu, L. Fei-Fei, and S. Savarese. Keto: Learning keypoint representations for tool manipulation. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 7278-7285. IEEE, 2020.
[9] M. Vecerik, J.-B. Regli, O. Sushkov, D. Barker, R. Pevceviciute, T. Rothörl, R. Hadsell, I.. Agapito, and J. Scholz. S3k: Self-supervised semantic keypoints for robotic manipula- tion via multi-view consistency. In Conference on Robot Learning, pages 449-460. PMIR, 2021.
[10] L. Manuelli, Y. Li, Florence, and R. Tedrake. Keypoints into the future: Self-supervised correspondence in model-based reinforcement learning. In Conference on Robot Learning, pages 693-710. PMIR, 2021.
[11] R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. In Conference Robot Learning, pages 373-385. PMLR, 2018
[12] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitz- mann. Neural descriptor fields: Se (qu-vuvivr object representations for manipulation. In 2022 International Conference on Robotics and Automation (ICRA), pages 2022.
IEEE,
[13] J. Besl and N. D. McKay. Method for registration of 3-d shapes. In Sensor fensor IV: control paradigms and data structures, volume 1611, pages 586-606. Spie, 1992.
[14] S. Bouaziz, A. Tagliasacchi, and M. Pauly. Sparse iterative closest point. In Computer graphics forum, volume 32, pages 113-123. Wiley Online Library, 2013.
[15] Rusinkiewicz and M. Levoy. Efficient variants of the icp algorithm. In Proceedings third international conference 3-D digital imaging and modeling, pages 145-152. IEEE, 2001.

---

[16] A. Segal, D. Haehnel, and S. Thrun. Generalized-icp. In Robotics: science and systems, volume 2. page 435, Seattle, WA, 2009.
[17] G. Agamennoni, Fontana, R. Y. Siegwart, and D. G. Sorrenti. Point clouds registration with probabilistic data association. In 2016 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1092-4099 IEEE, 2016.
[18] T. Hinzmann, T. Stastny, G. Conte, P. Doherty, P. Rudol, M. Wzorek, E. Galceran, R. Siegwart, and I. Gilitschenski. Collaborative 3d reconstruction using heterogeneous uavs: System and experiments. In International Symposium on Experimental Robotics, pages 43-56. Springer, 2016.
[19] D. Hähnel and W. Burgard. Probabilistic matching for 3d scan registration. Proc. of the VDI-Conference Robotik, volume 2002. Citeseer, 2002.
[20] Y. Wang and J. M. Solomon. Deep closest point: Learning representations for point cloud registration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3523-3532, 2019.
[21] A. V. Phan, M. Le Nguyen, Y. L. H. Nguyen, and L. T. Bui. Dgemn: A convolutional neural network over large-scale labeled graphs. Neural Nenworks, 108:533-543, 2018.
[22] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. N. Gomez, t. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
[23] T. Papadopoulo and M. I. Lourakis. Estimating the jacobian of the singular value decomposi- tion: Theory and applications. In European Conference on Computer Vision, pages 554-570. Springer, 2000.
[24] S. Hinterstoisser, V. Lepetit, S. llie, S. Holzer, G. Bradski, K. Konolige, and N. Navab. Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In Asian conference computer vision, pages $48-562. Springer. 2012.
[25] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning, 2016-2020.
[26] F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, and Oth- ers. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11097-11 107,, 2020.
[27] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, al. Transporter networks: Rearranging the visual world for robotic manipulation. arXiv preprint arXiv:2010. /4406, 2020.
[28] D. Q. Huynh. Metrics for 3d rotations: Comparison and analysis. Journal of Mathematical Imaging and Vision, 35(2):155- 164, 2009.
[29] R. Hartley, J. Trumpf, Y. Dai, and H. Li. Rotation averaging. International vision, 103(3):267-305, 2013.
journaa journal of computer
10


Reviews:
[{"review_id": "bly3hLd1CSa", "reviewer": "Reviewer_d5cK", "summary": "This paper tackles the problem of estimating the relative pose between two objects conditioned to a specific manipulation task. This pose is then used to guide a robot manipulator to pick and place one object in a desired location relative to the other.\nThe authors given a formal definition of the estimation task by introducing the concept of \"cross-pose\".\nThe proposed system, called TAX-Pose, is built upon Deep Closest Point (DCP)[20], with the addition of specific components required by the manipulation task, and OMPL for motion planning.\nThe different components of the system are validate in ablation studies.\nExperiments show that TAX-Pose achieves state-of-the-art performance and it is able to generalize to different objects of the same category.\nSim-to-real transfer is achieved without fine-tuning on some of the manipulation tasks tested in simulation. The proposed system is able to achieve much higher success rate than the baseline in the real-world.\n\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "-- Strengths\n\n- The paper is well-written and easy to read. The authors were able to describe the problem statement and methodology in a clear and descriptive way in spite of the little space available.  The paper contains all the information and references that the reader needs to understand the proposed method and make connections to the current state-of-the-art.\n\n- The definition of the term of \"cross-pose\" is intuitive and its mathematical formulation is sound. The reader needs to check the Appendix to get the full idea behind this definition.\n\n- The proposed system, TAX-Pose, is a slightly adapted version of DCP for the task of \"cross-pose\" estimation. The authors propose to add a residual net block, which applies a residual vector to each point correspondence, to account for the case when the object are not in contact or overlap. I find this an elegant solution.\n\n- The experiments are relevant. The different components of the system are validated in ablation studies. TAX-Pose achieves state-of-the-art results in simulation. Simulation to real-world transfer is achieved without fine-tuning, thanks to, according to the authors, the translation-equivariant property of TAX-Pose.\n\n-- Weaknesses\n\n- The originality of this work is a weakness. TAX-Pose heavily builds upon DCP for the relative pose estimation and completely rely on OMPL for planning. The main difference with respect to DCP is a revised version of the block that in DCP is called \"Pointer\". In this work a residual net is used to account for the case when the object are not in contact or overlap. Although this is an elegant solution, it is a minor increment with respect to the DCP architecture.\n\n- I would suggest  to give more insights about the results. In particular:\nDo the authors have an intuition why the rotation results of TAX-Pose are much better that the baselines? \nWhy the sim-to-real transfer capabilities of TAX-Pose are much better than Goal-Flow? I would expect that the translation-equivariant would improve the sim-to-real transfer capabilities of TAX-Pose but the results of Tab. 2 show a big gap with respect to Goal-Flow. Could the authors explain their intuition about the reason behind this result?\nWhy was it not possible to evaluate the mug hanging experiments in the real-world?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper has some potentials to have an impact in the field.\nAlthough the methodology is an incremental contribution with respect to DCP, the authors have shown that DCP can be used to solve a relative pose problem rather than point-cloud registration problem.\nThe experiments are relevant and the ablation studies useful to validate the different components of the system. However, further discussions on the results would improve the quality of the work. \n\n**After Rebuttal**\nI appreciate the authors' response to my concerns. In my opinion, this work is a good contribution for the conference. I confirm my initial recommendation of weak accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Besides the comments regarding weaknesses in the corresponding section, I include some other minor issues below:\n\n- The introduction of the variables $T_{\\alpha}$ and $T_{\\beta}$ in the section \"Problem Statement\" seems not necessary since such variables are not used anymore in the text. \n\n- It seems that \\tilde{P}_{A} in Fig. 3 is call \\tilde{V}_{A} in the text. Same for B.\n\n- Reference [34] and [35] cite the ArXiv version of the papers. However, the works have been published in ICRA 22 and CoRL 18 respectively. I would suggest to update these references."}}, {"review_id": "S3sILl0YFc", "reviewer": "Reviewer_r914", "summary": "The paper proposes an approach to estimate the appropriate transformation to apply to an object in order to achieve the desired relative pose between two objects in a given task. Experimental evaluation shows convincing results in two different settings. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper studies an interesting perspective in manipulation, treating a manipulation task as achieving a desired relative pose between objects. The proposed method is novel.\n2. The paper is well-written (despite some minor notation issues) and technically sound. \n3. The experimental results are compelling.\n\nWeaknesses:\n\n1. The framework seems hard to extend to tasks involving more than two objects. What if a task requires five objects to achieve some desired relative poses?\n2. There are some minor notation issues (see the ‘Issues’ section below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper studies an important problem in manipulation, defining the task as achieving desired relative pose between objects, and proposes a novel framework to solve it. The paper is well-organized. Although it seems like the framework is constrained in a two-object scenario, I believe the paper is a strong contribution and can potentially have a major impact.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. Can the authors comment on if it is possible to extend the framework to tasks that involve more than two objects?\n2. I wonder if the authors have thought about using some SE(3)-equivariant network architecture like the vector neuron used in NDF [34]. Will that simplify the training process?\n3. There are some minor notation issues:\n    1. Should $\\tilde{\\mathbf{V}}_\\mathcal{A}$ and $\\tilde{\\mathbf{V}}_\\mathcal{B}$ in Equation 6 and 7 be swapped? If I understand line 167 and 168 correctly, $\\tilde{\\mathbf{V}}_\\mathcal{A}$ is the correspondence for  $\\mathbf{P}_\\mathcal{B}$. In other words, $\\mathbf{P}_\\mathcal{A} \\in \\mathbb{R}^{3\\times N_A}$ , then $T^{GT}_\\mathcal{AB}\\mathbf{P}_\\mathcal{A}\\in \\mathbb{R}^{3\\times N_A}$. However, based on the definition of $\\tilde{\\mathbf{V}}_\\mathcal{A}$ in line 167 and 168, $\\tilde{\\mathbf{V}}_\\mathcal{A} \\in \\mathbb{R}^{3\\times N_B}$.\n    2. What are $\\tilde{\\mathbf{P}}_\\mathcal{A}$ and $\\tilde{\\mathbf{P}}_\\mathcal{B}$ in Figure 3? Those symbols does not exist in the paper."}}, {"review_id": "AAtxMOJS9HF", "reviewer": "Reviewer_4CrR", "summary": "The paper proposed a vision system that uses pose relationships between two objects for robot manipulation. Point clouds of two interacting objects are retrieved through DGCNN and the proposed TAX-Pose method learns to predict a task-specific pose transformation from demonstrations. The prediction is then used to plan a trajectory that accomplishes the task. The system has been tested in both simulation and real-world environments and shown that the model sample efficiency that can learn from a small number of demonstrations. Dense soft correspondence + residuals allows the method to generalize to novel object instances.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The author designed a cross-pose estimation system that helped with object manipulation. Comprehensive experiments were provided to show that the proposed method is beneficial in both simulation and real world. \nThe downside of the paper is that the proposed method injects a significant amount of inductive biases to solve pick-and-place-ish tasks. The scalability of the method is questionable. First the method is heavily rely on a relatively good pretrained model for point cloud extraction e.g. DGCNN as used in the paper. It turns the problem setup from image space to feature space that makes the claim of a “vision-based system” a bit weak. In addition, the feature of zero-shot sim to real transfer and generalization to novel objects of the same category likely comes from the use of DGCNN. Secondly, the scalability is significantly limited once a model is trained. It only works for the exact pair of objects and for one specific task. It is not so clear by reading the paper, but it seems a model trained for “Left” task would not be able to do “Right” task. Or a model trained for “cube-left-oven” might not be able to do “cube-left-microwave”. Thirdly, the proposed method works only for pairs of objects. It seems hard to scale the system to support multiple objects or even generalize to the number of objects. In practice, real-world tasks often involve multiple objects and the complex dynamic among objects interaction. \n\nAs the author mentioned in the limitation section, symmetry is not handled. However, Rotational errors (ER) are reported in the evaluation metric, which might not make much sense when objects like cubes, bottles, plates etc are involved.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The scalability of the method is questionable and is seems the model can only operate in a very constrained scenario. As symmetry is not haddled, the comparision between TAX-Pose and baseline methods are a bit weak. Rotational errors (ER) is no longer a fair metric. In terms of the translational errors (Et), TAX-Pose does not show much advantages. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "See above", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "0PXEAuXLIgv", "reviewer": "Reviewer_gUTd", "summary": "The paper introduces the notion of “cross-pose”, a task-specific relationship between objects that captures a desired goal configuration. The proposed “cross-pose” function maps pairs of object pointclouds to a relative pose in SE(3); the function value is defined to be identity when the objects satisfy the relative pose goal relationship. The model is trained on a dataset of demonstrations of the goal condition, e.g. a mug hung on a rack, and is designed to operate at the category level for “action” and “anchor” objects. \n\nThe main contribution of the paper is the formulation of cross-pose and an architecture for learning it. The central novelty in the problem formulation (in particular relative to Neural Descriptor Fields) is that both the action and anchor object are encoded, enabling generalization at the category level for both (in contrast to NDF, in which only the “action” object is embedded).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper studies an important problem. The description and analysis of the cross-pose function are thorough. The paper achieves impressive results on the simulated mug-hanging task in comparison to a recent state-of-the-art system, and the ablation results motivate the design decisions. \n\nThe paper has some limitations which the authors are transparent about - the strongest assumption is that there is one such “cross-pose” which satisfies the relationship. In reality, there are a manifold of poses which satisfy the goal condition for these tasks, though this is a reasonable first approximation inherited from previous work. A larger conceptual limitation is the restriction of tasks to pick and place. While a large number of household tasks may be specified by a static observation, many cannot - e.g. “carry this mug of tea” is not well specified this way. Another limitation present from the evaluation is the need to train one separate semantic model for each variant of a task - i.e. each semantic placement (right, left, in, …) requires a unique model to be trained. We’ve seen methods like CLIPort and Socratic Models use natural language in conjunction with vision to handle such variation. This isn’t the focus of this work but it warrants mention.\n\nThe related work section is relatively weak. In particular, it fails to mention the baselines compared to in the mug hanging experiment section, the most similar related work, as well as related ideas such as keypoint methods or canonicalization (e.g. NOCS). From the text alone, the novelty of the proposed method relative to these baselines is not clearly conveyed. \n\nThe comparison to existing methods is also a bit lacking. The related methods are absent from the PartNet evaluation without mention. I believe they are omitted because the anchor object pose is also varied, where DON and NDF would require a fixed anchor. This at least warrants clarification in the text (if not outright comparison in a \"fixed anchor\" experiment). Moreover, only one object category experiment from NDF was replicated - it's difficult to assess the improvement in performance offered by the proposed approach with a limited evaluation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I'm recommending weak rejection on the basis of the paper needing substantial improvements to the quality of presentation and potentially additional experimentation. At the moment, the novelty is not clear relative to existing baselines and the related work, experiment setup, and results lack sufficient discussion. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Please see above recommendations regarding the related work and result sections. \n\nI would recommend moving the success rate tables to the main paper - the error in rotation/translation is difficult to assess task performance with. \n\nQuestion regarding the partnet simulation experiment: is the anchor object pose randomized, or just the target object? The description in 6.1 is ambiguous. \n\nThe hardware results describe at length the modifications required to achieve the real world experiments, though the mug-hanging experiments demonstrate that the method is able to learn relative grasp poses. Is there a particular reason why this was forgone for the hardware experiment?\n\nMinor typos:\n\nAppendix 7.1 \"quarry\"\nSection 5.1 “a semantic goal positions”"}}]

Author rebuttal:

Decision: Accept (poster)