Deep SE(3)-Equivariant Geometric Reasoning for Precise Placement Tasks
Authors: Ben Eisner
Yi Yang
Todor Davchev
Mel Vecerik
Jonathan Scholz
David Held
Venue: ICLR 2024 poster (2024)
Keywords: Learning from Demonstration
Manipulation
3D Learning
SE(3) Equivariance
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Many robot manipulation tasks can be framed as geometric reasoning tasks, where an agent must be able to precisely manipulate an object into a position that satisfies the task from a set of initial conditions. Often, task success is defined based on the relationship between two objects - for instance, hanging a mug on a rack.  In such cases, the solution should be equivariant to the initial position of the objects as well as the agent, and invariant to the pose of the camera. This poses a challenge for learning systems which attempt to solve this task by learning directly from high-dimensional demonstrations: the agent must learn to be both equivariant as well as precise, which can be challenging without any inductive biases about the problem. In this work, we propose a method for precise relative pose prediction which is provably SE(3)-equivariant, can be learned from only a few demonstrations, and can generalize across variations in a class of objects. We accomplish this by factoring the problem into learning an SE(3) invariant task-specific representation of the scene and then interpreting this representation with novel geometric reasoning layers which are provably SE(3) equivariant. We demonstrate that our method can yield substantially more precise placement predictions in simulated placement tasks than previous methods trained with the same amount of data, and can accurately represent relative placement relationships data collected from real-world demonstrations. Supplementary information and videos can be found at https://sites.google.com/view/reldist-iclr-2023.

Published as a conference paper at ICLR 2024
DEEP SE(3)-EQUIVARIANY GEOMETRIC REASONING FOR PRECISE PLACEMENT TASKS
Ben Eisner
Yi Yang, Todor Davchev, Mel Vecerie, Jon Scholz Google DeepMind
Carnegie Mellon University
David Held
Carnegie Mellon University
ABSTRACT
Many robot manipulation tasks can be framed as geometric reasoning tasks, where agent must be able to precisely manipulate object into position that satisfies the task from a set of initial conditions. Often, task success is defined based on the relationship between two objects- for instance, hanging mug on rack. In such cases, the solution should be equivariant to the initial position of the objects as well as the agent, and invariant to the pose of the camera. This poses a challenge for learning systems which attempt to solve this task by learning directly from nigh-dimensional demonstrations: the agent must learn to be both equivariant as well as precise, which can be challenging without any inductive biases about the problem. In this work, we propose a method for precise relative pose prediction which is provably SEuuv-aqui,,riant, can be learned from only a few demonstra- tions, and can generalize across variations in a class of objects. We accomplish this by factoring the problem into learning SE(3) invariant task-specific represen- tation of the scene and then interpreting this representation with novel geometric reasoning layers which are provably SF(3) equivariant. We demonstrate that method can yield substantially more precise predictions in simulated placement tasks than previous methods trained with the same amount of data, and can ac- curately represent relative placement relationships data collected from real-world demonstrations. Supplementary information and videos can be found at this URL
INTRODUCTION
critical component of many robotic manipulation tasks is deciding how objects in the scene should move to accomplish the task. Many tasks are based on the relative relationship between set of objects, sometimes referred to as "relative placement" tasks (Simeonov et al. (2022); Pan et al. (2033 Simeonov al. (2023): Liu et al. 2022)). For instance, in order for a robot to set table, the position of the silverware has desired relationship relative to the plate. These types of problems can be described as geometric reasoning tasks- you know the locations each object in the scene, and you know what kind of relationship the objects should have to accomplish the task, then you can logically infer the target position of the objects. We would like our robotic agents to also possess these geometric reasoning faculties to solve such relative placement problems.
In this work, we consider the task of training robotic agents to perform relative placement tasks di- rectly from high-dimensional inputs by watching small number of demonstrations. Although there have been many recent advances predicting complex robot behaviors from raw sensor observa- tions (Lee et al. (2020); Miki et al. (2022); Yang al. (2023); Ha & Song (2022); AAkkaya et al.] (2019). high-dimensional observations pose particular challenges in geometric reasoning tasks. For example, the amount by which each object should be moved to reach the goal configuration should be (2023) demonstrates that if general- purppoe neural network is trained on small number of high- dimensional demonstrations with no additional inductive biases, it will typically not learn to be
to the initial locations of the object. However, prior work (Pan et al.
"Corresponding author: baei sner@andrew cmm.

---

Published as a conference paper at ICLR 2024
robust to the initial object configurations. Prior work to address this issue (Pan et al. (2023) incor- porates geometrie reasoning into the network, but is not provably equivariant. Other work that is provably equivariant (Simeonov al. (022)) is outperformed both by (Pan et al. (0223) and by our method.
To achieve both strong empirical performance and provable equivariance, propose a visual rep- resentation which can be used for equivariant manipulation in relative placement tasks. Our key insight is decouple representation learning into an invariant representation learning step and an equivariant reasoning step. We achieve this using a novel layer for differentiable multilateration, in- spired by work in true-range multilateration (Zhou (200)). In this work, we propose the following contributions:
A novel dense representation for relative object placement tasks, which is geometrically inter- pretable and fully invariant under SE(3) transformations to objects in the scene.
method for solving relative placement tasks with differentiable geometric reasoning using a novel layer for differentiable multilateration, which can be trained end-to-end from observa- tions of small number of demonstrations without additional labels.
A provably Squiv-eqivvariant neural network architecture predicting this representation di- rectly from high-dimensional raw inputs.
set of simulated experiments which demonstrate superior placement performance in several relative placement tasks both on precision metrics and on overall success. Our experiments also demonstrate that our method generalizes within a class of objects with reasonable varia- tion, and can be applied to real-world demonstration datasets.
2 RELATED WORK
One approach to solving relative placement tasks is via object pose esti- mation. Recently several approaches have proposed using test-time optimization or correspondence detection to align current observations with demonstration observations with known pose (Florence) al. (2018; Simeonov et al. (2023 2022)) However, (Simeonov et al. (2022) requires one of the objects to remain in a fixed location, while (Simeonov et al. (2023) requires user input to spec- ify relationships of interest our method has no such restriction and requires no user input. Most similar to our work is TAX-Pose (Pan et al. (2023)), which computes cross-object correspondences to estimate task-specific alignment between a pair of objects, correcting these correspondences with a learned residual. Both TAX-Pose and our work attempt to learn to predict, for each point on an object, where that point should end up to accomplish the relative placement task. However, we compute this mapping quite differently while TAX-Pose establishes "cross-correspondence"" and updates with a residual vector to regress the next point's location, we learn an SE(3)-invariant representation (which we call RelDist) of the pair of objects, and infer the point's 's location from this representation using differentiable multilateration. By construction, our framework is fully SE(3)- invariant by construction, whereas TAX-Pose is not.
Object Pose Estimation:
SE(3) Equivariance in Visual Prediction: Equivariance of geometric predictions under SE(3) transformations of objects in scene (including invariance camera transformations) is desirable property of many visual prediction systems. Several works design provably equivariant prediction methods (Cohen & Welling (2016); Weiler & Cesa (2019); van der Pol et al. (2020)) and use such methods for robot manipulation (Wang et al. (2022b);; Huang et al. (2022); Wang & Walters (2022); Wang et al. (2022a)). Vector Neurons (Deng et al. (202I) are used to design provably-SEi in- variant and equivariant versions of standard point cloud analysis architectures like DGCNN (Wang al. (2099) and PointNet (Qi et al. (2017)), which we incorporate in the feature-encoder in this work. Neural Descriptor Fields (NDF) (Simeonov et (202)) also use Vector Neurons (Deng et al.) 2021 to achieve provable equivariance, but suffer from suboptimal performance; NDF is outper- formed by TAXPose (Pan et al. (2023) which we compare to in this work. Other methods attempt to learn equivariance via training (Pan et al. (2023). Another approach is to search over SO(2) or SO(3) (Zeng et al. 2021); Lin al. (2023)), evaluating candidates with scoring network to align objects in a scene for a given task. However, such methods have not been demonstrated to perform the precise SE(3) relative placement tasks used in this work.
2

---

Published as a conference paper at ICLR 2024
3 BACKGROUND
We consider deep geometric reasoning tasks based on point cloud observations, and build on top of several separate lines of prior work:
Given a receiver with an unknown position p in 3D, a set of K beacons with known positions 4i in 3D space, and set of K measurements r of the scalar distance from each beacon to the receiver, the multilateration task is to estimate the position of the receiver Multilateration similar triangulation, except that the direction of the beacons to the receiver is unknown, If the squared error cost is used, this reasoning task reduces to non-linear least-squares optimization:
Multilateration:
min Σ (11g p1 -1))
There are several different approaches to solving this problem, discussed in survey paper multilateration (Sirola (2010)). We leverage the closed-form solution to this problem; see (Zhou for details.
(2099)
The orthogonal Procrustes problem: Given two sets of N corresponding 3D points A and B, the orthogonal Procrustes problem is to find rigid transform Tan E (Ran,tab) which best aligns them. Specifically, if the squared error cost is used, this reasoning task reduces to a constrained least-squares optimization task:
min RABIAB
Σ || +LB B,ll2. i=1
s.t. RAB E SO(3)
This problem has well-known closed- dloeed-form solution based on Singular Value Decomposition (SVD) (Sorkine-Hornung & Rabinovich (2017).
Equivariance//tvaiinne: This work considers two geometric functional properties: SFF(3 equiv- ariance (Eq. 3) and invariance (Eq. 4. under transform T E SE(3), which respectively satisfy:
f(z) = = f(Tr)= = Ty
(3)
f(x) = - f(Tz) =
=
(4)
Point Cloud Analysis Architectures: We make use of Dynamic Graph Convolutional Neural Net- works (DGCNN) (Wang & Solomon| 2019)) as the primary backbone architecture for learning per- point features. Roughly, each layer of DGCNN performs two-step operation: 1) it computes per-point connectivity graph, roughly its neighborhood (in either Euclidean space or latent space), typically using K-Nearest Neighbors; and 2) for each point, performs some sort of learned aggrega- tion of the local neighborhood, which is permutation invariant. Stacking such layers, an architecture can flexibly capture global and local context of point cloud.
On top of DGCNN, we also make use of Vector Neurons (Deng et al. (2021), which a set of deep learning primitives which are designed to be EquiiaEqui These primitives can be directly dropped into DGCNN to replace existing layers. Roughly, Vector Neurons accomplish SE(3) equiv- ariance by treating each point's Euclidean coordinate independently (through a batching operation), and constructing primitives that operate on the coordinates independently.
4 PROBLEM STATEMENT
Relative Placement Tasks:
In this work, we con-
T T
sider relative placement tasks, as defined in prior work (Simconov et al. (202)); Pan et al. (2023): :
Simeonov et al.] (2033) ). We borrow mathematical definitions of the task from (Pan et al. (2023)). ). Rel-
T
ative placement tasks require predicting a rigid body transform which transports a specific rigid object A from an initial position to a desired, task-specific fi- nal position with respect to another rigid object B. Suppose T and T are SE(3) poses for objects A and B respectively such that the placement task is
TE
Figure 1: Invariance of relative placement tasks under transformations. In this case, ring on peg maintains the same relative posi- tion under rigid transformation T.
3

---

Published as a conference paper at ICLR 2024
Invaaanvarraat Prediction
SE(3)- Equivarrna Reasoning
PA
Y
InOy Encodar
MDL
P PA
FRD
TL
P
q
SR/3-lo Encoder
:
MBL
T A8
Ksle)
TNNI
No..No
1
RAB
P3
Figure 2: Method overview. First, the point clouds A, P8 are each encoded with a dense S3) equivariant encoder, after which cross-attention is applied to yield task-specific dense representa- tions. Then, the kernel matrix RAB is constructed through the learned kernel Kg. This matrix is then passed into MUL infer the desired final point clouds, and then passed into PRO to extract final transform which moves object A into its goal position.
complete, expressed with respect to some arbitrary world frame. Following Pan et al. (2023), the task also considered complete if A and positioned such that, for some T E SE
RelPlace(TAr, TB) SUCCESS iff TE E SE(3) A To and Ts T TB.
(5)
Cross-Pose: We are interested in predicting a transform which brings object A to goal position relative to object B, to satisfy the placement condition defined in Eq. Rather than estimating each object's 's pose independently and computing a desired relative transform, we would like to learn function AB f(P A. P3) which takes input point cloud observations of the two objects, P... PB and outputs an (3) transform which would transport P a into their desired goal position with respect B. Pan al. (2023) define this transform the cross-pose for specific instance of a relative placement task, as follows: when objects A and have been rigidly transformed from their goal configuration by T, and T3, respectively, the cross-pose is defined as:
f(T. o TA Ta T3) = TAB = To T
(6)
meaning that applying this transformation to object A will then transform it back into goal con- figuration relative to object B. Note that the cross-pose function is equivariant in the sense that J(TA AA, Ti Pσ) TB J(PA,
T
Our method assumes that we receive point cloud observations of the scene (e.g. from LiDAR/depth cameras/stereo), and that the pair of objects being manipulated (e.g. the mug and the rack) have been segmented. For training the cross-pose prediction module, we assume access tions of pairs of objects in their final goal configuration (e.g. demonstrations in which objects A and have poses TA, Tg such that ReIPlace Tg) SUCCESS, according to Eqn. 6).
set of demonstra-
5 A FRAMEWORK FOR SE(3) EQUIVARIANT MANIPULATION
Method Overview: We now describe our method for precise relative object placement which is provably -uquvvaaaa under transformations to individual objects. At high level, we: 1) Generate 3E(3)-invarian per-point features for objects in the scene; 2) Use cross-attention to aug- ment these features with task-specific information: 3) Apply a learned kernel function, which takes in pair of points two different objects and predicts how far apart they should be to satisfy the task; 4) Use a differentiable closed-fone estimator of the least-squares solution to the multilateration problem to estimate the desired task-specific final location of each point; and 5) Compute the least- squares transform between the estimated point locations and the input locations, using differentiable SVD. See Figure 2 for high-level overview of approach.
4

---

Published as a conference paper at ICLR 2024
5.1 RELDIST: AN SE(3)-INVARIAN) REPRESENTATION FOR RELATIVE PLACEMENT
The key insight of this work for solving relative placement tasks is as follows: suppose that we wish to predict the desired goal position of object A, which should be placed in some location relative to object B. One approach is to predict the desired pose of object A directly, relative to object B; previous work Pan al.] (2023) shows that this approach has poor performance. Another approach PenterPan
used in prior work that performs better Pan et al. (2023) is to predict the desired location of cach point pp in object A. Instead, our approach is to predict set of desired distance relationships between Ps and at least points on object B. Specifically, for each point y in object A, and for some set of points {pf,... (pi....,PK .Pk on object B, we predict the desired Euclidean distance raj = d(ppt pf) between points p and of in the desired goal configuration. As long as the number of points K used from object B is at least 3, we can use multilateration (Section 3 to estimate the desired location of Ps
Note that these desired distance between point p. on object A and some point pf on object B is invariant to the current poses of objects A and B. Specifically, if object A is transformed by Ta and object B is transformed by Te, then the desired distance between these points constant:
My = d(p = d(Ta o p. T3 )
Further, these distances are scalar values, so they are not defined in any reference frame, which means that they are invariant to shifts in the origin of the coordinate system. We refer to rij as RelDist. This representation gives a convenient target for neural network prediction one simply needs to predict set of pairwise invariant relationships d(p,", .p))) across two objects.
5.2 PREDICTING RELDIST
We now describe a provably SE(3)-invariant architecture for predicting RelDist
Dense SE(3)-Invariant Features: We assume that we are given point cloud for objects A and B, given by PA E NAx3 and Pg NR respectively. The first step of our approach is to compute an invariant feature for each point in these point clouds. In other words, the feature should not change if either object is transformed by an SE(3) transformation. Further, we wish each point to have unique feature. Given point clouds PA and Pg. we use provably SE(3) (Iv-arrai point cloud neural network such as DGCNN Wang & Solomon (2019) implemented using Vector Neuron layers Deng al. (2021). Alternatively, one could train a standard DGCNN produce SE(3)- Invariant representations, although the resulting features would not be guaranteed to be invariant. We consider both approaches in experiments.
Specifically, we learn functions fea and fea which map the point clouds PA and Pg to a set of invariant d-dimensional per-point features A and 4g:
4 = fea (PA) E Naxd 4B =
f (PB) E Nexd
(8)
Task-Specific Cross-Attention: Although the representations A ogg are object specific, the rel- ative placement task requires reasoning about cross-object relationships. For instance, for hanging mug on a rack, the network needs to reason about the relative location of the mug handle and the peg on the rack. To reason about these cross-object relationships, we learn a cross-attention mod- ule, using multi-head cross attention Vaswani et al. (0177), which sum with the pre-attention embeddings from Eq. g
T'A 4' X-Attn, ( +8) 4 - 2 +
(9)
$ 2g + L'B
(10)
A Kernel Function for predicting RelDist: Let dxt and of represent the learned embeddings for point psd on object A and point D on object B, respectively. We now wish define a symmetric function IC which takes pair of embeddings de SB and computes a positive scalar Tij E R+ which predicts the desired distance between points p.d and p when objects A and B are placed in the goal configuration: ray K(pg (.)). This function K. can be thought of as type of kernel, for
5

---

Published as a conference paper at ICLR 2024
which there various choices of functions. In this work, select simple learned kernel with the following form:
Ko softplus (" of (o (off(())))))))
(11)
where 2 are of learned parameters. Details can be found in the Appendix to build the kernel matrix RAs:
C.1
This allows us
(op of))
A
E (R+)(NaxNN))
(12)
Kol o6 Kipld
2
which can be interpreted exactly the of RelDist values for all pairs of points pe and p between objects A and B. Note again that this prediction is SE(3) for any individual transformations of objects A or B.
5.3 FROM RELDIST TO CROSS-POSE
berk
block prppor
pper
Initial observation.
Sample points.
Estimate riy for point pairs.
Blok impaer
(preikzad)
(d) Infer A (Ja, Ps). (e) Repeat MUL Ps) for all i.
(1) Infer gripper goal pose.
Figure 3: Reasoning with multilateration (a) 2D environment with block and gripper. (b) For each sampled point on the gripper, (c) estimate the desired distances between the gripper point and points on the block. We then use multilateration (d) to extract least-squares solution to compute the desired gripper point location. Doing this for every point the gripper, (e) we reconstruct the desired position for each gripper point. (f) These corresponding points can be used to infer rigid transform which brings the gripper the final goal position.
We now describe how we can turn our invariant representation into Cross-Pose prediction with the desired properties above. A visual walkthrough can be found in Figure
Estimating Corresponding Points with Differentiable Multilateration: Given the estimated set of RelDist values R, would like estimate the intended position of each point in object A

---

Published as a conference paper at ICLR 2024
with respect to object B (or vice versa). The ith Ra of RAs contains set of scalars indicating the desired distances from points on object to the point p, on object A when placed in the goal configuration. We would like to these relative distances estimate the goal position of this point. Since As is a predicted quantity and may have some noise, we wish to find the point p¹A which minimizes the Mean-Squared error with respect that row:
Ns
P/A
= arg min X/A
Σ
|p - p.
(13)
This describes nonlinear least-squares optimization problem, class of problems which has general closed-form global minimizer. However, it has been shown that there differentiable, closed-form global minimizer [Zhou (2009) for this specific problem in Equation [3] which will refer to as the function MUL: ps = MUL( Ps). See Figures 3c and 3d for a visual- ization of this point estimation process, and Appendix A. for details. By computing this func- tion on each row of RAB. we can compute the desired goal position for each point on A, as: P [MUL PR))EE (see Figure Bep. Importantly, this solution to the multilateration problem is provably SE(3)-equivarian transtormations on object B, given fixed RAS (see Ap- pendix B for proof).
Estimating Cross-Pose with SVD: Given a of initial points PA and their corresponding esti- mated final task-specific goal positions PA. can set up classic weighted Procrustes problem to compute the estimated cross-pose T between objects A and B, as defined in Section This has known closed-fore solution based on the Singular Value Decomposition (SVD) (see Appendix (A.2) which can be implemented to be differentiable Papadopoulo & Lourakis (2000); Pan et al. (2023). For the relative placement task, once we estimate AB, we use motion planning to move object A by Aa into the goal pose. Because MUL and PRO are all differentiable functions, the whole
method can be trained end-to-end, using the same loss functions as defined in Pan et al. (2023) (see Appendix for details).
EXPERIMENTS
6.1 RLBENCH TASKS ACHIEVING PRECISE PLACEMENT
slack wine
loilet roll
phone on base
insert peg
hang hanger
Figure 4: RLBench (James (2020) relative placement tasks. Top: the initial state of a demon- stration. Bottom: the final state of a demonstration, where successful placement has been achieved.
To assess the ability our method to make precise predictions relative place- ment tasks, we select 5 manipulation tasks tasks from the RLBench benchmark (James et al. (2020) which require varying amounts of precision to successfully accomplish each task. Specifically, we choose the following tasks: stack_wine toilet.roll. phone.on.bar
Experiment Setup:
insert_pest and hang hanger. Each of these tasks involves placing singular object in a task-specific configu- ration relative some other object. We collect 10 expert demonstrations for each task, and extract

---

Published as a conference paper at ICLR 2024
point clouds for initial and final keyframes using ground truth segmentation from multiple simu- lated RGB-D images. This forms the basis for a per-task relative placement task, where the goal is to predict the desired goal configuration for each object given the initial point cloud. To evaluate the performance of a prediction system, we generate 1000 unseen initial starting positions for the object in the scene, and measure the both the rotation error 6 and the centroid translation error 6p of the predictions compared to the goal poses reached in expert demonstrations, We evaluate both TAX-Pose Pan al. (2023) and our method on each task. Results are presented in Table
RLBench Placement Tasks
stack_wing
toilet_roll
hang_hanger
phone_on_base
insert.peg
Method
4 (9)
Cp (m)
& (")
Ep (m)
& (")
Cp (m)
(?)
Ep (m)
(*) 6p (m)
TAX-Pose
1.485
0.003
1.173
0.001
5,471
0.012
4.144
0.005
7.098
0.004
Ours
0.764
0.001
1.150
0.001
0.624
0.002
0.804
0.001
1.209
0.003
Table 1: RLBench Placement Tasks, prediction error (4 is better): We measure the precision of our method when predicting the cross-pose which brings each object into the goal position. We report both the angular error 6 (")and the translational error Cp (m), compared to the goal poses achieved by expert demonstrations.
Analysis: For every task in our evaluation, our method achieves highly precise placements, with rotational errors less than 1.25%and translational errors less than 3 millimeters (and frequently only lmm). For each task, this level of precision is well within the tolerances for successfully completing the task. Additionally, we substantially outperform our closest baseline (TAX-Pose) in nearly every metric (which matches our translational precision in only on toil toilet_roll). In particular, our method is 2-9x more precise in rotation on the tasks. Because each task includes only a single object which varies only in initial pose, this evaluation can be interpreted roughly as ceiling for how precisely these relative relationships can be represented. We believe that this is strong evidence that invariant-equee equivariant architecture is capable of representing significantly more precise lationships in general than prior work. Furthermore, we show in Appendix D that our method can learn highly precise relationships from only single demonstration.
6.2 NDF TASKS CATEGORY-LEVE GENERALIZATION
To evaluate the ability of our method to generalize precise placements across a class of objects with reasonable variation, we evaluate our method on the NDF relative placement tasks, as proposed in Simeonov et al. (022).. These tasks have the following structure: an object is positioned in tabletop scene, along with a robot arm with parallel-jaw gripper. The task is to predict two poses for the robot to execute: first, a Grasp pose which will lead to a stable grasp on the object when the gripper is actuated; second, Place pose, where the grasped object should be placed to accomplish the task. At training time, the agent is given 10 demonstrations of each stage of the task (grasp and place), with variety of objects from the same object category. The agent must then learn a relative pose prediction function for each scenario which can generalize to novel instances of objects. There are two variants of each task evaluation: when the object is initialized to be resting upright on the table, bottom-side down (" ("Upright"), ") and when the object is initialized at an arbitrary orientation above the table for the robot to grasp ("Arbitrary")). At test time, we randomly sample 100 initial configurations from each distribution and report average metrics across all trials. We report success rates at various thresholds of penetration between the placed object and the scene as proxy for precision. See Appendix [E for more details and motivation for this metric.
Experiment Setup:
We compare our method to the following baselines: Dense Object Nets (Florence et al. 2018 Neural Descriptor Fields (Simeonov et al. (2022)), )) and TAX-Pose (Pan al. (2023)) (see Appendix for details).
Our numerical results are presented in Table 3 Our method achieves substantially more precise placements in both the upright and arbitrary mug hanging tasks: while other methods per- form well when geometric feasibility is not considered in the evaluation (infinity column), method yields substantially more feasible predictions in both settings, when penalizing for colli- sions (penetration thresholds of cm). Results for the remaining NDF tasks can be found in
Analysis:
8

---

Published as a conference paper at ICLR 2024
Mug Hanging (Upright/Arbitrryy)
Grasp
Place
Overall
Method
<1em
<3cm
<lem
<3cm
DON
0.91/0.35
0.50/0.45
0.45/0.17
NDF
0.96/0.78
0.92/0.75
0.88/0.58
TAX-Pose
1.00/0.50
0.15/0.32
0.70/0.75
0.93/0.85
0.15/0.14
0.70/0.40
0.93/0.44
Ours
1.00/0.43
0.35/0.36
0.78/0.66
0.99/0.88
0.35/0.19
0.78/0.32
0.99/0.41
Table 2: NDF Mug Hanging, Success Rate (t is better): The success rate of each method when running the relative placement tasks with the objects starting either upright on the table (left number) or in arbitrary poses above the table (right number). Each column group shows the success rates for the Grasp phase and Place phase, as well as Overall performance (when both grasp and place successful in trial). Additionally, for Place and Overall we report success when maximum penetration is thresholded at the given distance (Iem, 3cm, 00).
Appendix E, however, our results on the Bottle and Bowl tasks are inconclusive, as implementations techniques proposed by Pan al. (023) which are crucial to good per-
of the symmetry-breat formance for our closest baseline, TAX-PPsse were not released, and we were not able to reproduce the performance reported by Pan et al. (2023). We also performed multiple ablations to understand the effect of different design decisions of our method; see Appendix D for details.
6.3 REAL-WORLD SENSOR EXPERIMENTS
Experiment Description: To demonstrate the abil- ity of our method to generalize to real-world sen- sors, we design an experiment that closely fol- lows the real-world experiments proposed in Pan et al.2)), Specifically, we choose the real-world mug-hanging experiment from this work. The au- thors from Pan et al. (2023) have provided us with a dataset real-world mug-hanging demonstrations, collected in the setup displayed in Figure / We then train our method in the same way as we did simulated mugs. See Figure [0a and [1 and Ta- ble [ in the Appendix for qualitative and quanti- tative evaluations of this offline dataset. We find that our method produces predictions consistent with successful mug-hange hanging.
Figure 5: A real-world demonstration of mug-hanging.
CONCLUSION & LIMITATIONS
In this work, we present provably Eqqiiaaana system for predicting task-specific object poses for relative placement tasks. We propose a novel cross-object representation. RelDist, which an -3naa-anaaii geometric representation for cross-object relationships. We also propose a pair of -Equivraaa geometric reasoning layers that use multilateration and SVD, respectively, to extract relative pose prediction. Finally, we demonstrate that this representation provides superior performance in high-precision tasks in simulated environments, and is applicable to point cloud data collected in real-worle manipulation demonstrations.
This method has several limitations. First, it cannot handle symmetric objects or multimodal place- ment tasks without an external mechanism for cummetry-breaking as only a single pose predicted even when multiple poses may be valid. This might be addressed by exploring generative versions of this model. Further, our method assumes that we have segmented the two task-relevant objects (e.g. the mug and the rack). Still, hope that our method provides foundation for future work on learning for relative placement tasks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This material is based upon work supported by the National Science Foundation under Grant No. DGE-1745016 (NSF GRFP), as well as by NIST under Grant No. 70NANB23H178.
REFERENCES
1lge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint arXiv: 13, 2019.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- ence on machine learming, pp. 2990-2999. PMLR, 2016.
Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas Guibas. Vector neurons: A general framework for SQ(3)-equivarian networks. In 2021 IEEE/CVF International Conference Computer Vision (ICCV). IEEE, October 2021.
Peter R Florence, Lucas Manuelli, and Russ Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto (eds.), Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pp. 373-385. PMLR, 2018.
Huy Ha and Shuran Song. Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding. In Conference on Robot Learning, pp. 24 33. PMLR, 2022.
Haojie Huang, Dian Wang, Robin Walters, and Robert Platt. Equivariant transporter network. arXiv preprint arXiv:2202.094000 2022.
S James, Z Ma, D R Arrojo, and A J Davison. RLBench: The robot learning benchmark learning environment. IEEE Robotics and Automation Letters, 5(2):3019-3026 April 2020.
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47)xeabc5986, 2020.
Yen-Chen Lin, Pete Florence, Andy Zeng, Jonathan T Barron, Yilun Du, Wei-Chiu Ma, Anthony Simeonov, Alberto Rodriguez Garcia, and Phillip Isola. MIRA: Mental imagery for robotic affor- dances. In Karen Liu, Dana Kulic, and Jeff Ichnowski (eds.), Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pp. 1916-1927. PMLR, 2023.
Weiyu Liu, Tucker Hermans, Sonia Chernova, and Chris Paxton. Structdiffusion: Object-centric diffusion for semantic rearrangement novel objects. arXiv preprint ar arXiv:2211.04600 2022.
Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hut- ter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 782)) 2022.
Chuer Pan, Brian Okorn, Harry Zhang, Ben Eisner, and David Held. TAX-Pose: Task-Specific Cross-Pose estimation for robot manipulation. In Karen Liu, Dana Kulie, and Jeff Ichnowski (eds.), Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pp. 1783-1792. PMLR, 2023.
Théodore Papadopoulo and Manolis IA Lourakis. Estimating the jacobian of the singular value decomposition: Theory and applications. In Computer Vision-ECCV 2000: 6th European Con- ference Computer Vision Dublin, Ireland, June 26-July 1, 2000 Proceedings, Part I 6, pp- 554-570. Springer, 2000.
Charles R Qi. Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017.
10

---

Published as a conference paper at ICLR 2024
Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Equii-Equ object rep- resentations for manipulation. In 2022 International Conference Robotics and Automation (ICRA), pp. 6394-6400, May 2022.
Anthony Simeonov, Yilun Du, Yen-Chen Lin, Alberto Rodriguez Garcia, Leslie Pack Kaelbling, Tomás Lozano-Pärne Pérez, and Pulkit Agrawal. Equivvaqua relational rearrangement with neu- ral descriptor fields. In Karen Liu, Dana Kulic, and Jeff Ichnowski (eds.), Proceedings of The 6th Conference on Robot Learning. volume 205 of Proceedings of Machine Learning Research, pp- 835-846. PMLR, 2023.
Nillo Sirola. Closed-form algorithms in mobile positioning: Myths and misconceptions. In 2010 7th Workshop on Positioning. Navigation and Communication, pp. 38-44, March 2010.
Olga Porkine-Hornung and Michael Rabinovich. Least-squares rigid motion using svd. Computing, 1(1):1-5, 2017.
Elise van der Pol, Thomas Kipf, Frans A Oliehoek, and Max Welling. Plannable approximations to mdp homomorphisms: Equivariance under actions. arXiv preprint arXiv:2002. / 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I Guyon, U Von Luxburg, Bengio, H Wallach, R Fergus, S Vishwanathan, and R Garnett (eds.), Advances in Neural Infor- mation Processing Systems, volume 30. Curran Associates, Inc., 2017.
Dian Wang and Robin Walters, So (2) equivariant reinforcement learning. In International Confer- ence on Learning Representations, 2022.
Dian Wang, Jung Yeon Park, Neel Sortur, Lawson LS Wong, Robin Walters, and Robert Platt. The surprising effectiveness of equivariant models in domains with latent symmetry. arXiv preprint arXiv:2211.09231, 2022a.
Dian Wang, Robin Walters, Xupeng Zhu, and Robert Platt. Equivariant q learning in spatial action spaces. In Conference Robot Learning, pp. 1713-1723. PMLR, 2022b.
Yue Wang and Justin M Solomon. Deep closest point: Learning representations for point cloud registration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 3523-3532, 2019.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael Bronstein, and Justin M Solomon. Dynamic graph CNN for learning on point clouds. ACM Trans. Graph., 38(5):1-12, October 2019.
Maurice Weiler and Gabriele Cesa. General e (2)-equivariant steerable enns. Advances in Neural Information Processing Systems, 32, 2019.
Ruihan Yang, Ge Yang, and Xiaolong Wang. Neural volumetric memory for visual locomotion con- trol. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1440-1440. 2023.
Andy Zeng. Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, and Johnny Lee. Transporter networks: Rearranging the visual world for robotic manipulation. In Jens Kober, Fabio Ramos, and Claire Tomlin (eds.), Proceedings of the 2020 Conference Robot Learning, volume 155 of Proceedings of Machine Learning Research, pp. 726-747. PMLR, 2021.
Yu Zhou. An efficient least-squares trilateration algorithm for mobile robot localization. In 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3474-3479, October 2009.
11

---

Published as a conference paper at ICLR 2024
Appendix
A EEUVVRQQU GEOMETRIC REASONING
In the following sections, we provide the closed-form solutions used for the multilateration and SVD layers, respectively.
A.1 MULTIILATERATION
In Zhou (209). the problem of multilateration (i.e. estimating the position of receiver given set of distances r to & beacons with known positions qı) is considered:
min
Ip. - -211 ri11
(14)
Zhou (2009) derives two closed-form, differentiable minimizers to this problem, one of which is for the overspecified case (more than 3 non-co-planar beacons) and for the degenerate case where there are many co-planar beacons. In our approach, we utilize the first solution, as is simpler to implement and the point distributions for objects are almost never coplanar in our experiments. See Algorithm for pseudocode of this approach. This function is called as MUL(R, P).
Algorithm Multilateration (MUL)
Input: R E {ri|ra E + JNy, P (palp: E RR
a N Σ
Ee (p.p.
N
P: -r.p.)
B < Σ (- 2p.p. + +P
C N ΣN Pi=
f a + Bc N
2cc c
H 2 Σ
Li=1 p.p, 2cc
N
q H -f'f
p' q
Output: p'
As each operation the algorithm differentiable with respect to R, the entire function MUL is differentiable.
A.2 PROCRUSTES
The problem of finding the (weighted) least-squares minimizing rigid transformation to align two sets points is the well-known Procrustes problem Sorkine-Horung & Rabinovich (2017)
N
min R.
w Rp, +
4e112
s.t. Re 50(3)
(15)
This problem has a well-known closed-form minimizer, given by the pseudocode in Algorithm Because this algorithm solves the Procrustes problem, refer to PRO for short. This function called as PRO(P, Q, w) where w are an optional set of weights and may be omitted if the weights all equal to
12

---

Published as a conference paper at ICLR 2024
Algorithm 2 Least-squares solution to the Procrustes problem (PRO)
Input:
P {pilPy E R [N]. Q {qiq. E [M) w E (wsler E RR [N]
p wi
9- Miq!
Dist wy
X < (P: PNNN
Y + {q. M
W < diag(w)
< XWY"
U E, V + SVD(S)
> Singular- Value Decomposition
D+I
D NJ t- deU
R VDU
t q Rp
Output: R,
All operations in Algorithm 2 differentiable (including SVD, if differentiable implementation
used Papadopoulo & Lourakis (2000); Pan et al. (003))) ); therefore PRO also differentiable.
B PROOF OF FULL-SYSTEM EQUIVARIANCE IN PLACEMENT TASKS
We now offer a proof that in relative placement tasks for objects AA and B, our entire proposed system equivariant under SE(3) transforms to either or B. As a reminder, SE(3) equivariance (Eq. and invariance (Eq. 4). under transform T E (E)),, are defined as follows:
f(z) y
f(Tx= Ty
(16)
= - =
(17)
Let us now define what it means for the cross-poss function = f(PA. to be equivariant. Recall that we are focusing on relative placement tasks, in which the goal is move objects A and B into desired relative transformation (defined formally in Section The cross pose TAS defines how object needs to be transformed move into the goal pose relative to object B. For example, suppose T and T3 are SE(3) poses for objects A and B respectively such that the placement task complete. Now suppose that objects A and B have been rigidly transformed by Ta and T3, respectively. Then the cross-pose is given by
f(T. TA T TB) = TAB = Tg T
(18)
which is the transform one would need to apply to object to move it back into the goal pose, which defined relative object B. If we futher transform object A by TA and object B by Tg, then the cross-pose becomes
J T TB T3 T
(TB o TT) (TA TT)
(19)
= Ts o (T, T T
(20)
TB TAB T
(21)
Thus, we will define a cross-poss function f as equivariant if satisfies
J(TA o P Tg f(P T
(22)
We prove our theorem:
Theorem 1. Let f be the method defined in Section given by
TAS f(P P8) = PRO A MUL PB))
(23)
in which RA is computed from PA and Pg using Equations
10
12
MUL is described in
Sec. 5.3 and defined formally in Appendix and PRO is described in Sec. 5.3 and defined formally
in Appendix A.2 Then f is equivariant under SE(3) transformations to either of its inputs P. P or P 8-
13

---

Published as a conference paper at ICLR 2024
Proof. Assuming:
J0a: Sog (Eqn. are SF(3)-invarian)
MUAB PB) is SE(3) -qqiivaaaa with respect to Ps
PRO(PA, Pa) is left- and right- SE(iiaaauivarian with respect to its inputs PA. P.s respectively, in the sense defined by Equation 24
If fox=fos are change with any transformations of the input; we thus refer to 4 and &g as "invariant features." Thus, RAB (Eqn. 12 must also be an SE(3)-invariant feature, because it is function exclusively of invariant features @ and Qg.
then the features A and 2g output by these functions do not
Therefore:
J(TA PA. PRO (TA o PA, MU TB o Ps))
PRO (Ta PA, TB MUL RR. AB, PB))
=TB PRO WUL((RAA PB)) Tā
=TB fP, PAP)) T
where the first line follows from the definition of f in Eqn. ture explained above; the second line follows from the second assumption; the third line follows from the third assumption; and the fourth line is just the definition of f in Eqn. 23
Sqqn.23jam 23d because RAA is an invariant fea-
noted in the main text, make Sox: los to be S (F(3)-invariant we can either use Vector Neuron layers Deng et al. (2021) )) or we can train standard network to produce $ SF(33-Invariant represen- tations, althouuh the resulting features would not be guaranteed to invariant. We consider both approaches in our experiments.
The function MUL is discussed Sec. 5.3 and defined formally in Appendix A.I As can seen, this function consists of linear operations and thus is trivially (B(3)--quivariant with respect to its second input.
The third assumption follows by the linear algebra properties of Singular Value Decomposition; fol- lowing the steps of Algorithm [ if we compute PRO(TA Q), then X, will be transformed to RA X, Rg Y respectively, where A.A. RB the rotation components of T,, Tg, since PRO first subtracts off the mean of and Q. Since SVD computes set of matrices (U, Σ, VT) where U and V rotation matrices, then U and V will then be transformed RA U and RB V. We then compute the rotation matrix R that is output by the algorithm as VDU" so the output is then transformed RB R R2 which is equivalent to Rg R RA The translation component is similarly transformed, which leads us to the property that
PRO(TA P. TB Q) Ts PRO(P, (P)Q) oT
(24)
C TRAINING DETAILS
We present various implementation details for our method.
C.I NETWORK ARCHITECTURES
Point-Cloud Encoders: We use standard segmentation-styll DGCNN Wang et al. (019)) and VN-
DGCNN Deng et al.) (2021) (DGCNN with Vector Neuron layers) architectures for our encoders.
Pytorch implementations with identical network structure can found (DGCNN) here and (VN- DGCNN) here
Cross-Attention Layer: We implement a single multi-head attention block with 4 heads, with a similar network structure as implemented in Pytorch here
Kernel Network: Our implementation of the kernel network is a small MLP with 2 hidden layers of size [300, 100), ReL.U activation, and BatchNorm.
14

---

Published as a conference paper at ICLR 2024
Figure 6: Examples from the dataset of mugs used to train our method. Notice that the mugs vary in geometry, but have similar overall structure.
C.2 SAMPLING IN MULTILATERATION
In Equation 12 we described RA NA x N8 matrix. In practice, NA and NB can be quite large (> 1000), and each pair must be passed into an MLP to evaluate the kernel. For sufficiently large point clouds, this leads to explosion in GPU memory, where will not fit GPU memory To address this issue, we instead sample K points from each point cloud P A and Pg to construct K x K kernel matrix. In this work, we sample uniformly at random. Importantly, this sampling happens only during the construction of the kernel matrix everything upstream (encoders and cross-attention)
single training example
computed the full-resolution point clouds. In our results, we report model where K = 256, and we show an ablation in Appendix Fn example of performance where K 100.
C.3 PRETRAINING
Following Pan et al. (003), we pretrain our encoders (both DGCNN and VN-DGCNN) to produce SEn3)-invariant encodings using contrastive learning. This effectively learns to map each unique region of mug, for instance, unique representation space (similar to Dense Object Nets Flo- rence et al. (2018) representations). The pretraining involves sampling from SE(3), yielding an SE DGCNN encoder. However, VN-DGCNN is already pretraining step has effect of producing contrastive embeddings which improve performance and convergence speed when training method using VN-DGCNN encoders. See Appendix | for comparison of how pretraining affects model performance.
this
C.4 SUPERVISION
We use identical supervision as proposed in Pan et al.] (2023), combining the Point Displacement Loss, Direct Correspondence Loss, and Correspondence Consistency Loss. See|Pan al. (2023) for details. Examples training mugs can be found in Figure
'Our implementation may not he perfectly efficient with its GPU utilization, but the 2 relationship is fundamentally bottleneck for this method.
15

---

Published as a conference paper at ICLR 2024
D ADDITIONAL RLBENCH EXPERIMENTS
Ablation: Sample Efficiency: To
Angle Error
Transfation Error
Pore
understand how the number of train- ing demonstrations affects the preci- sion of our method, we train vari- ants of our method and TAX-Pose with 1, 5, and 10 training demos from the stack wine task. Results are found in Table 7 We find that our method achieves very high levels of precision with only a single demon- stration, and has substantially higher sample efficiency than TAX-Pose.
Guts
5.010
5.000
5.004
5.002
10
30
humber Demonatradion
Number Demrnntradioe
Figure 7: A comparison between number of training demon- strations and prediction precision on the s stack-wing task.
E ADDITIONAL NDF TASK INFORMATION AND RESULTS
E.I EVALUATION METRICS
In the original NDF and TAX-Pose papers, the performance of each method is evaluated based on whether the object is successfully grasped by the gripper ("Grasp Success"), successfully placed in goal location ("Place Success"), and both successfully grasped and successfully placed in the same episode ("Overall Success").
When considering the precision of a pose pre- diction method for task, however, there is one critical detail of the original evaluation metric which impedes our ability to measure preci- sion: specifically, in the original evaluation pro- tocol, placement success measured by tele- porting an object (without collision) to its pre-
Figure 8: Comparison of a mug placement with and without penetration. Left: placement with 3.4cm of collision. Right: a placement with 0. 0.7cm of collision.
dicted pose, enabling collisions, stepping physics simulator for 5 seconds, and detecting whether or not the object has hit the floor at the end of this period. This gives rise to the following degenerate case: if prediction is made which would result in an intersection of the object with the gripper or environment (for instance, the mug is in direct intersection with the base of the rack), the object may simply stick in place because of the interscction and not fall to the ground (despite being geometri- cally invalid). See Figure 8 for visualization of geometrically invalid and valid placement, both of which are measured as a "success" in the original paper.
To get some sense of how precise these methods actually are, we propose simple modification to the success metric: after the mug has been teleported to a placement location, we detect whether any penetrations between the mug and the rack have occurred (a trivial computation in our simulated environment), and mark the placement as a failure if the penetration magnitude exceeds a certain threshold (we evaluate on thresholds 1cm, 3cm, and unbounded). Like before, we also mark the placement as a failure if the object falls on the floor after being released by the gripper.
16

---

Published as a conference paper at ICLR 2024
Mug Hanging (Uprigh//Arbitrry))
Grasp
Place
Overall
Method
<1em
<3cm
lem
<3em
DONS
0.91/0.35
0.50/0.45
0.45/0.17
NDF
0.96/0.78
0.92/0.75
0.88/0.58
TAX-Pose
1.00/0.50
0.15/0.32
0. 0.70/0.75
0.93/0.85
0.15/0.14
0.70/0.40
0.93/0.44
Ours
1.00/0.43
0.35/0.36
0.78/0.66
0.99/0.88
0.35/0.19
0.78/0.32
0.99/0.41
Bowl Placement (Upright/Arbitrary)
Grasp
Place
Overall
Method
<1cm
<3cm
<lem
<3cm
OC
DON
0.50/0.08
0.35/0.20
0.11/0.00
NDF
0.91/0.79
1.00/0.97
0.91/0.78
TAX-Pose
0.87/0.66
0.47/0.40
0.53/0.56
0.57/0.64
0.37/0.26
0.43/0.37
0.47/0.44
Ours
0.67/0.59
0.36/0.19
0.49/0.42
0.66/0.68
0.26/0.08
0.35/0.23
0.46/0.35
Bottle Placement (Upright/Arbitrary)
Grasp
Place
Overall
Method
<lem
<3cm
0
1cm
<3cm
OC
DON
0.79/0.05
0.24/0.02
0.24/0.01
NDF
0.87/0.78
1.00/0.99
0.87/0.77
TAX-Pose 0.24/0.16
0.89/0.58
0.92/0.65
0.99/0.72
0.22/0.14
0.23/0.14
0.24/0.14
Ours
0.65/0.42
0.25/0.19
0.47/0.42
0.52/0.53
0.17/0.08
0.35/0.22
0.37/0.25
Table 3: NDF Tasks, Success Rate (f is better): The success rate of each method when running the relative placement tasks with the objects starting either upright the table (left number) or in arbi- trary poses above the table (right number). Each column group shows the success rates for the Grasp phase and Place phase, as well as Overall performance (when both grasp and place are successful in a trial). Additionally, for Place and Overall we report success when maximum penetration is thresholded the given distance (1em, 3cm, 00). In Mug hanging, our method outperforms nearly every baseline.
E.2 BASELINES
We compare method to the following baselines:
Dense Object Nets Florence et al. 2018): Dense Object Nets learns representation which is viewpoint-invariant, and then attempts to match novel image with an existing demon- stration. The approach to using Dense Object Nets to solve the relative placement task is given in Simeonov et al. (2022).
Neural Descriptor Fields Simeonov al. (2022): Learns an SE())-eqiivaiiat field, which is then optimized at test time to match set of demonstrations.
TAX- -osee Pan et al. (2023): This approach shares some similarities with ours, except that this approach learns a residual flow field which is not SE(3)-invariant by construction, instead of our method which estimates pairwise distances.
E.3 ADDITIONAL EXPERIMENTS
In this section, we present additional simulated results on two tasks presented in the NDF paper Simeonov al. 2022): Bottle Placement and Bowl Placement. Training datasets for these tasks
17

---

Published as a conference paper at ICLR 2024
generated in an identical fashion to the NDF Mug-Hanging task described in Section 6 the evaluation procedures and metrics.
In this task, the robot must grasp bottle. and place it shelf in its workspace. The bottle is frequently symmetric. Following Pan et al. (2023), we add an additional color channel to the object to break this symmetry.
NDF Bottle Placement Task:
NDF Bowl Placement Task: Similar to bottle, this task, the robot must grasp a bowl, and place on shelf in its workspace. The bowl is frequently symmetric. Following Pan et al. (023). we add an additional color channel to the object to break this symmetry.
E.4 ANALYSIS
We find that our method performs comparably to the closest baseline, TAX-Pose, when trained using the exact code and training procedure provided by Pan et al. (2023). However, performance for both our retrained version of TAX-Pose and our own method are substantially worse (by roughly the same amount) than the original reported TAX-Poss results this is primarily due to the fact that no implementation of symmetry- breakkngg technique described in Pan et al. (2023) was provided in the initial code release, and we were not able reproduce reported results with our own implementation. being very important for good performance. We believe
Pan et al. (2023) report this technique that additional details are needed to reproduce the strong performance reported in Pan et (2023), and will update this manuscript once we obtain the implementation from the authors.
F ABLATIONS
We further perform the following ablations to understand the effect of changing different parts of our method. Ablation results can be found in Tables 4
5 and
Mug Hanging (z/ SE(3))
Grasp
Place
Overall
Method
<1em <3em 00
<1em <3em
00
TAX-Pose, no SE(3)
0.29
0.01
0.35
0.37
0.00
0.05
0.06
Ours, SE(3)
0.35
0.24
0.77
0.87
0.05
0.27
0.31
Table 4: Ablation: Removing SE(3) Data Augmentation Arbitrary Mug Hanging, Success Rate (t is better): Compared to the standard training proposed by TAX-Pose (Pan et al.] (2023)), we remove all SE(3) data augmentations from the training pipeline. We train only on demonstration poses z directly (which are mostly vertical). TAX-Pose performance degrades significantly, while our method's performance remains strong.
Impact of SE(3) Data Augmentations: In the original training procedure for TAX-Pose (Pan al. (203)), ) the authors introduce substantial data augmentation in the form of random SE(3) transforms inputs in order to achieve neural networks which are approximately SE(3)--quivariant. To demon- strate that our method can achieve high success rates without these complex data augmentations, we train version of both our provable-qqiivariant model, as well the standard TAX-Pose model. We report our findings in Table 4 We find that while TAX-Pose fails entirely to generalize to mugs in novel configurations, because our method is provably SE(3)- Equivvaaui,,,,,, demonstrates strong performance across SE(3) even when trained on mugs in a narrow range of starting poses.
Pretraining: We evaluate the effect of pretraining on performance for both tasks. The optimal which the pretraining process approximately initializes for
setting of foa. for are DGCNN (our VN-DGCNN is guaranteed to be invariant). In Pan et al. (2023), the authors show that pretraining the standard DGCNN encoder improves test-time performance. However, we want to whether or not this pretraining procedure improves performance for the VN-DGCNN version (which is guaranteed to be invaraint). Intuitively, this pretraining would simply initialize the representation for all points on each object to be far from one another. Interestingly, in addition to helping training converge faster, pretraining substantially improves performance.
18

---

Published as a conference paper at ICLR 2024
Sampling: We evaluate whether the value of K in the sampling procedure for computing the ker- nel matrix RA detailed in Appendix C affects performance (see "VN-DGCNN (K = 100)" in Tables and 6 We find that increasing the number of sampled points improves the performance, which intuitively suggests that the multilateration optimization will be able to reduce the variance of predictions (and thus increase prediction precision) with more samples.
We evaluate how training a standard DGCNN network to predict $ SE(3) -invarann features for each object encoder compares to using the fully S SE(3)-nnvrriant VN-DGCNN predict those same features. We find that the standard DGCNN network is superior on some precision metries, while VN-DGCNN has superior overall performance. Because we are using networks of a similar size, this suggests that there may a representational penalty for limiting network to be fully--quivariant. Additionally, many of the mugs are perfectly or nearly symmetrical this means that the VN-DGCNN network may not have sufficient representational flexibility to differentiate between two instances, i.e. to "tiebreak" depending on the object's initial orientation.
Vector Neurons:
Grasp
Place
Overall
Method DON
Oem < <lem <2em <3em oo
Oem < lem <2cm <3em oc
0.91
0.50
0.45
NDF
0.96
0.92
0.88
TAX-Pose (release)
1.00
0.01
0.07
0.26
0.61
0.95
0.01
0.07
0.26
0.61
0.95
TAX-Pose (retrained)
1.00
0.04
0.24
0.44
0.67
0.96
0.04
0.24
0.44
0.66
0.96
VN-DGCNN (K 1KK1100
1.00
0.02
0.16
0.36
0.64
0.97
0.02
0.16
0.36
0.64
0.97
VN-DGCNN (no-pretrain)
0.99
0.03
0.21
0.49
0.79
0.93
0.03
0.20
0.48
0.78
0.92
DGCNN VN-DGCNN (Ours)
0.98
0.05
0.31
0.64
0.83
0.97
0.05
0.30
0.63
0.82
0.96
1.00
0.03
0.28
0.57
0.80
0.98 0.03
0.28
0.57
0.80 0.98
Table 5: Ablations Upright Mug Hanging, Success Rate (f is better): The success rate of each method when running the mug-hanging task with the mugs starting upright on the table. Each column group shows the success rates for the Grasp phase and Place phase, as well Overall performance (when both grasp and place are successful in trial). Additionally, for Place and Overall report success when maximum penetration is thresholded the given distance. Across nearly all metrics, our method outperforms the baselines.
Grasp
Place
Overall
Method DON NDF
Oem <lem <2em <3em
=0em <lem <em <3em
0.35
0.45
0.17
0.78
0.75
0.58
TAX-Pose (release)
0.74
0.04
0.18
0.48
0.74
0.89
0.03
0.14
0.35
0.55
0.65
TAX-Pose (retrained)
0.68
0.03
0.20
0.42
0.61
0.86
0.01
0.14
0.30
0.42
0.58
VN-DGCNN (K 100)
0.61
0.04
0.22
0.39
0.63
0.91
0.02
0.12
0.22
0.39
0.56
VN-DGCNN (no-pretrain)
0.44
0.05
0.28
0.52
0.71
0.82
0.02
0.13
0.24
0.32
0.37
DGCNN
0.68
0.04
0.30
0.51
0.70
0.89
0.03
0.20
0.35
0.48
0.61
VN-DGCNN (Ours)
0.67
0.04
0.29
0.55
0.72
0.92
0.03
0.18
0.36
0.47
0.62
Table 6: Ablations Arbitrary Mug Hanging, Success Rate (1 is better): The success rate of each method when running the mug-haud hanging task with the mugs starting in arbitrary orientations above the table.
19

---

Published as a conference paper at ICLR 2024
G REAL-WORLD EXPERIMENTS
Experiment Description: To demonstrate the abil- ity of our method generalize real-world robotics scenarios, we design an experiment that closely fol- lows the real-world experiments proposed in Pan et (003). Specifically, we choose the real-world mug-hanging experiment from this work. The thors from Pan et al. (2023) have graciously pro- vided us with a dataset real-world mug-hanging demonstrations, collected in the setup displayed in Figure This dataset consists 20 different single
Method Trans, Error (m) Rot. Error (*)
Ours
0.1
35
Table 7: Average prediction error un- seen, real-world mugs. The metrics are com- puted comparing the predicted position of the mug to the actual position of the mug in the demonstration.
demonstrations of various mugs placed rack, where the rack is placed in different places in the workspace of a robot. The demonstrations consist of initial and final observations of the mug and rack. We split this dataset into 10 training mugs and 10 test mugs. We then train our method the exact same way as we did our simulated mugs, and evaluate the test mugs.
Our evaluation is both quantitative and qualitative. First, our quantitative evaluation procedure is randomly perturb a real test demo capture, and attempt predict its success configuration. We then compute a translational and rotational errоr, to meausre how far the mug was placed relative the test demonstrations. See Table 7 Because the mug can rotate freely around the peg, the relatively high rotation error is not indicative of poor predictions in fact, these metries are frequently within the tolerance for successful mug hanging given scene geometry.
Analysis of Results:
To further qualitatively support this claim, consider real predictions we provide in Figure 11 We see that predictions are consistent with successful mug-hanging with the handle placed over the peg of the rack. Furthermore, the format of these predictions (a simple rigid transform) is identical to real system. those in Pan et al. (2023), which led in their work to successful robotic placement robot arm, While we do not execute motion policy based these predictions believe that this qualitative and quantitative evidence provides strong support for our claim that our model could be successfully used in real-world robot system similar to that proposed by Pan et al. 2023)
(a) A real training mug
(b) A demonstration of mug- aug-hanging
Figure 9: An example how demonstrations were collected real-world robot workspace. Im- ages provided by the authors of Pan et al. (2023).
20

---

Published as a conference paper at ICLR 2024
11
(a) Captured mug point clouds
(b) Captured rack point clouds
(c) Captured mug-hanging demo
Figure 10: Examples of training data collected with the setup in Figure The mugs are segmented
from the rack by the procedure outlined in Pan al. (2023), and then used as training data for method.
Figure 11: Images of distinct, unseen real-world mugs in their predicted goal configurations. Our method makes high-quality predictions novel mug instances in unseen configurations from real-world sensor data. Because we make outputs the same format as in Pan al. (2023), we believe that these predictions are strong evidence that our method could successfully be used to solve relative placement tasks on a real-world robot system similar to that proposed Pan et al. 2023
21


Reviews:
[{"review_id": "MCMV0mh6An", "reviewer": "Reviewer_5GvH", "summary": "The paper presents a system that is provably SE(3)-Equivariant for predicting task-specific object poses for relative placement tasks. It introduces a new cross-object representation called RelDist, an SE(3)-Invariant geometric reasoning framework, and employs multilateration and Singular Value Decomposition (SVD) to extract relative pose predictions. The study validates the representation’s performance through simulated high-precision tasks and real-world manipulation demonstrations, emphasizing its applicability to point cloud data.", "questions": "- Could experimental results be further augmented with a more diverse set of tasks or scenarios? What could be the choices?\n- How does the system handle noisy data or incomplete point clouds, common in real-world scenarios?\n- Could the authors elaborate on potential strategies for overcoming the limitations related to symmetric objects or multimodal placements?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper's originality lies in its novel representation for cross-object relationships and the formulation of a problem-solving approach that is SE(3)-Equivariant. The quality of the work is high, evidenced by the clear methodology and promising experimental results. The clarity of the presentation is commendable, with complex concepts and processes being explained with precision. The experiment results are supportive to the precision requirement of the tasks and explained by the algorithm design. Lastly, the significance of the work is underlined by its practical applications in robotics and potential to influence future research in the area.", "weaknesses": "Although the experimental results are positive and supportive to the claims on e.g., precision and the algorithm design, more tasks or scenarios could be evaluated. They can still be precise pick and place but should at least be different sets of objects, such as peg-in-hole.  The methodology looks pretty promising (equivariant + differentiable optimization process) and generic, as a submission to a ML conference, one would expect to see a more diverse evaluation of the approach.\n\nOther weaknesses identified pertain to limitations in handling symmetric objects or multimodal placement tasks, as acknowledged by the authors. This could restrict the system's application in scenarios where multiple correct poses exist. Additionally, the requirement for segmented task-relevant objects can be a significant limitation in unstructured environments. The paper could be improved by exploring these aspects further, possibly by integrating generative models or unsupervised segmentation methods.\n\nHowever, overall, I would still think the paper is slightly above the threshold, while a more complete evaluation could well strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IFNwEFSYWB", "reviewer": "Reviewer_5kHb", "summary": "This paper propose a method for precise relative pose prediction which is provably SE(3)-equivariant, can be learned from only a few demonstrations, and can generalize across variations in a class of objects. The core technical contribution is to transform the optimization of two SE(3) fields into a differentiable multilateration problem.", "questions": "- Will there be genuine real-world experiments conducted in the future?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper proposes a method that tackles SE(3)-equivariant learning by estimating the  corresponding point pairs with differentiable multilateration. The method provides the community with some fresh new ideas.\n- This paper is in general written in a clear way, and is easy to comprehend.", "weaknesses": "- Presentation\n    - The problem statement section as well as some figures are directly borrowed from TAX-Pose. I think it severely damages the presentation of this paper.\n- Performance\n    - The result in Table 1 is margianl improvement compared with PAX-Pose, though the proposed method does exhibit some advantages in higher-precision settings.\n- Real-world experiments\n    - This method is evaluated with offline real-world trajectories collected by TAX-Pose. I won't accept this as real-world experiments.\n    - In figure, the rotational error is 35 degrees. While the authors have made some explanations, I won't regard it as a satisfactory result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i1rz2afAcn", "reviewer": "Reviewer_Te9m", "summary": "This paper proposes a novel model for implementing the SE(3)-equivariance in robotic pick and place problems. Given a task involving placing an object relative to another object, the method first uses an equivariant network to calculate the desired distance between points on the two objects. The core innovation of the method is that given the desired distances from a point on object A to a set of points on object B, one can use multilateration to locate the desired location of this point relative to object B. Repeating this process for multiple points on object A, the relative pose between A and B could then be confirmed. The proposed method is evaluated against a number of baselines, where it demonstrates commendable performance, especially when the task requires high precision.", "questions": "1. As is mentioned in Weakness, my main comment is about the experiment section. The paper would be much stronger if there is an environment that actually requires high place precision (e.g., gear/kit assembly) rather than merely adjusting the precision requirement in the mug-hang task. Moreover, incorporating the performance of NDF under the 1cm and 3cm precision requirements in Tables 1 and 3 would likely strengthen the paper as well.\n2. Does the proposed method have a faster runtime compared with the baselines? Does the proposed method have a higher sample efficiency? Since the idea of using multilateration reduces the equivariant problem into an invariant problem, I am curious about if there are benefits from this angle.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of using multilateration to calculate the relative pose is a compelling aspect of this paper. It transforms the equivariant problem of calculating the desired relative pose into an invariant problem of calculating the desired relative distance, which could be useful to reduce the complexity of a model.\n2. The paper is well-written with intuitive examples to illustrate the idea.", "weaknesses": "My main concern with the paper is that the experimental evaluation is not strong enough. In the main paper, the experiments are mainly conducted in the Mug Hanging domain. In the two other domains in Table 3 in the appendix, the proposed method’s performance is worse than the baselines. Though the authors discuss that the underperformance compared with TAX-Pose could be due to the lack of the implementation of the symmetry-breaking technique, the proposed method also lags behind NDF. Additionally, the performance of DON and NDF under 1cm and 3cm distance thresholds is not reported in either Table 1 or Table 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)