RiEMann: Near Real-Time SE(3)-Equivariant Robot Manipulation without Point Cloud Segmentation
Authors: Chongkai Gao
Zhengrong Xue
Shuying Deng
Tianhai Liang
Siqi Yang
Lin Shao
Huazhe Xu
Venue: CoRL 2024 (2024)
Keywords: SE(3)-Equivariance
Manipulation
Imitation Learning
TL;DR: This paper proposes RiEMann, an efficient and near real-time SE(3)-equivariant robot imitation learning framework for manipulation tasks framework without point cloud segmentation.

Abstract: We present RiEMann, an end-to-end near Real-time SE(3)-Equivariant Robot Manipulation imitation learning framework from scene point cloud input. Compared to previous methods that rely on descriptor field matching, RiEMann directly predicts the target actions for manipulation without any object segmentation. RiEMann can efficiently train the visuomotor policy from scratch with 5 to 10 demonstrations for a manipulation task, generalizes to unseen SE(3) transformations and instances of target objects, resists visual interference of distracting objects, and follows the near real-time pose change of the target object. The scalable SE(3)-equivariant action space of RiEMann supports both pick-and-place tasks and articulated object manipulation tasks. In simulation and real-world 6-DOF robot manipulation experiments, we test RiEMann on 5 categories of manipulation tasks with a total of 25 variants and show that RiEMann outperforms baselines in both task success rates and SE(3) geodesic distance errors (reduced by 68.6%), and achieves 5.4 frames per second (fps) network inference speed.

RiEMann: Near Real-Time SE(3)-Equivariant Robot Manipulation without Point Cloud Segmentation
Chongkai Gao
Zhengrong Xue Tsinghua IIIS
National University of Singapore gaochongkai @u. nus edu
x2r230mai iss. ts tsinguua.... edu en
Shuying Deng
Tianhai Liang
Tsinghua University
Tsinghua University
starrism1 4128 eetzziimmmi2@gmaa....... meriammmal..oo com
tianhai liang.inngg....... on gmail com
Siqi Yang
Lin Shao
Tsinghua University yang-sa2lemaile tsinghua. edu. en
National University of Singapore linshao@nus edu. sg
Huazhe Xu
Tsinghua IIIS Shanghai AI Lab Shanghai Zhi Institute
huazhe.xu@mail. a weinnnuaa.deee edu. en
Abstract: We present RiEMann, an end-to-end near Real-time (qui-raquva Robot Manipulation imitation learning framework from scene point cloud input. Compared previous methods that rely on descriptor field matching. RiEMann directly predicts the target actions for manipulation without any object segmen- tation. RiEMann can efficiently train the visuomotor policy from scratch with 5 to 10 demonstrations for manipulation task, generalizes to unseen SE(3) trans- formations and instances of target objects, resists visual interference of distract- ing objects, and follows the near real-time pose change of the target object. The scalable SE(3)-equivarian action space RiEMann supports both pick-and-place tasks and articulated object manipulation tasks. In simulation and real-world 6- DOF robot manipulation experiments, we test RiEMann on 5 categories of ma- nipulation tasks with total of 25 variants and show that RiEMann outperforms baselines in both task success rates and SE(3) geodesic distance errors (reduced by 68.6%), and achieves 5.4 frames per second (fps) network inference speed. Keywords: Equivaaiaa,,,,,,,, Manipulation, Imitation Learning
1 Introduction
Learning from demonstrations is an effective and convenient mechanism for visual robot manipu- lation tasks [1, 2]. However, most current algorithms for learning from demonstrations suffer from low data efficiency and generalization ability to new situations. For example, current visual imita- tion learning algorithms require around 100 demonstrations [3, 5] lear simple manipulation task such picking up mug and placing yond the training distribution. Although previous works have tried to tackle these problems by data augmentation [6, 7] or contrastive learning on demonstrations [8, 9, 10]. they rely on task-specific domain knowledge and have no algorithmic guarantees to generalize to unseen object poses.
rack, and cannot generalize to new object poses be-
Exploiting symmetries in the 3D world improve the sample efficiency and generalization ability of robot learning algorithms [11]. Roto-translation equivariance is of the most common types
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

(b)
Figure 1: Overview of RiEMann. (a) Given 5 to 10 demonstrations of restricted object poses (The mug remains standing and only rotates in 90 degrees around the z-axis) of the task Mug on Rack and (b) with the full scene point could input without segmentation, (c) RiEMann can generalize to local SE(3)-equivarian transformations of target objects, to new instances of target objects, be robust distracting objects, and (d) has the near real-time following ability of target objects.
of symmetry for robot manipulation tasks. It denotes that robot actions can transform with the same SE(3) transformations as the target objects. Pioneering works like NDFs [12, 13] and EDFs [14, 15] propose to first learn an SEu3)-eqiivarian object descriptor field with SE(3)-qqiivarian back- bones [16, 17, 18] and then output robot actions with energy-based optimization to match differ- ent descriptor fields. They achieve -quivvaquve on pick-and-place tasks such as picking up the mug the rack [14, 12, 13, 15]. However, these methods share the same limitations of the field-matee feeld-macching mechanism: 1) the matching process is time-consuming preventing it from being a real-time method: 2) the matching process can only yield one SE(3) transformation, which only works on pick-and-place tasks and precludes its application to articulated object manipulation tasks.
Recently, some works have been trying to solve these issues. Fourier Transporter [19] and TAX- Pose [20] try to tackle the first issue by cross correlation of Fourier transformation on SO(3) [19] or differential SVD [20] to replace the time-consuming optimization process, but they either require exorbitant computation to discretize the SO(3) group at high resolution, require pre- trained encoder on an extra well-segmented dataset. EquivAct [21] tries to tackle the second problem by directly predicting robot actions with neural networks, but they require well-segmented point clouds and can only predict 3-DOF actions rather than 6-DOF actions. These works inspire us to design an end-to-end -uuuivaquv policy that enables 6-DOF manipulation tasks without segmentation to replace the field matching process, as well an efficient network structure and training algorithm to alleviate complexity brought by equivariant backbones improve the inference speed.
In this work, we present RiEMann, the first near Real-time (S(3)-Eqqivariant robot Manipulation framework from point cloud inputs without any segmentation. RiEMann leverages local SE(3)- actions. For equivariant backbones [17] and learns a policy that directly outputs -E(3)-invariant vector field the 6-DOF action space parameterization problem, RiEMann uses an as the target point affordance map for translational actions, and uses three -Euuvvequi vec- tor fields as three bases of the target rotation matrix well as Gram-Schmine orthogonalization operation for the rotational actions. This action space also be extended to articulated object manipulation tasks by adding an SE(3)- directional action. Theoretically. we prove that other common parameterizations for the SO(3) group, such as axis-angle, quaternion, and Euler an- gle, are not trainable SE(3) eequivariat representations. For the computational complexity problem, RiEMann firstly learns an SE(3)-invariant saliency map on the input scene point cloud to extract the region of interest, then trains the main Squuieequ policy on the extracted point cloud. Since the equivariant policy consumes the majority of computing and memory resources, reducing the number of points to be processed can significantly decrease computational and memory burden.
In 5 manipulation tasks with a total of 25 different task settings from simulation and real-world experiments, we demonstrate that RiEMann can solve various manipulation tasks with only 5 to 10 demonstrations for each task, generalize to unseen poses and instances of target objects, resist visual interference of distracting objects, have the near real-time (5.4 FPS) inference speed as illustrated in Figure 1, and outperforms baselines both success rates well the SE(3) geodesic distance errors (reduced the geodesic error by 68.6%), In summary, main contributions are follows:

---

We propose the first end-to-end -quivvaaiia visuomotor policy learning framework for 6-DOF manipulation tasks by our specially designed SE(3)-eqiivariant action space;
We design a saliency map network together with position and orientation networks as the policy model to achieve efficient SE(3) -quiivrauaa training and near real-time inference;
We achieve state-of-the-art success rates and pose estimation accuracy on a set of pick- and-place tasks under various settings, and we are the first to perform -quiievuiaa articulated object manipulation tasks through end-to-end training.
2 Related Works
2.1 Group Equivariant Neural Networks
Exploiting groups of symmetry pervading in data and incorporating them into deep neural networks has been the focus of many studies since can improve generalization and data efficiency. Equiv- ariant neural networks, which are first introduced into CNNs [22], extract symmetries from various kinds of data. According to Bekkers [11], Han et al. [23]. most equivariant models can be di- vided into two categories: regular group representation networks and steerable group representation networks. The former seeks to define equivariant convolution filters [22, 24], attention mecha- nism [25], or message passing mechanism [26] as functions on groups, while the latter uses irre- ducible group representations [27] with spherical harmonics as an equivariant basis to perform mes- sage passing [28, 29, 17, 18]. Some other works [16] design special non- non-linear kemells achieve equivariance the SO(3) group. The theory of equivariant networks on homogeneous spaces is formalized in Cohen et al. [30] with the vector bundle theory.
2.2 Equivariant Robot Manipulation
Researchers have been exploring equivariant models for robotic manipulation tasks to improve gen- cralizability and sample efficiency. Pioncering works [31, 32, 33, 34, 35, 36] learn equivariant rep- resentations of objects the scene with equivariant networks to get SO(2)- or SE(()-equivariance for desktop manipulation tasks with imitation learning or reinforcement learning. NDFs [12, 13, 21] and TTA--Pose [20] leverages Vector Neurons [16] or SO(3)-data augmentation to acquire SE(3)- cquivariant category-level object representations from point cloud of objects for downstream imi- tation learning, while EDFs [14, 15] uses SE(3) Transformer [17] and Equiformers [18] to directly learn -quivaranu representations from point clouds of the scene. Huang et [19] leverages discrete approximation of steerable representation SO(3) and uses cross correlation to tackle the optimization problem, but still requires huge computation for high-resolution discretization. In this work, we leverage (E(3)-transformes [17] and design an end-to-end learning paradigm to predict target actions from scene point cloud input in near real-time.
Background and Problem Formulation
3.1 Problem Formulation
Let a colored point cloud with N points be P ((I1,c1), (.N,,..),,,,,,,... (IN, cN)] } E Nx6 where x; E R3 the position and e E 3 the RGB color of the i-th point. For a manipulation task T, policy fa parameterized by is trained to predict the target pose of robot end-effector T [R. t} SE(3)
with a set of demonstrations D {(P,, ? that consists of m pairs, where R E SO(3) is the 3D rotation action and 3 is the 3D translation action. We use T = (P) to denote the predicted output. We assume a motion planner with collision avoidance is used execute the action T, as in Shridhar et al. [37], Simeonov et al. [12], Ryu et al. [14]. The trained policy will be tested on unseen target object poses, distracting objects, and new object instances, which relies the following local SE(3) property of our policy.
Let g E SE(3) be an element of the Special Euclidean Group in three dimensions (SE(3) and T, X > X be transformation of g on a space X. function X + y is called SE(3)- equivariant if there exists transformation Sy y + y such that:
S,1f(z)] ((T,r) Vgg 2E(3) X.
(I)
3

---

In our robot manipulation tasks, f is the trained policy, X' is the space of the scene point cloud P, y is the space of target pose T, 7g is the space of any SE(3) transformation T3 = {R, t) on P, and Sg is the SE(3) transformation on the target pose T and we set S = Tg following [30, 11, 38).
Local Seuia--quivaeiaace [14, 15] refers that T3 can be applied on only part of the input scene point cloud P rather than the whole scene imput, as illustrated in Figure 1. This is more practical for robot manipulation tasks, in which we want the policy to be only equivariant to the target objects. Local equivariance can be achieved by equivariant networks with local mechanisms. In this work, we choose to use transfrrrssssssmrs [17] that belong to the latter category, as introduced below.
3.2 Group Representations and SE(3)-Transfrmmer
Tg and Sg above are called group representations. Formally, group representation D of a group G is a map from group G to the set of x N invertible matrices GL( N). and it satisfies D ((h) = D(gh), Vg.h E G. Specifically,, any representation of SO(3) group ((R) for vR E SO(3) can be block dook-diigon as the direct sum of orthogonal Wigner-D matrices (R)
((21+))x((+1)
Vectors transforming according to (R) are called type-/ vectors [11]. Type-O vectors are invariant under rotations (D R I) and type-1 vectors (3d space vectors) rotate according to 3D rotation matrices. Note, type-/ vectors have length 2/ 1. Given a point cloud P. we can assign a type-l vector to each point x E P to get type- vector field f) RR3 + 2 Same or different type-! vector fields can be stacked to get complex vector field.
(E(3))transformee [17] are neural networks that map point cloud vector fields to point cloud vector fields with the same point number, and they are designed to achieve local -equivraaa for any learned type- field:
Dr(R)((z) ((Ta), Yo E P, = (R. t) E SE(3).
(2)
To leverage this property for robot manipulation tasks, we need to design special input and output vector fields to satisfy task requirements, as discussed in the following section.
4 (B(3)-Equivarian) Robot Manipulation
Previous works like NDFs [12, 13] and EDFs [14, 15] output several type-0 or higher type-l vector fields as the object descriptor for field feld-matching to get the desired robot action. In contrast, we aim to re-design the outputted vector fields and directly use them as actions. This can transform the generative modeling problem of the descriptor field into a supervised learning problem that enables higher training precision and a faster inference speed.
4.1 Design Choices of Vector Fields
There are two main requirements for the design of input and output vector fields: a) all the input (seene point cloud) and output (robot actions) can be incorporated into the vector fields; b) the predicted actions parameterized by output vector fields are -Euui--qqi to the input transfor- mation. For the input design, since SE(3)-tranfformmr is actually -(3)-invariant, we should only use the T(3)-invariant features, or visual features, as the input, ensure T(3)- iivariavae, Thus here we only use color information for input. RGB information can be seen type-O vectors, our input is direct sum of three vector fields:
P.n (x)
(3)
For the output, model predicts the end-effector target pose T= E SE(3), which can be decomposed into target position t. and target rotation R. For target position t, we cannot directly output the 3D target position vector since transforrrss are only able to predict Tn3)-invariant features. Thus we choose to learn one type-0 vector field as an affordance map [39, 40] on point clouds, and use softmax on the affordance map to perform weighted sum on all point positions to get the translational action É. We also assume an offset t, is provided if the target position is out of the convex hull of the object point cloud, and in this case the output t to.

---

For the target orientation R, people usu- ally use Fuler angle, quaternion, rotation matrix, or axis-angle as robot action pa- rameterization. Although all these pose parameterizations can satisfy the first re- quirement (representing output action in- formation), we need to ensure they are Squii--quivariant to the input transfor- mation after parameterizing them with different type- vectors. We answer this question with the following theorems. All proofs are in Appendix A.3.
X2 Rx
31
11/2
[0,0.1]
[10.0]
P.
8g
02 n/2
Y2
P2
(0..82:12) R(.y.
(o)
( 02) R .,,
(d)
Figure 2: Illustration of different 3D rotation represen- tations under type-l parameterizations. (a) The initial point cloud is transformed to æ2 with 31) rotation R; (b) Using three type-1 vectors to represent a Rotation Matrix. It transforms with the same transformation R as the input; (c) (d) Using three type-O vectors to represent Euler Angles, and using one type-1 vector and one type- O vector to represent Axis-angle. They cannot be trans- formed with the same transformation R as the input, so they are not SE(3) -qquivrrrra parameterizations.
Theorem 1 Rotation matrices, repre- sented by three type-1 vectors, are SE(3)- equivariant vector field parameteriza- tion.
Theorem 2 There is no SE(3)- equivariant vector field parameteri- zation for Euler angle, quaternion, and axis-angle.
The intuitions behind these theorems are illustrated in Figure short, rotation matrices, repre- sented by three type-1 vectors, are the only SEu3)-qquivariant rotation representation the input transformation.
In summary, RiEMann learns one type-0 vector field (target point heatmap) for target position t and three type-1 vector fields for target rotation R. which is direct sum of vector fields:
foue(r) = (o(z) + θ Vo
(4)
As comparisons, EquivAct [21] uses one type-1 vector field as output (end-effector velocity in Carte- sian space), which only supports 3-DOF manipulation tasks. Instead, RiEMann supports 6-DOF ma- nipulation tasks. The next question is how use networks to implement equation 3 and 4 efficiently and transform the output vector fields to single output action while ensuring
4.2 Network Design
The first problem of leveraging equivariant networks for scene point cloud input the heavy com- putational and memory cost. A scene point cloud for robot manipulation tasks usually contains thousands to tens of thousands of points (in our case, 8192 points), and end-to-end learning on such large number of points brings huge burdens for GPU memory and long training time. For exam- ple, EDFs [14] only support batch size = for training on an NVIDIA RTX3090 GPU. Meanwhile, the training difficulty also increases since most of the points are not informative. In this work, we employ an (E(3))transformers d(x) to output one type-O vector field f, (z) = fz))), æ e P for each point as a saliency map, and extract relatively small region of point cloud B Roz with a predefined radius 71 centered on the point with the highest value, as illustrated in the left part of Figure
With Rot as input, we then employ another two SE(3) tran))))ranformer networks to learn a transla- tional action network ub (I) and an orientation network (2) for Vz E BRo as policy networks for action prediction. For the translational part, 인 outputs one type-0 affordance f, (x) = fz(, e BROI. We then perform SoftMax on (z) as weight and multiply them to the positions of all points of Bror to get the output translational action f. For the orientation part, 4/2 outputs three type-1 3 ) (((x) x e BROS. Then we select the points from a smaller region that vector fields fR A
centered on & with radius r2, and perform mean pooling on these points to get the output orien- the translation action t in tation action R. as in the right side of Figure 3. During training,
5

---

Softrae Juii gitted
Softrar weighted
Pouitio
type-O vecttr feld
Sallency Net
Orientation Net
ocoling
IMGS
type
backpropagateable
(Salency map)
Three type-1 vector fields
MSt
Figure 3: Pipeline of RiEMann. type-O saliency map firstly outputted by an SE(3)-invariant backbone Φ to get small point cloud region Roor, and an Squ))-aquivarian policy network that contains 1/ and t/2 predicts the action vector fields on the points of B Bnor. Finally, we perform softmax, region mean pooling, and IMGS to get the target action T.
Figure 4: Simulation and world environments. Top: the training environment settings of simu- lation tasks and real world tasks. Bottom: the ALL testing case tasks, where the target object is a new instance and in pose, and distracting objects are added in the environments.
the demonstrations as supervision for both Φ(x) and (x)), and three orientation axes of R to train (2)) The training algorithm of RiEMann is shown in Algorithm in Appendix A.1. Note the three type-1 vectors from 1/2 are not necessarily orthogonal each other, which case they cannot form a legal rotation matrix. Here we perform Iterative Modified Gram-Schmidt orthogonalization (IMGS) [41] to get an orthogonal matrix R, for actual robot control, as Appendix A.2.
For articulated object manipulation, we assume that given a target pose T (R. E SE(3) and a target direction d E R3, the robot can accomplish the task by first going to the target pose T just as done pick-and-place tasks, and then moving along the target direction d while keeping orientation not changed, as illustrated Figure We expend the action space with another type-1 vector field on the orientation network u22 fR ff(x), E BROI- The final output directional action d is also calculated through mean pooling on points in the radius r2. Note the policy needs to continuously predict the output direction during the opening process, which shows that RiEMann can capture the local SE(3)-equivariance of the handle part of the faucet and has near real-time inference speed. Detailed implementation details of model are listed in Appendix A.4
Experiments
We systematically evaluate RiEMann in both simulation and real-world experiments. This evalua- tion includes types of manipulation tasks simulation environments and types of manipulation tasks in the real world, with different settings and 10 demonstrations for each task. The ablation studies are in Appendix A.9.
5.1 Simulation Environments and Tasks
We build all simulation manipulation environments based on Sapien [42] for point-cloud ma- nipulation. We build a ring-table experiment platform with uneven tabletop for all tasks simula- tion, shown in Figure 4. We put 6 RGBD cameras around the table to fuse their captures get the point cloud input, and downsample the scene point cloud to 8192 points. For each manipula- tion task, we collect demonstrations and train our policy under the training setting (T), and test the

---

Table 1: Success rates of different tasks in simulation. Evaluated under 20 random seeds.
Mug on Rack
Plane on Shelf
Turn Faucet
Method
T
N1
NP
DO
ALL
T
NI
NP
DO
ALL
T
NI
NP
DO
ALL
PerAct [37]
0.85
0.00
0.70
0.00
0.00
0.90
0.00
0.80
0.00
0.00
0.45
0.00
0.50
0.00
0.00
R-NDF [13)
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
u/a
u/a
n/a
EDF [14]
1.00
0.85
1.00
0.95
0.80
0.90
0.75
0.80
0.85
0.70
n/a
n/a
n/a
n/a
n/a
D-EDF [15]]
1.00
0.85
0.95
0.95
0.75
1.00
0.80
0.95
0.95
0.75
n/a
n/a
n/a
IVa
n/a
RiEMann (Ours)
1.00
0.90
0.95
1.00
0.85
1.00
0.90
1.00
1.00
0.90
1.00
0.75
1.00
1.00
0.65
Table 2: SE(3) Geodesic distances of tasks in simulation. Evaluated under 20 random seeds.
Mug on Rack
Plane on Shelf
Turn Faucet
Method
T
NI
NP
DO
ALL.
T
NI
NP
DO
ALL.
T
NI
NP
DO
ALI.
PerAct [37]
0.393
4.086
0.698
4.166
4.375
0.431
4.806
0.469
4.752
4.993
0.457
4.365
0.382
4.218
4.039
R-NDF
4.855
4.298
4.178
4.509
4.662
4.277
4.361
4.179
4.466
4.989
4.996
4.374
4.278
4.229
4.560
EDF [14]
0.249
0.429
0.347
0.252
0.501
0.333
0.872
0.461
0.337
0.985
0.188
1.478
0.448
0.242
2.049
D-EDF [15]
0.312
0.545
0.425
0.337
0.682
0.328
0.966
0.417
0.345
1.024
0.304
2.047
0.567
0.488
2.249
RiEMann (Ouns)
0.053
0.066
0.069
0.056
0.068
0.101
0.120
0.117
0.099
0.122
0.079
0.159
0.098
0.082
0.197
trained policy on T and four extra settings: 1) New Instance (NT): the target object will be replaced by different objects of the same category; 2) New Poses (NP): the initial and target poses of target objects will be under SE(3) transformations on the table; 3) Distracting Objects (DO): there will be additional distracting objects in the scene; 4) ALL: the combination of NI, NP, and DO.
We design three tasks for evaluation. More detailed task descriptions can be found in A.5: 1) Mug on Rack: For T, we put a mug the left-down quarter of the table, and put the rack on the right-down quarter of the table, allowing them to rotate within 90 degrees only around the z-axis randomly, For NI, we use a new mug and keep the rack the same, For NP, we allow the mug and the rack on any quarter of the table, and rotate the mug along all axes with arbitrary degrees; 2) Plane on Shelf: This task is designed for the evaluation of objects with complex geometry, as evaluated in [43]. The setting of T. NI, and NP are the same above: 3) Turn Faucet: This is an articulated object manipulation task and the robot must predict -Euui-equi opening direction to turn the faucet. For NP, rotate the faucet along the z-axis, and also change the initial position of the handle.
5.2 Baselines and Metrics
We compare RiEMann with four baselines: 1) PerAct [37]: a point-cloud- based imitation learn- ing method. This is for the comparison between the SEu3)-equivariant method and the non Non-SE equivaraint method. We perform SE(3) data augmentation for PerAct; 2) R-NDFs [13]: the strongest SE(3)-equivariant method that rely on NDFs [12] and Vector Neurons [16]: 3) EDFs [14]: an SE(3)- equivariant baseline that use SE(3) transfrrrass for manipulation tasks. This is for the comparison between directly regressing target poses (ours) and field matching on type-l fields. Note EDFs re- quire segmented point clouds of the scene and the robot end-effector respectively. Here we manually extract the end-effector point cloud out from the scene for EDFs; 4) D-EDFs [15]: the SOTA SE(3)- equivariant baseline for object rearrangement tasks. This is for the comparison of performances as well as training and inference speeds.
We report the success rates of all methods all five settings (T. NI, NP, DO, ALL) with the average success rates across 20 different random seeds. The success rates are the total success rates of the two stages of each task. Additionally, to better evaluate the quantitative results and climinate the factors, we propose to evaluate the SE(3) geodesic distance [44] of the
influences of
11
predicted target pose T and the ground truth T: D geo (T, f)
v log (R. (RTR)
+ lli t11
where V is the logmap. We report D Dgeo in the same manner as success rates.
5.3 Results
Table and Table 2 show the success rates and Dgea respectively. We can see that PerAct [37] generally performs worse than Squii-equivariant methods, which shows using equivariant networks better than doing SE(3) data augmentation for a non-equivariant backbone. NDFs [13] with Vector Neurons [16] totally fail unsegmented point cloud input, which shows the necessity of using local SEu3)-qquv modules. EDFs [14] and D-EDFs [15] achieved mediocre results in both
7

---

(b) The BRor and the local -Fquivauuiariian fea- ture visualization of the ALL test cases. The left shows the position (one type-O vector) field pre- dicted from (three type-1 vectors) fields from Φ2.. We can see that type-1 vectors away from the target position are different from the correct orientation, which shows the necessity of using (the red circle).
The right shows the orientation
ALL
(a) Pose predictions of the task Mug on Rack of four Test cases NI, NP, DO, and ALL in the real world.
Figure 5: Test pose predictions and feature visualization the real-world task mug-on-rack.
success rate and D gee. which shows that they can generalize to local SE(3) transformations of the target objects certain degree. However, their performances are worse than RiEMann, especially in gea+ which shows that the field-matching mechanism performs worse than end-to-end policy training. Meanwhile, they cannot support any articulated object manipulation tasks because their output space contains only one target pose. Instead, RiEMann consistently achieves competitive results in success rates significantly better results in Dyea (reducing the average Dyeo by 68.6%) under all test settings, which shows the superiority of our end-to-end learning paradigm. Note Dges = 0.05 means an error around lem + 2", which is low enough for a successful manipulation. We also visualize the four test case pose predictions in Figure Sa and the local -qquivaruaee features in Figure 5b. We can see that the position network 1/1 can attend the correct region (the rim of the mug), and the orientation network 1/2 can predict correct and consistent rotation matrices around points the rim of the mug, which shows the effectiveness of RiEMann under local SE(3)- transformations of target objects and new instances, and resist the distraction of other objects.
5.4 Real World Evaluation
Finally, we evaluate RiEMann in two real-world experiments: Mug on Rack and Plane on Shelf. as illustrated in Figure 4. We use four RealSense D435i cameras for point cloud fusion and Franka Panda arm for execution. We provide 10 demonstrations for each task. We show quantitative results of success rates of pick (P) and full task (A) respectively of each task in Table 5. We can see that the real-world performance of RiEMann is generally the same within the simulation, which shows RiEMann can resist the relatively low-quality point cloud input from the real world. The performance of NP notably suboptimal, primarily due the partial absence of the object's point cloud real-world scenarios. This deficiency stems from the cameras' inability capture the lower section of the object, resulting in missing portion that faces downwards. Consequently, in both T and NP, these specific parts exhibit noticeable disparities, illustrated in Figure in Appendix 1.7.1. Please also check supplementary videos for more results and the real-time following video.
Discussion and Future Works
This paper proposes RiEMann, an efficient and near real-time (eq()---quivaiiant robot manipulation framework without point cloud segmentation. One limitation that equivariant models usually bring more computation and memory costs, which hinders their application in large-scale robot learning scenarios. Another limitation is that RiEMann does not perform well for occluded point cloud input or symmetric objects. Future works can focus on these two problems, and seek to apply RiEMann on other tasks that require SE(3)-equivariant outputs such as forces.

---

Acknowledgments
We would like to thank Tianren Zhang, Yuanchen Ju, and Chenrui Tie for their helpful discussions.
References
[1] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-48,, 2009.
[2] H. Ravichandar, A. Polydoros, S. Chernova, and A. Billard. Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous systems, 3: 297-330, 2020.
[3] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L., Fei-Fei, S. Savarese, Y. Zhu, and R. Martin- What matters learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108. 03298, 2021.
[4] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and Stone. Libero: Benchmarking knowl- edge transfer for lifelong robot learning. arXiv preprint arXiv:2306.033: 03330000031,, 2023.
[5] C. Chi, S. Feng, Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04133, 2023.
[6] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. Advances in neural information processing systems, 33: 19884-99889,, 2020.
[7] I. Kostrikov, D. Yarats, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13649, 2020.
[8] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pages 1134-1141. IEEE, 2018.
[9] M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.
[10] P. R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. arXiv preprint arXiv: 1806 08756, 2018.
[11] E. J. Bekkers. An introduction to equivariant convolutional neural networks for continuous groups. httpa//wwg.ggiiibbbboooooooooooe
2021.
[12] A. Simeonov, Y. Du, A. Tagliasacchi, B. Tenenbaum, Rodriguez, P. Agrawal, and V. Sitz- mann. Neural descriptor fields: Se (3)-equivariant object representations for manipulation. In 2022 International Conference Robotics and Automation (ICRA), pages 2022.
IEEE,
[13] A. Simeonov, Y. Du, Y.-C. Lin, A. R. Garcia, L. P. Kaelbling, T. Lozano-Pérez and P. Agrawal. Se (3)-cquivariant relational rearrangement with neural descriptor fields. In Conference Robot Learning, pages 835-846 PMLR, 2023.
[14] H. Ryu, H.-i Lee, J.-H. Lee, and J. Choi. Equivariant descriptor fields: Se (3)-equivariant energy- based models for end-to-end visual robotic manipulation learning. arXiv preprint arXiv:2206.0833 2022.
[15] H. Ryu, J. Kim, J. Chang, H. S. Ahn, Seo, T. Kim, J. Choi, and Horowitz. Diffusion-edfss: Bi-equivariant denoising generative modeling (3) for visual robotic manipulation. arXiv preprint arXiv:2309. 230..0269.... 2023.

---

[16] C. Deng, O. Litany, Y. Duan, A. Poulenard, A. Tagliasacchi, and L. J. Guibas. Vector neu- rons: A general framework for so (3)eqqivvriiant networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12200-12209, 2021.
[17] F. Fuchs, D. Worrall, V. Fischer, and M. Welling. Se (3)-transformers: 3d roto-translation equivariant attention networks. Advances in neural information processing systems, 33:1970- 1981, 2020.
[18] Y.-L. Liao and T. Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. arXiv preprint arXiv:2206. /9900, 2022.
[19] H. Huang, O. Howell, X. Zhu, D. Wang, R. Walters, and R. Platt. Fourier transporter: Bi- equivariant robotic manipulation in 3d. arXiv preprint arXiv:2401. 2024.
[20] C. Pan, B. Okorn, H. Zhang, B. Eisner, and D. Held. Tax-pose: Task-specific cross-pose esti- mation for robot manipulation. In Conference on Robot Learning, pages 1783-1792, PMLR, 2023.
[21] J. Yang. C. Deng. J. Wu, R. Antonova, L. Guibas, and J. Bohg. Equivact: Sim (3)-equivariant visuomotor policies beyond rigid object manipulation. arXiv preprint arXiv:2310.16050 2023.
[22] Cohen and M. Welling. Group equivariant convolutional networks. In International confer- ence machine learning, pages 2990-29909 PMLR, 2016.
[23] J. Han, Y. Rong, T. Xu, and W. Huang. Geometrically equivariant graph neural networks: A survey. arXiv preprint arXiv:2202.072300 2022.
[24] M. Finzi, S. Stanton, P. Izmailov, and A. G. Wilson. Generalizing convolutional neural net- works for equivariance to lie groups on arbitrary continuous data. In International Conference on Machine Learning, pages 3165-3176. PMLR, 2020.
[25] M. J. Hutchinson, C. Le Lan, Zaidi, E. Dupont, Y. W. Teh, and H. Kim. Lietransformer: Equivariant self-attention for lie groups. In International Conference on Machine Learning, pages 4533 4533. PMLR, 2021.
[26] J. Gasteiger, F. Becker, and S. Günnemann. Gemnet: Universal directional graph neural net- works for molecules. Advances in Neural Information Processing Systems, 34:6790-6802, 2021.
[27] C. Esteves. Theoretical aspects of group equivariant neural networks.
arXiv preprint
arXiv:200..05154, 2020.
[28] T. S. Cohen and M. Welling. Steerable cnns. arXiv preprint arXiv: 2016.
[29] N. Thomas, T. Smidt, S. Kearnes, L. Yang, L. Li, K. Kohlhoff, and Riley. Tensor field networks: Rotation-and translation-equivarinn neural networks for 3d point clouds. arXiv preprint arXiv:1802.082111 2018.
[30] T. S. Cohen, M. Geiger, and M. Weiler. A general theory of equivariant cnns on homogeneous spaces. Advances in neural information processing systems, 32, 2019.
[31] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, pages 726-747, PMLR, 2021.
[32] D. Seita, Florence, J. Tompson, E. Coumans, V. Sindhwani, K. Goldberg, and A. Zeng- Learning rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 4568-4575. IEEE, 2021.
10

---

[33] H. Huang, D. Wang. R. Walters, and R. Platt. Equivariant transporter network. arXiv preprint arXiv2200.009400, 2022.
[34] D. Wang, R. Walters, and R. Platt. So (2)-equivariant reinforcement learning. In International Conference Learning Representations, 2022.
[35] D. Wang, M. Jia, Zhu, R. Walters, and Platt. On-robot learning with equivariant models. arXiv preprint arXiv: 20233444.,, 2022.
[36] X. Zhu, D. Wang, O. Biza, G. Su, R. Walters, and R. Platt. Sample efficient grasp learning using equivariant models. arXiv preprint arXiv:2202.09468, 2022.
[37] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799, PMLR, 2023.
[38] H. Huang, D. Wang, X. Zhu, R. Walters, and R. Platt. Edge grasp network: A graph- based se (3)-invariant approach grasp detection. arXiv preprint arXiv:2211.0019/ 2022.
[39] D. Kim and G. S. Sukhatme. Semantic labeling of 3d point clouds with object affordance for robot manipulation. 2014 IEEE International Conference on Robotics and Automation (ICRA), pages
IEEE, 2014.
[40] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category- level robotic manipulation. In The International Symposium of Robotics Research, pages 132- 157. Springer, 2019.
[41] W. Hoffmann. Iterative algorithmen für die gran-schnidd-orhhggnaiissruu Computing, 41: 335-348, 1989.
[42] Xiang, Y. Qin, K. Mo, Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang. L. Yi, A. X. Chang, L. J. Guibas, and H. Su. SAPIEN: A simulated part-based interactive environ- ment. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[43] Z. Xue, Yuan, J. Wang, X. Wang, Y. Gao, and H. Xu. Useek: Unsupervised se (3)- equivariant 3d keypoints for generalizable manipulation. In 2023 IEEE International Con- ference Robotics and Automation (ICRA), pages 1715-1722. IEEE, 2023.
[44] D. Bump et al. groups, volume 225. Springer, 2004.
[45] J. Gu, F. Xiang, X. Li, 7. Ling, X. Liu, T. Mu, Y. Tang, S. Tao, X. Wei, Y. Yao, al. Maniskil12: A unified benchmark for generalizable manipulation skills, arXiv preprint arXiv:2302.0465,, 2023.
[46] haosulab. Mplib: lightweight python package for motion planning, 2023, URL https: //github.
/bb..... GitHub repository.

---

A Appendix
A.1 RiEMann Trainig Algorighm
Following the notations in Section 4.2, the training algorithm of RiEMann is summarized as follows:
Algorithm RiEMann Training
Input: Demonstrations {(P,, T.) initialized models @, 4/1, th2, hyperparameters rı and r2.
epochs n.
for iter 0 to n do
2: Sample batch of m demonstrations ((P), T:)) where T, = -R,, t;)
3: Predict the saliency map f.()) ((x), E P;
4: Get by doing weighted sum on P with the softmax weight from fg (x)
5: Get B Ros centered on Te with radius T1
6: Predict (x) = ti (x), fR(x) 1/2 (z), Va E BROT
7: Get é as the weighted position of (x) and get R by mean pooing fR(()) on points centered
at È with the radius r2
8: Normalize each type-1 vector of R
9: Update Φ. 1/1 and ths with Σ E E (t; 1 Ei=1(tt ((t, t,) +R, R3)
end for Output: Trained models 6, 1/1, and 1/2
A.2 Iterative Modified Gram-Schmidt Orthogonalization
We use Iterative Modified Gram-Schmidt Orthogonalization [41] make the outputted rotation matrix R legal. IMGS works much more stable than the vanilla Gram-Schmidt Orthogonalization. The algorithm is summarized follows.
Algorithm 2 Iterative Modified Gram-Schimidt
Input: R that contains column vectors up, U1, and U2 E R
1: for iter 1 to 2 do
2: for 0 to 2 do
3:
Ut = 1/
4:
for j 0 to do
(v,,0s)
5:
1/ U/ (1). Ui
6: end for
7:
Wi 8/ end for
8:
9:
end for
Output: A legal rotation matrix R that contains updated column vectors up. Vi, and E R3
A.3 Proofs of Theories
First, let's review the definition Suiiv-qque our point cloud P. Given outputted vector field fout (æ) Di_ f'(x), Ve P from transforrer [17] where is the total types of vectors, they are -qqiivaaaa that means:
D (.(R)((()) = (Tx), Vz E P.T PPT=(R,t) e 1E(3),EEM
(5)
where (R) is the Wigner-D matrix. For entities living in the usual 3D physical world, the angular momentum quantum number of the Wigner-D matrix is 1, thus for vectors with | 1. we have:
D(R)
e- -im' d (8)e imy
(6)
12

---

where a, 7 are the Euler angle representation of R that satisfies R (3 m, m E {-1,0, O, 1}, and d m (3) the matrix element. Since D() is also a unitary matrix, we can find a set of basis [vo, U1 02) that makes D(R) R.
For type-0 vector fields, (R) is one-dimensional identical scale factor 1, Let's begin proofs. Theorem Rotation matrices, represented by three type-1 vectors, are -exuivariva parame-
terization of rotation actions.
Proof: For rotation matrix [mo, U1, U2l Rº, where eo. v1, and U2 are the three column vectors, use three type-1 vectors represent these three column vectors. Thus the output vector of the network is follows:
four (x)
A (c).
When the input point cloud is transformed by SE(3) transformation T (R, t), the rotation
matrix representation of the target object also transforms by R, that is [v6, v/] = R/vo, 11 U2]. According to Equation and 6, the outputted type-1 vector fields are transformed by D, (R) R. thus we have:
10 01. [22 = Di (R) [oo, U1 U2]
(8)
Theorem 2 There is no SE(3)-equivarinn vector field representation for Euler angle, quaternion, and axis-angle.
Proof: We use proof by contradiction to prove theorem Consider the example illustrated in Figure 2.
1) For quaternion, we define a quaternion q = [cos(0/2), sin(0/))), sin 2) sin(0/ (0/2) 2 up]
where the rotation angle and [u, ug Uq] is the rotation axis. For the initial pose, we have q [1, O, 0). For the end pose, have q'
v2 v22 0, thus:
1
2 2 1, v2
-
2 0.00.
There are two options for quaternion type-/ parameterization: 1) using four type-O vectors; 2) using one type-1 vector and one type-0 vector. For both cases, the +1 1, x 0, 0] operation cannot be represented by a Wigner-D matrix. Thus there is no SF(3)-equivariant vector field representation for quaternion. Axis-angle can be proven in the same way.
2) For Euler angles, we define an Euler angle E (a, 8) For the initial pose, we have Euler angles equal to E (0, 0, 0). For the end pose, we have Euler angles equal to E' (- 0,
Thus we have:
E'= 0.0 0,
(10)
There are two options for Euler angles' type-l parameterization: 1) using three type-0 vectors; 2) using one type-1 vector. For both cases, the +[- /. 0, 0] operation cannot represented by Wigner-D matrix. Thus there is no SE3))-equivarian vector field representation for Euler angles.
A.4
Training Details
A.4.1 Point Cloud Preprocessing
In the real-world experiments, we perform point cloud voxel downsampling before feeding the point cloud into the network with the voxel size equal to lem for the Mug Rack task and 2em for the task Plane on Shelf.
13

---

After this, we perform color jittering by adding Gaussian noise on each point's 's color with a standard variance equal to 0.005. We also perform random color dropping that replaces 30% points' color to zero, and HSV transformation that randomly transfers the hue, saturation, and brightness of each point by 0.4, 1.5, and times respectively.
A.4.2 Implementation Details
For RiEMann, We use r1 0.2m, r2 0.02m for all the tasks in the simulation. We use T1 0. 0.16m,, r2 0.02m for the real world Mug on Rack task, and rı 0.2m, r2 0.02m for the Plane on Shelf task. Other network hyperparameters of RiEMann are listed in Table 3. We do not use any kind of prior knowledge for the training of RiEMann such as object segmentation, pertaining, or pose augmentations. We also set the robot to first reach some pre-defined pre-grasp pose (e.g., above the mug) and a pre-place pose (e.g., in front of the rack) to eliminate the unnecessary influence of the motion planners.
The full training of RiEMann takes 200 epochs single NVIDIA A40 with batch size of 4 and learning rate of le-4 for each network module o, th, and l/2 However, we find that for Mug Rack, they only need about 50 epochs to converge, which takes about 47 minutes.
Table 3: Network hyperparameters of RiEMann.
Network Layer
Max Type-l
Head Number
Channels
Message Passing Distance
I
8
0.1m
11
1
0.07m
2/2
1
8
0.07m
For PerAct [37), the language descriptions of our tasks are: Put the mug on the rack, Place the plane on the shelf. Turn the faucet. We follow the original 3D voxel grid size (1003) and the patch size (5"). We use Euler angles as the rotational action representations for PerAct. For fairness, we do train the gripper action for PerAct. We use self self-attention layers for the perceiver Transformer module. The other hyper-parameter:
the same with the original paper.
For R-NDF [13], since there is no pre-trained weight for the plane and the faucet, we here pre- train the NDFs using the reconstructed meshes from the point clouds in our demonstrations, and use these model as the NDFs module to run R-NDFs. We observe that R-NDFs fail to accomplish all of the tasks when testing, which shows that R-NDFs cannot perform well without object segmentation, because of the locality requirements of R-NDFs. We also tried use the original pre-trained weights from the original paper [13] for the task Mug on Rack, but we found that the performances were worse because of the discrepancy of the specific object shapes in the test experiments and the pertaining datasets. Other network hyper-parameters are the same as in the original paper.
For EDF [14]and D-EDF [15], we manually separate the robot end-effector and the grasped object point cloud from the scene rather than setting a series of separate cameras to capture their point cloud. For EDFs, we run the MH for 1000 steps, run the Langevin algorithm for 300 steps, and optimize the samples for 100 steps. We use query point for picking and three query points for placing. We train EDFs for 200 epochs, the same epochs with RiEMann. For D-EDFs, we train the networks for hour with parallel training of the low-resolution and the high-resolution networks and the energy-based critic network. Other network hyper-parametent are the same as in the original paper.
A.5 Simulation Experiments
A.5.1
Detailed Descriptions of Simulation Tasks
For all simulation tasks, the radius the table is 0.75m, and we put Franka Panda robot arm in the center of the table. We divide the table into semicircle part and two quarter parts and make them different heights with height difference of 0.1m. This is designed to conveniently apply SE(3)
14

---

(a) First step: the robot to the pre- (b) Second step: the robot opens the faucet dicted target pose of end-effector. along the predicted direction (red arrow).
Figure 6: The articulated object manipulation task Turn Faucet.
transformations on target objects. We cut the input point cloud into a cube with a side length of 2m centered on the center of the table. Details of different tasks are introduced here:
Mug on Rack: A mug and rack placed on the table. The robot has to pick up the mug by the rim and then hang it the rack by the handle. This is the most representative object rearranging task that is also evaluated in [12, 14, 13, 15]. For the training set T, we use mug in blue with side length of about 17 cm, and rack with height of 67cm. The mug must be hung on the highest peg of the rack. For the new instance set NI, we use a patterned red mug with a height of about 19cm and base diameter of about 10cm.
Plane Shelf: A plane model and box-shape shelf are placed on the table, and the robot has to pick the middle part of the body of the plane and place it the shelf. For T, we use grey plane model with length of about 20cm. For NI, we use a blue plane with the same size.
Turn Faucet: As shown in Figure 6, a faucet is placed on the table, and the robot has to turn on the faucet by first moving to the handle of the faucet and then moving along the opening direction. For T, the NO. 5004 faucet model in ManiSkil12 [45]. For NI, the No. 5005 faucet model.
For the open-faucet task, we assume that given a target pose T = (R,t) E SE(3) and target direction d E R3, the robot can accomplish the task by first going to the target pose T just as done pidk-end tasks, and then moving along the target direction d while keeping the orientation not changed, illustrated in Figure 6. To encode the extra directional action d, we add another type-1 vector field the orientation network v/2 that is: fn O (x) B The final output directional action d is also calculated through mean pooling on points in the radius r2. Note the policy needs continuously predict the output direction during the opening process, which shows that RiEMann can capture the local SE(())-qqivarriane of the handle part of the faucet.
In this task, we give demonstrations of not only the target pose T. but also the opening direction d at the first frame. Here there exists two kinds of -quivaeauv the target pose should be equivari- ant to the pose of the faucet at the first frame, and the opening direction should be equivariant to the handle during the opening process, where the second equivariance requires both local-equivariance and the real-time performance.
A.5.2 Demonstration Collection
We provide the ground truth pose point cloud based motion planner MPlib [46] generate the demonstration trajectory for training. We transform all point cloud input to the end-effector coor- dinate system. We collect 10 demonstrations for each setting for the evaluation of SE(3) geodesic
15

---

Table inference time and the training GPU memory usage (on NVIDIA A40) of RiE- Mann and D-EDFs on the task Mug on Rack.
Table Success rates of Mug on Rack and Plane on Shelf in the real world. Each value is the average tests.
Inference Time
Memory Usage
N7
NP
DO
ALL
RiLMann (Ours)
0.19%
11GB
Task
G
G
A
D-EDFs [15]
15.2s
42GB
Mug Rack
1.0
0.83
distance. We manually exclude those situations that cannot support successful collision- mo- tion planning trajectory, well as in the testing cases.
A.6 Real World Experiments
A.6.1 Environment Setup
We use Franka Emika Panda robot arm with four RealSense D435i RGB-D cameras for the real- world experiments, as shown in Figure 4. The cameras are calibrated relative the robot's base frame. We fuse the point clouds from all four cameras and transform the point cloud into the end- effector frame of the robot for control. We crop the scene cube with a side length of 1.5 meters and downsample the point to get 8192 points in the scene.
For real-world tasks, since the point cloud is noisy and usually part-occluded, we do not perform the pose transformation calculation T place T-1 object for the task, i.e., we directly use the predicted
target pose the final action.
A.6.2 Detailed Task Descriptions
Mug on Rack: The task similar to the version in the simulation. For T, we use a pink mug with side length about 10cm. We use rack with a height of 35cm and a base diameter about 15cm. We split the table into four equal areas, in the simulated version of this task. In we only collect demonstrations in quarter of the desktop area and let the mug rotate along the 7-axis for 90 degrees in top pose. For NI, we use yellow new mug with similar size to the pink mug. For NP, we let the mug on all table regions and rotate in 3 dimensional with any degree.
Plan on Shelf: The task similar to the version in the simulation. For T. we use a blue plane model, and collect demonstrations in the same manner as above. For NI, we use a green plane with a similar size. The shelf is set of discrete racks that can support the plane it is placed in the correct pose.
A.6.3 Demonstration Collection
We use the teaching mode of the robot arm to give demonstrations, as illustrated Figure and in supplementary videos. We transform all the point clouds into the end-effector coordinate system.
A.7 More Illustrations
A.7.1 Failure Case Illustrations
We show some failure case point clouds of RiEMann in both tasks in Figure 7. We can see that the lower section of the object is not captured by the cameras, which leads to incomplete geometries.
A.7.2 Results Illustrations
We here illustrate the results and the features of the plane-on-shelf task in Figure 8.
16

---

(b) Low- point clouds in the real-world. Left big part is missed. Right up: the plane head is missed. Left down: big part is missed. Right down: noisy points the mug. These low-quality point clouds can lead to failure RiEMann.
(a) High-quality point clouds in the real world. There still some flaws in the point clouds, which shows robustness of RiEMann.
Figure 7: Visualization of the low-quality data in the real-world experiments that cause the failure of experiments.
NP
DO
ALL
(a) Pose predictions of the task Plane on Shelf of four NI. NP, DO. and ALL in the real world.
(b) The B and the local (E(3))-quivarian) fea- visualization of the ALL test cases.
Figure 8: Test pose predictions and feature visualization of real-world evaluations of the plane on shelf task.
A.8
More Results
A.9
Ablation Studies
A.9.1 Ablation of Hyperparameters
We test the trained model on the NP case the task Mug on Rack in simulation. Results are shown in Figure 9. We can see that with more points in the scene input point cloud, the performance of the model better, and the same is the number of demonstrations. Our model can achieve competitive results with less than 10 demonstrations. For the hidden layer type-/ experiment, higher max type-l, the model can work better. However, in practice, higher type-l will extremely
that with

---

SE(3) Geodesic Distances the NP setting of the Mug on Rack task
0.071
0.07
0.06
0.056
0.05
0.047
0.044
Values
0.04
0.03
0.025
0.02
0.018
0.016
0.013
0.011
0. 0.011
0.011
0.011
0.01
0.00
512 2048 4096 8192 Number of Points
2
4
10 Number Demonstrations
Hidden Layer Type-l
Figure 9: Ablation studies of different hyperparameters of RiEMann. Each value is the average result of 20 random seeds.
increase the computational cost and GPU memory usage. In the Mug On Rack task, network with maximum number equals 5 only supports batch size during training Lastly, the last group of Figure 9 shows that the saliency map network can not only reduce the training burden of the policy network but also improve the final pose estimation results.
NVIDIA A40.
A.9.2 Ablation of Radius 71 and
Here we add additional experiments to answer this question. We train the mug-on-rack task in the simulation with different r1 and r2. Results are in Table
Note in our paper, choose the mug is about 0.22m and the width of the mug is about 0.20m. Note 0.01m means that there is only one point for 1/2 to perform mean pooling, because we perform point cloud voxelization for the point cloud with a voxel size of 0.01m.
0.16m and r2 0.02m. In the mug-on-rack task, the height of
From the results, we can see that:
should be large enough (2 0.16m) to capture all points of the target objects to make sure the position and orientation network can understand the full object geometry. If r1 is less than the radius of the mug, the success rate will drop quickly, and the SE(3) geodesic error will increase quickly. If r1 is too small (r1 = 0.05m), the success rate becomes zero. The reason is that the training becomes very unstable because every time the position and the orientation network may choose different parts of the point cloud for training since the saliency map network is not that strong give very precise position predictions.
T2 should capture more than point (> 0.01m) to avoid noisy and accidental errors caused by too few points.
Although 12 0.04m also works well, it brings more burden for training since the orien- tation network uh outputs the highest type-/ vectors in our whole pipeline, and using fewer points for training is better for reducing the memory usage for 4/2.
For 0.01m, the results are significantly worse than other situations in DO and ALL. This shows that noisy points will influence the results we only use one point for predic- tion.
The worst Seode-ge error of RiEMann (rı 0.05m) is smaller than the R-NDF baseline [13] Table 2. This because the saliency map network will restrict the position error be certain range.
18

---

Table 6: Success rates and SE(3)-geodesic distances of different ri and r2 of the mug-on-racee task in simulation. Each value is evaluated under 20 random seeds. In our original paper, we choose
0.16m and r2 0.02m.
ri/m
0.05
0.10
0.28
ra/m
0.01
0.02
0.04
0.02
0.04
0.02
0.04
0.01
0.01
0.02
T
0.00
0.00
0.00
0.50
0.85
0.50
0.90
1.00
1.00
0.90
1.00
1.00
0.95
1.00
1.00
NI
0.00
0.00
0.00
0.10
0.50
0.75
0.80
0.90
0.95
0.90
0.95
0.90
0.90
0.90
0.95
SR
NP 0.00
0.00
0.00
0.45
0.73
0.80
0.85
0.95
1.00
0.90
0.95
0.95
0.95
0.95
1.00
DO
0.00
0.00
0.80
0.70
1.00
1.00
0.85
0.80
1.00
ALL
0.00
0.00
0.00
0.00
0.50
0.55
0.50
0.85
0.85
0.70
0.85
0.90
0.70
0.85
0.80
T 1.366
1.478
1.175
0.688
0.178
0.186
0.278
0.053
0.056
0.067
0.064
0.062
0.059
0.000
0.055
NI 1.159
1.759
1,311
0,982
0.220
0.201
0.321
0.066
0.061
0.068
0.061
0.062
0.066
0.063
0.062
Dyco
NP
1.668
1.076
1.079
0.804
0.123
0.202
0.339
0.069
0.063
0.066
0.055
0.064
0.058
DO
1.460
1.298
1, 1.290
0.866
0.175
0.199
0.297
0.058
0.056
0.082
0.062
0.059
0.079
0.064
0.063
ALL
1.982
1.333
1.077
1.390
7.797
0.639
0.427
0.071
0.070
0.079
0.077
0.008
0.088
0.079
0.072
A.9.3 The Geometry Generalization Ability RiEMann
The generalization ability of RiEMann on the new geometry comes from the neural network struc- ture and its inductive bias. We think it is the locality property of the message- paasage-passing mechanism in the equivariant backbone that brings this geometry-level generalization. We perform the new stance experiments because, in related works [14, 15], they found that equivariant networks have certain degree of geometric generalization ability only with 10 demonstrations, and we firmed this point in experiments. Yes, compared to NDFs [12, 13], geometry generalization ability is worse because their encoder is trained on a large-scale in-category dataset.
To better demonstrate how strong generalization ability of RiEMann on new object geometries, here we add an additional experiment. We gradually deform the mesh of the mug trained in the mug- on-rack task, put them into the NP setting, and record the corresponding SE(3)-geodesic error of the predictions from the original trained model. We keep the longest side of the mug the same. The results are in Table 7.
Table 7: SE(3)-goodesic distance evaluations of the trained model on the training mug. Each value evaluated under 20 random SE(3) poses.
Picture Description
original
flat
tall
fat
new
Dgeo
0.055
0.187
0.127
0.094
0.395
We that if the general shape of the mug remains (especially for the rim of the mug). the D Dyeo remains at low level, which shows that RiEMann has certain power of geometry generalization ability. However, if the geometry of the mug is deformed too much, the result becomes worse.
19


Reviews:
[{"review_id": "8rDYusdKqm", "reviewer": "Reviewer_HXLL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposed a robot object manipulation learning method given scene point clouds. The key idea is to directly regress an action field on the point cloud, as opposed to pose estimation from feature matching optimization, so that the inference efficiency can be largely improved. This paper claims to be the first near real-time point-cloud-based object manipulation framework.  SE(3)-equivariant networks are used to enable generalization to pose changes of the target object. Experimental results show good performance and generalizability to pose and instance variations.", "summary_of_recommendation": "This work provided innovative insights to the manipulation problem. Method and experiments convincing overall. Some clarification needed.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Please see above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "799UVlZNr0", "reviewer": "Reviewer_9VAx", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes RiEMann a SE(3) equivaraint open-loop control method for tabletop robotic manipulation. The method boasts an improved evaluation speed by reducing the overall size of the pointcloud used for policy learning. The authors evaluate the method in simulation and in the real world on a small set of tabletop manipulation tasks.", "summary_of_recommendation": "While the proposed method does show improved performance when compared to the baselines its overall novelty is a underwhelming. The authors state that their method is the first end-to-end SO(3)-equivaraint model but I do not belive this true and the theoritical contributions have been known for a long time in group theory. Additional model details are needed to properly assses the novely of the model architecture itself.", "technical_quality": 2, "clarity_of_presentation": 2, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "1. What is T(3)-invariance (Line 153)?\n2. Additional details on the model would be appreciated. For example, I assume the saliency network is smaller than the policy network in order to speed up evaluation but these details appear to be missing.\n3. How does the speed of this method compare to other more modern SO(3) equivariant model architectures such as Equiformer-v2?\n4. Why not compare to Fourier Transporter?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZYEkT9Kkme", "reviewer": "Reviewer_UjoM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "RiEMann is a locally SE(3) equivariant open loop imitation learning framework that predicts gripper pose, given point cloud observation.  RiEMann pipeline includes two stages. First, a saliency network take as input the full point cloud, then selects the region of interest. Second, the position and the orientation are regressed by position and orientation nets, respectively. Since all the networks as well as the action presentations are equivariant, RiEMann achieves end-to-end equivariance.  The local equivariance brings the benefits of generalizing to out-of-distribution scenes. Moreover, the action regression alleviates expensive SE(3) action inference, compares to previous works that used field-matching or steerable-convolution mechanisms.", "summary_of_recommendation": "The proposed method achieves SE(3) local equivariance, which is sound and efficient for a subset of tasks. Nevertheless, the method is limited to uni-model policy learning.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "In line 176-178 states that the rotation matrix is the only SE(3)-equivariant rotation representation. This statement is too strong (maybe wrong). I.e., 5D and 6D representation is also SE(3)-equivariant [1].\n\n[1] Zhou Y, Barnes C, Lu J, Yang J and Li H (2019) On the continuity of rotation representations in neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5745–5753.\n\nIn Appendix Theorem2, there seems missing the proof for that axis-angle is not SE(3) equivariant.\n\nHow can one extend the method to multi-model policies? I.e., how could the method be improved to learn grasping a ball?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: PRE REBUTTAL:

High level summary of reviews:

Strengths:

- SE(3)-equivariant networks enable generalization to pose changes, distractors and novel instances of the class. The method was called out as being relatively novel.
- The method is well-written and easy to follow (with some caveats about clarity of exposition, see below).
- RiEMann highlighted as maintaining low computation cost (via reduction in point cloud size).
- The experimental results validate the merit of the proposed framework and demonstrate the effectiveness of the method on two challenging tasks in the real world.

Weaknesses:

- Some technical details and heuristic designs in the framework need further clarification and justification. See specific reviewer feedback for details.
- The method relies on manually tested hyperparameters and does not predict pre-pick/pre-place actions, limiting its scope.
- Local equivariance may not be suitable for more complex tasks that require global information, such as sweeping dust with a broom (see reviewer UjoM’s comment).
- Reviewer 9VAx questions whether certain theoretical contributions, like the theorems on SE(3)-equivariance, are not novel and are actually well-established in group theory discussions. Additionally the authors use language to describe the group theory behind the equivariance that is a bit non-standard and this should be fixed.
- The clarity of presentation could be improved, particularly in describing the model details and experimental setup.

POST REBUTTAL:

The authors provided a thorough rebuttal and addressed many of the concerns raised by reviewers. This paper crosses the threshold for acceptance. Please note that Fourier Transporter is unlikely concurrent work (there was at least 3 months between) and the AC agrees this should have been a baseline. However, even with this missing the experimental results are strong enough to justify acceptance. The authors should make sure all the reviewer suggested changes (and their author's own clarifications) are reflected in the final manuscript for the camera ready.
Decision: Accept