RiEMann: Near Real-Time SE(3)-Equivariant Robot Manipulation without Point Cloud Segmentation

Authors: chongkai  gao
zhengrong  xue
shuying  deng
tianhai  liang
siqi  yang
lin  shao
huazhe  xu
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-30 18:01:52

Abstract: We present RiEMann, an end-to-end near Real-time SE(3)-Equivariant Robot Manipulation imitation learning framework from scene point cloud input. Compared to previous methods that rely on descriptor field matching, RiEMann directly predicts the target actions for manipulation without any object segmentation. RiEMann can efficiently train the visuomotor policy from scratch with 5 to 10 demonstrations for a manipulation task, generalizes to unseen SE(3) transformations and instances of target objects, resists visual interference of distracting objects, and follows the near real-time pose change of the target object. The scalable SE(3)-equivariant action space of RiEMann supports both pick-and-place tasks and articulated object manipulation tasks. In simulation and real-world 6-DOF robot manipulation experiments, we test RiEMann on 5 categories of manipulation tasks with a total of 25 variants and show that RiEMann outperforms baselines in both task success rates and SE(3) geodesic distance errors (reduced by 68.6%), and achieves 5.4 frames per second (fps) network inference speed.

Main paper: 
 introduction: 
 Learning from demonstrations is an effective and convenient mechanism for visual robot manipulation tasks [1, 2] . However, most current algorithms for learning from demonstrations suffer from low data efficiency and generalization ability to new situations. For example, current visual imitation learning algorithms require around 100 demonstrations [3, 4, 5] to learn a simple manipulation task such as picking up a mug and placing it on a rack, and cannot generalize to new object poses beyond the training distribution. Although previous works have tried to tackle these problems by data augmentation [6, 7] or contrastive learning on demonstrations [8, 9, 10] , they rely on task-specific domain knowledge and have no algorithmic guarantees to generalize to unseen object poses.Exploiting symmetries in the 3D world can improve the sample efficiency and generalization ability of robot learning algorithms [11] . Roto-translation equivariance is one of the most common types of symmetry for robot manipulation tasks. It denotes that robot actions can transform with the same SE(3) transformations as the target objects. Pioneering works like NDFs [12, 13] and EDFs [14, 15] propose to first learn an SE(3)-equivariant object descriptor field with SE(3)-equivariant backbones [16, 17, 18] and then output robot actions with energy-based optimization to match different descriptor fields. They achieve SE(3)-equivariance on pick-and-place tasks such as picking up the mug to the rack [14, 12, 13, 15] . However, these methods share the same limitations of the field-matching mechanism: 1) the matching process is time-consuming, preventing it from being a real-time method; 2) the matching process can only yield one SE(3) transformation, which only works on pick-and-place tasks and precludes its application to articulated object manipulation tasks.Recently, some works have been trying to solve these issues. Fourier Transporter [19] and TAX-Pose [20] try to tackle the first issue by cross correlation of Fourier transformation on SO(3) [19] or differential SVD [20] to replace the time-consuming optimization process, but they either require exorbitant computation to discretize the SO(3) group at a high resolution, or require a pre-trained encoder on an extra well-segmented dataset. EquivAct [21] tries to tackle the second problem by directly predicting robot actions with neural networks, but they require well-segmented point clouds and can only predict 3-DOF actions rather than 6-DOF actions. These works inspire us to design an end-to-end SE(3)-equivariant policy that enables 6-DOF manipulation tasks without segmentation to replace the field matching process, as well as an efficient network structure and training algorithm to alleviate complexity brought by equivariant backbones to improve the inference speed.In this work, we present RiEMann, the first near Real-time SE(3)-Equivariant robot Manipulation framework from point cloud inputs without any segmentation. RiEMann leverages local SE(3)equivariant backbones [17] and learns a policy that directly outputs SE(3)-equivariant actions. For the 6-DOF action space parameterization problem, RiEMann uses an SE(3)-invariant vector field as the target point affordance map for translational actions, and uses three SE(3)-equivariant vector fields as three bases of the target rotation matrix as well as a Gram-Schmidt orthogonalization operation for the rotational actions. This action space can also be extended to articulated object manipulation tasks by adding an SE(3)-equivariant directional action. Theoretically, we prove that other common parameterizations for the SO(3) group, such as axis-angle, quaternion, and Euler angle, are not trainable SE(3)-equivariant representations. For the computational complexity problem, RiEMann firstly learns an SE(3)-invariant saliency map on the input scene point cloud to extract the region of interest, then trains the main SE(3)-equivariant policy on the extracted point cloud. Since the equivariant policy consumes the majority of computing and memory resources, reducing the number of points to be processed can significantly decrease computational and memory burden.In 5 manipulation tasks with a total of 25 different task settings from simulation and real-world experiments, we demonstrate that RiEMann can solve various manipulation tasks with only 5 to 10 demonstrations for each task, generalize to unseen poses and instances of target objects, resist visual interference of distracting objects, have the near real-time (5.4 FPS) inference speed as illustrated in Figure 1 , and outperforms baselines on both success rates as well as the SE(3) geodesic distance errors (reduced the geodesic error by 68.6%). In summary, our main contributions are as follows:• We propose the first end-to-end SE(3)-equivariant visuomotor policy learning framework for 6-DOF manipulation tasks by our specially designed SE(3)-equivariant action space;• We design a saliency map network together with position and orientation networks as the policy model to achieve efficient SE(3)-equivariant training and near real-time inference;• We achieve state-of-the-art success rates and pose estimation accuracy on a set of pickand-place tasks under various settings, and we are the first to perform SE(3)-equivariant articulated object manipulation tasks through end-to-end training.2 Related WorksWe test the trained model on the NP case of the task Mug on Rack in simulation. Results are shown in Figure 9 . We can see that with more points in the scene input point cloud, the performance of the model is better, and the same is the number of demonstrations. Our model can achieve competitive results with less than 10 demonstrations. For the hidden layer type-l experiment, we can see that with a higher max type-l, the model can work better. However, in practice, higher type-l will extremely increase the computational cost and GPU memory usage. In the Mug On Rack task, a network with a maximum number l equals 5 only supports batch size = 1 during training on an NVIDIA A40. Lastly, the last group of Figure 9 shows that the saliency map network can not only reduce the training burden of the policy network but also improve the final pose estimation results.A.9.2 Ablation of Radius r 1 and r 2Here we add additional experiments to answer this question. We train the mug-on-rack task in the simulation with different r 1 and r 2 . Results are in Table 6 .Note in our paper, we choose r 1 = 0.16m and r 2 = 0.02m. In the mug-on-rack task, the height of the mug is about 0.22m and the width of the mug is about 0.20m. Note r 2 = 0.01m means that there is only one point for ψ 2 to perform mean pooling, because we perform point cloud voxelization for the point cloud with a voxel size of 0.01m.From the results, we can see that:• r 1 should be large enough (≥ 0.16m) to capture all points of the target objects to make sure the position and orientation network can understand the full object geometry. If r 1 is less than the radius of the mug, the success rate will drop quickly, and the SE(3) geodesic error will increase quickly. If r 1 is too small (r 1 = 0.05m), the success rate becomes zero. The reason is that the training becomes very unstable because every time the position and the orientation network may choose different parts of the point cloud for training since the saliency map network is not that strong to give very precise position predictions.• r 2 should capture more than one point (> 0.01m) to avoid noisy and accidental errors caused by too few points.• Although r 2 = 0.04m also works well, it brings more burden for training since the orientation network ψ 2 outputs the highest type-l vectors in our whole pipeline, and using fewer points for training is better for reducing the memory usage for ψ 2 .• For r 2 = 0.01m, the results are significantly worse than other situations in DO and ALL. This shows that noisy points will influence the results if we only use one point for prediction.• The worst SE(3)-geodesic error of RiEMann (r 1 = 0.05m) is smaller than the R-NDF baseline [13] in Table 2 . This is because the saliency map network will restrict the position error to be in a certain range.Table 6 : Success rates and SE(3)-geodesic distances of different r 1 and r 2 of the mug-on-rack task in simulation. Each value is evaluated under 20 random seeds. In our original paper, we choose r 1 = 0.16m and r 2 = 0.02m. T 0.00 0.00 0.00 0.50 0.85 0.80 0.90 1.00 1.00 0.90 1.00 1.00 0.95 1.00 1.00 NI 0.00 0.00 0.00 0.10 0.80 0.75 0.80 0.90 0.95 0.90 0.95 0.90 0.90 0.90 0.95 SR NP 0.00 0.00 0.00 0.45 0.75 0.80 0.85 0.95 1.00 0.90 0.95 0.95 0.95 0.95 1.00 DO 0.00 0.00 0.00 0.35 0.80 0.70 0.70 1.00 1.00 0.85 0.95 1.00 0.80 1.00 1.00 ALL 0.00 0.00 0.00 0.00 0.50 0.55 0.50 0.85 0.85 0.70 0.85 0.90 0.70 0.85 0.80 TThe generalization ability of RiEMann on the new geometry comes from the neural network structure and its inductive bias. We think it is the locality property of the message-passing mechanism in the equivariant backbone that brings this geometry-level generalization. We perform the new instance experiments because, in related works [14, 15] , they found that equivariant networks have a certain degree of geometric generalization ability only with 5 to 10 demonstrations, and we confirmed this point in our experiments. Yes, compared to NDFs [12, 13] , our geometry generalization ability is worse because their encoder is trained on a large-scale in-category dataset.To better demonstrate how strong the generalization ability of RiEMann is on new object geometries, here we add an additional experiment. We gradually deform the mesh of the mug trained in the mugon-rack task, put them into the NP setting, and record the corresponding SE(3)-geodesic error of the predictions from the original trained model. We keep the longest side of the mug the same. The results are in Table 7 . We can see that if the general shape of the mug remains (especially for the rim of the mug), the D geo remains at a low level, which shows that RiEMann has a certain power of geometry generalization ability. However, if the geometry of the mug is deformed too much, the result becomes worse.i=1 f i 1 (x), x ∈ B ROI . Then we select the points from a smaller region that are centered on t with a radius r 2 , and perform mean pooling on these points to get the output orientation action R, as in the right side of Figure3. During training, we use the translation action t in 
group equivariant neural networks: 
 Exploiting groups of symmetry pervading in data and incorporating them into deep neural networks has been the focus of many studies since it can improve generalization and data efficiency. Equivariant neural networks, which are first introduced into CNNs [22] , extract symmetries from various kinds of data. According to Bekkers [11] , Han et al. [23] , most equivariant models can be divided into two categories: regular group representation networks and steerable group representation networks. The former seeks to define equivariant convolution filters [22, 24] , attention mechanism [25] , or message passing mechanism [26] as functions on groups, while the latter uses irreducible group representations [27] with spherical harmonics as an equivariant basis to perform message passing [28, 29, 17, 18] . Some other works [16] design special non-linear kernels to achieve equivariance on the SO(3) group. The theory of equivariant networks on homogeneous spaces is formalized in Cohen et al. [30] with the vector bundle theory.Researchers have been exploring equivariant models for robotic manipulation tasks to improve generalizability and sample efficiency. Pioneering works [31, 32, 33, 34, 35, 36] learn equivariant representations of objects or the scene with equivariant networks to get SO(2)-or SE(2)-equivariance for desktop manipulation tasks with imitation learning or reinforcement learning. NDFs [12, 13, 21] and TAX-Pose [20] leverages Vector Neurons [16] or SO(3)-data augmentation to acquire SE(3)equivariant category-level object representations from point cloud of objects for downstream imitation learning, while EDFs [14, 15] uses SE(3)-Transformer [17] and Equiformers [18] to directly learn SE(3)-equivariant representations from point clouds of the scene. Huang et al. [19] leverages discrete approximation of steerable representation of SO(3) and uses cross correlation to tackle the optimization problem, but still requires huge computation for a high-resolution discretization. In this work, we leverage SE(3)-transformers [17] and design an end-to-end learning paradigm to predict target actions from scene point cloud input in near real-time.3 Background and Problem Formulation 
problem formulation: 
 Let a colored point cloud with N points beP = {(x 1 , c 1 ), • • • , (x N , c N )} ∈ R N ×6, where x i ∈ R 3 is the position and c i ∈ R 3 is the RGB color of the i-th point. For a manipulation task T , a policy f θ parameterized by θ is trained to predict the target pose of robot end-effector T = {R, t} ∈ SE(3) with a set of demonstrations D = {(P i , T i )} m i=1 that consists of m pairs, where R ∈ SO(3) is the 3D rotation action and t ∈ R 3 is the 3D translation action. We use T = f θ (P) to denote the predicted output. We assume a motion planner with collision avoidance is used to execute the action T, as in Shridhar et al. [37] , Simeonov et al. [12] , Ryu et al. [14] . The trained policy will be tested on unseen target object poses, distracting objects, and new object instances, which relies on the following local SE(3)-equivariance property of our policy.Let g ∈ SE(3) be an element of the Special Euclidean Group in three dimensions (SE(3)) and T g : X → X be a transformation of g on a space X . A function f : X → Y is called SE(3)equivariant if there exists a transformation S g : Y → Y such that:EQUATIONIn our robot manipulation tasks, f is the trained policy, X is the space of the scene point cloud P, Y is the space of target pose T, T g is the space of any SE(3) transformation T g = {R, t} on P, and S g is the SE(3) transformation on the target pose T and we set S g = T g following [30, 11, 38] .Local SE(3)-equivariance [14, 15] refers that T g can be applied on only part of the input scene point cloud P rather than the whole scene input, as illustrated in Figure 1 . This is more practical for robot manipulation tasks, in which we want the policy to be only equivariant to the target objects.Local equivariance can be achieved by equivariant networks with local mechanisms. In this work, we choose to use SE(3)-transformers [17] that belong to the latter category, as introduced below. 2l+1) . Vectors transforming according to D l (R) are called type-l vectors [11] . Type-0 vectors are invariant under rotations (D(R) = I) and type-1 vectors (3d space vectors) rotate according to 3D rotation matrices. Note, type-l vectors have length 2l + 1. Given a point cloud P, we can assign a type-l vector to each point x ∈ P to get a type-l vector field f l : R 3 → R 2l+1 . Same or different type-l vector fields can be stacked to get a complex vector field. SE(3)-transformers [17] are neural networks that map point cloud vector fields to point cloud vector fields with the same point number, and they are designed to achieve local SE(3)-equivariance for any learned type-l field:(R) ∈ R (2l+1)×(EQUATIONTo leverage this property for robot manipulation tasks, we need to design special input and output vector fields to satisfy task requirements, as discussed in the following section. 
se(3)-equivariant robot manipulation: 
 Previous works like NDFs [12, 13] and EDFs [14, 15] output several type-0 or higher type-l vector fields as the object descriptor for field-matching to get the desired robot action. In contrast, we aim to re-design the outputted vector fields and directly use them as actions. This can transform the generative modeling problem of the descriptor field into a supervised learning problem that enables higher training precision and a faster inference speed.There are two main requirements for the design of input and output vector fields: a) all the input (scene point cloud) and output (robot actions) can be incorporated into the vector fields; b) the predicted actions parameterized by output vector fields are SE(3)-equivariant to the input transformation. For the input design, since SE(3)-transformer is actually T(3)-invariant, we should only use the T(3)-invariant features, or visual features, as the input, to ensure T(3)-invariance. Thus here we only use color information for input. RGB information can be seen as 3 type-0 vectors, so our input is a direct sum of three vector fields:EQUATIONFor the output, our model predicts the end-effector target pose T = {R, t} ∈ SE(3), which can be decomposed into target position t and target rotation R. For target position t, we cannot directly output the 3D target position vector since SE(3)-transformers are only able to predict T(3)-invariant features. Thus we choose to learn one type-0 vector field as an affordance map [39, 40] on point clouds, and use softmax on the affordance map to perform weighted sum on all point positions to get the translational action t. We also assume an offset t o is provided if the target position is out of the convex hull of the object point cloud, and in this case the output is t + t o . For the target orientation R, people usually use Euler angle, quaternion, rotation matrix, or axis-angle as robot action parameterization. Although all these pose parameterizations can satisfy the first requirement (representing output action information), we need to ensure they are SE(3)-equivariant to the input transformation after parameterizing them with different type-l vectors. We answer this question with the following theorems. All proofs are in Appendix A.3.Theorem 1 Rotation matrices, represented by three type-1 vectors, are SE(3)equivariant vector field parameterization.Theorem 2 There is no SE(3)equivariant vector field parameterization for Euler angle, quaternion, and axis-angle.The intuitions behind these theorems are illustrated in Figure 2 . In short, rotation matrices, represented by three type-1 vectors, are the only SE(3)-equivariant rotation representation to the input transformation.In summary, RiEMann learns one type-0 vector field (target point heatmap) for target position t and three type-1 vector fields for target rotation R, which is a direct sum of 4 vector fields:EQUATIONAs comparisons, EquivAct [21] uses one type-1 vector field as output (end-effector velocity in Cartesian space), which only supports 3-DOF manipulation tasks. Instead, RiEMann supports 6-DOF manipulation tasks. The next question is how to use networks to implement equation 3 and 4 efficiently and transform the output vector fields to a single output action while ensuring SE(3)-equivariance.The first problem of leveraging equivariant networks for scene point cloud input is the heavy computational and memory cost. A scene point cloud for robot manipulation tasks usually contains thousands to tens of thousands of points (in our case, 8192 points), and end-to-end learning on such a large number of points brings huge burdens for GPU memory and a long training time. For example, EDFs [14] only support batch size = 1 for training on an NVIDIA RTX3090 GPU. Meanwhile, the training difficulty also increases since most of the points are not informative. In this work, we employ an SE(3)-transformers ϕ(x) to output one type-0 vector field f s (x) = f 0 (x), x ∈ P for each point as a saliency map, and extract a relatively small region of point cloud B ROI with a predefined radius r 1 centered on the point with the highest value, as illustrated in the left part of Figure 3 .With B ROI as input, we then employ another two SE(3)-transformer networks to learn a translational action network ψ 1 (x) and an orientation network ψ 2 (x) for ∀x ∈ B ROI as policy networks for action prediction. For the translational part, ψ 1 outputs one type-0 affordance f t (x) = f 0 (x), x ∈ B ROI . We then perform SoftMax on f t (x) as weight and multiply them to the positions of all points of B ROI to get the output translational action t. For the orientation part, ψ 2 outputs three type-1 vector fields f R = For articulated object manipulation, we assume that given a target pose T = {R, t} ∈ SE(3) and a target direction d ∈ R 3 , the robot can accomplish the task by first going to the target pose T just as done in pick-and-place tasks, and then moving along the target direction d while keeping the orientation not changed, as illustrated in Figure 6 . We expend the action space with another type-1 vector field on the orientation networkψ 2 : f R = 4 i=1 f i 1 (x), x ∈ B ROI .The final output directional action d is also calculated through mean pooling on points in the radius r 2 . Note the policy needs to continuously predict the output direction during the opening process, which shows that RiEMann can capture the local SE(3)-equivariance of the handle part of the faucet and has near real-time inference speed. Detailed implementation details of our model are listed in Appendix A. 4  
experiments: 
 We systematically evaluate RiEMann in both simulation and real-world experiments. This evaluation includes 3 types of manipulation tasks in simulation environments and 2 types of manipulation tasks in the real world, with 5 different settings and 10 demonstrations for each task. The ablation studies are in Appendix A.9.We build all simulation manipulation environments based on Sapien [42] for point-cloud-based manipulation. We build a ring-table experiment platform with uneven tabletop for all tasks in simulation, as shown in Figure 4 . We put 6 RGBD cameras around the table to fuse their captures to get the point cloud input, and downsample the scene point cloud to 8192 points. For each manipulation task, we collect demonstrations and train our policy under the training setting (T), and test the PerAct [37] 0.85 0.00 0.70 0.00 0.00 0.90 0.00 0.80 0.00 0.00 0.45 0.00 0.50 0.00 0.00 R-NDF [13] 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 n/a n/a n/a n/a n/a EDF [14] 1.00 0.85 1.00 0.95 0.80 0.90 0.75 0.80 0.85 0.70 n/a n/a n/a n/a n/a D-EDF [15] 1.00 0.85 0.95 0.95 0.75 1.00 0.80 0.95 0.95 0.75 n/a n/a n/a n/a n/a RiEMann (Ours)1.00 0.90 0.95 1.00 0.85 1.00 0.90 1.00 1.00 0.90 1.00 0.75 1.00 1.00 0.65 trained policy on T and four extra settings: 1) New Instance (NI): the target object will be replaced by different objects of the same category; 2) New Poses (NP): the initial and target poses of target objects will be under SE(3) transformations on the table; 3) Distracting Objects (DO): there will be additional distracting objects in the scene; 4) ALL: the combination of NI, NP, and DO.We design three tasks for evaluation. More detailed task descriptions can be found in A.5: 1) Mug on Rack: For T, we put a mug on the left-down quarter of the table, and put the rack on the right-down quarter of the table, allowing them to rotate within 90 degrees only around the z-axis randomly. For NI, we use a new mug and keep the rack the same. For NP, we allow the mug and the rack on any quarter of the table, and rotate the mug along all 3 axes with arbitrary degrees; 2) Plane on Shelf : This task is designed for the evaluation of objects with complex geometry, as evaluated in [43] . The setting of T, NI, and NP are the same as above; 3) Turn Faucet: This is an articulated object manipulation task and the robot must predict SE(3)-equivariant opening direction to turn the faucet. For NP, we rotate the faucet along the z-axis, and also change the initial position of the handle.We compare RiEMann with four baselines: 1) PerAct [37] : a point-cloud-based imitation learning method. This is for the comparison between the SE(3)-equivariant method and the non-SE(3)equivaraint method. We perform SE(3) data augmentation for PerAct; 2) R-NDFs [13] : the strongest SE(3)-equivariant method that rely on NDFs [12] and Vector Neurons [16] ; 3) EDFs [14] : an SE(3)equivariant baseline that use SE(3)-transformers for manipulation tasks. This is for the comparison between directly regressing target poses (ours) and field matching on type-l fields. Note EDFs require segmented point clouds of the scene and the robot end-effector respectively. Here we manually extract the end-effector point cloud out from the scene for EDFs; 4) D-EDFs [15] : the SOTA SE(3)equivariant baseline for object rearrangement tasks. This is for the comparison of performances as well as training and inference speeds.We report the success rates of all methods in all five settings (T, NI, NP, DO, ALL) with the average success rates across 20 different random seeds. The success rates are the total success rates of the two stages of each task. Additionally, to better evaluate the quantitative results and eliminate the influences of policy-irrelative factors, we propose to evaluate the SE(3) geodesic distance [44] of the predicted target pose T and the ground truth T:D geo (T, T) = log R ⊤ R ∨ 2 + t -t 2 ,where ∨ is the logmap. We report D geo in the same manner as success rates.Table 1 and Table 2 show the success rates and D geo respectively. We can see that PerAct [37] generally performs worse than SE(3)-equivariant methods, which shows using equivariant networks is better than doing SE(3) data augmentation for a non-equivariant backbone. NDFs [13] with Vector Neurons [16] totally fail on unsegmented point cloud input, which shows the necessity of using local SE(3)-equivariance modules. EDFs [14] and D-EDFs [15] achieved mediocre results in both success rate and D geo , which shows that they can generalize to local SE(3) transformations of the target objects to a certain degree. However, their performances are worse than RiEMann, especially in D geo , which shows that the field-matching mechanism performs worse than end-to-end policy training. Meanwhile, they cannot support any articulated object manipulation tasks because their output space contains only one target pose. Instead, RiEMann consistently achieves competitive results in success rates significantly better results in D geo (reducing the average D geo by 68.6%) under all test settings, which shows the superiority of our end-to-end learning paradigm. Note a D geo = 0.05 means an error around 1cm + 2 • , which is low enough for a successful manipulation. We also visualize the four test case pose predictions in Figure 5a and the local SE(3)-equivariance features in Figure 5b . We can see that the position network ψ 1 can attend to the correct region (the rim of the mug), and the orientation network ψ 2 can predict correct and consistent rotation matrices around points on the rim of the mug, which shows the effectiveness of RiEMann under local SE(3)transformations of target objects and new instances, and resist the distraction of other objects.Finally, we evaluate RiEMann in two real-world experiments: Mug on Rack and Plane on Shelf, as illustrated in Figure 4 . We use four RealSense D435i cameras for point cloud fusion and a Franka Panda arm for execution. We provide 10 demonstrations for each task. We show quantitative results of success rates of pick (P) and full task (A) respectively of each task in Table 5 . We can see that the real-world performance of RiEMann is generally the same within the simulation, which shows RiEMann can resist the relatively low-quality point cloud input from the real world. The performance of NP is notably suboptimal, primarily due to the partial absence of the object's point cloud in real-world scenarios. This deficiency stems from the cameras' inability to capture the lower section of the object, resulting in a missing portion that faces downwards. Consequently, in both T and NP, these specific parts exhibit noticeable disparities, as illustrated in Figure 7 in Appendix A.7.1. Please also check supplementary videos for more results and the real-time following video. 
discussion and future works: 
 This paper proposes RiEMann, an efficient and near real-time SE(3)-equivariant robot manipulation framework without point cloud segmentation. One limitation is that equivariant models usually bring more computation and memory costs, which hinders their application in large-scale robot learning scenarios. Another limitation is that RiEMann does not perform well for occluded point cloud input or symmetric objects. Future works can focus on these two problems, and seek to apply RiEMann on other tasks that require SE(3)-equivariant outputs such as forces.Following the notations in Section 4.2, the training algorithm of RiEMann is summarized as follows:Algorithm 1 RiEMann Training Input: Demonstrations {(P i , T i )} Mi=1 , initialized models ϕ, ψ 1 , ψ 2 , hyperparameters r 1 and r 2 , epochs n.1: for iter = 0 to n -1 do 2:Sample a batch of m demonstrations {(P i , T i )} m i=1 , whereT i = (R i , t i ) 3:Predict the saliency map f s (x) = ϕ(x), x ∈ P i 4:Get x t1 by doing weighted sum on P with the softmax weight from f s (x)5:Get B ROI centered on x t1 with radius r 1 6:Predict f t (x) = ψ 1 (x), f R (x) = ψ 2 (x), ∀x ∈ B ROI 7:Get t as the weighted position of f t (x) and get R by mean pooing on f R (x) on points centered at t with the radius r 2Normalize each type-1 vector of R 9:Update ϕ, ψ 1 , andψ 2 with L = m i=0 [ N j=1 (t i -ti ) 2 + N B k=1 ((t i -ti ) 2 + (R i -R i )2 )] 10: end for Output: Trained models ϕ, ψ 1 , and ψ 2We use Iterative Modified Gram-Schmidt Orthogonalization [41] to make the outputted rotation matrix R legal. IMGS works much more stable than the vanilla Gram-Schmidt Orthogonalization. The algorithm is summarized as follows.Input: R that contains column vectors v 0 , v 1 , and v 2 ∈ R 3 1: for iter = 1 to 2 do 2:for i = 0 to 2 do 3:u i = v i 4:for j = 0 to i -1 do 5:v i = v i - ⟨vi,uj ⟩ ⟨uj ,uj ⟩ u j 6:end for 7:u i = v i 8:end for 9: end for Output: A legal rotation matrix R that contains updated column vectors v 0 , v 1 , and v 2 ∈ R 3First, let's review the definition SE(3)-equivariance on our point cloud P. Given an outputted vector field f out (x) = n i=1 f i (x), ∀x ∈ P from SE(3)-transformer [17] where n is the total types of vectors, they are SE(3)-equivariant that means:EQUATION)where D l (R) is the Wigner-D matrix. For entities living in the usual 3D physical world, the angular momentum quantum number j of the Wigner-D matrix is 1, thus for vectors with l = 1, we have:EQUATIONwhere α, β, γ are the Euler angle representation of R that satisfies R = R z (α)R x (β)R z (γ), m, m ′ ∈ {-1, 0, 1}, and d j m ′ ,m (β) is the matrix element. Since D(R) is also a unitary matrix, we can find a set of basis[v 0 , v 1 , v 2 ] that makes D(R) = R.For type-0 vector fields, D 0 (R) is a one-dimensional identical scale factor 1. Let's begin our proofs.Theorem 1 Rotation matrices, represented by three type-1 vectors, are SE(3)-equivariant parameterization of rotation actions.Proof: For a rotation matrix R = [v 0 , v 1 , v 2 ] ∈ R 9 , where v 0 , v 1 , and v 2 are the three column vectors, we use three type-1 vectors to represent these three column vectors. Thus the output vector of the network is as follows:EQUATION)When the input point cloud is transformed by a SE(3) transformation T = (R, t), the rotation matrix representation of the target object also transforms by R, that is[v ′ 0 , v ′ 1 , v ′ 2 ] = R[v 0 , v 1 , v 2 ]. According to Equation 5 and 6, the outputted type-1 vector fields are transformed by D 1 (R) = R, thus we have:EQUATIONTheorem 2 There is no SE(3)-equivariant vector field representation for Euler angle, quaternion, and axis-angle.Proof: We use proof by contradiction to prove theorem 2. Consider the example illustrated in Figure 2 .1) For quaternion, we define a quaternion q = [cos(θ/2), sin(θ/2)u i , sin(θ/2)u j , sin(θ/2)u k ] where θ is the rotation angle and [u i , u j , u k ] is the rotation axis. For the initial pose, we have q = [1, 0, 0, 0]. For the end pose, we haveEQUATION2 , 0, 0], thus:EQUATIONThere are two options for quaternion type-l parameterization: 1) using four type-0 vectors; 2) using one type-1 vector and one type-0 vector. For both cases, the +[EQUATION2 , 0, 0] operation cannot be represented by a Wigner-D matrix. Thus there is no SE(3)-equivariant vector field representation for quaternion. Axis-angle can be proven in the same way.2) For Euler angles, we define an Euler angle as E = (α, β, γ). For the initial pose, we have Euler angles equal to E = (0, 0, 0). For the end pose, we have Euler angles equal to E ′ = (-π 2 , 0, 0). Thus we have:EQUATIONThere are two options for Euler angles' type-l parameterization: 1) using three type-0 vectors; 2) using one type-1 vector. For both cases, the +[-π 2 , 0, 0] operation cannot be represented by a Wigner-D matrix. Thus there is no SE(3)-equivariant vector field representation for Euler angles.In the real-world experiments, we perform point cloud voxel downsampling before feeding the point cloud into the network with the voxel size equal to 1cm for the Mug on Rack task and 2cm for the task Plane on Shelf.After this, we perform color jittering by adding Gaussian noise on each point's color with a standard variance equal to 0.005. We also perform random color dropping that replaces 30% points' color to zero, and HSV transformation that randomly transfers the hue, saturation, and brightness of each point by 0.4, 1.5, and 2 times respectively.For RiEMann, We use r 1 = 0.2m, r 2 = 0.02m for all the tasks in the simulation. We use r 1 = 0.16m, r 2 = 0.02m for the real world Mug on Rack task, and r 1 = 0.2m, r 2 = 0.02m for the Plane on Shelf task. Other network hyperparameters of RiEMann are listed in Table 3 . We do not use any kind of prior knowledge for the training of RiEMann such as object segmentation, pertaining, or pose augmentations. We also set the robot to first reach some pre-defined pre-grasp pose (e.g., above the mug) and a pre-place pose (e.g., in front of the rack) to eliminate the unnecessary influence of the motion planners.The full training of RiEMann takes 200 epochs on a single NVIDIA A40 with a batch size of 4 and a learning rate of 1e-4 for each network module ϕ, ψ 1 , and ψ 2 . However, we find that for Mug on Rack, they only need about 50 epochs to converge, which takes about 47 minutes. For PerAct [37] , the language descriptions of our tasks are: Put the mug on the rack, Place the plane on the shelf, Turn the faucet. We follow the original 3D voxel grid size (100 3 ) and the patch size ( 53 ). We use Euler angles as the rotational action representations for PerAct. For fairness, we do not train the gripper action for PerAct. We use 6 self-attention layers for the perceiver Transformer module. The other hyper-parameters are the same with the original paper.For R-NDF [13] , since there is no pre-trained weight for the plane and the faucet, we here pretrain the NDFs using the reconstructed meshes from the point clouds in our demonstrations, and use these model as the NDFs module to run R-NDFs. We observe that R-NDFs fail to accomplish all of the tasks when testing, which shows that R-NDFs cannot perform well without object segmentation, because of the locality requirements of R-NDFs. We also tried to use the original pre-trained weights from the original paper [13] for the task Mug on Rack, but we found that the performances were even worse because of the discrepancy of the specific object shapes in the test experiments and the pertaining datasets. Other network hyper-parameters are the same as in the original paper.For EDF [14] and D-EDF [15] , we manually separate the robot end-effector and the grasped object point cloud from the scene rather than setting a series of separate cameras to capture their point cloud. For EDFs, we run the MH for 1000 steps, run the Langevin algorithm for 300 steps, and optimize the samples for 100 steps. We use one query point for picking and three query points for placing. We train EDFs for 200 epochs, the same epochs with RiEMann. For D-EDFs, we train the networks for 1 hour with parallel training of the low-resolution and the high-resolution networks and the energy-based critic network. Other network hyper-parameters are the same as in the original paper.For all simulation tasks, the radius of the table is 0.75m, and we put a Franka Panda robot arm in the center of the table. We divide the table into a semicircle part and two quarter parts and make them different heights with a height difference of 0.1m. This is designed to conveniently apply SE(3) transformations on target objects. We cut the input point cloud into a cube with a side length of 2m centered on the center of the table. Details of different tasks are introduced here:Mug on Rack: A mug and a rack are placed on the table. The robot has to pick up the mug by the rim and then hang it on the rack by the handle. This is the most representative object rearranging task that is also evaluated in [12, 14, 13, 15] . For the training set T, we use a mug in blue with a side length of about 17 cm, and a rack with a height of 67cm. The mug must be hung on the highest peg of the rack. For the new instance set NI, we use a patterned red mug with a height of about 19cm and a base diameter of about 10cm.Plane on Shelf : A plane model and a box-shape shelf are placed on the table, and the robot has to pick the middle part of the body of the plane and place it on the shelf. For T, we use a grey plane model with a length of about 20cm. For NI, we use a blue plane with the same size.Turn Faucet: As shown in Figure 6 , a faucet is placed on the table, and the robot has to turn on the faucet by first moving to the handle of the faucet and then moving along the opening direction.For T, we use the NO. 5004 faucet model in ManiSkill2 [45] . For NI, we use the No. 5005 faucet model.For the open-faucet task, we assume that given a target pose T = {R, t} ∈ SE(3) and a target direction d ∈ R 3 , the robot can accomplish the task by first going to the target pose T just as done in pick-and-place tasks, and then moving along the target direction d while keeping the orientation not changed, as illustrated in Figure 6 . To encode the extra directional action d, we add another type-1 vector field on the orientation network ψ 2 , that is:f R = 4 i=1 f i 1 (x), x ∈ B ROI .The final output directional action d is also calculated through mean pooling on points in the radius r 2 . Note the policy needs to continuously predict the output direction during the opening process, which shows that RiEMann can capture the local SE(3)-equivariance of the handle part of the faucet.In this task, we give demonstrations of not only the target pose T, but also the opening direction d at the first frame. Here there exists two kinds of SE(3)-equivariance: the target pose should be equivariant to the pose of the faucet at the first frame, and the opening direction should be equivariant to the handle during the opening process, where the second equivariance requires both local-equivariance and the real-time performance.We provide the ground truth pose to a point cloud based motion planner MPlib [46] to generate the demonstration trajectory for training. We transform all point cloud input to the end-effector coordinate system. We collect 10 demonstrations for each setting for the evaluation of SE(3) geodesic A.9 Ablation Studies A.9. 
 
 Appendix:  


Decision: PRE REBUTTAL:

High level summary of reviews:

Strengths:

- SE(3)-equivariant networks enable generalization to pose changes, distractors and novel instances of the class. The method was called out as being relatively novel.
- The method is well-written and easy to follow (with some caveats about clarity of exposition, see below).
- RiEMann highlighted as maintaining low computation cost (via reduction in point cloud size).
- The experimental results validate the merit of the proposed framework and demonstrate the effectiveness of the method on two challenging tasks in the real world.

Weaknesses:

- Some technical details and heuristic designs in the framework need further clarification and justification. See specific reviewer feedback for details.
- The method relies on manually tested hyperparameters and does not predict pre-pick/pre-place actions, limiting its scope.
- Local equivariance may not be suitable for more complex tasks that require global information, such as sweeping dust with a broom (see reviewer UjoM’s comment).
- Reviewer 9VAx questions whether certain theoretical contributions, like the theorems on SE(3)-equivariance, are not novel and are actually well-established in group theory discussions. Additionally the authors use language to describe the group theory behind the equivariance that is a bit non-standard and this should be fixed.
- The clarity of presentation could be improved, particularly in describing the model details and experimental setup.

POST REBUTTAL:

The authors provided a thorough rebuttal and addressed many of the concerns raised by reviewers. This paper crosses the threshold for acceptance. Please note that Fourier Transporter is unlikely concurrent work (there was at least 3 months between) and the AC agrees this should have been a baseline. However, even with this missing the experimental results are strong enough to justify acceptance. The authors should make sure all the reviewer suggested changes (and their author's own clarifications) are reflected in the final manuscript for the camera ready.

Reviews: review: title: title: Innovative work with some details to be clarified
paper_summary: paper_summary: This paper proposed a robot object manipulation learning method given scene point clouds. The key idea is to directly regress an action field on the point cloud, as opposed to pose estimation from feature matching optimization, so that the inference efficiency can be largely improved. This paper claims to be the first near real-time point-cloud-based object manipulation framework.  SE(3)-equivariant networks are used to enable generalization to pose changes of the target object. Experimental results show good performance and generalizability to pose and instance variations.
main_review: main_review: This paper presented a promising object manipulation framework with action field regression. At first glance, the main idea of this paper (action regression) is very simple, and the claim that this simple idea can realize superior performance, generalizability, and efficiency seems too good to be true. I think this paper resolved two major challenges in making this simple idea work. 

First, how to map the scene point cloud to an action in a generalizable way? The action prediction is designed as a field regression task, where the translation is estimated through a heatmap regression of the target position, and the rotation is estimated through vector field prediction at the salient area. By casting the action prediction as a dense prediction problem, the framework naturally associates the action to the local part of objects most relevant to the interaction. Importantly, the translation is not regressed as a vector field but as a scalar field,  allowing the generalization benefit from the translation invariance. It requires the translation target to be in the convex hull of the point cloud, which explains why the point cloud of the whole scene instead of the segmented object is input to the network. It also forms a superiority of the proposed method, with less preprocessing required on the input. 

Second, how to learn the action field from demonstration efficiently and without explicit labels? There is no ground truth action field to supervise the training. Only a few demonstrations are provided. This paper takes a weighted sum of the dense prediction to supervise the saliency, translation, and rotation estimation. On the one hand, the single action ground truth signal can be propagated to the dense field. On the other hand, the weight naturally modulates the supervision to the most relevant areas of the object. 

However, the paper also employed several heuristic designs to make the framework practical. For example, the two scalar field predictions (saliency and translation) seem duplicate and redundant in their function. They both highlight a focus area and constrain the input to this area in downstream processing. Furthermore, there are two predefined radius hyperparameters for the local area selection. These ad-hoc designs show that the action field learning idea needs careful tuning in the implementation details, but I think they are acceptable. 

Overall, I think the presentation is clear and easy to follow. The experimental results validate the merit of the proposed framework. However, I have several important questions regarding the clarity of the method and its limitations. 

1. How are the demonstration data pair (P_i, T_i) defined? Does the point cloud include the robot arm/gripper, or do you manually remove them for the framework? Since the framework runs near real-time and updates the target continuously, how exactly is the target pose defined? Is it defined with a certain time horizon or with respect to the predefined stages of a certain task? Take the pick-and-place task for example, are there two target ground truth poses for pick and place respectively, or more than that? If it is defined with respect to the stages, do we need to switch stages manually or the network learns it from the point cloud configuration? For the pick-and-place task, the scene remains static before the robot lifts the mug, so how does the robot switch from the "pick" target pose to the "place" target pose? 

2. How much data is used for training? Is it only using the data of 5-10 demonstrations in the simulation environment? Are they sliced into the data points per second for training? How many data points are there in total for the network? 

3. From my understanding, the success of the pose-equivariance and the action prediction in this framework largely relies on the locality of the learning, given the two-level local area confinement and the SE(3)-transformer's property. Will it restrict the capacity of the network and the potential of this framework to tackle more complicated manipulation tasks?
questions: questions: Please see above.
limitations: limitations: 3
review_summary: review_summary: This work provided innovative insights to the manipulation problem. Method and experiments convincing overall. Some clarification needed.
title: title: RiEMann demonstrates decent performanace but lacks novellty.
paper_summary: paper_summary: This paper proposes RiEMann a SE(3) equivaraint open-loop control method for tabletop robotic manipulation. The method boasts an improved evaluation speed by reducing the overall size of the pointcloud used for policy learning. The authors evaluate the method in simulation and in the real world on a small set of tabletop manipulation tasks.
main_review: main_review: **Strengths:**
- The paper is well written.
- The saliency map reduction to reduce the pointcloud size utilized for policy learning is a good design choice to alleviate some of the issues with equivariant models.
- Authors demonstrate the effectiveness of their method on two challenging tasks in the real world. 
- Method is applicable to articulated tasks (i.e. turn faucet) which is a under-explored area of robotic manipulation. 

**Weaknesses:**
- The authors present two theorems/proofs as contributions but neither are novelties and are commonly discussed topics in group theory.
- The authors use language to describe the group theory behind the equivariance is a bit non-standard. I.e. T-equivariance, type-l vectors, etc.
- Fig.2 is difficult to understand due to the amount of information crammed into a relatively small figure.
- Policy evaluation time should be given in Hz not FPS.
- The authors state that their method is much faster than other equivariant methods but other SO(3) model architectures such as Equiformer-v2 have been shown to be fairly fast. 
- Authors state that they ran ablations in the main body but do no report any results. I understand this is due to the page limit but they should not reference them in the main paper unless they report *some* results at the very least in the main body of the work.
- Minor gripe but proposed method acronym (RiEMann ) is both unintuitive and difficult to remember.
questions: questions: 1. What is T(3)-invariance (Line 153)?
2. Additional details on the model would be appreciated. For example, I assume the saliency network is smaller than the policy network in order to speed up evaluation but these details appear to be missing.
3. How does the speed of this method compare to other more modern SO(3) equivariant model architectures such as Equiformer-v2?
4. Why not compare to Fourier Transporter?
limitations: limitations: 1
review_summary: review_summary: While the proposed method does show improved performance when compared to the baselines its overall novelty is a underwhelming. The authors state that their method is the first end-to-end SO(3)-equivaraint model but I do not belive this true and the theoritical contributions have been known for a long time in group theory. Additional model details are needed to properly assses the novely of the model architecture itself.
title: title: The proposed method demonstrates strong generalization ability towards novel scenes, but is limited to uni-model policies.
paper_summary: paper_summary: RiEMann is a locally SE(3) equivariant open loop imitation learning framework that predicts gripper pose, given point cloud observation.  RiEMann pipeline includes two stages. First, a saliency network take as input the full point cloud, then selects the region of interest. Second, the position and the orientation are regressed by position and orientation nets, respectively. Since all the networks as well as the action presentations are equivariant, RiEMann achieves end-to-end equivariance.  The local equivariance brings the benefits of generalizing to out-of-distribution scenes. Moreover, the action regression alleviates expensive SE(3) action inference, compares to previous works that used field-matching or steerable-convolution mechanisms.
main_review: main_review: quality

The paper proposes a solid local SE(3) equivariant imitation learning method that brings the benefits of generalizing to out-of-distribution scenes. Moreover, the action regression alleviates expensive SE(3) action inference, compares to previous works that used field-matching or steerable-convolution mechanisms.

clarity

Generally the presentation is clear.  Two small suggestions are: briefly highlight the contributions in the abstract and switch the order of 3.2 and 3.3, so that the group theory backgrounds comes before the equivariance property of the method.

originality

The SE(3) local equivariance and inferring SE(3) action by regression over 3 type0 and 3 type1 features are novel and valid.

significance of this work

-strengths

The proposed method achieves local SE(3) equivariance, thus is robust to novel position, distractors, and novel instance of the class, compares to the training scenes, while maintaining low computation costs.

-weaknesses

The method regresses action pose, thus is incompatible with multi-model actions (i.e., there are 4 orientations to grasp a cube by a parallel jaw gripper).

The method relied on manually tested hyperparameter r1, r2.

The method do not predict pre-pick/ pre-place actions.

Moreover, local equivariance is not always preferred, since more complex tasks requires global information, i.e., to sweep dust using a boom, though the action is related to the gripper and the handle of the boom (local information), the policy needs to reason about the head of the boom and the dust (global information).
questions: questions: In line 176-178 states that the rotation matrix is the only SE(3)-equivariant rotation representation. This statement is too strong (maybe wrong). I.e., 5D and 6D representation is also SE(3)-equivariant [1].

[1] Zhou Y, Barnes C, Lu J, Yang J and Li H (2019) On the continuity of rotation representations in neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5745–5753.

In Appendix Theorem2, there seems missing the proof for that axis-angle is not SE(3) equivariant.

How can one extend the method to multi-model policies? I.e., how could the method be improved to learn grasping a ball?
limitations: limitations: 2
review_summary: review_summary: The proposed method achieves SE(3) local equivariance, which is sound and efficient for a subset of tasks. Nevertheless, the method is limited to uni-model policy learning.