PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation
Authors: Shizhe Chen
Ricardo Garcia Pinel
Cordelia Schmid
Ivan Laptev
Venue: CoRL 2023 Poster (2023)
Keywords: Robotic manipulation
3D point clouds
language-guided policy

Abstract: The ability for robots to comprehend and execute manipulation tasks based on natural language instructions is a long-term goal in robotics. The dominant approaches for language-guided manipulation use 2D image representations, which face difficulties in combining multi-view cameras and inferring precise 3D positions and relationships. To address these limitations, we propose a 3D point cloud based policy called PolarNet for language-guided manipulation. It leverages carefully designed point cloud inputs, efficient point cloud encoders, and multimodal transformers to learn 3D point cloud representations and integrate them with language instructions for action prediction.
PolarNet is shown to be effective and data efficient in a variety of experiments conducted on the RLBench benchmark. It outperforms state-of-the-art 2D and 3D approaches in both single-task and multi-task learning. It also achieves promising results on a real robot.

PolarNet: 3D Point Clouds for Language-Guided
Robotic Manipulation
Shizhe Chen; Ricardo Garcia", Cordelia Schmid, Ivan Laptey Inria, École normale supérieure, CNRS, PSL Research University https: wwww di ens. h/t////oowwrreeerrcc//oooo....aa
The ability for robots to comprehend and execute manipulation tasks based on natural language instructions a long-term goal in robotics. The domi- nant approaches for language-guin guided manipulation use 2D image representations, which face difficulties combining multi-view cameras and inferring precise 3D positions and relationships. To address these limitations, we propose 3D point cloud based policy called PolarNet for language-guides manipulation. It leverages carefully designed point cloud inputs, efficient point cloud encoders, and multi- modal transformers to learn 3D point cloud representations and integrate them with language instructions for action prediction. PolarNet is shown to be effective and data efficient a variety experiments conducted on the RLBench bench- mark. It outperforms offte--f-hheann art 2D and 3D approaches in both single-task and multi-task learning. It also achieves promising results real robot.
Abstract:
Keywords: Robotic manipulation, 3D point clouds, language-guided policy
Introduction
People are able to perform a wide range of complex manipulation tasks 3D physical environments. One efficient and intuitive way specify different tasks through natural language instructions. Consequently, long-term goal in robotics is to develop robots that can follow language instructions to perform various manipulation tasks.
Most existing work on language-guided robotic manipulation uses 2D image representations [1, 2, 3, 4]. BC-Z [1] applies ResNet [5] to encode a single-view image for action prediction. Hiveformer [3] employs transformers [6] to jointly encode multi-view images and all the history. Recent advances in vision and language learning [7, 8] have further paved the way amage-basee manipulation [4]. CLIPort [4] and InstructRI. [9] take advantage of pretrained sssion-and-languae models [8, 10]
to improve generalization in multi-task manipulation. GATO [11] and PALM-E [12] jointly train robotic tasks with massive web image-text data for better representation and task reasoning.
Although 2D image-basee policies have achieved promising results, they have inherent limitations for manipulation in the 3D world. First, they do not take full advantage of multi-view cameras for visual occlusion reasoning, as multi-view images
explicitly aligned with each other, as shown in Figure 1. Second, accurately inferring the precise 3D positions and spatial relations [13] from 2D images is significant challenge. Current 2D approaches mainly rely on extensive pretraining and sufficient in-domain data to achieve satisfactory performance.
To overcome the limitations of 2D-based manipulation policy learning, recent research has turned 3D-based approaches. The use of 3D representations offers a natural way to fuse multi-view ob- servations [14] and facilitates more accurate 3D localization [15], see Figure I (right). For example, PerAct [14] adopts an action-centric approach that takes digh-ddmenniioal input of over million voxels to classify the next active voxel, achieving ofte-off-hheart results for multi-task language-
"The authors contributed equally to this work.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

(a) Reach and Drag Task
(b) Multi-view RGB-D Observation
Left shoulder
Wrist
Right shoulder camera
amo the rcd target.
square on top.
[c) Merged 3D Point Cloud from Multi-view Cameras
Design Choices
coordinates
RGB color
Normal
Gripper frame
up push pul the cube onto the maroon target.
stick by pick the black onto the green target.
Removiig background / table
Figure 1: (a): Variations the "Reach and Drag" task RLBench [19], with different target colors per variation. (b): Although different views are complementary to represent the scene, they not explicitly aligned with each other. (c) Merging multi-view cameras to construct a unified point cloud in 3D space. Design options of the point cloud input are carefully investigated in this work.
guided manipulation. However, such action-centric 3D voxels suffer from quantization errors and computational inefficiency. Alternative 3D representations in the form of point clouds have been successfully explored for 3D object detection [16], segmentation [17] and grounding [13]. How- ever, the effective and efficient processing of 3D point clouds for robotic manipulation [18] still underexplored. In addition, focus has primarily been on single-task manipulation, lacking the
versatility to incorporate language instructions for completing multiple tasks simultaneously.
This paper proposes PolarNet, point dloud-based language- guided robotic manipulation network. We explore various design options for 3D point cloud representation, including the composition of point cloud features, the coverage space of the point cloud, the use of multiple camera views, and the choice of coordinate frames for point clouds. Our method builds on the PointNext architecture [17] to efficiently encode point cloud inputs. multimodal transformer is employed to fuse the encoded point cloud with language instructions an intermediate level. Finally, our PolarNet predicts 7-DoF (degrees of freedom) actions encompassing position, rotation and open state of a parallel gripper, which are executed by motion planer to complete the task.
We conduct extensive experiments on RLBench [19] in three language-guiin guideed manipulation setups: single-task single-variation (one policy for each 10 tasks), multi-takk single- variation (one policy for all 10 tasks), and multi-task illustrates variations for the task "reach and drag". Our model significantly outperforms state-of-the- art 2D and 3D based models for all the three settings, demonstrating the effectiveness of PolarNet in integrating language guidance with 3D point cloud based representation for robotic manipulation. PolarNet also achieves 60% average multi-task success rate on tasks on the real robot.
single-variation (one policy for tasks with 249 variations). Figure
2 Related Work
Language-guided Robotic Manipulation. iaoguage-guided policy learning has gained significant attention in robotics due to its potential for convenient human-robot interaction and skill general- ization across diverse tasks [20, 21, 22, 23, 24, 25, 26, 27]. Several multi-task benchmarks have been established recently [28, 19, 29, 30, 31]. In our work, we use RLBench [19] since provides hundreds of challenging manipulation tasks with demonstrations and instructions. Leveraging pre- trained large language models [12, 32, 32, 3, 4, 11] has shown be beneficial for language-guided manipulation. Most existing works are based on 2D images [2, 34, 1,27,3, 3], while few recent works explore the potential of 3D representations [14, 35]. Both C2F-ARM [35] and PerAct [14] utilize

---

3D voxel representation. C2F-ARM proposes to represent the 3D scene in a coarse-to-fine manner to balance quantization error and computation cost. PerAct [14] uses an action-centric representation that encodes the language and all 3D voxels in the workspace via a Perceiver transformer [36] and classifies which voxel is the next gripper position, making it less efficient. In contrast, we explore the point cloud representations and predict actions in a more efficient "point-centric" manner.
Manipulation Learning with Point Clouds. Due to the advantages of 3D representations, research has increasingly focused on vision- vision-guided robotic manipulation from 3D point clouds [37, 18, 38, 39, 40, 41, 42, 43, 44]. The design choice of the point cloud representation plays an important role for the task performance. Strudel et al. [45] shows the benefits from normals for motion planning. Liu et al. [18] finds that normalizing point clouds in the end-effector and target object frames work significantly better than the vanilla world and robot base frames. Seita et al. [46] adds per-point flow vectors to improve manipulation of tools. Qin et al. [47] further shows direct sim-to-real transfer is possible with point clouds. However, existing work mainly utilizes point clouds to train a single- task policy and does not consider any language interactions. In this work, we jointly represent point clouds and language to perform various manipulation tasks, and systematically evaluate design choices for point cloud representations, such as the importance colors.
Point Cloud Networks. Point cloud- vision approaches have studied the question of 3D rep- resentations for 3D object detection [16], segmentation [48] and 3D object grounding [13, 15]. PointNet [48] and its extension PointNet++ [49] are the most popular networks for processing point clouds. Although more advanced networks based on transformers [6] have been proposed recently [50, 51], PointNext [17] demonstrates that small modifications on PointNet++ can even out- perform the newer architectures. Our approach PolarNet is based on the state-of-the-art PointNext model [17] and evaluates different design choices in the context of language-gun guided manipulation.
3 Method
3.1 Problem Definition
We aim to learn a visual policy x(a O;) to perform robotic manipulation tasks following natural language instructions, where O, E 0.a, E A are the observation and action at step r, and A are the observation and action space respectively.
this work, the observation space includes: 1) a language instruction (x), .xx, where x; is a tokenized word, 2) RGB images from K cameras ( 1 rgb E Rxx and 3) aligned depth images ( ( dep E HR We use K = 3 with cameras the left shoulder, right shoulder and wrist of the robotic arm and H = W 128 following previous work [2, 3]. The intrinsic and extrinsic camera parameters are known. The action space A consists of the pose and open state of the parallel gripper. The pose is composed of the Cartesian coordinates a; E R and rotation described with a quaternion aj E R in the world frame. The open state af E {0, 1} indicates whether the gripper is open or closed. We follow the standard setup in RLBench [34, 2, 3, 14] to predict action of key steps and use motion planner execute the action.
More details about the key step presented in Section B of the appendix.
3.2 PolarNet: Point Cloud-based Language-guided Robotic Manipulation Policy
Figure presents an overview of PolarNet for language-guided manipulation with 3D point clouds. More detailed architecture is presented in Figure in the appendix. We first describe the design choice of point cloud representations in Sec 3.2.1, and then introduce the details of the model archi- tecture in Sec 3.2.2 followed by the training objectives in Sec 3.2.3.
3.2.1 Point Cloud Inputs
Preprocessing, Given 1 dep together with known camera instrincs and extrinsics at step 1, we project each pixel in heep to the 3D world coordinates. Since 1hg aligned with Sep the RGB color of pixel can be appended to the corresponding point. In this way, obtain point cloud V Hxx
3

---

Merged point cloud
Residual connection
Position ot
camera
ff 82
of
PointNext Encoder
Self-Attn
Cross Cross-Attn Attn
PointNext Decoder
cameea 2
FFN
heatmap offset
Ng
Ng
camera 3
1 2Nz
Text Encoder
Max Pool & Concat
rotation of open state az
Instruction: Grasp the red cup and lift it.
MLP
Figure 2: PolarNet for language-guided manipulation. The approach takes as imput the merged
point cloud obtained from multi-view RGB-D images and language instruction, and uses Point- Next [17] for efficient point cloud encoding and CL.IP text encoder [8] for language encoding. The point cloud and language are integrated via a multi-layer transformer at the intermediate level. Po- larNet predicts the position (cyan node) using an integral over the heatmap of the point cloud with offset per point, and also rotation and open state of the gripper using global features.
for each camera in the world frame, where each point consists of the XYZ coordinates and its RGB color. is possible to merge point clouds from different cameras, as they are in the same coordinate system. To reduce redundancy, we use Open3D toolkit [52] to uniformly downsample the merged point cloud to one point per voxel. We use voxel size of lem2 following previous work [14]. We also estimate the normal of each point via Open3D, The geometric structure of the point cloud makes it straightforward to select relevant points, which is difficult for 2D images. For example, the background points like wall and floor are far away from the robot's workspace, and the manipulated objects are usually above the table surface Therefore, we define a 3D bounding box that covers the workspace above the table to crop the point cloud (Figure 5 in the appendix), only keeping points of the objects and the robotic arm. Our empirical results in Table show the effectiveness of removing irrelevant points. We randomly sample N, 2048 points the processed point cloud as V/-
Point cloud features. The type of point cloud representation [45] and normalization [18] sig- nificantly impact the performance. Our paper is the first to study this impact systematically. In the experimental section 4.2, we first investigate different compositions of the point cloud inputs, in- cluding: 1) XYZ coordinates, 2) RGB color which seldom used for robotic manipulation because most works focus on single task requiring geometry structures only to be solved, 3) normal which has been shown effective avoid obstacles [45], and 4) the height with respect to the manipulation table which helps infer the object location. We then compare two types of coordinate frame to nor- malize the point cloud into unit ball: 1) using the gripper position the coordinate origin which has the benefits to infer the object-robot relation [18], and 2) using the center of the point cloud as the origin which is a common practice in 3D scene understanding tasks [15]. Our empirical results in Table and indicate that all the four point cloud representations are beneficial with color of particular importance, while the two coordinate frames perform similarly.
3.2.2 Model Architecture
Language encoding. We employ the language encoder from the CLIP model [8] to tokenize and encode the language instructions. It is pre-trained on large-scale image-text pairs, enabling to ef- fectively understand visually relevant instructions. We freeze the language encoder and add lincar layer to it to obtain the language embeddings X (f,,...,,,,,
W, CLIP((x)........ (xx,))..
Point cloud encoding. We encode the point cloud V, with the state-of-the-art PointNext coder [17]. It consists of number of set abstraction (SA) blocks to hierarchically abstracts features
'Thre are few exceptions such as the "sliding block" task where the target is green area on the table. We manually select those tasks and do not remove the table points for them.

---

v. Each SA block contains a subsampling layer, grouping layer, a multilayer perceptron (MLP), and a reduction layer. The subsampling layer at the /-th block uses farthest point sampling to select N points from V,, and then the grouping layer finds neighbors for each selected point which are points within radius r' to the query. The MLP is shared for all the points to generate features. Finally, the reduction layer utilizes max pooling to aggregate features within the neighbors for the N₁ query points. Assume v/ is the imput feature of query point i at the (-th block and p) is the XYZ coordinates of the point, the computation is formulated as:
|+ MaxPool j(ij)eu (MLP((V) -/)
(1)
where N denotes the set of neighbor points to the query point i, and [-] is the feature concatenation. Each SA block decreases the number of points by 2 times, while increases the dimensionality of the features by 2 times. We denote the encoded point cloud as V, {uz y where NØ N v RR2 and de is the hidden size in the first layer.
No
Vision-and-language fusion. We utilize multi-layer transformer architecture [53] to integrate the point cloud features v, and the language embeddings X. To be specific, we first add the XYZ position and step id embedding to the point cloud feature as follows:
p W, v/- + Wp P (t),,
where E-) is the sinusoidal positional encoding [6] to denote different steps. Then, in each trans- former layer, we sequentially perform self-attention on the point cloud features, cross-attention from the point cloud to the language followed by a feedforward network, which is:
Wo Vd WWV,
Aut(e, V) = Softmax
(3)
D' - Aut (M,,
"'==Am(() (2,8), pI+ LN(W2 GeLU(WiP"))
(4)
where d is the hidden size in the transformer layers and LN denotes layer normalization. We stack layers to obtain the language-conditioned point cloud features D,c E M./2///
Action decoding. Since the gripper position 4,41 is in continuous space, previous work [14] discretizing the action space can suffer from quantization error. However, direct regression ap- proach is often difficult in training leading to unsatisfactory performance [54]. Therefore, we take inspiration from the integral human pose regression [55] to get the best of the both worlds. For the position prediction, we use the PointNext decoder [17] with residual connection and upsampling to generate a heatmap over the input point cloud H/ E Noxl and an offset at each point A E MR Albeit the point cloud only contains points of visible objects, the action position can cover the whole workspace. Thus we predict an offset to shift each point. The final predicted position is as follows:
Σ H.(p: AA)-
(5)
It allows continuous output and the underlying heatmap representation makes it easy to train. The rotation and open state are more discrete in nature, thus we directly predict them as follows:
d 1. 24 = MLP((MaaPool((y')
MaxPoll
(6)
3.2.3 Training Objective
We use behavioral cloning for model training. Given the dataset D {(,,, +)) with N pairs of successful demonstration v and the corresponding instruction X, where T composed of sequence of T key steps with visual observations fo, and actions {a, )/ The training objective is minimize the following loss, including mean square error (MSE) the gripper's position and rotation and binary cross-entropy (BCE) loss on the open state classification:
Σ TED
Σ MSE (a, ",,, + MSE (a) .af) + BCE (a,
(7)
NT
5

---

Table 1: Comparison of point cloud representations on the single-task single-variation evaluation.
Pick & Pick-Up
Put Knife
Put
Push
Reach
Slide
Stack
Take Wine Money Umbrella
Take
RGB Normal Height
Avg.
Lift
Cup
Money Button Target Block
x
26.2
44.0
81.1
95.9
99.6
27.8
89.3
91.0
70.3
95.3
72.1 14.4
x
97.9
94.7
79.5
95.8
100.0
100.0
91.0
91.1
65.9
97.3
91.3 11.6
x
94.9
94.2
77.1
95.9
90.3
100.0
93.1
94.1
69.4
94.4
90.3 #3.1
V
96.2
94.3
82.6
95.3
100.0
99.9
91.5
90.3
67.5
97.7
91.5 #14
96.7
91.9
82.5
96.1
99.9
99.9
93.5
94.1
68.6
97.5
92.1 40.4
4 Experiments
4.1 Experimental Setup
Evaluation setup. We consider three evaluation setups on RLBench [19]. The single-task single- variation [3] consists of N tasks with one variation per task. We generate 100 demonstrations for each task variation. Separate models are trained for each task. The multi-takk single-variane the same N tasks the single-task single-variation setup, but unified model is trained solve all the tasks. We consider [= 74) following the previous work [3], where 74 is the maximum number of tasks that can be successfully executed in RLBench. The multi-task multi-variation includes 18 tasks where each task has multiple variations, resulting in 249 variations in total [14]. We use the same 100 demonstrations for each task as [14]. A single model is trained for all the task variations, which more challenging due to the larger set of task variations and fewer training data per task variation. More details of the three setups are presented in Section in the appendix.
uses
Evaluation metrics. We use the task success rate (SR) measure the performance which is for complete success and 0 for failure per episode with no partial credits. For the first two setups, we evaluate on 500 unseen episodes per task (500 x 10 = 5000 evaluation episodes in total) follow- ing [3]. We run experiments three times with three seeds and report the mean and standard deviation. For the last setup, we use the released dataset from [14] and evaluate on 25 episodes per task as [14] with total number of 25 x 18- 450 evaluation episodes.
Implementation details. For the model architecture, we use the PointNext-S model [17] with L, 44 SA blocks and compare de {32, 64). We initialize it with the weights trained on ShapeNet. The transformer layers have d = 512 and we compare L {1,2,4). We use the Adam W optimizer with a learning rate of 5 10-4 and a batch size demonstrations. We train for 20K. 200K and 600K for the single-task single-variation, multi-task single-variation and multi-task multi-variation setups respectively. We use a single NVIDIA V100 GPU for training except in the multi-task multi- variation setup where we use 4 GPUs for acceleration. Our model is efficient train, taking about 1 hour for single-task training and 30 hours to train for multi-task multi-variation.
4.2 Ablations
We study the impact of point cloud inputs and model archi- tectures. Unless stated otherwise, we point cloud repre- sentation combining XYZ, RGB, normal and height, remove background and table points, and normalize them with the gripper position as the origin. The model uses de 32, L= 1. The ablations are performed in the single-task single-variation setup with 10 tasks.
Table Comparison of point cloud processing on the single-task Linglevarraatinn evaluation.
Coord origin
Remove
Avg.
Table Background
Center
92.1 =2.0
x
81.6 43.2
Point cloud representation. Table 1 compares point cloud input representations. The vanilla point cloud with only XYZ.
Gripper
X
89,9 12.8
92.1 .0.4
performs worst on average. The RGB color plays an important role for tasks requiring to distinguish colors such as "Pick-Up Cup" where the goal is to pick up the red cup amongst differently colored cups. The improvement from normal of the point less stable, which might result from noisy normal estimation. The height of point relative to the table can also slightly improve the performance.

---

Table 5: Comparison with state-of-the-an methods on single-task single-variation, multi-task single-variation and multi-task vurlti-ariiaion setups.
10 Tasks
74 Tasks
Tasks Multi- task Multi-var
Single-task
Multi task
Single-task
Multi-task
Auto-à [2]
55.0
69.3
PerAct [14]
42.7
Hiveformer [3]
88.4
83.3
66.1
49.2
45,3
PolarNet (Ours)
92.1
89.8
69.0
60.3
46.4
Table 3: Comparison of cam- era views on the single-task single-variation evaluation.
Point cloud processing. We compare two coordinate frames to normalize the point cloud, using the gripper position the points center the origin. As shown in Table 2, the two ways perform similarly while using the gripper position is more robust with lower standard deviation. More important in point cloud processing is to remove irrelevant points. Since we have a limited capacity to sam- ple points, the irrelevant points make the interesting points sparser and harm the performance. Results on the multi-task setup also shows the same trend in Table 12 in the appendix.
Left
Right
Wrist
Avg.
37.6 14.8
V
x
48.0 14.5
X
35.0 =5.5
V
67.0 #4.7
X
80.2 +3.0
Multi-view cameras. Table 3 analyzes the contributions from multi-view cameras for point cloud construction. A single cam- era alone is insufficient to accomplish a task due to occlusions, decreasing the performance by more than 44%. The wrist camera performs worst among the three views. However, is more com- plementary to the other two cameras, with more than 10% boost compared the combination of left and right shoulder cameras. Using all the three cameras achieves the best performance with significant margin (+11.9%, +15.5% + 25.1% depending the resp. two cameras).
X
V
76.6 +5.6
V
92.1 +0.4
Model capacity. We explore the influence of model capacities to the task performance in single- and multi-task setups in Table In the single-task setup, small network with de 32 performs slightly better than larger network with de 64. We suspect that the larger model may require more demonstrations However, the same model capacity does not perform well in the multi-task setup. It is essential to increase the model size to learn multiple manipulation tasks together. We also notice that the best model multi-task still underperforms that in single-task setup by 2.3%, indicating that better multi-task learning strategy is in need. In the following, we use = 32,L = for single-task setting and d. 64.1.= 2 for the multi-task settings.
Table 4: Comparison of model capacity on single-task and multi-task (MT) setups.
train.
MT
de
L
Avg.
92.1 20.4
x
64
91.0 10.9
77.4 18.6
64
1
89.3 12.0
64
2
89.8
11.5
64
4
86.3
=4.8
4.3 Comparison with State the Art
Evaluation three setups. We compare PolarNet with state-of-ther thettttt-rt approaches single- task single-variation, multi-task single-variation and multi-task multi-variation setups in Table 5. More detailed results on each task of the three setups are provided in Table 9, 10, and 11 respectively in the appendix. The compared methods include 2D image based models such as Auto-2 [2] and Hiveformer [3] and 3D voxel based model PerAct [14]. Our PolarNet consistently outperforms prior models on all the evaluation setups. It achieves considerable improvement with more tasks in the multi-task setup (+11.1), demonstrating the advantages of 3D point clouds to solve multiple tasks simultaneously. In addition, our model is more computationally efficient compared to previous 3D based model PerAct [14] which requires 16 days for training with 8 V100 GPUs. We only use 4 V100 GPUs to train the same amount of iterations in 30 hours, which is roughly 13 times faster. We analyze the success and failure cases in Section D in the appendix.
Robustness to viewpoint perturbation. We perform new evaluation on robustness viewpoint variances demonstrate the advantages of the point cloud model compared to 2D image based meth- ods. In the evaluation, change the camera positions and rotations to obtain new RGB-D images.
7

---

We only apply the change to the two side cameras but keep the wrist camera the same, because real world experiments the side cameras are easy to place in different locations but the wrist cam- era is always attached to the robotic hand. We denote the per- turbation factor f as randomly selecting = cm shift and = degrees for the modified camera pose. We directly larNet and Hiveformer models trained on the original cameras in the multi-task setup of 10 tasks with the new camera poses. The results are presented in Figure 3. The performance of PolarNet drops much less than Hiveformer for all the perturbation factors, demonstrating the robustness of our PolarNet on camera pertur- bation. The detailed results on each of the 10 tasks are presented in Figure 9 in the appendix.
ooee Polartet
0.6
0.5
Relatve
03
the Po-
02
01
05
10 Camera partubation factor
Comparison of dif- ferent models the robustness to camera perturbation, where the lower the better.
Figure 3:
4.4 Real-robot Experiments
We further evaluate PolarNet with real visual sensors on a real robot. We 6-DoF URS robotic arm and adopt real-world tasks. For each task, we collect 20 human demonstrations and fine- tune language-conditioned multi-task model trained on RLBench on the collected data. More details are presented in Section E in the appendix. Table presents the real-robot results. We report the success rate on episodes per task. Our PolarNet achieves 60% success rate on average for the real-world tasks. We summa- rize two prevalent reasons of failures in PolarNet. First, the model confuses the target object such "strawberry" and "apple" since the sparse point cloud may not provide sufficient semantic information. Secondly, the predicted grasping pose lacks precision. We conjec- ture that this is due to partial views and the multimodal nature of the target action distribution. To
Table 6: Performance of Po- larNet on 7 real-world tasks.
Task
PolarNet
Stack cup Put fruit in box Put plate on table
8/10
8/10
3/10
Open drawer Put item in drawer
9/10
4/10
Put item in cabinet
4/10
Hang mug
6/10
Average
6/10
tackle these limitations, further research could be conducted on pretraining semantic-aware 3D rep- resentations [56], combining with pretrained 2D models, and improving the complexity of action policy. Please see our project website [57] for examples of the policy execution in the real world.
Conclusion
This work addresses language-guided robotic manipulation using 3D point clouds. The proposed PolarNet employs carefully designed point cloud inputs, an efficient point cloud encoder, and a multimodal transformer to predict 7-DoF actions for language-condttioned manipulation. We find that it is critical to use point color with colors, filter irrelevant points and merge multiple views. Extensive experiments on the RLBench benchmark demonstrate that PolarNet outperforms state-of- the-art models variety of tasks, from single-task kingle-variation to multi-task The PolarNet also achieves promising results on a real robot to solve multiple tasks.
Our multi-task model still does not perform as well as the best single-task model, requiring more advanced multi-task learning algorithms. Additionally, while our policy can perform multiple tasks, we have not studied the generalization new scenes, objects, and tasks.
Limitations:
Acknowledgments
This work was partially supported by the HPC resources from GENCI-IDRIS (Grant 20XX- AD011012122). was funded in part by the French government under management of Agence Nationale de la Recherche as part of the "Investissements d' a'aveni"" program, reference 19- P3IA-0001 (PRAIRIE 3IA Institute), the ANR project VideoPrediet FANR-21-FAI1-0002-0 and by Louis Vuitton ENS Chair on Artificial Intelligence.

---

References
[1] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. BC-z: Zero-shot task generalization with robotic imitation learning. In CoRL, 2021.
[2] S. Liu, S. James, A. Davison, and E. Johns. Auto-Lambda: Disentangling dynamic task relationships. TMLR, 2022.
[3] P.-L. Guhur, S. Chen, R. Garcia-Pinel, M. Tapaswi, I. Laptev, and C. Schmid. Instruction- driven history-aware policies for robotic manipulations. CoRL., 2022.
[4] M. Shridhar, I.. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In CoRL, 2021.
[5] K. He, X. Zhang, Ren, and J. Sun. Deep residual learning for image recognition. 2016.
[6] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, u. Kaiser, and
I. Polosukhin. Attention all you need. NeurIPS, 2017.
[7] J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, NAACL-HLT, 2019.
[8] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, al. Learning transferable visual models from natural language supervi- sion. In ICML, 2021.
[9] H. Liu, I.. Lee, K. Lee, and P. Abbeel. Instruction-followinng agents with jointly pre-trained vision-language models. arXiv preprint arXiv:2210.1343) 2022.
[10] X. Geng, H. Liu, L. Lee, D. Schuurams, S. Levine, and P. Abbeel. Multimodal masked autoen- coders learn transferable representations. arXiv preprint arXiv: 14204, 2022.
[11] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-maron, M. Giménez, Y. Sulsky, J. Kay, J. T. Springenberg, T. Eccles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y. Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas. A generalist agent. TMLR, 2022.
[12] D. Driess, F. Xia, M. M. Sajjadi, C. C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.03338, 2023.
[13] D. 7. Chen, A. X. Chang, and M. Nießner. Scanrefer: 3d object localization in rgb-d scans ing natural language. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX, pages 202-221. Springer, 2020.
[14] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: multi-task transformer for robotic manipulation. In CoRL, 2022.
[15] S. Chen, P.-. Guhur, M. Tapaswi, C. Schmid, and Laptev. Language conditioned spatial relation reasoning for 3d object grounding. In NewrIPS 2022-36th Conference on Neural In- formation Processing Systems, 2022.
[16] Y. Zhou and O. Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4499, 2018.
[17] G. Qian, Y. Li. H. Peng, J. Mai, H. Hammoud, M. Elhoseiny, and B. Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In NeurIPS, 2022.

---

[18] M. Liu, X. Li, Z. Ling, Y. Li, and H. Su. Frame mining: free lunch for learning robotic manipulation from 3d point clouds. In 6th Annual Conference on Robot Learning, 2022.
[19] S. James, Ma, D. R. Arrojo, and A. Davison. RLBench: The robot learning benchmark & learning environment. RA-L,, 2020.
[20] L.. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg. Concept2robot: Learning manipula- tion concepts from instructions and human demonstrations. In RSS, 2020.
[21] M. Shridhar and D. Hsu. Interactive visual grounding of referring expressions for human-robot interaction. In Proceedings of Robotics: Science and Systems, 2018
[22] brian ichter, A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Ir- pan, E. Jang, R. Julian, D. Kalashnikov, S. Levine, Y. Lu, C. Parada, K. Rao, P. Sermanet, A. T. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, M. Yan, N. Brown, M. Ahn, O. Cortes, N. Sievers, C. Tan, S. Xu, D. Reyes, J. Rettinghouse, J. Quiambao, P. Pastor, L. Luu, K.-H. Lee, Y. Kuang, S. Jesmonth, K. Jeffrey, R. J. Ruano, J. Hsu, K. Gopalakrishnan, B. David, A. Zeng, and C. K. Fu. Do as can, not as i say: Grounding language in robotic affordances.
In CoLR, 2022.
[23] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence. Interactive language: Talking to robots in real time. In NeurlPs, 5th Robot Leurning Workshop: Trustworthy Robotics, 2022.
[24] S. Stepputtis, Camphell, M. Phielipp, S. Lee, C. Baral, and H. Ben Amor Language- conditioned imitation learning for robot manipulation tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, NeurlPs, 2020.
[25] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. VIOLA: Object-centric imitation learning for vision- based robot manipulation. CoRL, 2022.
[26] A. P. Shah, G. A. D. Lopes, and E. Najafi. Contact-based language for robotic manipulation planning. 2016 IROS, 2016.
[27] C. Lynch and Sermanet. Language conditioned imitation learning over unstructured data. RSS, 2021.
[28] C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martín-Martin C. Wang, G. Levine, M. Lingelbach, J. Sun, M. Anvari, M. Hwang, M. Sharma, A. Aydin, D. Bansal, S. Hunter, K.-Y. Kim, A. Lou, C. R. Matthews, I. Renterea, J. H. Tang, C. Tang, F. Xia, S. Savarese, H. Gweon, K. Liu, Wu, and L. Fei-Fei. BEHAVIOR-IK: benchmark for embodied AI with 1,000 everyday activities and realistic simulation. In CoRL, 2022.
[29] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRL, 2019.
[30] K. Zheng, X. Chen, O. Jenkins, and X. E. Wang. VLMbench: A compositional benchmark for manipulation. In NeurlPs Datasets and Benchmarks Track, 2022.
[31] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and I.. Fan. Vima: General robot manipulation with multimodal prompts. ICMI. 2023, 2022.
[32] S. Huang, Z. Jiang, H. Dong, Y. Qiao, P. Gao, and H. Li. Instruct2act: Mapping multi- moddalto instructions to robotic actions with large language model, 2023.
[33] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, Haus- man, A. Herzog, J. Hsu, Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. Ju- lian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao,
10

---

M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q- Vuong, F. Xia, Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-
1: Robotics transformer for real-world control at scale. In arXiv preprint arXiv:2212.06817, 2022.
[34] S. James and A. J. Davison. Q-Attention: Enabling efficient learning for vision-based robotic manipulation. RA-L, 2022.
[35] James, Wada, Laidlow, and A. Davison. Coarse-to to-eieto-eneeeee q-attention: Efficient learning for visual robotic manipulation via discretisation. In CVPR, 2022.
[36] A. Jaegle, F. Gimeno, A. Brock, A. Zisserman, O. Vinyals, and J. Carreira. Perceiver: General perception with iterative attention, 2021.
[37] X. Lin, Y. Wang, Z. Huang, and D. Held. Learning visible connectivity dynamics for cloth smoothing. In CoRL, 2021.
[38] A. Alliegro, M. Rudorfer, F. Frattin, A. Leonardis, and T. Tommasi. End-to-end learning to grasp via sampling from object point clouds. RA-L, 2022.
[39] J. Lv, Q. Yu, L. Shao, W. Liu, W. Xu, and C. Lu. Sagci-system: Towards sample-efficient, generalizable, compositional, and incremental robot learning. In ICRA, 2022.
[40] S. James and A. J. Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Robotics and Automation Letters, 2022.
[41] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes. 2021.
[42] B. Eisner, H. Zhang, and D. Held. FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects. In RSS, 2022.
[43] Y. Qin, B. Huang, Z.-H. Yin, H. Su, and X. Wang. Dexpoint: Generalizable point cloud reinforcement learning for sim-to-real dexterous manipulation. CoRL, 2022.
[44] S. Song, A. Zeng, J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations. RA-L, 2020.
[45] R. Strudel, R. Garcia, J. Carpentier, J. Laumond, I. Laptev, and C. Schmid. Learning obstacle representations for neural motion planning. In CoRL, 2020.
[46] D. Scita, Wang. S. J. Shetty, E. Y. Li, Z. Erickson, and D. Held. Toolflownet: Robotic manipulation with tools via predicting tool flow from point clouds. In Conference on Robot Learning, pages 1049.1049, PMLR, 2022.
[47] Y. Qin, B. Huang, Z.-H. Yin, H. Su, and X. Wang. Dexpoint: Generalizable point cloud rein- forcement learning for sim-to-real dexterous manipulation. Conference on Robot Learning, pages 594-605. PMLR, 2022.
[48] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. 2017.
[49] C. R. Qi, L. Yi, H. Su, and J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. NewrIPS, 2018.
[50] H. Zhao, L. Jiang, J. Jia, P. H. Torr, and V. Koltun. Point transformer. In Proceedings of the JEEE/CVF international conference on computer vision, pages 16259-16268, 2021.
[51] M.-H. Guo, J.-X. Cai. Z.N. Liu, T.J. Mu, R. R. Martin, and S.-M. Hu. Pet: Point cloud transformer. Computational Visual Media, 7:187-199, 2021.

---

[52] Q.-Y. Zhou, J. Park, and V. Koltun. Open3D: A modern library for 3D data processing. 180.00847, 2018.
[53] J. Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic repre- sentations for vision-and-land language tasks. Advances in neural information processing systems, 32, 2019.
[54] X. Sun, J. Shang, S. Liang, and Y. Wei. Compositional human pose regression. In Proceedings of the IEEE international conference on computer vision, pages 2602-2611, 2017.
[55] X. Sun, B. Xiao, Wei, S. Liang, and Y. Wei. Integral human pose regression. In Proceedings of the European conference on computer vision (ECCV), pages 529-545, 2018.
[56) L. Xue, N. Yu, S. Zhang, J. Li, R. Martín-Martin, J. Wu, C. Xiong, R. Xu, J. C. Niebles, and S. Savarese. Ulip-2: Towards scalable multimodal pre-training for 3d understanding. arXiv preprint arXiv:2305.08275. 2023.
[57] S. Chen, R. Garcia, C. Schmid, and I. Laptev. Polarnet project, 2023. URL https: // //ww. di. ens. fr ft/////lwwwreesaacc//oooaarrrr/r
12

---

A RLBench Tasks
We consider RLBench tasks for three different learning setups: single-task single-variation multi- task single-variation and multi-task multi-variation.
In the first two setups, we adopt 10-task setting and 74-task setting following [3]. Figure 4 depicts the 10 tasks and the corresponding instructions. Table presents the details of all 74 tasks that can be successfully executed in RLBench. The 74 tasks are manually grouped into 9 categories by [3]. We use the original RLBench code to collect training data and run evaluation.
In the multi
setup, use 18 tasks with 249 variations following [14]. Table 8 lists the task names, variation type, the number of variations and examples of instructions. We use RLBench codebase" from [14] which modified some tasks to have more variations. To match the training and evaluation setup from [14] we use the datasets provided in PerAct code repository.
Pick and
Pick up
Put knife
Put money
Push button
"pick up the red block the
"gresp cup
the knite holder chopping
"pul money away
maroon button"
Reach target
Slide block
Take money
Stack wine
Take umbrella
"tauch the red ball with grisper"
"side the block towards the target"
"take the money off ot the first shelf the safe"
"siide the bottle onto the wine rack"
"netrieve the umbrella from the stand"
Figure 4: Examples of the selected 10 tasks with corresponding instructions in RLBench.
Model Details
steps [34] key turning points in action trajectories where the gripper changes its state (open/close) velocities of joints close to zero. In this way, the sequence length of an episode is significantly reduced from hundreds of small steps to typically less than 10 macro steps.
Keysteps actions. For robotic control,
Point removal in point cloud preprocessing. In Figure we show the raw point cloud generated from multi-view cameras, the point cloud with background and table removal. As the background such wall and floor geometrically far away from the robot workspace and the table is in fixed height, we could simply pre-define 3D bounding box to select relevant points from the raw point cloud. This is more efficient than performing object segmentation to obtain points of objects.
PolarNet architecture. Figure presents detailed architecture of our PolarNet, which uti- lizes PointNext [17] backbone for point cloud encoding, multimodal transformers for vision-and- language interaction and PointNext decoder for action prediction.
13

---

Table 7: Details of the selected 74 tasks in RLBench.
Group
#Tasks
Task Names
fikketball__n_hoopp change_channel. -neat_off-grill, meat_on.grill, push.buttons, put.rbbiihh.in bin, stack_wine, tower3, tv_on
Planning
hang frame hanger, hanger, place hanger rack,
reach_and_drag, scoop_with_spatula screw_nail, slide_block_to.targ sweep_to sweep_to_dustpan, take_frame_pp _hanger,
Tools
take -platt.ttteee ooff pooloore_dish.nnoe water-plants
4 slide cabinet open and place cups, stack blocks, take sshshhoessso out of box, wipe_desk
Long Term
Rotation-
lamp_off, lamp _on, pick_and lift, push button, reach_targer lake_lid lof. saucepan
invariant
Motion
close_box, close drawer, close laptop lid, open_box open drawer, phone_on mhhone.on_bae,, base, woo__boos_oonnooo boooosseell roilet_seat_down toilet_seat_up
close_laptop
planner
Multimodal
5
beat the buzz, lift numbered block, pick up cup, stack cups, turn tap
insert.onto insert_onto_squarc. insert_usb comomuteer
aaddaanndd.lss.. Slace_sapee www.iinnesssoseeee _sortrr play-jenga, pnif-nife_on chhopiiingnchcggggeeeeeeeee chopping.bbardd board, put umbrella in umbrella stand, setup checkers, straighten rope, take toilet-roo off stand, take umbrella out of umbrella stand, take ush..ubb.... out of computer
Precision 12
12
Screw
change_clock, open_window, open_wine__bbleeoo.p.peee bottle, turnooeen_on
close _osee close fridge, close_grill, close micrrwwve,, open. open_door, fridge, open. grill, open _eccrowwve open_oven plug charger in power power.supply, press swiich, put money safe, site_money_outsall unplug.charger
Visual occlusion
14
Background and table removal
Raw point cloud
Background removal
Figure 5: Illustration of point cloud processing. We represent the point cloud, point cloud with background removal and point cloud with background and table removal for put knife task.
14

---

Table Details of the tasks with 249 variations in the multi-task multi-variation setup.
Task
Var. type
# vars.
Example of language template
open drawer
placement
"open the drawer"
slide block
color
4
"slide the block - target"
sweep to dustpan
size
sweep dirt to the dustpan"
meat off grill
category
2
"take the off the grill"
turn tap
placement
2
"turn tap"
put in drawer
placement
3
"put the item in the drawer"
close jar
color
20
"close the jarj "use the stick drag the cube onto the - target"
drag stick
color
20
stack blocks
color, count
60
"stack blocks"
screw bulb
color
20
"screw in the light bulb" "put the money away in the safe on the shelf"
put in safe
plaeeeent
3
place wine
placement
3
"stack the wine bottle to the - of the rack
put in cupboard
category
9
"put the in the cupboard"
sort shape
shape
"put the in the shape sorter" "push the button, [then the button]"
push buttons
color
50
insert peg
color
20
"put the ring on the spoke" "stack the other cups top of the cup" "place cups on the cup holder"
stack cups
color
20
place cups
count
Rest. Propapation
of zy2
Abstraction
Prosapatton
MUP
Set Abstraction
Propapatoon
(i An.)
Ret. Propaption
Set Abstractioo
IN 6]
IN.. 10] [N dg]
Abstracton
Multimode
Tnnnformen
N₂ 2
[N,.dg]
[ 2dgl
2d,]
N
N2 .4d,]
No 8d,]
Subsmpling
Grouping
Redurtion
Interpobeee
MLPs
16d,]
Max Pool
rotation of open state of
X
Encoder
Concat
Figure 6: Detailed architecture of PolarNet for language guided robotic manipulation. The model takes the input of point cloud {PI..... PN, and instruction X. where N, is the number of points and de is the dimensionality of the feature. It outputs the position a; rotation of and open state a" of the gripper for the next keystep. To compute the position 22 first predicts a heapmap H; and position offset A; for each point and then compute the integral.
Hiveformer [3] re-implementation. We implemented Hiveformer [3] by ourselves and achieve similar performance they reported in the paper the multi-task dingle-variation setting. To use Hiveformer the challenging multi-task setups, we enlarge the model capacity by doubling the hidden size and train the same number of iterations with PolarNet, which leads to better performance than using the prree-pprameters as in the original paper.
C Quantitative Results in Simulation
Results on 10 tasks. Table presents the detailed results for 10 tasks in the single-task and multi- task single-variatio setups. We compare our model with ARM [34], Auto-2 [2] and Hiveformer [3]. All the three methods are based on multi-view 2D RGB-D images for action prediction. Our point cloud based model obtains better and more stable results compared to the previous best model Hive- former (92.110.44 vs. 88.414.9) in the single-task setup, demonstrating the effectiveness of the 3D representations. The improvements consistent across tasks except for "Take Money". We ob- serve that the main problem the "Take Money" task is due to the imperfect motion planner. As
15

---

Table 9: Comparison with state-of-the methods on 10 tasks in single-tast and multi tallit-ask setups. All the methods are trained on 100 demonstrations task. We report the success rate (%) of our method on 500 episodes per task averaged over 3 seeds.
Pick & Pick-Up Put
Put
Push Reach Slide
Stack
Take Take Wine Money Umbrella
Avg.
Lift
Cup
Knife Money Button Target Block
Single-task single-variation
ARM [34]
70
80
100
70
70
Auto-2 [2]
82
72
36
31
95
100
36
23
38
37
55.0
Hiveformer
92.2
77.1
69.7
96.2
99.6
100.0
95.4
81.9
82.1
90.1
88.4
PolarNet (Ours)
96.7
91.9
82.5
96.1
99.9
99.9
93.5
94.1
68.6
97.5
92.1
Multi-task single-variation
Auto-A [2]
87
78
31
62
95
100
77
19
64
80
69.3
Hiveformer [3]
88.9
92.9
58.2
100.0
100.0
78.7
71.2
79.1
83.3
PolarNet (Ours)
97.8
86.0
80.5
94.1
99.6
100.0
93.4
80.5
68.1
97.8
89.8
Table 10: Comparison with state-of-the-art methods on 74 tasks in single-task and multi-task setups. We evaluate 500 episodes per task. Besides reporting averaged success rate (%), we group tasks into 9 categories and report the averaged success rate per group. (f denotes our own implementation.)
Long
Rot.
Motion
Multi
Visual
Avg.
Planning
Tools
Screw
Precision
Term
Invar.
Planning
Modal
Occlusion
tasks
of tasks
11
4
9
4
12
14
74
Single-task single-variation
Hiveformer [3]
81.1
60.6
7.6
95.6
75.8
82.4
64.0
53.6
65.4
66.1
PolarNet (Ours)
90.5
66.8
5.5
96.4
75.6
73.1
61.7
59.0
69.1
69.0
Multi-task single-variation
Hiveformer [3] PolarNet (Ours) 75.5
65.9
27.7
3.4
78.9
72.8
66.1
48.5
35.5
47.8
49.2
46.3
6.9
87.8
83.3
63.0
54.5
49.0
61.0
60.3
shown Figure 7, when the check is placed inside of the safe, the robotic arm often collides with the safe door even if the predicted action is reasonable. Because this, the groundtruth keystep action only achieves 89% success rate. In the multi-task setup, our model also surpasses the late-of-the-an methods with 6.5% absolute gains compared to Hiveformer [3].
Results on 74 tasks. In Table 10, we provide the comparison with Hiveformer [3] on 74 tasks in the single-task and multi-task single- variation setup. We achieve 2.9% absolute gains on average for the single-task learning. The improvement becomes more considerable in the multi-task setup (+11.1), demonstrating the advantages 3D point clouds to solve multiple tasks simultaneously. Our 3D model outperforms the 2D model tasks that require planning, use tools, need precise control, and have visual occlusion.
Rigtt shoulder
Left Srouter
Results on 18 tasks in the multi-task multi-variation setup. In Table 11, we compare with state-of-the-ar methods C2FARM- BC [35], PerAct [14] and Hiveformer [3]. For fair comparison, we add an additional front view camera following [14]. Both C2FARM-BC and PerAct utilize 3D voxels as inputs to train the manipulation policy. Our proposed point cloud based model is more efficient than the two 3D based models. It also improves PerAct by 3.7% on average over 18 tasks. Our 3D-based approach also achieves comparable performance as the state-of-the-and 2D-based approach Hiveformer on the setup which uses histories. It is interesting to further employ histories in our method and unify the 2D and 3D representations in the future work.
Figure 7: Failure case of "Take Money" task due to im- perfect motion planner.
16

---

Table 11: Comparison with state-of-the-aas art methods on 18 tasks with 249 variations in total. All the methods are trained on 100 demonstrations per task. We report the success rate of our method on 25 episodes per task by default. (f denotes our own implementation.)
Open
Slide
Sweep to
Meat off
Turn
Put in
Close
Drag
Stack Blocks
Drawer Block Dustpan
Grill
Tap
Drawer
Jar
Stick
C2FARM-BC [35]
20
16
0
20
68
24
24
0
PerAct [14]
80
72
56
84
80
68
60
68
36
Hiveformer' [3]
52
64
28
100
80
68
52
76
8
PolarNet (Ours)
84
56
52
100
80
32
36
92
4
Screw Bulb
Put in
Place Wine
Put in
Sort Shape
Push Buttons
Insert
Stack
Place Cups
Avg.
Safe
Cupboard
Peg
Cups
C2FARM-BC [35]
12
0
8
0
16
PerAct [14]
24
44
12
16
20
48
0
0
0
42.7
Hiveformer [3]
32
45.3
PolarNet (Ours)
44
84
40
12
12
96
8
0
46.4
Table 12: Comparison of point removal on the single-task and multi-task setups with 10 tasks.
Remove
Pick & Pick-Up
Put Knife
Put
Push
Reach
Slide
Stack
Take Wine Money Umbrella
Take
Avg.
Table Backgroud
Lift
Cup
Money
Button
Target Block
Single-tase single-variants
94.6
89.2
48.5
48.7
99.9
100.0
91.9
80.0
74.9
88.4
81.6
95.1
86.3
71.9
89.9
99.9
100.0
93.5
95.5
72.3
89.8
96.7
91.9
82.5
96.1
99.9
99.9
93.5
94.1
68.6
97.5
92.1
Multi-task single-variie variation
92.2
87.0
75.4
63.6
97.2
100.0
86.6
76.4
73.2
97.0
84.9
92.0
82.6
77.2
95.4
97.4
100.0
88.2
68.2
71.6
86.4
/
97.8
86.0
80.5
94.1
99.6
100.0
93.4
80.5
68.1
97.8
89.8
95
Data efficiency. To demonstrate the data efficiency of our model, reduce the number of demonstrations per task from 100 10 in training. We evaluate the multi-task learning setup with 10 tasks and compare model with the state-of-the-arr 2D-based model Hiveformer [3]. The results in Figure show that our 3D-based approach is also data efficient.
Hiveformer
90
PolanNet (Ours)
65
80
75
Point removal in PolarNet. Table 2 has shown that re- moving irrelevant points in the point cloud processing is critical to the performance. To gain more comprehensive insight into the improvements, we provide more detailed results point removal in Table 12 on both single-task and multi-task setups with 10 tasks. The trend in multi- task setup is similar to that in single-task setup, demon- strating it is beneficial remove irrelevant points from the point cloud. We can that removing background points mainly improves Put Knife, Put Money, Stack Wine and Take Umbrella. In these tasks, the robotic arm rotates a lot at some steps shown in Figure 5, resulting in more background points. Since the background points are geometrically far away from the workspace of the robot, they drastically affect the normalization of the point cloud and make the PointNext encoding less effective. Removing table points bring more improvements for tasks where the contact areas in the objects are small such as Pick-Up Cup and Put Knife. We also observe that all the tasks benefit from the point removal. It would be interesting to investigate automatic point removal in the future.
10 demos
100 demos
Figure Performance using differ- ent numbers of demonstrations per task for training on the multi-task single- variation setup.
17

---

Pateritt
Pacchet
01
13
Noadaa
Narkaa
Norto
320
Figure 9: Comparison of different models on the robustness to camera perturbation. The x-axis the level of perturbation. The y-axis is the relative performance drop compared to the results without any perturbations where the lower the better.
Table 13:
Comparing with an equíva- lent point removal in Hiveformer on the multi-task setup of 10 tasks.
Comparing with Hiveformer with equivalent point re- moval. For 2D-based models such Hiveformer, it is non-trivial remove the background and table pixels from the image compared to the simple approach used in 3D-based PolarNet. Segmentation models are required to mask the background and table. Since we are able to obtain the groundtruth segmentation mask in the simula- tion, we use the groundtruth mask as an upper bound for 2D models. We mask pixels of the background and table as 0 in the image as the imput the model as well as the final predicted heatmap. The results presented in Table 13. We see that removing those pixels does not improve the performance. We conjecture that those pixels might be helpful for the 2D-based model to align multi-view images.
Method
Point removal
Avg.
Hiveformer
X
82.8
73.9
X
84.9
PolarNet
V
89.8
Robustness to viewpoint perturbation. Figure 9 presents the relative performance drop different multi-task policies under different camera perturbation factors for each of the 10 task. We can see that PolarNet is more robust all the 10 tasks than the 2D>bbaed state-of-the-ar model Hive- former, demonstrating the advantage of 3D-based representations. Particularly, PolarNet achieves stable performance for Push Button, Reach Target and Take Umbrella where the relative perfor- mance drops are less than 20% even under the large camera perturbation factor of 30.
D Qualitative Results in Simulation
Success cases. We show successful cases for multi-variation multi-task setting in Figure 10. In drag stick task, the policy is able to use a stick to drag the cube to the colored target given by the language instruction. We observed that policy can adapt to the cube trajectory drift to complete the task. Policy is also able to generalize placement variation such as place wine task. We provide more complex and additional success cases the supplementary video.
Failure cases. In Figure 11 we show different failure cases of our PolarNet. We can observe how the policy fails to insert the ring on the blue spoke due to imprecise position prediction in insert peg task. This task requires fine-grained manipulation which difficult to have when controlling the robot with keysteps. Place cups is a task that can be solved by placing the cups on the tree in any order. Due the multimodality nature of this task, we observe that the robot fails grasp cup going in between two cups. Finally, in some tasks such sort shape, the motion planner fails causing the gripper to collide with other objects in the scene failing to complete the task.
18

---

"drag the block towards the gray square on the table top"
drag stick
"use the stick to drag the cube onto the red target"
"slide the bottle onto the left part of the rack"
place wine
'stack the wine bottle to the middle of the rack"
Figure 10: PolarNet success cases on multi-task multi-variation setting. We show policy running examples of the drag stick and place wine tasks.
Real-robot Experiments
Hardware setup. We conduct real-robot experiments 6-DoF URS robotic equipped with RG6 gripper. Two Intel RealSense D435 RGB-D cameras are the front and lateral sides of the scene as illustrated in Figure 12. To be noted, our gripper is larger than the one used in RLBench, which makes it more challenging to predict precise gripper positions. Furthermore, our robotic arm's 's base is situated besides a column instead of directly on the table, resulting in additional difficulties in motion planning. For example, the robot needs perform substantial rotations to prevent collisions with the column.
Tasks and data collection. We adopt 7 real-world tasks including stack cup, put fruit in box, open drawer, hang mug. put item in drawer, put item in cabinet and put plate on table as illustrated in Figure 13. We randomly place the target objects and multiple distractors on the table for each episode of a task. Due to a large gap between simulated and real environments, we collect a few real-robot demonstrations finetune the policy trained the simulator. Specifically, we use joystick controller manually control the robot. We define the keysteps for the tasks and save RGB-D images and proprioceptive information of the gripper each keystep. Finally, collected 20 demonstrations per task.
19

---

insert peg
imprecise position
place cups
Incorrect grasp
sort shape
collision
Figure 11: PolarNet failure cases. We illustrate failure cases for insert peg, place cups and sort shape.
UR5 robot arm RG6 gripper
Realsense d435
Objects
Figure 12: Robotic setup. Our setup includes RealSense D435 cameras, the objects and the URS robotics arm equipped with a RG6 gripper.
20

---

Stack cup
Put fruit in box
Open drawer
Hang mug
Put item in drawer
Put item in cabinet Put plate on table
Figure 13: Illustration of the real-world tasks.
Training and execution. We finetune the PolarNet trained in the multi-tass setup in RLBench with the collected real-robot data. Though the depth images are noisy in the real-world setup, we do not perform other preprocessing algorithms improve the obtained point cloud except the outlier moving We train NVIDIA V100 GPU for 100K iterations which takes around 9 hours. During evaluation, we simply the last checkpoint from training. We a NVIDIA GeForce GTX 1080 Ti GPU for inference. To move between keysteps, we use Movelt motion planner framework.
'nttp://www httt:::/www.opendd.ooo http:///wwcccoeeee/e/e/////////a outlier_ removal. html
21


Reviews:
[{"review_id": "mMSBr1J7lV", "reviewer": "Reviewer_dUV7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposed a framework for language-guided robot manipulation leveraging 3D point cloud representations and the text embeddings from large pretrained language models. Point-cloud representation is the highlight of this work, as it benefits efficiency, sensor fusion, and accurate 3D localization.", "summary_of_recommendation": "This work is a pioneer in combining language models and RGBD-based manipulations leveraging point cloud features. The proposed framework is simple and reasonable. I believe this work could have major impact in its field. \n\nAfter rebuttal:\n\nThe rebuttal resolved most of my concerns in the review. I'd like to keep my recommendation at weak accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Could you explain more on the variations involved in the experiments?\n\n2. What are the possible ways to enable better generalization?\n\n3. What are the possible ways to remove some requirements on the input other than the images?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Originality:\n\nThe problem and the method are both relatively new. It proposed to use point clouds as the feature space for the manipulation problem with RGBD inputs. It incorporated representative solutions in point cloud learning and large language models and leveraged transformers to fuse cross-modal features. \n\nQuality:\n\nThe proposed technique is presented clearly, correct, and reasonable. The presented work is a complete piece as a conference paper. The experiment section is thorough. It not only compared with existing work, but detailed ablation study in terms of point cloud normalization, combination of input information, point cloud preprocessing (removing irrelevant points), number of camera inputs, and model size are presented. One shortcoming is the lack of real-world experiments. \n\nOverall, while the components are mature results from several relevant fields, this paper presented an effective way of putting them together to fulfill the task, and the details in the realization are also validated through experiments. \n\nClarity:\n\nThe motivation, background, methodology, and experiments are presented mostly clearly. I suggest the author to more clearly explain what the \"variations\" involve, when mentioning \"multi-task multi-variation\", etc. \n\nSignificance:\n\nThe proposed framework for language-guided RGBD-based manipulation is valuable and researchers are likely to build improved solutions based on this method in the future. \n\nRelevance: \n\nThe problem addressed in this paper is of significant practical value and could attract the interest of CoRL's audience. \n\nLimitations:\nThe paper mentioned that this work did not explore the generalization of out-of-distribution scenes, objects, and tasks, which I agree. Also, the method requires a lot of supportive information to work properly, including the removal of irrelevant points, known camera poses, and dense depth images. In my understanding, this is a prototype work incorporating language models and point clouds for manipulation. \n\nStrengths:\n\n1. Proposed to combine LLM and point cloud representation for manipulation. \n\n2. Obtained superior performance in experiments. \n\nWeaknesses:\n\n1. No real-world experiments. \n\n2. Cannot generalize to unseen tasks or scenes despite the application of large language models.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "iYhfcs7O-P", "reviewer": "Reviewer_mm9b", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a policy called PolarNet that takes in 3D point clouds along with language instructions. A merged point cloud obtained from multi-view RGB-D cameras is encoded with a PointNext encoder, and a language instruction is encoded with the CLIP text encoder. These are given to a multi-layer transformer, and a PointNext decoder and an MLP together output the predicted action, which is an end effector pose along with gripper open state. The authors evaluate their approach on RLBench with 10 tasks under different settings, and compare their approach with baseline methods such as PerAct and Hiveformer. They show that their approach performs better and is more computationally efficient. They also run ablation studies of various components of their system.", "summary_of_recommendation": "The approach is interesing and shows good results in simulation but I am not convinced of the applicability to a real system if there is no hardware validation at all.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "While the existing simulation results show the efficacy of this approach, this paper would be greatly strengthened by compelling results on real hardware if that is possible in the revision period.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The authors propose a new method that addresses the weaknesses of both 2D-image-based approaches (lack of precision) and 3D-based approaches (quantization, computation inefficiency)\n- Ablations on the system design regarding the design choices are interesting\n\nWeaknesses:\n- No evaluation on real hardware. While simulation results can show whether an approach is promising, they don't always generalize well to the real world when it comes to perception or policy execution. It's hard to evaluate whether this approach shows promise on a real system.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "6oY7dS17Rka", "reviewer": "Reviewer_2cU3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a language-guided manipulation model that directly takes pointclouds and the authors show that a single model can handle multiple tasks with variations on the RLBench benchmark. It takes CLIP language embeddings and PointNext embeddings and uses a transformer to cross-attend for latent point feature to predict the next action with heatmap voting. The model was mainly compared with HiveFormer, a transformer-based approach with RGBD inputs, and PerAct, a voxel-based approach and showed better performance.", "summary_of_recommendation": "(Updated) I recommend a weak accept to the paper. A PointNet-based encoder is more efficient than the voxel-based approach and it may have a few advantages over the 2D or voxel-based approaches. After the rebuttal, the contribution from background/table removal became clear and showed the benefit of 3D model with camera perturbation along with a real-world experiment.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* How good the proposed method without background and table removal at the multi-task setup?\n* Could you specify the variances of the tasks and provide statistics of category of variances?\n* What would be the benefits of using the model that takes 3D pointclouds instead of RGBD images or voxels?\n  * Would it be possible to show some experimental results that pointcloud-models can be useful than models with different input modalities?\n* It is not clear what the caption of Figure 4 explains.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Strength\n* The paper is generally well-organized and clear to understand.\n* The proposed method is easy to understand.\n* Showed an elegant way of predicting action instead of discretizing the action space.\n* The paper claims to introduce the first language-guided manipulation model that takes 3D pointcloud.\n* Significantly compact/faster than the voxel-based approach of comparison.\n* Detailed ablation tables are provided.\n\n# Weakness\n* Hard to understand the actual contribution of the model from the experiment.\n  * For instance, Table 2 shows 10.5 percentage point difference with and without background/table removal. Without the removal, the proposed approach is outperformed by HiveFormer by 6.8 percentage point in the single task. No further information is provided for the multi-task setup. Based on the numbers in Table 2, seems like **the table removal contributed to the performance significantly**. For a fair comparison, **the empirical result of the approach without any removal should be included** or the equivalent removal should be provided to the 2D/3D models that are compared with.\n  * In addition to that, it is hard to understand the types of variations in the task. For instance, if most of them are about background colors, the manual removal benefits more and relative contribution of the approach might be reduced.\n* Hard to understand the benefit of using 3D pointcloud within the limited problem setup.\n  * Without handling occlusion, it was hard to understand the benefit of using directly 3D pointcloud if it only applies to the setup with a table-top with fixed cameras. For instance, the approach that understands 3D geometry can be more robust with viewpoint variances than the 2D models as pointcloud embedding models are permutation invariant.\n  * In comparison to the voxel-based approach, the proposed model shows similar constraints or limitation as those of the voxel-based \napproach. It uses a down-sampled pointcloud with known a bounding-box and a table volume.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "2gpAJSCGbPM", "reviewer": "Reviewer_ABTj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces PolarNet, a point-cloud and language conditioned network for robot manipulation. It is based on a PointNext network to encode the point cloud, obtained by three cameras (right-left shoulder + wrist), and a Transformer to compute features taking into account both the encoded point cloud and the tokenized language command.\nThe authors compare their method with other state of the art methods on RLBench, a popular benchmark for robot learning in simulation, in different settings and with different tasks. Their experiments demonstrate in general comparable performance to PerAct and Hiveformer, two state of the art method which also use a combination of 2D and 3D inputs. \nAdditionally, the authors provide a series of ablations, analyzing many design choices involving the input space modality and point cloud processing, among the others.", "summary_of_recommendation": "I'm on the edge, and I will certainly discuss this with the other reviewers and the Area Chair after the rebuttal phase. While the results are convincing, the novelty is limited, and the paper feels quite similar in design to the baselines they mention. The thorough evaluation and ablations of their design choices are a plus, but there is also a lack of real world experiments. \n**UPDATE**: after the rebuttal, I updated my score to weak accept.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "I invite the authors to better position the novelty of their paper with respect to the existing baselines, that also use point clouds in a way or another. The paper can be strong even without the need for big novelty, but it is important to justify the strong claims made in the introduction. \nWhile I realise it may be difficult to achieve during the short rebuttal phase, a few real world experiments may also make the paper stronger.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The paper performs a large series of experiments and ablations that can be useful for the reader, and any researcher that is designing point cloud-based robot manipulation.\n- Results are strong, and although not superior to baselines in a statistically significant way, the reduction in computational complexity with respect to e.g. PerAct is still interesting.\n\nWeaknesses:\n- Claiming that the paper is the first to use a point cloud and language conditioned network may be excessive: while they are not entirely based on that, Hiveformer also uses point clouds in their network computation, and PerAct also uses voxelised point clouds, which are quite similar.\n- The paper is not very clear in some parts, especially in the description of the network computation and its final output. If a reader is not familiar with the literature that uses point clouds, and the RLBench convention, it may be hard to follow at times.\n\nI found it to be an interesting paper but the novelty of it may not be particularly significant, considering also that the results are not very different from Hiveformer. The point cloud is encoded through an existing architecture, PointNext, and the use of Transformers to manipulate tokens of different modalities has been explored several times. \nI do believe however that the series of ablations are interesting and, although not all provide statistically significant results, they make the paper stronger.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers felt that the paper presented a new method for a 3D point cloud policy with language instructions.  Reviewers appreciated the extensive experiments and ablations, including the recently added real-world experiments. Other parts of the paper were also greatly improved in the rebuttal.
Decision: Accept (Poster)