Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation
Authors: Theophile Gervet
Zhou Xian
Nikolaos Gkanatsios
Katerina Fragkiadaki
Venue: CoRL 2023 Poster (2023)
Keywords: Learning from Demonstrations
Manipulation
Transformers

Abstract: 3D perceptual representations are well suited for robot manipulation as they easily encode occlusions and simplify spatial reasoning. Many manipulation tasks require high spatial precision in end-effector pose prediction, which typically demands high-resolution 3D feature grids that are computationally expensive to process. As a result, most manipulation policies operate directly in 2D, foregoing 3D inductive biases. In this paper, we introduce Act3D, a manipulation policy transformer that represents the robot’s workspace using a 3D feature field with adaptive resolutions dependent on the task at hand. The model lifts 2D pre-trained features to 3D using sensed depth, and attends to them to compute features for sampled 3D points. It samples 3D point grids in a coarse to fine manner, featurizes them using relative-position attention, and selects where to focus the next round of point sampling. In this way, it efficiently computes 3D action maps of high spatial resolution. Act3D sets a new state-of-the-art in RLBench, an established manipulation benchmark, where it achieves 10% absolute improvement over the previous SOTA 2D multi-view policy on 74 RLBench tasks and 22% absolute improvement with 3x less compute over the previous SOTA 3D policy. We quantify the importance of relative spatial attention, large-scale vision-language pre-trained 2D backbones, and weight tying across coarse-to-fine attentions in ablative experiments.

Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation
Theophile Gervet".
Zhou Xian
Nikolaos Gkanatsios
Katerina Fragkiadaki
Maachnne Learning Department Robotics Institute
School of Computer Science
Carnegie Mellon University
(tgervet, xianzl, ngkanats, katef @cs. cmu. cmu. edu
act3d. act3d.github. github. io
Abstract: 3D perceptual representations are well suited for robot manipulation as they easily encode occlusions and simplify spatial reasoning. Many manipu- lation tasks require high spatial precision in end-effector pose prediction, which typically demands high-resolution 3D feature grids that computationally pensive to process. As result, most manipulation policies operate directly in 2D, foregoing 3D inductive biases. In this paper, we introduce Act3D, a manipula- tion policy transformer that represents the robot's workspace using a 3D feature field with adaptive resolutions dependent on the task at hand. The model lifts 2D pre-trained features to 3D using sensed depth, and attends to them to com- pute features for sampled 3D points. It samples 3D point grids in a coarse fine manner, featurizes them using relative-position attention, and selects where focus the next round of point sampling. In this way, it efficiently computes 3D action maps of high spatial resolution. Actīl) sets a new state-of-the-art in RL- Bench, an established manipulation benchmark, where it achieves 10% absolute improvement over the previous SOTA 2D multi-view policy on 74 RLBench tasks and 22% absolute improvement with 3x less compute over the previous SOTA 3D policy. We quantify the importance of relative spatial attention, large-scale vision-language pre-trained 2D backbones, and weight tying across coarse-to-fine attentions in ablative experiments. Code and videos available project site: https://accta: gi thub. io/.
Keywords: Learning from Demonstrations, Manipulation, Transformers
Introduction
Solutions to many robot manipulation tasks can be modeled as a sequence of 6-DoF end-effector poses (3D position and orientation). Many recent methods train neural manipulation policies to predict 3D end-effector pose sequences directly from 2D images using supervision from demon- strations [1, 2, 3, 5, 6). These methods are typically sample inefficient: they often require many trajectories to handle minor scene changes at test time and cannot easily generalize across camera viewpoints and environments, as mentioned in the respective papers and shown in our experiments.
For robot policy to generalize under translations, rotations, or camera view changes, needs to be spatially equivariant [7], that is, map 3D translations and rotations of the input visual scene similar 3D translations and rotations for the robot's end-effector. Spatial equivariance requires predicting 3D end-efffcctor locations through 2D or 3D action maps, depending the action space considered, instead of regressing action locations from holistic scene or image features. Transporter
Equal contribution
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Environment
3D Feature Cloud
Coanse-io-ine 3D feature Teids
Action Prediction
Opcs
Ded Toet
Instuctioo
CUP
Oocipied DAR'T FASINA
Peplocpplee
Nioe azezn to kouah dizs 2000
(Grot Qupen Real?
Linguge
Figure I: Act3D is a iiguaae--onditionn robot action transformer that learns 3D scene feature fields of arbitrary spatial resolution via recurrent coarse-to-fine 3D point sampling and featurization using relative-position attentions. Act3D featurizes multi-view RGB images with a pre-trained 2D CLIP backbone and lifts them in 3D using sensed depth. It predicts 3D location of the end-effector using classification of the 3D points of the robot's workspace, which preserves spatial equivariance of the scene to action mapping.
networks [8] introduced a spatial equivariant architecture for 4-DoF robot manipulation: they re- project RGB-D input images to a top-down image and predict robot end-effector 2D translations through a top-down 2D action map. They showed better generalization with fewer training demon- strations than prior works. However, they are limited to top-down 2D worlds and 4-DoF manipula- tion tasks. This begs the question: how can we extend spatial equivariance in action prediction to general 6-DoF manipulation?
Developing spatially equivariant 6-DOF manipulation policies requires predicting 3D action maps by classifying 3D points in the robot's workspace as candidates for future 3D locations for the robot's end-effector. Predicting high-resolution 3D action maps, necessary for fine- fine-grained manip- ulation tasks, poses a computational challenge over their 2D counterparts due to the extra spatial dimension. Voxelizing the robot's 3D workspace and featurizing the 3D voxels at high resolution is computationally demanding [9]. The next end-effector pose might be anywhere in free space, which prevents the use of sparse convolutions [10, 11] to selectively featurize only part of the 3D free space. To address this, recent work of PerAct [1] featurizes 3D voxels using the latent set bottle- necked self-attention operation of Perceiver [12], whose complexity is linear to the number of voxels opposed to quadratic, the all-to-all self attention operations. However, it gives up spatial disentanglemen of features due to the latent set bottleneck. Other methods avoid featurizing points in 3D free space altogether and instead regress an offset for the robot's 's 3D locations from a detected 2D image contact point [2, 13, 14), which again does not fully comply with spatial equivariance.
In this paper, we introduce Act3D, language-conditioned transformer for multi-task 6 DoF robot manipulation that predicts continuous resolution 3D action maps through adaptive 3D spatial com- putation. Act3D represents the scene a continuous 3D feature field. It computes scene-level physical 3D feature cloud by lifting features of 2D foundational models from one or more views using sensed depth. It learns 3D feature field of arbitrary spatial resolution via recurrent coarse- to-fine 3D point sampling and featurization. At each iteration, the model samples 3D points in the whole workspace and featurizes them using relative spatial cross-attention [15] to the physical 3D) feature cloud. Act3D predicts 3D end-effector locations by scoring 3D point features, and then regresses the 3D orientation and opening of the end-effector. At inference time, we can trade-off compute for higher spatial precision and task performance by sampling more 3D points in free space than the model ever saw at training time.
We test Act3D in RLBench [16], an established benchmark for learning diverse robot manipulation policies from demonstrations. We set a new state-of-the-art in the benchmark in both single-task and multi-task settings. Specifically, we achieve 10% absolute improvement over prior SOTA on the single-task setting introduced by HiveFormer [2] with 74 tasks and a 22% absolute improvement over prior SOTA in the multi-task setting introduced by PerAct [1] with 18 tasks and 249 variations.

---

We also validate our approach on a Franka Panda with multi-task agent trained from scratch on 8 real-world tasks with a total of just 100 demonstrations (see Figure 2). In thorough ablations, we show the importance of the design choices of our architecture, specifically, relative spatial attention, large-scale vision-language pre-trained 2D backbones, high resolution featurization and weight tying across coarse-to-fin attentions.
In summary, our contributions are: 1. A novel neural policy architecture for language-condttioee multi-task 6-DoF manipulation that both reasons directly in 3D and preserves locality of computa- tion in 3D, using iterative coarse-to-fine vaanlation-invariant attention. 2. Strong empirical results on a range of simulated and real-world tasks, outperforming the previous SOTA 2D and 3D methods on RLBench by large absolute margins, and generalizing well to novel camera placements at test time. 3. Thorough ablations that quantify the contribution of high-resolution features, tied attention weights, pre-trained 2D features, and relative position attention design choices.
2 Related Work
Learning robot manipulation from demonstrations Many recent work train multi-task manip- ulation policies that leverage Transformer architectures [1, 2, 3, 5, 17, 18] to predict robot actions from video input and language instructions. End-to-end image-to-action policy models, such as RT- 1 [5]. GATO [18], BC-Z [19], and InstructRL [3], directly predict 6-DoF end-effector poses from 2D video and language inputs. They require many thousands of demonstrations to learn spatial rea- soning and generalize to new scene arrangements and environments. Transporter networks [8] and their subsequent variants [20, 21, 22] formulate 4-DoF end-effector pose prediction as pixel classi- fication in 2D overhead images. Thanks to the spatial equivariance of their architecture, their model dramatically increased sample efficiency over previous methods that regress end-effector poses by aggregating global scene features. However, they are limited to top-down 2D planar worlds with simple pick-and-place primitives. 3D policy models of C2F-ARM [4] and PerAct [1] voxelize the robot's workspace and are trained to detect the 3D voxel that contains the next end-effector key- pose. Spatially precise 3D pose prediction requires the 3D voxel grid to be high resolution, which comes at a high computational cost. C2F-ARM [4] uses coarse-to-fine voxelization to handle computational complexity, while PerAct [1] uses Perceiver's latent bottleneck [12] to avoid voxel- to-voxel self-attention operations. Act3l avoids 3D voxelization altogether and instead represents the scene continuous resolution 3D feature field. It samples 3D points in the empty workspace and featurizes them using cross -ross-attentions to the physical 3D point features.
Feature pre-training for robot manipulation Many 2D policy architectures bootstrap learning from demonstrations from frozen or finetuned 2D image backbones [23, 24, 19, 25] to increase experience data sample efficiency. Pretrained vasion-language backbones can enable generalization new instructions, objects, and scenes [26, 21J. In contrast, SOTA 3D policy models are typically trained from scratch from colored point clouds input [1, 4, 27]. Act3D
CLIP pre- pre-trained 2D backbones [28] to featurize 2D image views and lifts the 2D features in 3D using depth [29, 30]. We show that 2D feature pretraining gives a considerable performance boost over training from scratch.
Relative attention layers Relative attentions have shown improved performance in many 2D vi- sual understanding tasks and language tasks [31, 32). Rotary embeddings [33] implement relative attention efficiently by casting it as an inner-product in an extended position feature space. In 3D, relative attention is imperative as the coordinate system is arbitrary. 3D relative attentions have been used before in 3D Transformer architectures for object detection and point labelling [34, 35). We show in Section that relative attentions significantly boost performance of our model.
3
3D Feature Field Transformers for Multi-Task Robot Manipulation
The architecture of Act3D is shown in Figure 1. It is a policy transformer that, at given timestep predicts 6-DoF end-effector pose from one more RGB-D images, language instruction,
3

---

and proprioception information regarding the robot's current end-effector pose. Following prior work [36, 1, 2, 3]. instead of predicting an end-effector pose at each timestep, we extract a set of keyposes that capture bottleneck end-effector poses in a demonstration. A pose is keypose if (1) the end-effector changes state (something grasped or released) or (2) velocities approach near zero (a common occurrence when entering pre-grasp poses or entering new phase of task). The prediction problem then boils down predicting the next (best) keypose action given the current observation. At inference time, Act3D iteratively predicts the next best keypose and reaches it with sampling-based motion planner, following previous works [1, 2].
We assume access to a dataset n demonstration trajectories. Each demonstration is a sequence of observations O = {01, .z.... o paired with continuous actions A {@1, 02, Oe and, optionally, language instruction 1 that describes the task. Each observation on consists of RGB-D images from one camera views; more details are in Appendix 7.2. An action a consists of the 3D position and 3D orientation (represented quaternion) of the robot's end-effector, its binary open or closed state, and whether the motion planner needs to avoid collisions to reach the pose:
a = {Apos 1R3,, Arot E IHI, aopen E 10,11,.00 Gcol E [0,1]]
Arot
Next, we describe the model's architecture in detail.
Visual and language encoder Our visual encoder maps multi-view RGB-D images into a multi- scale 3D scene feature cloud. We use large-scale pre-trained 2D feature extractor followed by feature pyramid network [37] to extract multi-scale visual tokens for each camera view. Our input is RGB-D, each pixel is associated with depth value. We "lift" the extracted 2D feature vectors to 3D using the pinhole camera equation and the camera intrinsics, based on their average depth. The language encoder featurizes instructions with a large-scale pre-trained language encoder. We use the CLIP ResNet50 [28] visual encoder and language encoders to exploit their common vision- language feature space for interpreting instructions and referential grounding. Our pre-trained visual and language encoders are frozen, not finetuned, during training of Act3D.
Iterative 3D point sampling and featurization Our key idea is to estimate high resolution 3D action maps by learning 3D perceptual representations of free space with arbitrary spatial resolution, via recurrent coarse-to-fine 3D point sampling and featurization. 3D point candidates (which we will call ghost points) are sampled, featurized and scored iteratively through relative cross- attention [15] to the physical 3D scene feature cloud, lifted from 2D feature maps of the input image views. We first sample coarsely across the entire workspace, then finely in the vicinity of the ghost point selected the focus of attention in the previous iteration, as shown in Figure 1. The coarsest ghost points attend to global coarse scene feature cloud, whereas finer ghost points attend to a local fine scene feature cloud.
Relative 3D cross-attentions We featurize each of the 3D ghost points and a parametric query (used to select via inner-product one of the ghost points as the next best end-effector position in the decoder) independently through cross-attentions to the multi-scale 3D scene feature cloud, language tokens, and proprioception. Featurizing ghost points independently, without self-attentions one another, enables sampling more ghost points at inference time to improve performance, as we show in Section 4. Our cross- aosssnattnntoon use relative 3D position information and are implemented effi ciently with rotary positional embeddings [15]. The absolute locations of our 3D points are never used in our featurization, and attentions only depend on the relative locations of two features.
Decoding actions We score ghost point tokens via inner product with the parametric query to select one the next best end-effector position apos- We then regress the end-effector orientation arot and opening aopen. as well as whether the motion planner needs avoid collisions to reach the pose Gool, from the last iteration parametric query with 2-layer multi-laye layti-layer perceptron (MLP).

---

sboc
money
miccle
block'
bulam chowar
cubs onto lar Arr
the the duss insido
disickohh the spilled coffee
fuil bow?
"stack
transfer the yellow bowl the plate"
Figure 2: Tasks. We conduct experiments on 92 simulated tasks in RLBench [16] (only 10 shown), and real-world tasks (only 5 shown).
Training Act3D trained supervised from input-action tuples from dataset of manipulation demonstrations. These tuples are composed of RGB-D observations, language goals, and keypose actions {(o1, 4, k,), (02, 12, ...). During training, randomly sample tuple and supervise Act3D to predict the keypose action l given the observation and goal (o, l). We supervise position prediction apos at every round of coarse-to-fing with a softmax cross-entropy loss over ghost points, rotation prediction ayot. with a MSE loss on the quaternion prediction, and binary end-effector open- ing Oopen and whether the planner needs to avoid collisions acol with binary cross-entropy losses.
Implementation details We use three ghost point sampling stages: first uniformly across the entire workspace (roughly meter cube), then uniformly in 16 centimeter diameter ball, and finally in 4 centimeter diameter ball. The coarsest ghost points attend to global coarse scene feature cloud (32x32xnçam coarse visual tokens) whereas finer ghost points attend to local fine scene feature cloud (the closest 32x32xneam out of the total 128x128xXcanm fine visual tokens). During training, we sample 1000 ghost points in total split equally across the three stages. At inference time, we can trade-off extra prediction precision and task performance for additional compute by sampling more ghost points than the model ever saw at training time (10, 000 in our experiments). We'll show in ablations in Section 4 that framework is robust these hyper-parameters but tying weights across sampling stages and relative 3D cross coos-attention are both crucial for generalization. We use batch size 16 on Nvidia 32GB V100 GPU for 200k steps (one day) for single-task experiments, and a batch size 48 on 8 Nvidia 32GB V100 GPUs for 600K steps (5 days) for language-conditione multi-task experiments. At test time, we call upon a low-level motion planner to reach predicted keyposes. In simulation, we use native motion planner implementation provided in RLBench, which a sampling-basee BiRRT [38] motion planner powered by Open Motion Planning Library (OMPL) [39] under the hood. For real-world experiments, use the same BiRRT planner provided by the Movelt! ROS package [40]. please, see Appendix 7.4 for more details.
Experiments
We test Act3D learning from demonstrations single-task and multi-task manipulation policies in simulation and the real world. We conduct our simulated experiments in RLBench [16], an established simulation benchmark for learning manipulation policies, for the sake of reproducibility and benchmarking. Our experiments aim answer the following questions:

---

Auto-Lambca (100)
HiveFormer (100) InstruciRL (100) ActtD (10) Act3D (100)
Success (%)
Average
Planningg
Long
Rosation
Planning
Precsion
union
Figure 3: Single-task performance. On RLBench tasks across 9 categories, Act3l) reaches 83% success rate, an absolute improvement of 10% over InstructRI. [3], prior SOTA in this setting.
Perfect (10)
Perkat (103)
ActDD
(100)
open drawer
slide blockk
sweep to dustpan
meat off grill
turn
put in
close Jer
drag stick
stack blocks
Grawer
Method
10
100
10
100
100
10
100
10
100
100
100
10
100
10 100
PerAct
68
80
32
72
72
56
68
84
72
80
16 68
32
60
36
68
12
36
Success Rate (N)
Act3D
92
93
64
94
92
screw bulb
put
place wine
put
push buitons
insert
place
sale
cupboorr
shape
pog
cups
cupes
10
10K
10
10K
100
10
100
10
100
100
10
100
10
100
10
100
Act3D
26
47
75
95
32
27
51
7
98
9
Average
Figure 4: Multi-task performance. On 18 RLBench tasks with 249 variations, Act3D reaches 65% success rate, an absolute improvement of 22% over PerAct [1], prior SOTA in this setting.
How does Act3D compare against SOTA 2D multiview and 3D manipulation policies in single- task and multi-task settings with varying number of training demonstrations?
2. How does Act3D generalize across camera viewpoints compared to prior 2D multiview policies?
How do design choices such as relative 3D attention, pre-trained 2D backbones, weight-tied attention layers, and the number of coarse-to-fine sampling stages impact performance?
4.1 Evaluation in simulation
Datasets We test Act3D in RLbench in two settings: 1. Single-task manipulation policy learn- ing. We consider 74 tasks grouped into categories proposed by HiveFormer [2]. Each task includes variations which test generalization to novel arrangements of the same training objects. Each method trained with 100 demonstrations and evaluated on 500 unseen episodes. Multi-task manipu- lation policy learning. We consider tasks with 249 variations proposed by PerAct [1]. Each task includes 2-60 variations, which test generalization to new goal configurations that involve novel ob- ject colors, shapes, sizes, and categories. This is more challenging setting. Each method is trained with 100 demonstrations per task split across variations, and evaluated on 500 unseen episodes per task.
Baselines We compare Act3D with the following state-of-the-art manipulation policy learning methods: InstructRL [3], 2D policy that directly predicts 6 DoF poses from image and language conditioning with pre-trained
backbone. 2. PerAct [1], 3D policy that voxelizes the workspace and detects the next best voxel action through global self-attention. 3. HiveFormer [2] and Auto-A [13], hybrid methods that detect a contact point within an image input, then regress an offset from this contact point. We report numbers from the papers when available.
Evaluation metric We evaluate policies by task completion success rate, the proportion of execu- tion trajectories that lead to goal conditions specified in language instructions.

---

Single-task and multi-task manipulation results We show single-task quantitative results of our model and baselines in Figure Act3D reaches 83% success rate, an absolute improvement of 10% over InstructRL [3], prior SOTA this setting, and consistently outperforms it across all 9 categories of tasks. With only 10 demonstrations per task, Act3D is competitive with prior SOTA using 100 demonstrations per task. Act3D outperforms 2D methods of InstructRL and Hiveformer because it reasons directly in 3D. For the same reason, it generalizes much better than them novel camera placements, as we show in Table 3.
We show multi-task quantitative results of our model and PerAct in Figure 4. Act3D reaches 65% success rate, an absolute improvement of 22% over PerAct, prior SOTA in this setting, consistently outperforming it across most tasks. With only 10 demonstrations per task, Act3D outperforms PerAct using 100 demonstrations per task. Note that Act3D also uses less than third of PerAct's training computation budget: PerAct trained for 16 days on Nvidia V100 GPUs while train for 5 days on the same hardware. Act3D outperforms PerAct because its coarse-to-fine relative attention based 3D featurization of the 3D workspace is more effective than the perceiver's 's latent bottleneck attention in generating spatially disentangled features.
4.2 Evaluation in real-world
In our real-world setup, we conduct experiments with a Franka Emika Panda robot and a single Azure Kinect RGB-D sen- sor. We consider 8 tasks (Figure 2) that involve interactions with multiple types objects, spanning liquid, articulated ob- jects, and deformable objects. For each task, collected 10 to 15 kinesthetic demonstrations and trained languaged- conditioned multi-task model with all of them. We report the success rate on 10 episodes per task in Table 1. Act3D can cap- ture semantic knowledge in demonstration well and performs reasonably well on tasks, even with single camera input. One major failure case comes from noisy depth sensing: when the depth image is not accurate, the selected point results in imprecise action prediction. Leveraging multi-view input for error correction could improve this, and we leave this for future work. For videos of the robot executing the tasks, please see our project website.
Task
# Train
Success
reach target
10
10/10
duck in oven
15
6/10
wipe coffee
15
7/10
fruits in bowl
10
8/10
stack cups transfer beans
15
6/10
15
5/10
press handsan
10
10/10
uncrew cap
10
8/10
Table 1: Real-world tasks.
4.3 Ablations
We ablate the impact of our design choices in Table 3. We perform most ablations in the single-task setting on 5 tasks: pick cup, put knife on chopping board, put money in safe, slide block to target, take umbrella out of stand. We ablate the choice of pre-trained 2D backbone the multi-task setting with all 18 tasks.
Generalization across camera viewpoints: We vary camera viewpoints at test time for both Act3D and HiveFormer [2]. The success rate drops to 20.4% for HiveForrmrr, a relative 77% drop, while Act3D achieves 74.2% success rate, 24% relative drop. This shows detecting actions in 3D makes Act3I) more robust to camera viewpoint changes than multiview 2D methods that regress offsets.
Weight-tying and coarse-to-fine sampling: All 3 stages of coarse-to-fine sampling are neces- sary: a model with only 2 stages of sampling and regressing an offset from the position selected at the second stage suffers 4.5% performance drop. Tying weights across stages and relative 3D positional embeddings are both crucial; we observed severe overfitting without, reflected in respec- tive 17.5% and 42.7% performance drops. Fine ghost point sampling stages should attend to local fine visual features with precise positions: all stages attending to global coarse features leads to 8.3% performance drop. Act3D can effectively trade off inference computation for performance:
7

---

Table 2: Ablations.
Average success rate in single-task setting (5 tasks)
Full Act3D
98.1
Only 2 stages of coarse-to-fine sampling
93.6
No weight tying across stages
80.6
Core design choices
Absolute 3D positional embeddings
55.4
Attention to only global coarse visual features
89.8
Only 1000 ghost points at inference time
93.2
Act3D HiveFormer
74.2
Viewpoint changes
20.4
Multi-task setting (18 tasks)
CLIP ResNet50 backbone
65.1
Backbone
ImageNet ResNet50 backbone
53.4
sampling 10,000 ghost points, instead of the 1,000 the model was trained with, boosts performance by 4.9%.
Pre-training 2D features: We investigate the effect of the pre-trained 2D backhone in the multi-
task setting where language instructions are most needed. A ResNet50 [28] backbone pre-trained with CLIP improves success rate by 8.7% over ResNer50 backbone pre-trained on ImageNet.
For additional ablations regarding augmentations and sensitivity to hyperparameters, please see the Appendix section 7.6. We found Random crops of RGB-D images boost performance but yaw rotation perturbations did not help. The model is robust variations in hyperparameters such the diameter of ghost point sampling balls or the number of points sampled during training.
4.4 Limitations and future work
Our framework currently has the following limitations: 1. Act3D is limited by the motion planner used to connect predicted keyposes with straight trajectory segments. It does not handle manip- ulation of articulated object well, such as opening/closing doors, fridges, and ovens, where robot trajectories cannot be well approximated by few line segments.2. Act3D does not utilize any de- composition tasks into subtasks. hierarchical framework that would predict language subgoals policy would allow better re-
for subtasks [41, 42, 43] and feed those our language-conditnn usability of skills across tasks. Addressing these limitations is direct avenue for future work.
Conclusion
We presented Act3D, a language-conditione policy transformer that predicts continuous resolution 3D action maps for multi-task robot manipulation. Act3D represents the scene using continuous resolution 3D feature map, obtained by coarse-to-fine 3D point sampling and attention-based fea- turization. Act3D sets mark, and solves diverse manipulation tasks in the real world from single RGB-D camera view and handful of demonstrations. Our ablations quantified the contribution of relative 3D attentions, 2D) feature pre-training, and weight tying during coarse-to-fine iterations.
state-of-the-art in RLBench, established robot manipulation bench-
Acknowledgements
This work is supported by Sony AI, NSF award No 1849287. DARPA Machine Common Sense, an Amazon faculty award, and an NSF CAREER award.

---

References
[1] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799, PMLR, 2023.
[2] P.-L. Guhur, Chen, R. G. Pinel, M. Tapaswi, I. Laptev, and Schmid. Instruction-driven history-aware policies for robotic manipulations. In Conference Robot Learning, pages 175-187. PMLR, 2023.
[3] H. Liu, L.. Lee, K. Lee, and P. Abbeel. Instruction-followinng agents with jointly pre-trained vision-language models. arXiv preprint arXiv:2210.13431, 2022.
[4] S. James, K. Wada, T. Laidlow, and A. J. Davison. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13739-13748, 2022.
[5] A. Brohan, N. Brown, Carbajal. Y. Chebotar, J. Dabis, Finn, Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint vrXiv:2212.068177 2022.
[6] M. Sieb, Z. Xian, Huang, O. Kroemer, and K. Fragkiadaki. Graph-structured visual imita- tion. Conference on Robot Learning, pages 979-989, PMLR, 2020.
[7] X. Zhu, D. Wang, O. Biza, G. Su, R. Walters, and R. Platt. Sample efficient grasp learning using equivariant models, 2022.
[8] A. Zeng, P. Florence, J. Tompson, S. Welker, Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, pages 726-747. PMLR, 2021.
[9] H.-Y. F. Tung, Z. Xian, M. Prabhudesai, Lal, and K. Fragkiadaki. 3d-oes: Viewpoint- invariant object-factorized environment simulators. arXiv preprini urXiv:201 1.004604,,06444,, 2020.
[10] B. Graham. Sparse 3d convolutional neural networks, 2015.
[11] C. Choy, J. Gwak, and S. Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks, 2019.
[12] A. Jaegle, F. Gimeno, A. Brock, A. Zisserman, O. Vinyals, and J. Carreira. Perceiver: General perception with iterative attention, 2021.
[13] S. Liu, S. James, A. Davison, and E. Johns. Auto-lambda: Disentangling dynamic task relationships. arXiv preprint arXiv:2202.03091, 2022.
[14] P. Parashar, J. Vakil, S. Powers, and C. Paxton. Spatial- attention policies for efficient robot learning. arXiv preprint arXiv:2304. 11255, 2023.
[15] J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXhv:2104.09864, 2021.
[16] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. IEFE Roboties and Automation Letters, 5(2):3019-3026, 2020.
[17] N. M. Shafiullah, Z. Cui, A. A. Altanzaya, and L. Pinto. Behavior transformers: Cloning k modes with one stone. Advances in neural information processing systems, 35.22955-22968, 2022.
[18] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikow, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint drXiv:2205.06175, 2022.

---

[19] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learn- ing. pages 991-1002. PMLR, 2022.
[20] D. Seita, Florence, J. Tompson, E. Coumans, V. Sindhwani, K. Goldberg, and A. Zeng. Learning to rearrange deformable cables, fabries, and bags with goal-conditioned transporter networks, 2021.
[21] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Conference on Robot Learning, pages 894 906. PMLR, 2022.
[22] N. Gkanatsios, A. Jain, Z. Xian, Y. Zhang, C. Atkeson, and K. Fragkiadaki. Energy- based models as zero-shot planners for compositional scene rearrangement. arXiv preprint arXiv:2304. /439/, 2023.
[23] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation, 2022.
[24] S. Parisi, A. Rajeswaran, S. Purushwalkam, and A. Gupta. The unsurprising effectiveness of pre-trained vision models for control, 2022.
[25] L.. Yen-Chen, A. Zeng, S. Song, Isola, and T.-Y. Lin. Learning to see before learning to act: Visual pre-training for manipulation, 2021.
[26] A. Stone, Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong. P. Wohlhart, B. Zitkovich, Xia, Finn, and K. Hausman. Open-world object manipulation using pre-trained vision- language models, 2023.
[27] Z. Xian, S. Lal, H.-Y. Tung, E. A. Platanios, and K. Fragkiadaki. Hyperdynamics: Meta- learning object and agent dynamics with hypernetworks. arXiv preprint arXiv:2103.09439, 2021.
[28] A. Radford. J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021.
[29] F. Tung, R. Cheng, and K. Fragkiadaki. Learning spatial common sense with geometry- aware recurrent networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2595-2603, 2019.
[30] A. W. Harley, S. K. Lakshmikanth, Li, X. Zhou, H.-Y. F. Tung, and K. Fragkiadaki. Learn- ing from unlabelled videos using contrastive predictive neural 3d mapping. arXiv preprint arXiv: 1906.9966... 037644 2019.
[31] Shaw, Uszkoreit, and A. Vaswani. Self-attention with relative position representations, 2018.
[32] Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong, F. Wei, and B. Guo. Swin transformer v2: Scaling up capacity and resolution, 2022.
[33] J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding, 2022.
[34] X. Wu, Y. Lao, L. Jiang, X. Liu, and Zhao. Point transformer v2: Grouped vector attention and partition-based pooling, 2022.
[35] Y.Q.. Yang, Y.-X. Guo, J.-Y. Xiong, Y. Liu, H. Pan, P.-S. Wang, X. Tong, and B. Guo. Swin3d: A pretrained transformer backbone for 3d indoor scene understanding, 2023.
10

---

[36] S. James and A. J. Davison. Q-attention: Enabling efficient learning for vision- based robotic manipulation. IEEE Robotics and Automation Letters, 7(2::1112-1619, 2022.
[37] T.Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2117-2125, 2017.
[38] Kuffner and S. M. LaValle. Rrt-connect: An efficient approach to single-query path planning. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37063), volume 2, pages 995-1001. IEEE, 2000.
[39] I. Sucan, M. Moll, and L. E. Kavraki. The open motion planning library. IEEE Robotics & Automation Magazine, 19())72-82, 2012.
[40] D. Coleman, I. Sucan, S. Chitta, and N. Correll. Reducing the barrier to entry of complex robotic software: a moveit! case study, arXiv preprint arXiv: 1404.3785. 2014.
[41] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
[42] W. Huang, F. Xia, D. Shah, D. Driess, A. Zeng, Y. Lu, P. Florence, I. Mordatch, S. Levine, K. Hausman, et al. Grounded decoding: Guiding text generation with grounded models for robot control. arXiv preprint arXiv:2303.00855 2023.
[43] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Tex22motion: From natural language instructions feasible plans, arXiv preprint arXiv:2303.12153 2023.

---

7 Appendix
7.1 Real-world Setup
Our real-robot setup contains Franka Panda robotic arm equipped with parallel jaw grip- per, as shown in Figure 5. We get RGB-D in- put from a single Azure Kinect sensor at front view at 30Hz. The image input is of resolu- tion 1280 720, we crop and downsample 256 256. We calibrate the extrinsics of the camera with respect the robot base us- ing the haadeye ROS package. We ex- tract keyposes from demonstrations in the same was as in simulation. Our real-world multi-task policy is trained on 4 V100 GPUs for days, and we run inference on desktop with a single RTX4090 GPU. For robot control, we use the open-source frankapy' package to send real- time position-control commands to the robot.
Franka Emika
Azure Kinect
7.2 RLBench Simulation Setup
To ensure fair comparison with prior work, we use Ream E {3, 4} cameras for simulated ex- periments depending on the evaluation setting. In our single-task evaluation setting first pro- posed by HiveFormer [2],
Figure 5: Real-world setup.
the same 3 cameras they do {Oleft: Oright w In our multi-task evaluation setting first proposed by PerAct [1]. we use the same cameras they do (Oron Oleft Oright w
Figure 6: RLbench simulation setup.
12

---

7.3
RLBench Tasks
Task
Variation Type
# of Variations
Avg. Keyposes
Language Template
open drawer
placement
3.0
"opeen the drawer"
slide block
color
4.7
"slide the block to target"
sweep dustpan
size
2
4.6
"sweep dirt the dustpan"
meat off grill
category
2
5.0
"take the off the grill"
turn tap
placement
2
2.0
"turn tapp
put in draver
placement
3
12.0
"put the item in the drawer"
close jar
color
20
6.0
"close the - jar"
drag stick
color
20
6.0
"use the stick to drag the cube onto the target"
stack blocks
color, count
60
14.6
"stack - blocks"
screw bulb
color
20
7.0
"screw
- light bulb" "put the money away in the safe on shelf"
put in safe
placement
3
5.0
place wine
placement
3
5.0
"stack the wine bottle the the rack"
put in cupboard
category
9
5.0
"put the - in the cupboard
sort shape push buttons
shape
5
5.0
"put the in the shape sorter"
color
50
3.8
"push the button, [then the button]
insert peg
color
20
5.0
"put the ring the spoke
stack cups
color
20
10.0
"stack the other cups on top of the cup"
place cups
count
3
11.5
"place cups on the cup holder"
Figure 7: PerAct [1] tasks. We adopt the multi-task multi-variation setting from PerAct [I] with 18 tasks and 249 unique variations across object placement, color, size, category, count, and shape.
We adapt the single-task setting of HiveFormer [2] with 74 tasks grouped into categories according to their key challenges. The 9 task groups are defined as follows:
The Planning group contains tasks with multiple sub-goals (e.g. picking basket ball and then throwing the ball). The included tasks are: basketball in hoop, put rubbish in bin, meat off grill, meat on grill, change channel, tv on, tower3, push buttons, stack wine.
The Tools group is special case of planning where robot must grasp object to interact with the target object. The included tasks are: slide block to target, reach and drag, take frame off hanger, water plants, hang frame on hanger, scoop with spatula, place hanger on rack, move hanger, sweep to dustpan, take plate off colored dish rack, screw nail.
The Long term group requires more than 10 macro-steps to be completed. The included tasks are: wipe desk, stack blocks, take shoes out of box, slide cabinet open and place cups.
The Notation-invarian group can solved without changes in the gripper rotation. The included tasks are: reach target, push button, lamp on, lamp off, push buttons, pick and lift, take lid off saucepan.
The Motion planner group requires precise grasping. As observed in [81] such tasks often fail due to the motion planner. The included tasks are: toilet seat down, close laptop lid, open box, open drawer, close drawer, close box, phone base, toilet seat up, put books on bookshelf.
The Multimodal group can have multiple possible trajectories to solve a task due to large affordance area of the target object (e.g. the edge of a cup). The included tasks are: pick cup, turn tap, lift numbered block, beat the buzz, stack cups.
The Precision group involves precise object manipulation. The included tasks are: take usb out computer, play jenga, insert onto square peg, take umbrella out of umbrella stand, insert usb in computer, straighten rope, pick and lift small, put knife on chopping board, place shape in shape sorter, take toilet roll off stand, put umbrella in umbrella stand, setup checkers.
The Screw group requires screwing an object. The included tasks are: turn oven on, change clock, open window, open wine bottle.
The Visual Occlusion group involves tasks with large objects and thus there are occlusions from certain views. The included tasks are: close microwave, close fridge, close grill, open
13

---

grill, unplug charger, press switch, take money out safe, open microwave, put money in safe, open door, close door, open fridge, open oven, plug charger in power supply
7.4 Further Architecture Details
Relative 3D cross-attentions We featurize each of the 3D ghost points and parametric query (used to select via inner-product one of the ghost points as the next best end-effector position in the decoder) independently through cross-attentions to the multi-scale 3D scene feature cloud, lan- guage tokens, and proprioception. Featurizing ghost points independently, without self-attentions to one another, enables sampling more ghost points at inference time to improve performance, as we show in Section 4. Our cross-attentions use relative 3D position information and are implemented efficiently with rotary positional embeddings [15].
Given point p (ar, 2) R and its feature TRd. the rotary position encoding function PE is defined as:
M
PE(p, M(p)x=
x
(1)
Ma/6
cos w0k
sin 20g
0
0
sin z0n
cos x0k
0
0
cos y0
sin y0x
0
0
M
0
0
sin y0u
cos y0x
0
0
0
cos 20k
sin 20
0
O
0
sin 20,
20k
where 0s 1000gack 1372, k (l....d/6) The dot product of two positionally encoded features is PE(p,, PE(P, x,) x (P.) (((P.))M((P))) (x,,,)x,- x M(P, -Pi)x,
(3)
which depends only on the relative positions of points p; and Pj-
We extract two feature maps per 256x256 input image view: 32x32 coarse visual tokens and 128x128 fine visual tokens. We use three ghost point sampling stages: first uniformly across the entire workspace (roughly meter cube), then uniformly in a 16 centimeter diameter ball, and fi- nally in centimeter diameter ball. The coarsest ghost points attend a global coarse scene feature cloud (32x32xneam coarse visual tokens) whereas finer ghost points attend local fine scene feature cloud (the closest 32x32xncam out of the total 128x 128xncam fine visual tokens). During training, we sample 1000 ghost points in total split equally across the three stages. At infer- ence time, can trade-off extra prediction precision and task performance for additional compute by sampling more ghost points than the model ever saw at training time (10, 000 in our experiments). We'll 'll show in ablations in Section 4 that our framework is robust to these hyper-parameters but tying weights across sampling stages and relative 3D cross-attention are both crucial for generalization. We use 2 layers of cross-attention and an embedding size 60 for single-task experiments and 120 for multi-task experiments. Training samples are augmented with random crops of RGB-D images and +45.0 yaw rotation perturbations (only in the real world as this degrades performance in simulation as we'll show in Section 4). The cropping operation is performed on aligned RGB and depth frames together, thus maintain pixel-level correspondence. We use a batch size 16 on Nvidia 32GB V100 GPU for 200k steps (one day) for single-task experiments, and a batch size 48 on 8 Nvidia 32GB V100 GPUs for 600K steps (5 days) for iuguage-conditiones multi-task experiments. At test time, we call low-level motion planner to reach predicted keyposes. simulation, we use native mo- tion planner implementation provided RLBench, which sampling-bases BiRRT [38] motion planner powered by Open Motion Planning Library (OMPL) [39] under the hood. For real-world experiments, the same BiRRT planner provided by the Movelt! ROS package [40].
14

---

Wrist
Global Fine Feature Cloud
Average (x.yzz) of Token Pixels
Feature Pyramid Network
Flight
Frozen 2D Backbone
Lefft
with 3x1 28x128 tokens
Global Coarse Feature Cloud
with 3x32x32 tokens
keep only two feature maps coarse of size 32x32 fine of size 128x128
input images size 256x256
feature maps of size 128x128, 64x64, 32x32, 16x16
Figure 8: Scene Feature Cloud Generation. We encode each image independently with a pre- trained and frozen vision backbone get multi-scale feature maps, pass these feature maps through feature pyramid network and retain only two: coarse feature map (at a granularity that lets ghost points attend to all tokens within GPU memory) and fine feature map (as spatially precise as afforded by input images and the backbone). We lift visual tokens from these two feature maps for each image to 3D scene feature clouds by averaging the positions of pixels each 2D visual token.
ghost query crose-attend to the global coarse feature cloud independently
parametric
local ghost points and the parametric query cross-attend feature cloud inclependnnll
fine
Relatvv 3D Cross Attention Encoder
Relative 3D Cross Attemtion Encoderr
Detection Treneformer Decoder
Detection Trantformer Decoder
Ghost Embedding MAMMNROO
Ghost Point Embedcing MAMMAROM
Ghost Point Embedding MARRMRAA
Ghost Point Embedding
Parametric Query
Parametric Query
Parametric Quary
Parametric Query
Global Coarse Foature Cloud
Local Fine Feature Cloud
the parametric query scores each ghost point embedding by product; scoring paint as the center of the next round of ghost point sempling; sampla local ghost points and select local fine festure cloud
repeat this for many rounds of coarse-to-fin torre--to-fine ghost point sampling as needed to obtain the racessery precision
top
with 3x32x32 takens
Figure 9: Iterative Ghost Point Sampling, Featurization, and Selection.
7.5 High Precision Experiments
In this section, we further investigate the ability of Act3D to improve over existing 3D methods that voxelize the workspace for high-precision tasks. We compare two variants of Act3D against PerAct [1] on three high-precisio tasks in success rate. The first Act3D variant is the standard architecture used in the remainder our experiments operating on 256x256 input image views: the second operates higher resolution 512x512 input image views, from which extracts four times many visual tokens with more precise 3D positions. This further tests the ability of Act3D to provide high precision by processing higher-resolution RGB-D views at the cost of extra compute.
15

---

1=1
Action Predction
Figure 10: Iterative Ghost Point Sampling, Featurization, and Selection.
Method
insert peg
sort shape
screw nail
PerAct
16
31
12
Act3D (256x256)
29
31
Act3D (512x512)
47
43
55
Act3D improves over PerAct on high precision tasks and can further benefit from higher resolution RGB-D images, at the cost of extra compute.
7.6 Further ablations
Augmentations: Random crops of RGB-D images boost success rate by 6.5%, but yaw rotation perturbations drop it by 11.9%. This is in line with PerAct [1] results in RLBench.
Hyperparameter sensitivity: Act3l) robust to variations in hyperparameters. Doubling the di-
ameter ghost point sampling balls from (16 cm, 4 em) to (32 cm, cm) drops success rate by 1.5% and halving to em, em) by 6.9%. Halving the total number of ghost points sampled from 1,000 to 500 drops success rate by 2.3% whereas doubling it 2,000 increases success rate by 0.3%. We use 1, 1,000 ghost points in our experiments to allow training with single GPU per task.
16

---

Table 3: Ablations.
Average success rate in single-task setting (5 tasks)
Model
Best Act3D model (evaluated in Fig. 3)
98.1
Only 2 stages of coarsc-to fine sampling:
93.6
full workspace, 16 ball, regress an offset No weight tying across stages
Core design choices
80.6
Absolute 3D positional embeddings Attention only global coarse visual features Only 1000 ghost points inference time
55.4
89.8
93.2
Best Act3D model (evaluated in Fig. 3) HiveFormer
74,2
Viewpoint changes
20.4
No image augmentations
91.6
Augmentations
With rotation augmentations
86.2
Double sampling ball diameters: 32 and cm and cm
96.6
Halve sampling ball diameters:
91.2
Hyperparameter sensitivity
500 ghost points at training time
95.8
2000 ghost points at training time (need 2 GPUs)
98,4
Multi-task setting (18 tasks)
CLIP ResNer50 backbone
65.1
Backbone
ImageNet ResNet50 backbone
53,4
No backbone (raw RGB)
45.2
17


Reviews:
[{"review_id": "pK4GZWDz-35", "reviewer": "Reviewer_C6Eb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a novel manipulation policy transformer that leverages multi-view camera information and selects the end-effector action (SE(3) position and some other actions). The policy first extracts features from image observation and language goal with pre-trained models, projects the features to the scene point cloud, then evaluates sampled ghost points in the X-Y-Z- action space using a transformer in a coarse-to-fine style. Lastly, regress the other actions. Experiments indicate the transformer outperforms baselines in both single-task and multi-task in RLBench. Finally, real-world experiments verified the capability of the transformer in practice.", "summary_of_recommendation": "This paper proposes a manipulation policy transformer to address two issues in previous SOTA method PerAct, which are computation burden and action descretization granularity. The method effectively achieves better performance with less computation. However, the advantage of high-resolution is not explicitly demonstrated.\n\nUpdates after rebuttal:\nI appreciate authors' explanations and additional experiments. I keep my current rating (weak accept). These new experiments support the high resolution contributes to the task success rate. However, it remains a question that what could improve the task success rate from ~65%  to 100%.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. The major novelty ghost point evaluation, i.e., Detection Transformer decoder in line 141 is poorly presented. This is main learnable neural network in the method as well. But its architecture is unclear. Please add more description.\n2. Another concern is the method supports high-resolution, but some of the experiments (as mentioned in weaknesses) does not indicate the benefit of the high-resolution. It would be nice to have experiment show the benefit of high-resolution.\n3. In the ablations (Table 2), when the image backbone is removed from the method, performance dropped to 45.2%, which is similar to that of PerAct (43%). Does this mean the task successes rate advantage mainly comes from the pre-trained vision model?\n4. It is also worth mention how the ghost point is sampled.\n5. Minor: some figures are not properly referenced, for example Fig.4 is not mentioned in Multi-task manipulation results.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality, clarity, originality and significance of this work:\nThe major novelty is leverage ghost points instead of voxels (in PerAct) representing action candidates to reduce computation burden and improve resolution. Moreover, this work combines multiple idea in the literature (image and language pre-training, relative position embedding, key-point estimation, coarse-to-fine evaluation). Together, these novelties contribute to manipulation policy learning research.\nOverall the structure of the paper is well-organized.\nStrengths:\nPoint-cloud based transformer and pre-trained vision model effectively improves task success rate and reduces computation overhead.\nWeaknesses:\nInsufficient description for “Detection Transformer decoder”. \nAs described in the title, the method supports high-resolution. Nevertheless, the method does not achieves signification improvement in precision tasks, i.g., Precision in Figure. 3 and “sort shape”, “insert peg” in Figure. 4.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "eJySZ7bhyWv", "reviewer": "Reviewer_hSNU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "\nRecently keypoint-based manipulation has seen increasing attention in robotics, where a neural network (typically a transformer) is trained to predict a sequence of end-effector keypoints to be used as motion planner goals. Prior approaches suffer from poor spatial resolution or errors due to the scene representation to the transformer (e.g. voxels or images), of which voxels additionally impose a n^3 runtime. Ideally, the representation used should be sparse, and this paper suggests a featurized pointcloud. Unlike voxels, a point-cloud is only defined on surfaces, but end-effector poses are almost always in free space. Thus, a point cloud does not offer input tokens for the transformer at positions corresponding to output poses. This paper iteratively samples, computes and refines refines a set of featurized ghost points in free space to enable output of end-effector poses in continuous free space, not suffering from the limited resolution or view-dependency of prior methods", "summary_of_recommendation": "It is a good paper that proposed a useful method and is substantiated by sufficient experiments and ablations.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "generally the paper is good enough as-is and I would recommend acceptance without any additional work. It is easy to read, conveys a clear message, and is not unnecessarily overstuffed.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n- Solves a real problem in keypoint-based manipulation\n- Strong results on a standard benchmark for the task\n- The paper includes convincing RLBench experimental results showing significant improvements over prior single-task and multi-task keypoint based manipulation policies. Real-world experiments showing success rates across a set of tasks are also included, showing the ability to learn from limited data in real world.\n\nCons (not really cons, but thought I’d mention):\n- the main drawback is the problem formulation and not the paper - keypoint based manipulation is not able to solve dynamic dexterous manipulation tasks. However, the paper should not be penalized for it, since it is still very useful for a large amount of real world tasks, and is able to learn from limited data - a key requirement for real-world applicability.\n- Requires a calibrated setup with known camera poses. Again, this is currently a standard assumption in related work and still practical in uses like industrial work cells.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "YWnVlqR7xb", "reviewer": "Reviewer_4R1b", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces Act3D, a novel manipulation policy transformer, for performing 6D robot manipulation tasks given natural language description and multi-view RGB-D images of the workspace. The method uses a coarse-to-fine sampling method to create a 3D feature cloud of candidate end-effector poses for the task, which is later queried to predict the keyposes required to complete the task. A motion planner is used to find collision-free motion plans to connect these keyposes. The proposed method outperforms the state-of-the-art policy transformer methods on the RLBench robot manipulation benchmark for performing single and multi-task manipulation tasks.", "summary_of_recommendation": "The manuscript in its current form requires significant improvements. It fails to motivate the problem correctly and does not convey the crucial ideas about the approach comprehensively. It is unclear what are the main technical contributions of the work. Hence, I am inclined to reject the manuscript.\n\n**Post-rebuttal update:**\nThe authors have addressed my concerns satisfactorily. I appreciate the significant improvement in the quality of the manuscript. I have updated my recommendation accordingly.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. In the included video, excessive end-effector rotations were observed in some simulated experiments, e.g., \"screw the red light bulb\", and \"put the item in the bottom drawer\". Please discuss the observation. \n2. Please clarify the technical contributions of the work.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies an important problem in robot manipulation and proposes a novel solution to address it. However, the manuscript requires significant improvement in writing. In its current form, it fails to motivate the problem correctly and does not convey the crucial ideas about the approach comprehensively. It is unclear what are the main technical contributions of the work. Hence, I am inclined to reject the manuscript.\n\nMore detailed feedback is as follows:\n- Please add examples of \"... scene features...\" referred to in line 28 for more clarity.\n- It is unclear what is meant by the phrases \"... predicted on 3D points in free space...\" and \"...3D physical scene points\" in lines 31-32. Consider rephrasing the sentence for better readability.\n- Consider adding examples and more context with the phrase \"2D backbones and back-project to 3D\" in lines 35-37. Please rephrase line 37 for better readability.\n- It is difficult to follow the argument being presented in lines 39-40. Please rephrase.\n- In line 103 it is stated that Act3D predicts s 6-Dof end-effctor pose at each time step. However, in line 111 it is said that \"Following prior work [11, 1, 2, 3], instead of predicting an end-effector pose at each timestep, we extract a set of keyposes that capture bottleneck end-effector poses in a demonstration\", which seems to contradict the previous statement. Please clarify.\n- Please add further details on the motion planner used in the method.\n- Consider describing how the \"2D visual tokens are lifted to 3D by interpolating depth values\" in the text.\n- Consider adding more details on how the end-effector orientation a_{ori} and opening/closing action a_{open} are predicted.\n- In the included video, excessive end-effector rotations were observed in some simulated experiments, e.g., \"screw the red light bulb\", and \"put the item in the bottom drawer\". Please discuss the observation.\n- In the single-task and multi-task manipulation results in sections (lines 193-209), consider discussing which components of the proposed approach helped it to outperform SoTA methods for a better emphasis on the merits of the approach.\n- Consider adding evaluations to a more diverse set of baselines (e.g. [1], [2], [3]) to clearly demonstrate the effectiveness of the method.\n- Typographical:\n  - map -> maps [line 119]\n  - No line number is available for the paragraph between lines 117 and 118. Please check the paper format.\n  - Consider describing a range of suffix 'k' in the equation between lines 139-140.  Also, please add an equation number for easy reference.\n  - Rephrase \"...which test generalization to test goal...\" in line 200.\n  - Consider replacing the phrase \"... robust to hyperparameters\" --> \"robust to variations in hyperparameters\".\n\nReferences:\n- [1] Brohan, Anthony, et al. \"Rt-1: Robotics transformer for real-world control at scale.\" arXiv preprint arXiv:2212.06817 (2022).\n- [2] Driess, Danny, et al. \"Palm-e: An embodied multimodal language model.\" arXiv preprint arXiv:2303.03378 (2023).\n- [3] Lin, Kevin, et al. \"Text2motion: From natural language instructions to feasible plans.\" arXiv preprint arXiv:2303.12153 (2023).", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "Pe3e7iweAPz", "reviewer": "Reviewer_fEwK", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces Act3D, a manipulation policy Transformer designed to address the computational challenges associated with 3D representations. It proposes a novel approach by casting 6-DoF key pose prediction as 3D detection with adaptive spatial computation. It takes in multi-view RGB-D observations, language instruction, and proprioception data of the current end-effector as inputs to generate action in the form of end-effector pose, gripper state, and a binary describing the collision allowance. From the input RGB-D observations, it generates 3D feature clouds and employs iterative coarse-to-fine sampling of 3D point grids in free space. These grids are then featurized using relative spatial attention to the physical feature cloud, allowing Act3D to select the optimal end-effector position. The results in the paper indicate that Act3D achieves improvement over the previous SOTA with only one-third of the computational resources compared to the previous SOTA method.", "summary_of_recommendation": "I believe the work would be valuable to the community. I would consider increasing my recommendation if the questions for the rebuttal phase are addressed.\n\n**Post-rebuttal update:**\nThe authors have addressed my concerns satisfactorily. I appreciate the significant improvement in the quality of the manuscript. My recommendation to accept this paper remains.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Can the authors elaborate on 3D feature cloud generation in more detail?\n- Can the authors elaborate on the augmentation with random crops?\n- Can the authors explain the significance of the parametric query? And how this changes for every iteration?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strengths:**\n- The contribution of the paper - multi-task robot manipulation policy learning in 3D - addresses an important challenge of spatial precision required in manipulation tasks. \n- While some previous methods have addressed this challenge, this paper tackles the problem with lesser computational requirements and fewer amounts of data.  \n- The method is more general by predicting the key poses of the end-effector toward performing a task. It can easily plug the advancements in motion planning and trajectory optimization research areas into the pipeline.\n- Reported results substantiate the contributions listed. \n\n**Weaknesses:**\n- In the current version of the paper, 3D Feature cloud generation that lifts the 2D visual tokens into 3D needs to be clarified. Particularly the usage of the features from the feature pyramid and if multiple 3D points will hold the same features in this process. An additional diagram to illustrate this will be beneficial, as this is a crucial part of the proposed method. \n- In the current version of the paper, the description of augmentation with random crops needs to be clarified, as the association between RGB and depth will likely be lost. \n- As the robot is part of the observation space, the developed method is catered to the setup and the specific robot.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present Act3D, a method for 6D language-conditioned robot manipulation from multi-view RGB images. Compared to similar sparse-trajectory (open-loop) style approaches, the author’s course-to-fine method is able to infer actions continuously on SE(3), overcoming the inherent limitation of discretization of other approaches.

Owing to strong real-world results on benchmarks that matter, and a clear and concise manuscript (after revisions), reviewer sentiment was uniformly in favor of acceptance. Likewise, the authors presented a strong rebuttal, with additional experiments, a re-written introduction and clarifications on points raised by the reviewers.

A number of reviewers raised a weakness inherent to this style of approach; that it is not appropriate for high-rate dexterous tasks or (as the empirical results suggest) does not outperform baselines on tasks with high precision. However, such tasks are likely outside the scope of this work and the strong results on the evaluated tasks against fair baselines was deemed sufficient for acceptance, and generally for this paper to be of interest to the community.
Decision: Accept (Poster)