RVT: Robotic View Transformer for 3D Object Manipulation
Authors: Ankit Goyal
Jie Xu
Yijie Guo
Valts Blukis
Yu-Wei Chao
Dieter Fox
Venue: CoRL 2023 Oral (2023)
Keywords: 3D Manipulation
Multi-View
Transformer

Abstract: For 3D object manipulation, methods that build an explicit 3D representation perform better than those relying only on camera images. But using explicit 3D representations like voxels comes at large computing cost, adversely affecting scalability. In this work, we propose RVT, a multi-view transformer for 3D manipulation that is both scalable and accurate. Some key features of RVT are an attention mechanism to aggregate information across views and re-rendering of the camera input from virtual views around the robot workspace. In simulations, we find that a single RVT model works well across 18 RLBench tasks with 249 task variations, achieving $26\%$ higher relative success than the existing state-of-the-art method (PerAct). It also trains 36X faster than PerAct for achieving the same performance and achieves 2.3X the inference speed of PerAct. Further, RVT can perform a variety of manipulation tasks in the real world with just a few ($\sim$10) demonstrations per task. Visual results, code, and trained model are provided at: https://robotic-view-transformer.github.io/.

RVT: Robotic View Transformer for 3D Object Manipulation
Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, Dieter Fox NVIDIA
Abstract: For 3D object manipulation, methods that build an explicit 3D rep- resentation perform better than those relying only camera images. But using explicit 3D representations like voxels comes at large computing cost, adversely affecting scalability. In this work, we propose RVT, multi-view transformer for 3D manipulation that is both scalable and accurate. Some key features of RVT are an attention mechanism aggregate information across views and re-rendering of the camera input from virtual views around the robot workspace. In simula- tions, find that single RVT model works well across 18 RLBench tasks with 249 task variations, achieving 26% higher relative success than the existing state- of-the-art method (PerAct). It also trains 36X faster than PerAct for achieving the same performance and achieves 2.3X the inference speed of PerAct. Further, RVT can perform a variety of manipulation tasks in the real world with just a few (~10) demonstrations per task. Visual results, code, and trained model are provided at: https:/troootic-iieww http:://oboiieeiiiier.rrrmrrmmmp transformerrgihuub..o..
Keywords: 3D Manipulation, Multi- Transformer
Introduction
fundamental goal of robot learning is to build systems that can solve various manipulation tasks in unconstrained 3D settings. A popular class of learning methods directly processes image(s) viewed from single or multiple cameras. These view-based methods have achieved impressive success on variety of pick-and- pllceecc and object rearrangement tasks [1, 2, 3,4. However, their success on tasks that require 3D reasoning has been limited. As shown by James al. [5] and Shridhar al. [6], view-based methods struggle at 3D manipulation tasks on RLBench [7] with less than 2% success.
To address this, methods have been proposed that rea- son with explicit 3D representations of the scene. C2F- ARM [5] represents the scene with multi-resolution voxels and achieves strong performance on difficult RLBench tasks. PerAct [6] improves upon C2F-ARM in behavior cloning by using perceiver transformer [8] to process voxels. However, creating and reasoning over voxels comes at a higher computing cost com- pared to reasoning over images, since the number of voxels scales cubicly with the resolution as opposed squarely for image pixels. This makes voxel-based methods less scalable compared their view-based counterparts. In fact, training PerAct on 18 RLBench tasks takes 16 days using 8 V100 GPUs (3072 GPU hours). This hinders fast development and prototyp- ing. Moreover, such computing requirements become even more prohibitive when scaling to larger datasets with more tasks and diversity.
1.26X
60
36X Faster
Avg. Success
40
30
20
10
RVT
PerAct
0
4 6 8 1011 1213141516
Training Time (in Days)
Figure 1: RVT scales and performs better than PerAct on RLBench, achieving on- par performance in 36X less time (same hardware), and 1.26X peak performance.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Hence, key question is - can we build manipulation network that not only performs well but also inherits the scalability of view-based methods? To this end, we propose RVT (Robotic View Transformer) that significantly outperforms the SOTA voxel-based method both in terms of success rate and training time, as shown in Fig. 1. With the same hardware, RVT achieves the peak per- formance of PerAct 36X less time, decreasing the training time from 14 days to just 10 hours. Apart from being much faster to train, RVT also achieves 26% higher success rate than PerAct, averaged over 18 tasks (249 task variations)
RLBench. RVT outperforms PerAct on 88.9% % of tasks on RLBeecch while achieving 2.3X the inference speed (11.6 vs 4.9 fps). Further, we find that RVT also works well in the real world, where with only 51 demonstrations, single RVT model can learn to perform a variety of manipulation tasks (5 tasks, 13 variations) like opening a drawer, placing objects on shelf, pressing hand sanitizer, and stacking objects (see Fig. 4).
At its core, RVT is view-based method that leverages the transformer architecture. jointly attends over multiple views of the scene and aggregates information across the views. It then produces view-wise heatmaps and features that are used to predict robot end-effector pose. We extensively explore the design of the multi-view architecture and report several useful findings. For example, we observe a better performance when enforcing the transformer to first attend over patches within the same image before concatenating the patches for joint attention.
Another key innovation is that, unlike prior view-based methods, decouple the camera images from the images fed to the transformer, by re-rendering the images from virtual views. This allows us to control the rendering process and leads to several benefits. For example, we can re-render from viewpoints that are useful for the task (e.g., directly above the table) while not being restricted by real-world physical constraints. Also, since the multi-view input to RVT is obtained via re-rendering, we can use RVT even with single sensor camera as done our wealdworld experiments.
To summarize, contributions are threefold: first, we propose RVT, multi-view transformer for 3D object manipulation that is accurate and scalable; second, we investigate various design choices for the multi-view transformer that lead to better object manipulation performance; and finally, we present an empirical study for multi-task object manipulation in simulation and the real world.
2 Related Work
Vision-based Object Manipulation. The learning of robotic control policy has been tradition- ally studied with low-dimensional state observations [9. 10, 11, 12, 13]. Recently, vision-based policies [14, 15, 16, 17, 18, 19, 20, 21] have gained increasing attention since the high-dimensional visual sensory input provides more generalizable observation representation across tasks and is more accessible in real-world perception systems. Various forms of visual input have been explored. Prior work has directly encoded the RGB images into low-dimensional latent space and relied on model- based [22, 23] or model-free [24, 25] reinforcement learning (RL) to train policies to operate in this space. More recently, RT-1 [26] infers the robot's actions from a history of images by leveraging transformer architectures [27]. Our proposed RVT also uses transformer to predict actions, how- ever, unlike RT-1, we additionally leverage depth to construct a multi-view scene representation. The use of depth input has also been extensively studied. Methods like CLIPort [3] and IFOR [1] directly process the RGB-D images for object manipulation, and hence are limited to simple pick- and-place tasks 2D top-down settings. To overcome this issue, explicit 3D representations such as point clouds have been utilized. C2F-ARM [5] and PerAct [6] voxelize the point clouds and use a 3D convolutional network as the backbone for control inference. However, high-precision tasks typically require high resolution of voxelization, resulting in high memory consumption and slow training. Our approach falls into this category but addresses the scalability issue by transforming the point cloud into a set of RGB-D images from multiple views. We show that this significantly im- proves memory footprint and training efficiency, and leads to higher performance when compared to directly working with RGB(-D) or point cloud input (see Table. 1). Another relevant work is MIRA [28], which also uses novel view images to represent the 3D scene for action inference. MIRA achieves this by implicitly constructing neural radiance field representation (NeRF) of the scene

---

Multi-View Transformer
Sensor Point Cloud Vintull Cameras
Virtual Images
Predicted Healmaps
Prediction
Overview of RVT. Given RGB-D from sensor(s), we first construct point cloud of the scene. The point cloud is then used to produce virtual images around the robot workspace. The virtual images are fed to a multi-view transformer model to predict view-specific features, which are then combined predict action in 3D.
Figure 2:
from a set RGB images and then generating novel view images from the optimized NeRF model. However, the requirement of optimizing a scene NeRF model slows down the inference speed at test time and relies on RGB images from dense set of views as input. contrast, our approach can achieve significantly faster inference speed and can work with even single-view RGB image.
Multi-Task Learning in Robotics. Learning single model for many different tasks has been of particular interest to the robotics community recently. A large volume of work achieves the multi-task generalization by using a generalizable task or action representation such as object point cloud [18, 19), semantic segmentation and optical flow [1], and object-centric representation [29, 30]. However, the limited expressiveness of such representations constrains them only generalize within task category. Task parameterization [31, 32] and discrete task-dependent output layer [33, 34] approaches are investigated with reinforcement learning to learn policies for tasks in different categories. With the recent breakthrough in large language models, multi-task robot learning has been approached by using natural language to specify a broad range of tasks and learning the policy from large pre-collected datasets [35, 26, 36, 2, 37, 38, 39, 40, 41]. We are inspired by this success propose learn language-conditioned multi-tase policies with small demonstration dataset.
Transformers for Object Manipulation. The success of transformers in vision and NLP has led its way into robot learning [42, 43, 44, 17, 45, 46]. Especially in object manipulation, transformer- based models with an attention mechanism can be utilized to extract features from sensory inputs to improve policy learning [47, 48, 49, 50, 51]. Unlike most prior work, we do not use large datasets for training. RVT efficiently learns from a small set of demonstrations, handle multiple views as visual inputs, and fuses information from language goals to tackle multiple manipulation tasks.
Multi- View Networks in Computer Vision. Multi-view representations have been explored in various vision problems. For point cloud recognition, Simple [52] showed how simple view- based method outperforms sophisticated point-based methods. Follow-up works like MVTN [53] and Voint cloud [54] further improved upon SimpleView, Multi-view representations have also been used for other problems like 3D visual grounding [55], view synthesis [56] and depth prediction [57]. Unlike them, we focus on the problem of predicting robot actions for object manipulation.
3 Method
Our goal to learn a single model that can complete a wide range of manipulation tasks. The input consists of (1) a language description of the task, (2) the current visual state (from RGB-D camera(s)), and (3) the current gripper state (open or closed). The model should predict an action, specified by target end-effector pose and gripper state the next key-frame. The key-frames represent important or bottleneck steps of the gripper during the task execution [58], such pre- pick, grasp, or place pose. Given a target end effector pose, we assume low-level motion planner

---

and controller that can move the end effector to the target pose. To train the model, we assume dataset D = {D1, D2,.... Dn} of n expert demonstrations covering various tasks is given. Each demonstration D; = (foi...m; ), language description of the task, fof. 0h. ".,, )n
1,4)) is a successful roll-out of length m,, where I; is the sequence of the observations from RGB-D camera(s) with gripper state, and {aj 02: .... am, is the sequence of corresponding robot actions. seeeee
This demonstration dataset can be used to train models with behavior cloning.
Our proposed method (RVT) is transformer model [27] that processes images re-rendered around the robot workspace, produces an output for each view, and then back-projects into 3D to predict gripper pose actions, as shown in Fig. 2.
Rendering. The first step is the re-rendering of camera input. Given the RGB-D image(s) captured by one or multiple sensor cameras, we first reconstruct point cloud of the scene. The point cloud is then re-rendered from set of virtual viewpoints anchored in the space centered at the robot's 's base (see Fig. 2 and Fig. 3). Specifically, for each view, we render three image maps with a total of 7 channels: (1) RGB (3 channels). (2) depth (1 channel), and (3) (x, y, 2) coordinates of the points in the world frame (3 channels). The (Ir, y, z) coordinates help establish the correspondence of pixels across views, i.e., pixels from different views share the same (ar, y, 2), they correspond to the same point in 3D. We use PyTorch3D [59] for rendering. We empirically verify various design choices in our rendering pipeline (see Tab. 2 (left)).
The re-rendering process decouples the input images to the ones fed to the transformer. This offers several benefits such as: the ability to re-render at arbitrary and useful locations (c.g., directly above the table) while not being constrained by real-world camera placements; multi-view reasoning even with a single sensor camera; allowing the use of orthographic images instead of generally provided perspective ones; facilitating 3D point-cloud augmentations and enabling additional channels like point correspondence which are not natively presented in the sensor images. We empirically find that these contribute to achieving high performance with view-based networks (see Sec. 4.1).
Joint Transformer. The re-rendered images, the language description of the task, and the gripper state (open or close) are processed by a joint transformer model (see Fig. Al in the appendix). For language, we use pretrained CLIP [60] embeddings (ResNet-50 variant), which provide one token for each word. For the virtual images, we break each of them into 20 20 patches and pass through a multi-laye perceptron (MIP) to produce image tokens, similar to ViT [61]. For the gripper state, similar to PerAct [6], we pass it through an MLP and concatenate it to the image tokens. We also add positional embeddings to all the image and language tokens to preserve the positional information.
Overall, RVT has eight self-attention layers. In the first four layers, an image token is only allowed to attend to other tokens from the same image. This biases the network to process individual images first before sharing information across images. We concatenate all the image tokens along with the language tokens afterward. In the last four layers, we allow the attention layers to propagate and accumulate information across different images and text. Finally, the image tokens are rearranged back to the original spatial configuration, resulting in the feature channels of each image.
Action Prediction. The model outputs 8D action, including the 6-DoF target end effector pose (3-DoF for translation and 3-DoF for rotation), 1-DoF gripper state (open or close), and binary indicator for whether to allow collision for the low-level motion planner (see [6] for details). For translation, we first predict a heatmap for each view from the per-image features from the joint trans- former (as shown in Fig. A1 in the appendix). The heatmaps across different views are then back- projected to predict scores for discretized set of 3D points that densely cover the robot workspace (see See. A.3 in the appendix). Finally, the end effector translation is determined by the 3D point with the highest score. Note that this multi-view heatmap representation for translation prediction extends prior approaches in the 2D towndown view setting [4]. Hence, RVT inherits the benefit of su- perior sample efficiency by representing the visual input and action in the same spatial structure [4].
For end effector rotation, we follow PerAct to use the Huler angles representation, where each angle is discretized into bins 5" resolution. The gripper state and the motion planner collision indicator are represented as binary variables. To predict the rotations, gripper state, and collision indicator,

---

Avg. Success
Avg. Rank
Train time
Inf. Speed
Close Jar
Drag Stick
Insert
Meat ofl
Open Drawer
Place
Place
Models
(in days)
(in (ps)
Pog
Grill
Cups
Wine
Image-BC (CNN) [2, 6]
1.3
3.7
0
0
0
0
4
0
0
Image-BC (VIT) 6]
3.8
0
C2F-ARM-BC (5. 6]
20.1
3.1
24
24
4
20
20
PerAct
49.4
1.9
16.0
4.9
58.2 147
89.6
5.6 -4. 70.4 2.0 88.0 157 24 44,8 7.8
RVT (ours)
62.9
1.1
1.0
11.6
52.0 2,5
99.2 1.6 11.2 3.0 88.0 2,5 71.2 +. 4.0 25 91.0 5.2
Push Buttons
Put
Put
Put in
Screw
Slide Block
Sort
Stack
Stack
Sweep to Dustpan
Turn
Models
Cuphoord
Drwer
Safe
Bulb
Shape
Blocks
Cups
Tap
Image-BC (CNN) 6]
0
0
8
0
0
Image-Bo BC (2IT) 6]
0
0
D
0
0
0
0
C2U-ARM-BC 6]
72
PerAct [6]
92.8 3.0
28.0
44
$1.2 4.7
84.0 -3.0--36--- 366 17.6 20 74.0 13.0 16.8 4.7 26.4 3.2 24
+20 $2.0 - 0.0 88.0 4,4
(ours)
100.0 a0 49.6- 88.0
-57 91.2 -30 48.0 557 81.6 36.0 28.8 264 72.0
4.1
Table I: Multi-Tass Performance on RLBench. RVT outperforms state-of-the-art methods while being faster to train and execute. RVT has the best success rate and rank when averaged across all tasks. Performance for Image-BC (CNN), Image-BC (ViT) and C2F-ARM-BE Shridhar al. in [6]. We re-evalaute PerAct using the released final model and estimate mean and variance. RVT is 2.3X faster on execution speed than PerAct and outpeforms it on 16/18 18 tasks. The training time and inference speed of PerAct and RVT are measured on the same GPU model.
are as reported by
we use global features (G). The global features are a concatenation of (1) the sum of image features along the spatial dimensions, weighted by the predicted translation heatmap; and (2) the max-pooled image features along the spatial dimension. Specifically, let fs be the image feature and h2 be the predicted translation heatmap for the ith image. Then the global feature G given by 9 [é(fi @ h..;.... : oflk @ hε); ((fi): [fx)], where K is the number of images, @ denotes element-wiss multiplication, and & and a denote the sum and max max-pooling over the height and width dimensions. The weighted sum operation provides higher weights to image locations near the predicted end effector position.
Loss Function. We train RVT using a mixture of losses. For heatmaps, we use the cross-entropy loss for each image. The ground truth is obtained by truncated Gaussian distribution around the 2D projection of the ground-truth 3D location. For rotation, we use the cross-entropy loss for each of the Euler angles. We use binary classification loss for the gripper state and collision indicator.
4 Experiments
4.1 Simulation Experiments
Simulation Setup. We follow the simulation setup in PerAct [6], where CoppelaSim [62] is applied to simulate various RLBench [7] tasks. A Franka Panda robot with parallel gripper is controlled to complete the tasks. We test on the same 18 tasks as PerAct, including picking and placing, tool use, drawer opening, and high-accuracy peg insertions (see the appendix for a detailed specification of each task). Each task includes several variations specified by the associated language description. Such wide range of tasks and intra-task variations requires the model to not just specialize in one specific skill but rather learn different skill categories. The visual observations are captured from four noiseless RGB-D cameras positioned at the front, left shoulder, right shoulder, and wrist with resolution of 128 x 128. To achieve the target gripper pose, we generate joint space actions by using the same sampling- -hased motion planner [63, 64] as in [5, 6).
Baselines. We compare against the following three baselines: (1) Image-BC [2] an image-to- action behavior cloning agent that predicts action based on the image observations from the sensor camera views. We compare with two variants with CNN and ViT vision encoders respectively. (2) C2F-ARM-BB MARMMBBCBB [5] is a behavior cloning agent that converts the RGB-D images into multi-resolution voxels and predicts the next key-frame action using a coarse-to-fine scheme. (3) PerAcl [6] is the state-of-the-art multi-task behavior cloning agent that encodes the RGB-D images into voxel grid patches and predicts discretized next key-frame action using the perceiver [8] transformer.
Training and Evaluation Details. Just like the baselines, we use the RLBench training dataset with 100 expert demonstrations per task (1800 demonstrations over all tasks). Similar to PerAct, we apply
5

---

Rotated
RLBench Real
Perspective Proj.
Orthographic Proj.
Figure 3: We evaluate RVT with various camera locations for re-rendering (a-d) and find that loca- tions in (a) perform best. We also test various projection options (e-f) for rendering images and find that RVT works better with orthographic images.
translation and rotation data augmentations. For translation, we randomly perturb the point clouds in the range [+ 0.125m, a 0, 0.125m, = 0. 0.125m]. For rotation, we randomly rotate the point cloud around the 2-axis (vertical) in the range of +45 We train RVT for 100k steps, using the LAMB [65] optimizer as PerAct. We use batch size 24 and an initial learning rate of 2.4 x 10-4, We use cosine learning rate decay with warm-start for 2K steps.
For Image-BC and C2F-ARM-BC, we adopt the evaluation results from [6] since their trained mod- els have not been released. These results overestimate the performance of Image-BC and C2F-ARM- BC, as they select the best model for each of the 18 tasks independently based on the performance on validation sets. Hence, the reported performance does not reflect single multi-task model. Nev- ertheless, these baselines still underperform both PerAct and RVT (see Tab. 1). For PerAct, we evaluate the final model released by Shridhar al. [6]. We test models (including the models in the ablation study, Tab. 2 (left)) and PerAct on the same 25 variations for each task. Due to the randomness of the sampling- based based motion planner, we run each model five times on the same 25 variations for each task and report the average success rate and standard deviation in Tab. 1.
To fairly compare the training efficiency against PerAct, we train both PerAct and our model with the same GPU type (NVIDIA Tesla V100) and number GPUs (8), as reported by Shridhar et al. [6]. We report the total training time for both models in Tab. ("Training time"). We also evaluate the inference speed of PerAct and RVT models by running the prediction inferences for the same input data on the same GPU (NVIDIA RTX 3090).
Multi- Performance. Tab. 1 compares the performance between RVT and the baselines. We find that PerAct and RVT perform significantly better than the rest. Overall, RVT outperforms all baselines with the best rank and success rate when averaged across all tasks. It outperforms prior state-of-the-art methods, C2F-ARM, by 42 percentage points (213% relative improvement); and PerAct by 13 percentage points (26% relative improvement). RVT outperforms PerAct on 88.9% (16/18) of the tasks. More remarkably, RVT trains 36X faster than PerAct for achieving the same performance (see Fig. 1). We also observe that at inference time, RVT is 2.3X faster than PerAct. These results demonstrate that RVT is both more accurate and scalable when compared to existing state-of-the-art voxel-based methods.
Ablation Study. We conduct ablation experiments to analyze different design choices of RVT: (a) the resolution of the rendered images ("Im. Res." column in Tab. 2 (left)): (b) whether to include the correspondence information across rendered images ("View Corr."; (c) whether to include the depth channel ("Dep. Ch."); (d) whether to separately process the tokens each image before jointly processing all tokens ("Sep. Proc."); (e) the projection type for
or orthographic ("Proj. Type"); (f) whether to use rotation augmentation ("Rot. Aug."); "); (g) the number of views and camera locations for re-rendering ("# of View" and "Cam. Loc."); and (h) the benefit of using re-rendered images versus using real sensor camera images ("Real" for "Cam. Loc.").
Tab. 2 (left) summarizes the ablation experiment results. The same table along with the mean and standard dev. for each task can be found in the appendix Tab, A2. Below we discuss the findings:
(a) As expected, virtual images rendered at higher resolution help as RVT with virtual image reso- lution 220 outperforms the one with 100.

---

Im.
View
Dep.
Sep.
Proj.
Rot.
Cam
of Avg.
of of oof
PerAct
RVT
PerAct
RVT
Res.
Corr.
Type Aug.
View
Succ.
Task
vari.
train
test (+ mark.) t+ mark.) (- mark.) (- mark.)
220
/
Orth.
Cube
5
62.9
Stack
100
Orth.
Cube
5
51.1
blocks
10
50%
100%
50%
100%
220
Orth.
Cabe
5
59.7
Press
220
/
X
Orth.
Cube
5
60.3
sanitizer
7
10
40%
80%
40%
80%
220
Orth.
58.4
Put marker
220 /
Pers.
Cube
40.2
in mug/bowl
10
0%
0%
220 /
Cabe
60.4
Put object
230
/
Orth.
Cube
3
60.2
drawer
10
20%
50%
50%
100%
220 /
Orth.
Front
35.8
Put object
220
Orth.
Rot. 15
59.9
in shelf
10
50%
220 /
Pers.
Real
4
10.4
220
22.9
All tasks
13
51
50
28%
56%
42.5%
82.5%
Table 2: Left: Ablations on RLBench. A larger res., adding view correspondence, adding depth channel, separating initial attention layers, orthographic projection, using rotation aug., and re- rendered views around cube improve performance. Right: Success rate of RVT and Peract in the real-world. A single RVT model can perform well on most tasks with only a few demonstrations.
(b) Adding correspondence information for points across different views helps (see Sec. 3). This is likely because the network need not learn solve the correspondence problem and can predict more consistent heatmaps across views. Note that the view correspondence channel is not present in sensor images but is rendered along with RGB(D) images in RVT.
(c) Adding the depth channel along with RGB channels helps, likely because it aids 3D reasoning.
(d) Independently processing the tokens from a single image, before merging all the image tokens, helps. It is likely because this design expects the network to extract meaningful features for each image before reasoning over them jointly.
(e) Rendering images with orthographic projection performs better than rendering with perspective projection, for both the cube and real camera locations. We hypothesize that it is because ortho- graphic projection preserves the shape and size of an object regardless of its distance from the camera (see Fig. (e-f)). It also highlights the advantage of re-rendering, as real sensors generally render with perspective projections.
(f) As expected, using 3D rotation augmentation in the point cloud before rendering helps. To take advantage of 3D augmentations, the re-rendering process is necessary.
(g) The model with views around cube (Fig. 3 (a)) performs the best followed by the one with 3 views (front, top, left) around cube (Fig. 3 (b)). The single view model, where predict the third coordinate an offset like TransporterNet [4], performs substantially worse, calling for the need for multiple views for 3D manipulation. It also highlights the advantage of re-rendering as with re-rendering we can leverage multiple views even with single sensor camera. We also empirically find that rotating the location of the cameras by 15t (see Fig. 3) with respect to the table (and robot) decreases performance. This could be likely because views aligned with the table and robot might be easier to reason with (e.g., overhead top view, aligned front view).
(h) RVT performs better with re-rendered images as compared to using sensor camera images (Tab. 2 (left), second last row). The sensor camera images are rendered with perspective projection (physical rendering process) and are not straightforward to apply 3D augmentations (c.g., rotation) without re-rendering. Also, the location of sensor cameras may be sub-optimal for 3D reasoning, e.g., the views are not axially aligned with the table or robot (see Fig. (d)). All these factors contribute to RVT performing better with re-rendered images than with sensor camera images.
Notably, one might consider rearranging the sensor cameras to match the re-rendering views in order to bypass re-rendgging (see appendix A.2). However, this will void the gains from using orthographic projections, 3D augmentation, and adding correspondences (see appendix A.3). This also strictly requires multi-camera setup (Fig. 3 (a)), which is more costly and less portable in the real world than using one sensor camera. Finally, we have briefly explored view selection and found an option that works well across tasks. Further optimization of views, including the sensor and re- reneederedd ones, is interesting future direction.
7

---

Put bottle top drawer
Put yellow cube in the bottom shelf
Figure 4: Examples of RVT in the real world. A single RVT model can perform multiple tasks (5
tasks, 13 variations) in the real world with just 10 demonstrations per task.
4.2 Real-World
Real World Setup. We experiment on a table-top setup using a statically mounted Franka Panda arm. The scene is perceived via an Azure Kinect (RGB-D) camera statically mounted in third- person view. We calibrate the robot-camera extrinsies and transform the perceived point clouds to the robot base frame before passing into RVT. Given target gripper pose from RVT, we use FrankaPy [66] to move the robot to the target with trajectory generation and feedback control.
Tasks. We adopt a total of 5 tasks similar to the ones in PerAct [6] (see Tab. 2 (right)): stack blocks, press sanitizer, put marker in mug/bowl, put object in drawer, put object in shelf. Each task can be instantiated with different variations defined by the language description. For example, for stack blocks, some variations could be "put yellow block on blue block" and "put blue block on red block". Given task and variation, sample scene by placing the task-related objects and set of distractor objects on the table in random configuration.
Data Collection. We first collect a dataset for training RVT through human demonstration. Given a sampled task and scene configuration, we ask the human demonstrator to specify a sequence of gripper target poses by kinesthetically moving the robot arm around. Once we have the target pose sequence, reset the robot to the start pose, and then control it to sequentially move to each target pose following the specified order. We simultaneously record the RGB-D stream from the camera during the robot's motion to the targets. This provides us with dataset of RGB-D frames paired with target pose annotations. In total, we collected 51 demonstration sequences over all tasks.
Results. We train on real-world data for 10K steps, with the same optimizer, batch size, and learning rate schedule as the simulation data. We report the results in Tab. 2 (right). We find RVT outperforms prior method PerAct. It achieves high success rates for the stack block task (100%) and the press sanitizer task (80%). Even on longer horizon tasks such as putting objects in drawers and shelves (e.g., the robot has to first open the drawer/shelf and then pick up the object), our model achieves 50% success rates (see Fig. 4). We found RVT struggled with marker-related tasks, which is likely due to sparse and noisily sensed point clouds. We further divide the results into two sets: "+ markers" (full set) and ". markers". Our model overall achieves an 82.5% success rate on non-marker tasks. The marker issue can potentially be addressed by attaching the camera to the gripper to capture point clouds at higher quality. Another possibility is to zoom-in views similar to C2F-ARM [5].
5 Conclusions and Limitations
We proposed RVT, a multi-vier view transformer model for 3D object manipulation. We found that RVT outperforms prior state-of-the-an models like PerAct and C2F-ARM on variety of 3D manipula- tion tasks, while being more scalable and faster. We also found that RYT can work on real-world manipulation tasks with only few demonstrations.
Although we found RVT achieves state-of-the-art results on RLBench (62.9% success rate), there is scope for improvement. Following, we identify some limitations that present exciting directions for future research. We briefly explore various options for virtual view and found the orthogonal views work well across tasks, but it would be exciting if the virtual views can be optimized further or learned from data. Further, when compared prior view-based methods, RVT (as well explicit voxel-based methods like PerAct and C2F-ARM), requires the calibration of extrinsics from the camera to the robot base. It would be exciting explore extensions that remove this constraint.
8

---

References
[1] A. Goyal, A. Mousavian, C. Paxton, Y.-.. Chao, B. Okorn, J. Deng, and D. Fox. Ifor: Iterative flow minimization for robotic object rearrangement. In arxiv:2202.00733. 2022.
[2] E. Jang, A. Irpan, M. Khansari, D. Kappler, Ebert, C. Lynch, S. Levine, and C. Finn. BC-z: Zero-shot task generalization with robotic imitation learning. In Sth Annual Conference Robot Learning, 2021. URL https: / /oppnreeeeeiw net/ ft///oruuua Forum? dit.didddddkkkkk"""k00
[3] M. Shridhar, I.. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
[4] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, pages 726-747, PMLR, 2021.
[5] S. James, K. Wada, T. Laidlow, and A. Davison. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13739-13748, 2022.
[6] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-Actor; multi-task transformer for robotic manipulation. In CoRL, 2022.
[7] James, Z. Ma, D. R. Arrojo, and A. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Roboties and Automation Letters, 5(2):3019-300 2020.
[8] A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pages 4651-4664. PMLR, 2021.
[9] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
[10] Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learn- ing agile and dynamic motor skills for legged robots. Science Robotics, 4(26):aau55772, 2019.
[11] A. Kumar, Z. Fu, D. Pathak, and Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2 107.040344 2021.
[12] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:
2017.
[13] J. Xu, T. Du, M. Foshey, B. Li, B. Zhu, A. Schulz, and W. Matusik. Learning to fly: compu- tational controller design for hybrid uavs with reinforcement learning. ACM Transactions Graphics (TOG), 38(4):1-12, 2019.
[14] Anderson, A. Chang, D. Chaplot, A. Dosovitskiy, Gupta, V. Koltun, Kosecka, J. Ma- lik, R. Mottaghi, M. Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06751 2018.
[15] A. Goyal and J. Deng. Packit: A virtual environment for geometric planning. In International Conference on Machine Learning, pages 3700-3710. PMLR, 2020.
[16] D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and S. Levine. Ving: Learning open- open-world navigation with visual goals. In 202/ IEEE International Conference on Robotics and Automa- tion (CRA), pages 13215-13222. IEEE, 2021.
[17] R. Yang, M. Zhang, N. Hansen, H. Xu, and X. Wang. Learning vision-guided quadrupedal lo- comotion end-to-end with cross-modal transformers. arXiv preprint arXiv:2107.03996 2021.

---

[18] W. Huang. I. Mordatch, P. Abbeel, and D. Pathak. Generalization in dexterous manipulation via geometry-aware multi-task learning. arXiv preprint arXiv:211..0300 2021.
[19] T. Chen, J. Xu, and P. Agrawal. A system for general in ae-hand object re-orientation. Conference on Robot Learning. 2021.
[20] W. Yu, D. Jain, A. Escontrela, A. Iscen, P. Xu, E. Coumans, S. Ha, J. Tan, and T. Zhang. Visual- locomotion: Learning to walk on complex terrains with vision. In 5th Anmual Conference on Robot Learning, 2021. URL https::: /pwp:///eee./eeeeee///eem..o/ee. net met/forurd forum? wortummid-NDDA -d=-D
[21] D. Shah, A. Bhorkar, H. Leen, I. Kostrikov, N. Rhinehart, and S. Levine. Offline reinforcement learning for visual navigation. In 6th Annual Conference on Robot Learning, 2022. URL https: /ttp:////peeeeeeeeeeeeeeee net/ forum? net:/oouunidduhhhF1e. wiiiiiddhihiiiiie.
[22] D. Hafner, T. Lillicrap. J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.0166 2019.
[23] M. Watter, J. Springenberg, J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. Advances in neural information processing systems, 28, 2015.
[24] D. Yarats, I. Kostrikov, and R. Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021. URL https: http://pppeeeeeeeeeee net./ fe rum?
www.:mdddcoccooeooeooo d fvva...
[25] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved augteau reinforcement learning. In International Conference on Learning Representa- lions, 2022. URL
[26] A. Brohan, N. Brown, Carbajal, Y. Chebotar, J. Dabis, Finn, Gopalakrishnan, Haus- man, A. Herzog, Hsu, Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. Ju- lian, D. Kalashnikov, Y. Kuang. I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-
1: Robotics transformer for real-world control at scale. In arXiv preprint 22X1v:2212.06817, 2022.
[27] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E.. Kaiser, and I. Polo- sukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[28] Y.-C. Lin, P. Florence, A. Zeng, Barron, Y. Du, W.-C. Ma, A. Simeonov, A. R. Garcia, and Isola. Mira: Mental imagery for robotic affordances. In K. Liu, D. Kulic, and Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1916-1921 PMLR, 14-18 Dec 2023.
[29] I. Lenz, H. Lee, and A. Saxena. Deep learning for detecting robotic grasps. The International Journal of Robotics Research, 34(4-5):705-72) 2015.
[30] A. Saxena, Driemeyer, Kearns, and A. Ng. Robotic grasping of novel objects. Advances in neural information processing systems, 19, 2006.
[31] M. P. Deisenroth, Englert, Peters, and Fox. Multi-task policy search for robotics. In 2014 IEEE international conference on robotics and automation (ICRA), pages 3876-3881. IEEE, 2014.
[32] D. Kalashnikov. J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.08212, 2021.
10

---

[33] C. Devin, A. Gupta, T. Darrell, P. Abbeel, and S. Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In 2017 IEEE international conference on robotics and automation (ICRA), pages 2169-21777 IEEE, 2017.
[34] Y. Teh, V. Bapst, W. M. Czarnecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pas- canu, Distral: Robust multitask reinforcement learning. Advances neural information pro- cessing systems, 30, 2017.
[35] M. Ahn, Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey,, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, Xu, S. Xu, M. Yan, and A. Zeng. Do as can and not as i say: Grounding language in robotic affordances. In arXiv preprint arXiv:2204.01691, 2022.
[36] A. Goyal, K. Yang, D. Yang, and J. Deng. Rel3d: minimally contrastive benchmark for grounding spatial relations 3d. Advances in Neural Information Processing Systems, 33: 10514-10525, 2020.
[37] C. Lynch and P. Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.07764 2020.
[38] O. Mees, L. Hermann, and W. Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 7(4): 1205-11212, 2022.
[39] S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn, et al. Learning language-conditionee robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pages 1303-1315. PMLR, 2022.
[40] Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, Kay, J. T. Springenberg, et al. generalist agent. arXiv preprint arXiv:2205.06175, 2022.
[41] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. ICRA, 2022.
[42] D. S. Chaplot, D. Pathak, and J. Malik. Differentiable spatial planning using transformers. In International Conference on Machine Learning, pages 1484-1495. PMLR, 2021.
[43] H. M. Clever, A. Handa, H. Mazhar, K. Parker, O. Shapira, Q. Wan, Y. Narang, I. Akinola, M. Cakmak, and D. Fox. Assistive tele-op: Leveraging transformers to collect robotic task demonstrations. arXiv preprint arXiv:2112.05112,, 2021.
[44] J. Johnson, L. Li, A. H. Qureshi, and M. C. Yip. Motion planning transformers: One model plan them all. arXiv preprint arXiv:2106.00 02791, 2021.
[45] C. Chi, Feng, Du, Z. Xu, E. Cousineau, Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[46] Z. Zhao, V. Kumar, Levine, and Finn. Learning fine- fine-grained bimanual manipulation with low-cost hardware. arXiv preprint 23X75 2023.
[47] T. Cachet, J. Perez, and S. Kim. Transformer- based meta-imitation learning for robotic manip- ulation. In Neural Information Processing Systems, Workshop Robot Learning, 2020.

---

[48] H. Kim, Y. Ohmura, and Y. Kuniyoshi. Transformer-basee deep imitation learning for dual- arm robot manipulation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8965-8972. IEEE, 2021.
[49] S. Dasari and A. Gupta. Transformers for one-shot visual imitation. In Conference Robot Learning, pages 2071-2084. PMLR, 2021.
[50] R. Jangir, N. Hansen, S. Ghosal, M. Jain, and X. Wang. Look closer: Bridging egocentric and third-person views with transformers for robotic manipulation. IEEE Robotics and Automation Letters, 7(2):3046-3053, 2022.
[51] W. Liu, C. Paxton, T. Hermans, and D. Fox. Structformer: Learning spatial structure for gegguage-guided semantic rearrangement of novel objects. In 2022 International Conference on Robotics and Automation (ICRA), pages 6322-6329, IEEE, 2022.
[52] A. Goyal, H. Law, B. Liu, A. Newell, and J. Deng. Revisiting point cloud shape classification with simple and effective baseline. In International Conference on Machine Learning, pages 1809-3820. PMLR, 2021.
[53] A. Hamdi, S. Giancola, and B. Ghanem. Mvtn: Multi-view transformation network for 3d shape recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages
[54] A. Hamdi, S. Giancola, and B. Ghanem. Voint cloud: Multi-view point cloud representation for 3d understanding. arXiv preprint arXiv:2111. 153663 2021.
[55] S. Huang, Y. Chen, J. Jia, and L. Wang. Multi-view transformer for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15524-15533, 2022.
[56] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1))99-106, 2021.
[57] V. Guizilini, I. Vasiljevic, Fang, R. Ambru, Shakhnarovich, R. Walter, and A. Gaidon. Depth field networks for generalizable multi-viww scene representation. In European Confer- Computer Vision, pages 145-262. Springer, 2022.
[58] E. Johns. Coarse-to-fine imitation learning: Robot manipulation from a single demonstration. In 2021 IEEE international conference on robotics and automation (ICRA), pages 4613-4619. IEEE, 2021.
[59] N. Ravi, J. Reizenstein, D. Novotny, T. Gordon, W.-Y Lo, J. Johnson, and G. Gkioxari. Ac- celerating 3d deep learning with pytorch3d. arXiv:2007.08501, 2020.
[60] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, Askell, Mishkin, Clark, al. Learning transferable visual models from natural language supervi- sion. In International conference machine learning, pages 8748 8763. PMLR, 2021.
[61] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transform- ers for image recognition at scale, arXiv preprint arXiv:2010. 1929, 2020.
[62] E. Rohmer, S. P. Singh, and M. Freese. V-rep: A versatile and scalable robot simulation framework. In 2013 IEEE/RSJ international conference on intelligent robots and systems, pages 1321-1326. IEEE, 2013.
[63] G. Sánchez and J.-C. Latombe. A single-query bi-directional probabilistic roadmap planner with lazy collision checking. In Int. Symp. Robotics Research, pages 403-417, 2001.
12

---

[64] I. A. Sucan, M. Moll, and L. E. Kavraki. The open motion planning library. IEEE Robotics & Automation Magazine, 19(4):72-82, 2012.
[65] Y. You, J. Li, S. Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, X. Song, J. Demmel, K. Keutzer, and C.-J. Hsich. Large batch optimization for deep learning: Training bert in 76 minutes, arXiv preprint 194400990..00962, 2019.
[66] K. Zhang, M. Sharma, Liang, and O. Kroemer. A modular robotic arm control stack for research: Interf-e and FrankaPy. arXiv preprint arXiv:2011.02398, 2020.
13

---

A
Appendix
A.1
RLBench Tasks
We provide brief summary of the RLBench tasks in Tab. Al. There are 18 tasks with 249 variations. For more detailed description of each task, please refer to PerAct [6], Appendix A.
Task
Language Template
of Variations
open drawer
"open the drawer"
3
slide block
"slide the block to target"
4
sweep to dustpan
"sweep dirt to the dustpan"
2
meat off grill
"take the off the grill"
2
turn tap
"turn tap"
2
put in drawer
"put the item in the drawer" "close the jar" "use the stick to drag the cube onto the target"
3
close jar
20
drag stick
20
stack blocks
"stack blocks" "screw in the light bulb" "put the money away in the safe on the shelf"
60
screw bulb
20
put in safe
3
place wine
"stack the wine bottle the of the rack"
3
put in cupboard
"put the in the cupboard"
9
sort shape
"put the in the shape sorter"
5
push buttons
"push the button, [then the button]"
50
insert peg
"put the - peg in the spoke"
20
stack cups
"stack the other cups on top of the cup"
20
place cups
"place - cups the cup holder"
3
Table A1: Tasks in RLBench We evaluate on 18 RLBench tasks which are same as those used in PerAct [6]. For more details, check see PerAct [6]. Appendix A. For videos, visit https: //rooooiiii c- vivviw ew trans toretttansssrrrrroooo wowmer.githu.................... io/
A.2 Additional Experiments
Experiments on dataset with input cameras in orthogonal configuration. We created a new im- age dataset for RLBench with input cameras arranged orthogonal configuration like in Fig. 3a. Note that this dataset is not provided in PerAct. With this new image dataset, we did two experi- ments, one directly using input camera images as input and one with our pipeline (re-rendering with orthographic projection, 3D augmentation and correspondences) We find that results this new dataset consistent with the results on the dataset provided by PerAct, where pipeline works better (60.0% vs 27.2%) likely because it allows for orthographic projection, 3D augmentation, and point correspondence (Table Left).
How does the quality of input point cloud affeet performance of RVT? To investigate how the quality of the point cloud affects performance in RYT, we do experiments in simulation. Specifically, add Gaussian noise with varying standard deviation (2.5mm, 5mm, lem, 2cm, and 4cm) to the original point cloud. We add this noise both during training and evalua- tion to simulate sensor noise in both phases. The success rate is 62.9 for no noise, 62.0 for 2.5mm, 61.6 for 5mm, 56.4 for lcm, 58.7 for 2cm and 51.7 for 4em standard deviation noise. We find that RVT is robust to 2cm standard deviation noise in the point cloud and its perfor- mance degrades gracefully with more noise. For reference, the depth measurements in Intel RealSense D400 camera has an error of 2.5mm to 5mm for an object Im from the cam- http:///seeeesuupppprrrtrtttotet 000026260 6 /ttp:///ieeeeeooooooaaeeeeeeaaar
(source: https://www.nntel.. wwwp:/wwwwww......ooooot.eemeee. wwtt
14

---

A.3 Additional Explanation
How are the heatmaps of multiple virtual views back-projected to 3D? To calculate the heatmap value at a 3D location, we map 3D points to 2D pixels in the virtual views. We consider not just the points in the point cloud, but all points in the 3D scene bounds that are distributed at resolution of (h h x h), where (h x h) is the resolution of the virtual image. For each point, the heatmap value from multiple views are averaged.
Why only using input images (without re-renderning) voids the gains from using orthographic projections, 3D augmentation, and adding correspondences? As we see in Table 2 Left, or- thographic projections, 3D augmentation and adding xyz. image (correspondence) improve perfor- mance. However, these could only be added after re-rendering, because: First, real-world cameras generally provide only perspective projection and not orthographic projection. To obtain ortho- graphic projection, re-rendering is needed. Second, effects of 3D augmentation like rotation of the object cannot be trivially simulated in the image without re-rendering. We first create 3D represen- lation, apply the augmentation and re-render. Finally, adding xyz image between points in images first requires explicitly building the 3D point cloud from images, and rendering the xyz image.
In the real robot experiments, only one camera view is used. How carefully does this view have to be selected that the method performs well? Would the performance of the method improve if more cameras are placed in the workspace? We used standard third person view that is in front of the robot. We ensured that the workspace is visible in the camera but no particular effort was put in adjusting the view to make the method perform well. Potentially having more cameras could improve the method.
PerAct extracts set of keyframe actions from the demo by capturing bottleneck end-effector poses in the action sequence that has (1) near-zero joint velocities or (2) unchanged gripper open state. It seems from Sec 4.2 (Data Collection) that the keyframe actions are specified in a different way in RVT. We follow the same pipeline as PerAct to extract keyframes from demonstrations. Sec 4.2 (Data Collection) describes our scheme for collecting target poses for real world demonstrations. The difference that PerAct uses a VR controller to specify target poses while we do so by kinesthetically moving the robot. Once real-world demonstration is collected, the process of extracting keyframes is the same.
15

---

A.4 RVT Overview
Rotation
MLP
Attention X
Gripper
Attention
+
Collision 1
CLIP
Projoction
Insert peg in the blue spoke
Patchify
+
Virtual Image
Virtual Image
Virtual Image
Figure A1: Overview of the transformer used in RVT. The input to the transformer is language description of the task and virtual images of the scene point cloud. The text is converted into token embeddings using the pretrained CLIP [60] model, while the virtual images embeddings via patchify and projection operations. For each virtual image, tokens helonging to the same image are processed via four attention layers. Finally, the processed image tokens as well as the language tokens are jointly processed using four attention layers. The 3D action is inferred using the resulting image tokens.
converted into token
16

---

A.5
Ablations
We report the ablations mentioned in Tab. 2, along with the mean and standard deviations for each task Tab. A2.
Im.
View
Dep.
Bi-
Proj.
Rot.
Cam
of
Avg.
Close
Dong
Insert Pog
Meat off
Open Drawer
Place
Type
Aug.
Cups
Orth
100
Orlh.
Cabe
51.1
= 1.7
2.8
91 =
=
1.7
220
/
Ort.
Cabe
5
59.7
44 4
100 =00
4.4
90 6
71 9.1
7:5
220
Orth.
Cabe
60.3
37 3.3
96 0
11:33
9.7
3
220
220
Pens.
Cabe
40.2
2..
904
4:30
84.8 4:7
13.6
2.4
220
Onh.
Cahe
60.4
12.8 1.6
97.6
4.8
85.6
0:2.0
220
Omh.
Cabe
60.2
1..
75.2
1.33333
89.6
68.
3.26
220
Onh,
Hroet
35.8
36
1.7
2
220
Orth
Rot. 15
59.9
48.8 1.6
99.2 1.6
12.14.
2:
71.2.19.
0.100
220
Pers.
Real
10.4
14.4 1.6.5
=4.4
0=:0
0.00
2.
0.1.0
220
Onho.
43.2 4:77
4
Im..
View
Dep.
Bi-
Proj.
Rol.
Cam
o
Avg.
Place
Push
Put in
in
Put
Screw
Type
Safe
220
Onh.
Cabe
91.2
100
Orch.
Cabe
$1.1
8.7
100
86
77 1.7
4.5
220
/
Orth.
Cabe
39.7
96
99 = 1.7
48 + 6.9
6
79
36 36
220
Orch
Cabe
60.3
1.7
99
56
92 4.9
77.
4.4
220
220
Pens.
Cabe
40.2
5.7
91.2 =.6
26.4 4 2.5
64.8
$1.2
20
220
/
Orc.
Cube
60.4
84 $
96 a 2.5
88 8.2
90.4 4.11
8.4
220
Onh.
Cahe
60.2
8
97.6
94.4
220
7.1
220
Onth
Rot. 15
59.9
74.4 5.4
9..2 1.6
4:4.::4
81.6
80. 4.7
4..6.:488
220
Pens.
Real
10.4
11.2 1.99
26.4 241
0.00
0.00
0.10
0.00
220
Onhoo
Im..
View
Dep.
Bi-
Proj.
RoI
Cam
of
Avg.
Slide
Sort Stape
Stack
Stack
Sweep Dusipon
Tum
Block
Cups
220
Onth
Cabe
2.5
1001
Onh.
Cahe
1
1:77
36-
220
/
Orth.
Cabe
5
59.7
83 1.7
4.4
26.7 1.5
20 4.9
72 0
95 4.4
220
x
Onth.
Cúbe
60.3
4
37
29 =
92
9.4
220
Pers.
Cabe
40.2
4.4
19.2=4.
22.4
16
16=0
30.8
220
Onh.
Cabe
60.4
72.8 = 1.6
25.6
8.8
=4
92
220
Orc.
Cabe
60.2
95.2
37.6
29.6
8.6
8
92.8
220
220
Orh.
Rot. 15
59.9
1.7
30.4544
46.4 9.3
20.8 4.7
64
94.4
220
Pers
Real
4
10.4
37.6 10.6
2.4
0.
0:00
$6.8
220
Ortho.
Real
22.9
12.8.
1.2
11.2
12.0
132
Table A2: Ablations results for RVT on RLBench with metrics for each task.
17


Reviews:
[{"review_id": "UDY4X5ReKO", "reviewer": "Reviewer_CGaw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes a new framework (RVT) for 3D object manipulation. Instead of using voxel-based or NeRF-based methods, the authors propose to synthesize virtual 2D images from a point cloud of the scene and feed the virtual images into a multi-view transformer. The multi-view transformer predicts a 2D heatmap for each view, which is then back-projected to the 3D space to determine the next best end-effector position. The global features aggregated from the outputs of the transformer are used to predict the end-effector rotation, gripper actions, and collision indicator.\n\nThe authors performed thorough simulation experiments on RLBench, where results show that RVT not only outperforms PerAct in 16 of the 18 RLBench tasks but also achieves much faster training time and inference speed compared to PerAct. The authors also conducted a set of real-world experiments, showing that their proposed method could reliably perform a range of 5 manipulation tasks on a real robot platform.", "summary_of_recommendation": "The paper introduces a novel method for 3D object manipulation that could inspire future efforts in this area. The contributions are clear and the experiments are well-executed. The paper could improve with more real robot quantitative results and some minor writing fixes. If the authors could address those concerns, I would be happy to recommend the manuscript for publication.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Include a baseline for the real robot result.\n- Discussions of related works in learning 3D manipulation mentioned above.\n- Include explanations for the clarification questions in the paper or appendix.\n- Include more details about hyperparameter choices of the method and baselines, loss function, and real robot setup in the appendix.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\nThe paper presents an interesting method to improve the accuracy and speed of 3D object manipulation. The writing is clear and easy to read. The author also conducted a thorough literature review of related works in the field of visual object manipulation and multi-task policy learning.\n\nIssues:\n\nOne issue in the real robot experiments in the paper is that there seems to be no quantitative comparison between the proposed method and the baseline. It would be great if the authors could consider running PerAct on real robot environments to confirm that their method still outperforms the best baseline in the sim experiments.\n\nThe paper also seems to lack discussion of the relationship between their work and other recent works that make use of transformer-based or diffusion-based architectures for performing 3D object manipulation [1, 2] (note that those works went public on Arxiv at least a month before the CoRL deadline).\n\nSome clarification questions:\n- In Section 3 Line 160, the authors mention that the heatmaps of multiple virtual views are back-projected to 3D. Could the authors elaborate on how this process is done? Is there a mapping between the 2D pixels in the virtual views and the 3D points in the point cloud? What does the algorithm do to the parts of the image where there are no corresponding points in the 3D space? Do the heatmap values from multiple views get averaged after they are back-projected onto a set of points?\n- PerAct extracts a set of keyframe actions from the demo by capturing bottleneck end-effector poses in the action sequence that has (1) near-zero joint velocities or (2) an unchanged gripper open state. It seems from line 292 that the keyframe actions are specified in a different way in RVT. Could the authors clarify the difference between the two ways of collecting demos?\n- In the ablation experiments, the authors argued in lines 271-274 that rearranging sensor cameras to match re-rendering views in order to bypass re-rendering would \"void the gains from using orthographic projections, 3D augmentation, and adding correspondences\", but did not seem to actually run such an experiment. Could the authors add such an experiment in Table 2 left? Also, it would be interesting to see ablation experiments where only the sensor camera locations change on top of the original experiment (ie. Table 2 left first row).\n- In the real robot experiments, only one camera view is used. How carefully does this view have to be selected so that the method performs well? Would the performance of the method improve if more cameras are placed in the workspace?\n\n[1] Zhao et al. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. RSS 2023.\n\n[2] Chi et al. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS 2023.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "PliFo8vTnm", "reviewer": "Reviewer_sWvB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This manuscript introduces a multi-view transformer for 3D object manipulation.  The transformer contains an attention mechanism to gather information from multiple view images (re-rendering from virtual views) and output the target end-effort pose. The low-level controller or motion planner generates actions to achieve these target poses. The authors applied the proposed pipeline in the RLBench to demonstrate its effectiveness and also tested it in real-robot experiments.", "summary_of_recommendation": "The authors proposed a Robotic View Transformer framework for 3D object manipulation achieving significant improvements compared to other baselines. However, this pipeline requires first reconstructing a point cloud. How the quality of the reconstructed point cloud affects the performance is not clear to the reviewer. If multiple images are needed to reconstruct the point cloud, what are the advantages of re-rendering and how the quality of the rendered images affects the pipeline performance is not clear.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "See weakness ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n1. The paper is well-organized and easy to follow.\n2. The proposed Robotic View Transformer framework achieves significant improvements compared to other baselines\n3. The authors applied the proposed pipeline in the real robot experiments\n\nWeakness:\n1.  Can these multi-view images be directly achieved using multiple cameras, since the pipeline requires first reconstructing a point cloud of the scene? If real multiple-camera images are fed into the pipeline, will it improve the performance? \n2. How accurate are the re-rendered images? Does the quality of these re-rendered images affect the performance of the action prediction?\n3. What would happen if the predicted key frame(gripper poses) is not achievable due to collision or joint limit?\n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "LtqPSUf7O9", "reviewer": "Reviewer_E55B", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces the Robotic View Transformer (RVT) for manipulating objects in 3D. RVT leverages Behavioral Cloning to learn from a small number of demonstrations, enabling it to execute various manipulation tasks using a single policy. By utilizing a language description of the task and a set of RGB-D images as input, RVT predicts the target end-effector pose and gripper open/close state at each time step to generate actions. A notable architectural innovation of RVT is the incorporation of \"multi views,\" where virtual cameras are strategically positioned in the scene and aggregated through an attention mechanism to make predictions in a 3D space. Comparisons with Perceiver-Actor (PerAct) from CoRL 2022 show that RVT gets similar performance but trains *much* faster and has much faster inference speed.\n", "summary_of_recommendation": "Overall, this paper makes a strong contribution to the research community. PerAct was already one of my favorite papers due to its multi-task and manipulation performance, and I'm happy to see research that makes training and running it much faster. (I am not an author on PerAct nor do I collaborate with the authors.)\n", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "I don't have specific questions other than those implied by the review above. Please correct me if I happen to be mistaken in my understandings.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths of the paper and RVT:\n\n1. Perceiver-Actor (PerAct), a model proposed at CoRL 2022, is quickly getting a lot of traction in the robot learning community (with lots of citations for a recent paper). I view RVT as the next version of PerAct, and it has 36X faster training time and 2.6X faster inference time, for the same performance. This is a very compelling result because PerAct's training time and requirements are a huge limitation to its usage in the research community.\n\n2. RVT can use a small number of demonstrations to train a single multi-task policy for a wide variety of language- and image-conditioned 3D manipulation tasks, where the policy has to predict full 6D robot end-effector poses, as well as a gripper opening/closing choice.\n\n3. There are a lot of simulation experiments using RLBench and compelling real world results.\n\n4. There are lots of ablations and analysis of the design choices for RVT.\n\nMinor comments and questions:\n\n1. For Figure 1, why does the performance of RVT stagnate after 60%? Would it possibly decrease if it was trained longer, due to overfitting?\n\n2. In line 272: can the paper briefly clarify why \"this will void the gains from using orthographic projections, 3D augmentation, and adding correspondences\"?\n\nThere are a few limitations of the paper as mentioned in the conclusion. In addition to these, I would first suggest adding that performance is still only 60% (from Figure 1) so there's still more to go. Second, some might view this paper as \"only\" an engineering contribution built on top of PerAct. However, I think the usage of multiple views here results in sufficient technical novelty.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an extension of PerAct using a continuous representation introducing a multi-view transformer architecture that achieves significant improvements over the prior method, and the paper also provides real world evaluation of the method. Overall, they idea of keyframe prediction of end-effector poses for complex manipulation tasks is a method that  gains a lot of traction.

The reviewers have evaluated mostly positive the work, and the authors have done a good job in the rebuttal period, particularly providing additional results on the real world comparison of RVT and PerAct, showcasing the superior performance of PerAct. The addition of clarifications on the experimental process and the the additional real-world comparison are essential for the paper and I expect to see them added in the camera-ready version of the paper.
Decision: Accept (Oral)