Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation
Authors: Mohit Shridhar
Lucas Manuelli
Dieter Fox
Venue: CoRL 2022 Poster (2022)
Keywords: Transformers
Language Grounding
Manipulation
Behavior Cloning

Abstract: Transformers have revolutionized vision and natural language processing with their ability to scale with large datasets. But in robotic manipulation, data is both limited and expensive. Can manipulation still benefit from Transformers with the right problem formulation? We investigate this question with PerAct, a language-conditioned behavior-cloning agent for multi-task 6-DoF manipulation. PerAct encodes language goals and RGB-D voxel observations with a Perceiver Transformer, and outputs discretized actions by “detecting the next best voxel action”. Unlike frameworks that operate on 2D images, the voxelized 3D observation and action space provides a strong structural prior for efficiently learning 6-DoF actions. With this formulation, we train a single multi-task Transformer for 18 RLBench tasks (with 249 variations) and 7 real-world tasks (with 18 variations) from just a few demonstrations per task. Our results show that PerAct significantly outperforms unstructured image-to-action agents and 3D ConvNet baselines for a wide range of tabletop tasks.

PERCEIVER-ACTOR: A Multi-Task Transformer for Robotic Manipulation
Mohit Shridhar Lucas Manuelli 2
Dieter Fox 1.2
'University of Washington 2NVIDIA
mahrêca. washington. Imameell@@nvidia.comm foxêcs washington.
peract.githen git.ut.thbb.... io
Transformers have revolutionized vision and natural language process- ing with their ability to scale with large datasets. But in robotic manipulation, data is both limited and expensive. Can manipulation still benefit from Transformers with the right problem formulation? We investigate this question with ACRACT, a language-conditioned behavior-cloning agent for multi-task 6-DoF manipula- tion. PERACT encodes language goals and RGB-D voxel observations with a Perceiver Transformer [1], and outputs discretized actions by "detecting the next best voxel action". Unlike frameworks that operate on 2D images, the voxelized 3D observation and action space provides strong structural prior for efficiently learning 6-DoF actions. With this formulation, we train a single multi-task Trans- former for 18 RLBench tasks (with 249 variations) and 7 real-world tasks (with 18 variations) from just a few demonstrations per task. Our results show that PERACT significantly outperforms unstructured maage-to-action agents and 3D ConvNet baselines for a wide range of tabletop tasks.
Abstract:
Keywords: Transformers, Language Grounding, Manipulation, Behavior Cloning
1 Introduction
Transformers [2] have become prevalent in natural language processing and computer vision. By framing problems as sequence modeling tasks, and training on large amounts of diverse data, Trans- formers have achieved groundbreaking results in several domains [3, 4, 5, 6]. Even in domains that do not conventionally involve sequence modeling [7, 8). Transformers have been adopted as a gen- eral architecture [9]. But in robotic manipulation, data is both limited and expensive. Can we still bring the power of Transformers to 6-DoF manipulation with the right problem formulation?
Language models operate sequences of tokens [10], and vision transformers operate on sequences of image patches [4]. While pixel transformers [11, 1] exist, they are not as data efficient as ap- proaches that use convolutions or patches to exploit the 2D structure of images. Thus, while Trans- formers may be domain agnostic, they still require the right problem formulation to be data efficient. A similar efficiency issue is apparent in behavior-cloning (BC) agents that directly map 2D) images to 6-DoF actions. Agents like Gato [9] and BC-Z [12, 13] have shown impressive multi-task ca- pabilities, but they require several weeks even months of data collection. In contrast, recent works seiforreeent-learmiine space to efficiently learn visual representations of 3D actions with 3D ConvNets. Similarly, in this work, we aim to exploit the 3D structure of voxel patches for efficient 6-DoF behavior-cloning with Transformers (analogous to how vision transformers [4] exploit the 2D structure of image patches).
like C2FARM [14] construct voxelized observation and action
To this end, we present PERACT (short for PERCEIVER- ACTOR)), a language-cone conditioned BC agent that can learn to imitate wide variety of 6-DoF manipulation tasks with just few demonstrations per task. PERACT encodes a sequence of RGB-D voxel patches and predicts discretized translations, rotations, and gripper actions that are executed with a motion-planner in an observe-act loop. PER- ACT essentially a classifier trained with supervised learning to detect actions akin prior work like CLIPort [ [16, 17]. except our observations and actions are represented with 3D voxels instead of 2D image pixels. Voxel grids are less prevalent than images in end-to-end BC approaches often due
Work done partly while the author was a part-time nart-time intern at NVIDIA.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

(al
[b]
[c]
[c]
"open the middle drawer"
"lldd the block to pink target"
"put the moon the shape sorter"
"stack purple blocks"
"place the wine bottle on the middle of rack"
Chl
gray lightbull""
"turn right tap"
"sweop to the short dustpan"
"take offihe grill"
"use drag
the cube onto the rose target"
Iml
put the tomatoes
"put the top
"hit the ball the stick"
"place the blue whiteboard
"sweep the beans into the gray dustpan"
top
mug"
Language-Condtioone Conditioned Manipulation Tasks: PERACT is a Langunge-ooddttonnd multi-task agent capable of imitating wide range of 6-DoF manipulation tasks. We conduct experiments simulated tasks RLBench [15] (a-j: only shown), several pose and semantic variations. We also demonstrate approach with Franka Panda 7 roal-world tasks (k-o; only shown) with malti-task agent trained with just 53 demonstrations. See the supplementary video for simulated and real world rollouts.
Figure 1. Language-
to scaling issues with high-ding digeesinniio inputs. But in PERACT, we use Perceiver" Transformer [1] to encode very high-dimensional input up to million voxels with only small set of latent vec- tors. This voxel-based formulation provides strong structural prior with several benefits: natural method for fusing multi-view observations, learning robust action-centric representations [18, 19), and enabling data augmentation in 6-DoF all of which help learn generalizable skills by focusing on diverse rather than narrow multi-task data.
To study the effectiveness of this formulation, we conduct large-scale experiments in the RL- Bench [15] environment. We train single multi-task agent on 18 diverse tasks with 249 variations that involve range of prehensile and non-prehensile behaviors like placing wine bottles on rack and dragging objects with a stick (see Figure a-j). Each task also includes several pose and seman- tic variations with objects that differ in placement, color, shape, size, and category. Our results show that PER ACTACT significantly outperforms image-to-action agents (by 34x) and 3D ConvNet baselines (by 2.8x), without using any explicit representations of instance segmentations, object poses, mem- ory, or symbolic states. We also validate our approach on a Franka Panda with multi-task agent trained from scratch real-world tasks with total of just 53 demonstrations (see Figure k-o).
In summary, our contributions are as follows:
A novel problem formulation for perceiving, acting, and specifying goals with Transformers.
An efficient action-centric framework for grounding language 6-DoF actions.
Empirical results investigating multi-task agents on a range of simulated and real-world tasks.
The code and pre-trained models will be made available peract. .ee.ggitthi github.
2 Throughout the paper we refer to PerceiverlO [1] as Perceiver for brevity. Action-centric refers system that learns perceptual representations of actions; Appendix J.

---

2 Related Work
Vision for Manipulation. Traditionally, methods in robot perception have used explicit "object"
representations like instance segmentations, object classes, poses [20, 21, 22, 23, 24, 25]. Such methods struggle with deformable and granular items like cloths and beans that are hard to represent
with geometric models or segmentations. In contrast, recent methods [26, 17, 16, 27] learn action- centric representations without any "objectness" assumptions, but they are limited to top-down 2D settings with simple pick-and-place primitives. In 3D, James et al. proposed C2FARM [14), an action-centric reinforcement learning (RL) agent with coarse-to-fine-geee The coarse-to-fine fo-iie--rnnnnin scheme has a limited receptive field that cannot look at the entire scene at the finest level. In contrast, PERACT learns action-centric representations with a global-receptive field through Transformer backbone. Also, PERACT does BC instead of RL, which enables us to easily train& a multi-task agent for several tasks by conditioning it with language goals.
3D-UNet backbone.
Enndtt-EEnd Manipulation approaches [28, 29, 30, 31] make the least assumptions about objects and tasks, but are often formulated as an image-to-actee to-ainionn prediction task. Training directly on RGB images for 6-DoF tasks is often inefficient, generally requiring several demonstrations or episodes just to learn basic skills like rearranging objects. In contrast, AERACT uses a voxelized observation and action space, which is dramatically more efficient and robust in 6-DoF settings. While other works in 6-DoF grasping [32, 33, 34, 35, 36, 37] have used RGB-D and pointeloud input, they have not been applied to sequential tasks used with language-conditionin
Another line of work tackles data inefficiency by using pre-trained image representations [16, 38, 39] to bootstrap BC. Although our framework is trained from scratch, such pre-training approaches could be integrated together in future works for even greater efficiency and generalization to unseen objects.
Transformers for Agents and Robots. Transformers have become the prevalent architecture in several domains. Starting with NLP [2. 3, 40], recently in vision [4, 41], and even RL [8, 42, 43]. In robotics, Transformers have been applied to assistive teleop [44], legged locomotion [45], path- planning [46, 47], imitation learning [48, 49], morphology controllers [50]. spatial rearrange- ment [, and grasping [52]. Transformers have also achieved impressive results in multi-domain settings like in Gato [9] where a single Transformer was trained on 16 domains such as captioning, robotic control etc. However, Gato relies on extremely large datasets like 15K episodes for block stacking and 94K episodes for Meta-World [53] tasks. Our approach might com- plement agents like Gato, which could use our 3D formulation for greater efficiency and robustness.
Language Grounding for Manipulation. Several works have proposed methods for grounding language in robot actions [54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65]. However, these methods use disentangled pipelines for perception and action, with the language primarily being used to guide perception [66]. Recently, a number of end-to-end approaches [13, 12, 67, 68, 69] have been proposed for conditioning BC agents with language instructions. These methods require thousands of human demos or autonomous episodes that are collected over several days even months. In contrast, PERACT can learn robust multi-task policies with just a few minutes of training data. For benchmarking, several simulation environments exist [70, 17, 53], but we use RLBench [15] for its diversity of 6-DoF tasks and ease of generating demonstrations with templated language goals.
PERCEIVER-ACTOR
PERACT is a language-conditionee behavior-cloning agent for 6-DoF manipulation. The key idea is to learn perceptual representations of actions conditioned on language goals. Given a voxelized reconstruction of scene, we use a Perceiver Transformer [1] to learn per-voxel features. Despite the extremely large input space (100*), Perceiver uses a small set of latent vectors to encode the input. The vor-voxel features are then used to predict the next best action in terms of discretized translation, rotation, and gripper state at each timestep. PERACT relies purely on the current observation to determine what do next in sequential tasks. See Figure 2 for an overview.
Section 3.1 and Section 3.2 describe our dataset setup. Section 3.3 describes our problem formula- tion with PERACT, and Section 3.4 provides details training PERACT. Further implementation details are presented in Appendix B.
3

---

Vectors
2048
ldt
right
Vonel Decoder
best
PerceiveriO Transformer
Language Ericodar
Vaxel Encoder
open
rot
collide
t-3
open the middle diawer
"open the middle drawer"
trans
Figure 2. PERACT Overview. PERACT langunge-condrioond behavior-cloning agent trained with supervised learning deteet actions. PERACT takes as input a language goal and a vexel grid reconstructed from RGB-D sensors. The voxels are splat into 3D patches, and the language goal encoded encoded with Perceiver transformer [I]. Despite the extremely long input sequencc,, input (see Appendix Figure an illustration). These encodings are upsampled back to the original voxel dimensions with decoder and reshaped with lineur layers predict discretized translation. retation, gripper open, and collision avoidance action. This action is executed with motion planeer afler which the new observation is used to predict the next discrete action an observe act loop until lermiiation.
pre
language model.
are appended logether a sequence languuge Penceiver uses small set of latent vectors to encode the
3.1 Demonstrations
We assume access to a dataset D = S2...,,..., of expert demonstrations, each paired with English language goals [11, 12....... ..... In These demonstrations collected by an expert with the aid of a motion-planner to reach intermediate poses. Each demonstration Ç is a sequence of continuous actions A (01,00,..... a paired with observations O {ō1 o., )z). An action consists of the 6-DoF pose, gripper open state, and whether the motion- motion-planner used collision {apose: Gopen. Gcollide J. An observation ô consists
avoidance to reach an intermediate pose: of RGB-D images from any number of cameras. We use four cameras for simulated experiments Ösim (Oron Oten, Oright; Ownss). but just a single camera for weal-world experiments Öreal {Otrom)
3.2 Keyframes and Voxelization
Following prior work by James al. [14], we construct structured observation and action space through keyframe extraction and voxelization.
Training agent to directly predict continuous actions is inefficient and noisy. So instead, for each demonstration Ç. we extract a set of keyframe actions (kg kg.... km } C A that capture bottleneck action is keyframe if (1) the joint-velocities are near zero and (2) the gripper open state has not changed. Each datapoint in the demonstration ( can then be cast "predict the next (best) keyframe action" task [14, 72, 73). See Appendix Figure for an illustration of this process.
end-effector poses [71] in the action sequence with simple heuristic:
To learn action-centric representations [18] in 3D, we use voxel grid [74, 75] to represent both the observation and action space. The observation voxels v are reconstructed from RGB-D observations o fused through triangulation ō from known camera extrinsics and intrinsics. By default, we use voxel grid of 1003, which corresponds to a volume actions k are discretized such that training BC agent can be formulated classification task [14]. Translation simply the closest voxel to the center of the gripper fingers. Rotation is discretized into 5 degree bins for each of the three rotation axes. Gripper open state is binary value. Collide is also binary value that indicates if the motion-planner should avoid every- thing in the voxel grid nothing all; switching between these modes of collision avoidance crucial as tasks often involve both contact based (e.g.. pulling the drawer open) and non-contact based motions (e.g., reaching the handle without colliding anything).
1.0m in metric scale. The keyframe
"next best action"
3.3 PERACT Agent
PERACT is Transformer-based [2] agent that takes in voxel observation and language goal (v, I), and outputs a discretized translation, rotation, and gripper open action. This action is executed with motion-planned after which this process is repeated until the goal is reached.
The language goal is encoded with a pre-trained language model. We use CLIP's [76] language encoder, but any pre-trained language model would suffice [13, 69]. Our choice of CLIP opens up possibilities for future work to use pre-trained vision features that are aligned with the language for better generalization to unseen semantic categories and instances [16].

---

The voxel observation v is split into 3D patches of size 5 (akin to In implementation, these patches are extracted with 3D convolution layer with a kernel-size and stride of 5, and then flattened into sequence of voxel encodings. The language encodings are fine- tuned with a linear layer and then appended with the voxel encodings to form the input sequence. We also add learned positional embeddings to the sequence to incorporate voxel and token positions.
like ViT [4].
The input sequence of language and voxel encodings extremely long. A standard Transformer with O(²²) self-attention connections and an input of (100/5)3 = 8000 patches is hard to fit on the memory of commodity GPU, Instead, we use the Perceiver [1] Transformer. Perceiver is a latent- space Transformer, where instead of attending to the entire input, it first computes cross-attention between the input and a much smaller set of latent vectors (which are randomly initialized and trained). These latents are encoded with self-attention layers, and for the final output, the latents are again cross-attended with the input to match the input-size. See Appendix Figure for an illustration. By default, we use 2048 latents of dimension 512 22018612 but in Appendix G we experiment with different latent sizes.
The Perceiver Transformer uses 6 self-attention layers to encode the latents and outputs a sequence of patch encodings from the output cross-attention layer. These patch encodings are upsampled with a 3D convolution layer and tri-linear upsampling to decode 64-dimensional voxel features. The decoder includes a skip-connection from the encoder (like in UNets [77]). The per-voxel features are then used to predict discretized actions [14]. For translation, the voxel features are reshaped into the original voxel grid (100²) form 3D Q-function of action-values. For rotation, gripper open, and collide, the features are max-pooled and then decoded with linear layers to form their respective Q-function. The best action T chosen by simply maximizing the Q-functions:
Trrans arginax Qerans ((x, y, 2) 1,,1, (x.
Tro argmaa Qoor ((v, v,1), (v.0,)) Toollide argmax Qcollide K v,1),
Topon argmax Dopan w v,1),
where (a, y, 2) is the voxel location in the grid, (t), 0, の) are discrete rotations in Euler angles, w is the gripper open state and is the collide variable. See Figure 5 for examples of Q-predictions.
3.4 Training Details
PERACT is trained through supervised learning with discrete-time input- tuples from a dataset of demonstrations. These tuples are composed of voxel observations, language goals, and key teyframe actions ((v1, I k)), (v2, 12, k2), ... During training, we randomly sample tuple and supervise the agent to predict the keyframe action k given the observation and goal (v. 1). For translation, the ground-truth action is represented as one-hot voxel encoding Yrans R H Rotations also represented with a one-hot encoding per rotation axis with R rotation bins Y, (3600/ R) (R = 5 degrees for all experiments). Similarly, open and collide variables are binary one-hot vectors Yopen IR? Yoollide IR? The agent is trained with cross-entropy loss like classifier:
Listal Ey,.. (rog)) Ey,, [logVro] -Ey,,e [log)apen] Fyyaaa [logVeolide):
where Virans softmax Qieans ()) VeSt softmax Que 6, Φ) I)), Vopen 1)), Veollide = softmax (x/v,, respectively. For robustness, we also augment and k with translation and rotation perturbations. See Appendix E for more details.
softmax Copen
(wlv,, (ooen(w/lv,
By default, we use a voxel grid size of 1003. We conducted validation tests by replaying expert demonstrations with discretized actions to ensure that 100't is sufficient resolution for execution. The agent was trained with a batch-size of 16 on 8 NVIDIA V100 GPUs for 16 days (600K itera- tions). We use the LAMB [78] optimizer following Perceiver [1].
For multi-task training, we simply sample input-action tuples from all tasks in the dataset. To ensure that tasks with longer horizons are not over-represented during sampling, each batch contains a uniform distribution of tasks. That is, we first uniformly sample a set of tasks of batch-size length, then pick random input-action tuple for each of the sampled tasks. With this strategy, longer- horizon tasks need more training steps for full coverage of input-action pairs, but all tasks are given equal weighting during gradient updates.
5

---

4 Results
We perform experiments to answer the following questions: (1) How effective is PERACT com- pared to unstructured image-to-action frameworks and standard architectures like 3D ConvNets? And what are the factors that affect PERACT's performance? (2) Is the global receptive field of Transformers actually beneficial over methods with local receptive fields? (3) Can PERACT be trained on real-world tasks with noisy data?
4.1 Simulation Setup
We conduct our primary experiments in simulation for the sake of reproducibility and henchmarking.
Environment.
The simulation is set in CoppelaSim [79] and interfaced through PyRep [80]. All experiments use a Franka Panda robot with parallel gripper. The input observations are captured from four RGB-D cameras positioned at the front, left shoulder, right shoulder, and on the wrist, as shown in Appendix Figure 7, All cameras are noiseless and have resolution of 128 x 128.
Tasks, We train and evaluate on 18 RLBench [15] tasks. See per- act gith for examples and Appendix A for details on individual tasks. Each task includes several variations, ranging from 2-60 possibilities, e.g., in the stack blocks task. "stack 2 red blocks" and "stack 4 purple blocks" " are two variants. These variants are randomly sampled during data genera- tion, but kept consistent during evaluations for one-to-one comparisons. Some RI.Bench tasks were modified to include additional variations stress-test multi-task and language-groune groggdigggnninggg capabil- ities. There are a total of 249 variations across 18 tasks, and the number of extracted keyframes range from 2-17. All keyframes from an episode have the same language goal, which is constructed from templates (but Human-annotated for real-worle tasks). Note that in all experiments, we do not test for generalization to unseen objects, i.e., our train and test objects are the same. However during test time, the agent has to handle novel object poses, randomly sampled goals, and randomly sampled scenes with different semantic instantiations of object colors, shapes, sizes, and categories. The focus here is to evaluate the performance of single multi-task agent trained on all tasks and variants.
Language-Conditinn
Evaluation Metric. Each multi-task agent is evaluated independently on all 18 tasks. Evaluations scored either 0 for failures or 100 for complete successes. There are no partial credits. We report average success rates on 25 evaluation episodes per task (25 x 18 = 450 total episodes) for agents trained with n = 10, 100 demonstrations per task. During evaluation, an agent keeps taking actions until an oracle indicates task-completion or reaches a maximum of 25 steps.
4.2 Simulation Results
Table reports success rates of multi-task agents trained on all 18 tasks. We could not investigate single-task agents due to resource constraints of training 18 individual agents.
Baseline Methods. We study the effectiveness of our problem formulation by benchmarking against two snnguage-conditione baselines: Image- BC and C2FARM-BC. BC. Imaage-CCla is an imae-ttoaaction
agent similar BC-Z [12]. Following BC-Z, we use FiLM [81] for conditioning with CLIP [76] language features, but the vision encoders take in RGB-D images instead of just RGB. We also study both CNN and ViT vision encoders. C2FARM BC is a 3D nally-convllutional network by James et al. [14] that has achieved state-of-the-ar results on RLBench tasks. Similar to our agent, C2FARM- BC also detects actions in a voxelized space, however it uses coarse-to- tonetto---eeeeeee finegrrinn scheme to detect actions at two-levels of voxelization: 32 voxels with a 1'm grid, and 323 voxels with a 0.15'm grid after "zooming in" from the first level. Note that at the finest level, C2FARM-BC has higher resolution (0.47cm) than ACT (Iem). We use the same 3D ConvNet architecture as James et al. [14], but instead of training it with RL, we do BC with cross-cntropy loss (from Section 3.4). We also condition it with CLIP [76] language features at the bottleneck like in LingUNets [82, 16].
Multi-Task Performance. Table 1 compares the performance of Image-BC and C2FARM- BC against PERACT. With insufficient demonstrations, Image-BC BC has near zero performance on most tasks. Image-BC disadvantaged with single-view observations and has to learn hand-eye coordination from scratch. In contrast, PERACT's voxel-based formulation naturally allows for in- tegrating multi-view observations, learning 6-DoF action representations, and data-augmentation in 3D, all of which are non-trivial to achieve in image-based methods. C2FARM-BC is the most competitive baseline, but has limited receptive field mostly because of the coarse-to-fine- scheme and partly due to the convolution-onle architecture. PERACT outperforms C2FARM-BC in

---

opan draver
slide block
szeep to duatpan
neat off
turn tap
put in
clase
drag stick
stack blocka
grilll
jar
Method
10
100
10
100
10
100
100
10
100
10
100
100
10
100
10
100
Image-BC (CNN)
4
4
0
0
0
0
0
20
0
8
0
0
0
0
Image-BC (VIT)
16
0
0
CZHARM-BC
28
20
12
16
4
0
40
20
60
68
12
4
28
24
72
24
4
0
(w/o Lang)
20
48
16
PERACT
68
80
32
72
72
56
68
84
72
80
16
68
32
60
36
68
12
36
Scree
put in
place
put in
sort shape
push
insert
stack
place
bulb
cupboard
buttona
peg
cupa
cupa
10
100
10
100
10
100
100
10
100)
10
100
100
10
100
10
100
Imuge-BC BC (CNN)
Image-BC, BC (VIT)
0
0
0
4
0
4
0
0
16
0
0
0
0
0
0
C2FARM-BC
36
0
88
72
0
PERACT (wlo Lang)
0
24
20
8
20
0
0
60
68
0 0
0
PERACT
28
24
44
20
16
16
56
48
0
0
Table I. Multi-Task Test Results. Success rates (mean %%) of various multi task agents tasks trained with either 10 or 100 demonstrations per task and evalunted 25 episodes per task, Each evaluation episode is scored either for failure 100 succces. PERACT outperforms C2FARM-BC (14), the most competitive baseline, with an average improvement of 1.33x with 10 demos and 2.83x with 100 demos.
25/36 evaluations in Table I with average improvement of 1.33x with 10 demonstrations and 2.83x with 100 demonstrations. For a number of tasks, C2FARM-BB BC actually performs worse with more demonstrations, likely due to insufficient capacity. Since additional training demonstra- tions include additional task variants to optimize for, they might end up hurting performance.
In general, 10 demonstrations are sufficient for PERACT to achieve > 65% success on tasks with limited variations like open drawer (3 variations). But tasks with more variations like stack blocks (60 variations) need substantially more data, sometimes to simply cover all possible con- cepts like "teal color block" that might have not appeared in the training data. See the simulation rollouts in the supplementary video to get a sense of the complexity tasks: insert peg. stack cups, and pl ace cups, all agents achieve near success. These are very high-precision tasks where being off by a few centimeters or degrees could lead to unre- coverable failures. But in Appendix H we find that training single-task task agents, specifically for these tasks, slightly alleviates this issue.
these evaluations. For three
Ablations. Table reports PERACT w/o Lang, an agent without any language conditioning. Without language goal, the agent does not know the underlying task and
100
80
Success Rata
performs chance. We also report additional ablation results on the open drawer task in Figure 3. To sum- marize these results: (1) the skip connection helps train the agent slightly faster, (2) the Perceiver Transformer is crucial for achieving good performance with the global receptive field, and (3) extracting good keyframes actions essential for supervised training as randomly chosen or fixed-interval keyframes lead to zero-performance.
60
40
PerAct
Peckut wés silp
20
PerAct random keypoints
Pectut wis Perceiver
PerAct interval keypaints
0-
10000
20000 Training Steps
Figure Ablation Experiments. Success rate PER- ACT after ablating key components.
Sensitivity Analysis. In Appendix G we investigate factors that affect PERACT's performance: the number of Perceiver latents, voxelization resolution, and data augmentation. We find that more latent vectors generally improve the capacity of the agent to model more tasks, but for simple short- horizon tasks, fewer latents are sufficient. Similarly, with different voxelization resolutions, some tasks are solvable with coarse voxel grids like 323. but some high-precision tasks require the full 1003 grid. Finally, rotation perturbations in the data augmentation generally help improving robustness essentially by exposing the agent more rotation variations of objects.
4.3 Global vs. Local Receptive Fields
Packet
To further investigate our Transformer agent's global re- ceptive field, conduct additional experiments on the open drawer task. The open draver task has three variants: "open the top drawer". "open the middle drawer". and "open the bottom drawer". and with a lim- ited receptive field is hard to distinguish the drawer handles, which are all visually identical. Figure 4 re- ports PERACT and C2FARM-BC agents trained with 100 demonstrations. Although the open drawer tasks can be
Succiss Rate
CIARMAB (1636]
C2FARMBC 132.22
(2FRMBBC 164641
CHARMBC (161
C2FRM BC (52)
C2FAMM BC (64)
C2ARBABC (66)
10000
20000
30000
40000
Training Steps
Figure 4. Global
Local Receptive Field Ex-
periments. Success rales of PERACT against various
C2FARM-BC. [14] haselines
7

---

Q Preddiction
Expert Action
"place the wine bottle on the middle rack"
"sweep the dirt to the tall dustpan"
"ress the handsan"
"put the tape in the top drawer"
Qualitative examples translation PrPrdiictions from PER ACT along with expert actions, highlighted with dotted-circles. The loft two are simulated tasks, and the right two are real-world tasks. See Appendix J for more examples.
Figure 5. Q-Prediction Examples:
solved with fewer demonstrations, here we want to ensure that insufficient data is not an issue. We include several versions of C2FARM-BC with different voxelization schemes. For instance, [16, 16] indicates two levels of 16 voxel grids at Im and 0.15m', respectively. And [64] indicates a sin- gle level of 64 voxel grid without the boase-to-fine-grain scheme. PER ACT the only agent that achieves > 70% success, whereas all C2FARM-BC versions perform at chance with ~ 33%, indicating that the global receptive field of the Transformer crucial for solving the task.
4.4 Real-Robot Results
We also validated results with real-robot experiments Franka Emika Panda. See Appendix for setup details. With- any sim-to-real transfer or pre-training, we trained multi-task PERACT agent from scratch on 7 tasks (with 18 unique variations) from total of just 53 demonstrations, See the supplementary video for qualitative results that showease the diversity of tasks and ro- bustness scene changes. Table reports success rates from small- scale evaluations. Similar to the simulation results, we find that PERACT is able to achieve 65% success on simple short-horizon tasks like pressing hand-sanitizers from just handful number of demonstrations. The most common failures involved predicting incorrect gripper open actions, which often lead the agent into unseen states. This could be addressed in future works by using HG-DAgger style approaches to correct the agent [12]. Other issues included the agent exploiting biases in the dataset like in prior work [16]. This could be addressed by scaling up expert data with more diverse tasks and task variants.
Task
# Tminn
Test Suce. %
Press Handsan
5
10
Put Marker
8
10
Put in Drawer
8
10
40
Hit Ball
8
Stack Blocks
10
Sweep Beans
8
Success (mean of a'auiiii task model trained an evaluated real. tasks (see Figure 1).
Limitations and Conclusion
We presented PERACT, Transformer-based multi-task agent for 6-DoF manipulation. Our experi- ments with both simulated and real-world tasks indicate that the right problem formulation, i.e., de- tecting voxel actions, makes substantial difference in terms of data efficiency and robustness.
While PERACT is quite capable, extending it to dexterous continuous control remains challenge. PERACT is at the mercy of sampling-basee motion-planne to execute discretized actions, and is easily extendable N-DoF actuators like multi-fin finle--fi hands. See Appendix L. for extended discussion on PERACT's limitations. But overall, we are excited about scaling up robot learning with Transformers by focusing on diverse rather than narrow multi-task data for robotic manipulation.
Acknowledgments
We thank Selest Nashef and Karthik Desingh for their help with the Franka setup UW. We thank Stephen James for helping with RLBench and ARM issues. We are also grateful to Valts Blukis, Zoey Chen, Markus Grotz, Aaron Walsman, and Kevin Zakka, for providing feedback on the initial draft. And thanks to Shikhar Bahl for initial discussions. This work was funded in part by ONR under award 4057984405880. Mohit Shridhar supported by the NVIDIA Graduate Fellowship. and was also part-time intern at NVIDIA throughout the duration of this project.

---

References
[1] A. N.Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding. S. Koppula, D. Zoran,
A. Brock, E. Shelhamer, al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107. 147995 2021.
[2] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, L.. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeuRIPS), 2017.
[3] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Neural In- formation Processing Systems (NeurIPS), 2020.
[4] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transform- ers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2020.
[5] J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, Bates, A. Zidek, A. Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 2021.
[6] O. Vinyals, I. Babuschkin, J. Chung, M. Mathieu, M. Jaderberg, W. M. Czarnecki, A. Dudzik, A. Huang, P. Georgiev, R. Powell, et al. Alphastar: Mastering the real-time strategy game starcraft ii. DeepMind blog, 2, 2019.
[7] T. Chen, S. Saxena, L. Li, D. J. Fleet, and G. Hinton. Pix2seq: A language modeling frame- work for object detection. arXiv preprint 2rXiv:2109.18855 2021.
[8] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbecl, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Neural Information Processing Systems (NeurIPS), 2021.
[9] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, al. A generalist agent. arXiv preprint arXiv:2205.061
2022.
[10] J. Devlin, M. W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguisties (NAACL), 2018.
[11] Jacgle, Gimeno, A. Brock, O. Vinyals, A. Zisserman, and Carreira. Perceiver: General perception with iterative attention. In International Conference on Machine Learning (ICML), 2021.
[12] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference Robot Learning (CoRL), 2021.
[13] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
[14] S. James, K. Wada, T. Laidlow, and A. Davison. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. In Computer Vision and Pattern Recognition (CVPR), 2022.
[15] James, 7. Ma, D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters (RA-L.) 2020.
[16] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manip- ulation. In In Conference on Robot Learning (CoRL), 2021.

---

[17] A. Zeng, P. Florence, J. Tompson. S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRL.), 2020.
[18] J. Gibson. The ecological approach to visual perception: classic edition. Psychology Press, 2014.
[19] R. A. Brooks. New approaches to robotics. Science, 1991.
[20] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[21] Y. Xiang, Schmidt, Narayanan, and D. Fox. Posecnn: convolutional neural network for 6d object pose estimation in cluttered scenes. In Robotics: Science and Systems (RSS), 2018.
[22] M. Zhu, K. G. Derpanis, Y. Yang, S. Brahmbhatt, M. Zhang, C. Phillips, M. Lecce, and K. Danillidis. Single image 3d object detection and pose estimation for grasping. In 2014 IEEE International Conference on Robotics and Automation (ICRA), 2014.
[23] A. Zeng, K.-T. Yu, S. Song, D. Suo, E. Walker, A. Rodriguez, and J. Xiao. Multi-view self- supervised deep learning for 6d pose estimation in the amazon picking challenge. In 2017 IEEE international conference on robotics and automation (ICRA), 2017.
[24] X. Deng, Y. Xiang, A. Mousavian, C. Eppner, T. Bretl, and D. Fox. Self-supervised 6d object pose estimation for robot manipulation. 2020 IEEE International Conference on Roboties and Automation (ICRA), 2020.
[25] C. Xie, Y. Xiang. A. Mousavian, and D. Fox. The best of both modes: Separately leveraging rgb and depth for unseen object instance segmentation. In Conference on Robot Learning (CoRL), 2020.
[26] A. Zeng,
S. Song, K.TT. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, et al. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. The International Journal of Robotics Research (IJRR), 2019.
[27] E. Stengel-Eskin, A. Hundt, Z. He, A. Murali, N. Gopalan, M. Gombolay, and G. Hager. Guiding multi-step rearrangement tasks with natural language instructions. Conference on Robot Learning (CoRL), 2022.
[28] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotie manipulation. Conference on Robot Learning (CoRL), 2018.
[29] Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel. Learning to Manipulate Deformable Objects without Demonstrations. In Robotics: Science and Systems (RSS), 2020.
[30] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1): 1373, 2016.
[31] C. Finn and Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017.
[32] S. Song, A. Zeng, J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6dof closed- loop grasping from low-cost demonstrations. IEEE Robotics and Automation Letters (RA-L), 2020.
[33] A. Murali, A. Mousavian, Eppner, C. Paxton, and D. Fox. 6-dof grasping for target- driven object manipulation in clutter. In International Conference on Robotics and Automa- tion (ICRA), 2020.
[34] A. Mousavian, C. Eppner, and D. Fox. d- graspnet: Variational grasp generation for object manipulation. In International Conference Computer Vision (ICCV), 2019.
10

---

[35] Z. Xu, H. Zhanpeng, and S. Song. Umpnet: Universal manipulation policy network for articulated objects. Robotics and Automation Letters (RA-L), 2022.
[36] S. Agrawal, Y. J.-S. Liu, S. K. Feiner, and S. Song. Scene editing as teleoperation: A case study in 6dof kit assembly. arXiv preprint arXiv:21 fr.0422110004450 2021.
[37] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitz- mann. Neural descriptor fields: Se equ-equrra object representations for manipulation. arXiv preprint arXiv:2112.05114, 2021.
[38] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: universal visual represen- tation for robot manipulation. arXiv preprint 223. 2022.
[39] W. Yuan, C. Paxton, K. Desingh, and D. Fox. Sornet: Spatial object-centric representations for sequential manipulation. In In Conference Robot Learning (CoRL). PMLR, 2021.
[40] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprini arXiv: 1692, 2019.
[41] 7. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, 7. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In In International Conference on Computer Vision (ICCV), 2021.
[42] M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. Neural information processing systems (NeurIPS), 2021.
[43] K.-H. Lee, O. Nachum, M. Yang, L. Lee, D. Freeman, W. Xu, S. Guadarrama, I. Fis- cher, E. Jang, H. Michalewski, et al. Multi-game decision transformers. arXiv preprint arXiv:2205. 15241, 2022.
[44] M. Clever, Handa, H. Mazhar, Parker, Shapira, Q. Wan, Y. Narang, I. Akinola, M. Cakmak, and D. Fox. Assistive tele-op: Leveraging transformers to collect robotic task demonstrations. arXiv preprint arXiv:2112.05129 2021.
[45] R. Yang, M. Zhang, N. Hansen, H. Xu, and X. Wang. Learning vision-guided quadrupedal locomotion end-to-end with cross-modal transformers. arXiv preprint arXiv:2107.03996 2021.
[46] D. S. Chaplot, D. Pathak, and J. Malik. Differentiable spatial planning using transformers. In International Conference on Machine Learning (ICML), 2021.
[47] J. Johnson, L. Li, A. H. Qureshi, and M. C. Yip. Motion planning transformers: One model to plan them all. arXiv preprint arXiv:2106.02791, 2021.
[48] S. Dasari and A. Gupta. Transformers for one-shot visual imitation. arXiv preprint 1rXiv:2011.05977 2020.
[49] H. Kim, Y. Ohmura, and Y. Kuniyoshi. Transformer-based deep imitation learning for dual- robot manipulation. In International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021.
[50] Gupta, Fan, S. Ganguli, and L. Fei-Fei. Metamorph: Learning universal controllers with transformers. arXiv preprint arXiv:2203.1193: 2022.
[51] W. Liu, C. Paxton, Hermans, and D. Fox. Structforner: Learning spatial structure for Ianguage-giided semantic rearrangement of novel objects. In International Conference on Robotics and Automation (ICRA), 2022.
[52] Y. Han, R. Batra, N. Boyd, T. Zhao, Y. She, S. Hutchinson, and Y. Zhao. Learning general- izable vision-tactile robotic grasping strategy for deformable objects via transformer. arXiv preprint arXiv:2112.063777 2021.
11

---

[53] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. Conference on Robot Learning (CoRI.), 2020.
[54] M. Shridhar and D. Hsu. Interactive visual grounding of referring expressions for human- robot interaction. In Robotics: Science and Systems (RSS), 2018.
[55] C. Matuszek, L. Bo, L. Zettlemoyer, and D. Fox. Learning from unscripted deictic gesture and language for human-robot interactions. In AAAI Conference on Artificial Intelligncce, volume 28, 2014.
[56] M. Bollini, Tellex, T. Thompson, N. Roy, and D. Rus. Interpreting and executing recipes with cooking robot. In Experimental Robotics, pages 481-495. Springer, 2013.
[57] D. K. Misra, J. Sung, K. Lee, and A. Saxena. Tell me dave: Context-sensitive grounding of natural language to manipulation instructions. The International Journal of Robotics Re- search (IJRR), 2016.
[58] Y. Bisk, D. Yuret, and D. Marcu. Natural language communication with robots. In North American Chapter of the Association for Computational Linguistics (NAACL), 2016.
[59] J. Thomason, S. Zhang, R. J. Mooney, and P. Stone, Learning to interpret natural language commands through human-robot dialog. In Twenty-Fourth International Joint Conference Artificial Intelligence (ICCAI), 2015.
[60] J. Hatori, Y. Kikuchi, S. Kobayashi, K. Takahashi, Y. Tsuboi, Y. Unno, W. Ko, and J. Tan. Interactively picking real-world objects with unconstrained spoken language instructions. In International Conference on Robotics and Aulomation (ICRA), 2018.
[61] Y. Chen, R. Xu, Y. Lin, and P. A. Vela. A Joint Network for Grasp Detection Conditioned on Natural Language Commands. arXiv:2104.0049) [cs], Apr. 2021.
[62] V. Blukis, R. A. Knepper, and Y. Artzi. Few-shot object grounding for mapping natural language instructions to robot control. In Conference on Robot Learning (CoRL), 2020.
[63] C. Paxton, Y. Bisk, J. Thomason, A. Byravan, and D. Fox. Prospection: Interpretable plans from language by predicting the future. In International Conference on Robotics and Au- tomation (ICRA), 2019.
[64] S. Tellex, T. Kollar, S. Dickerson, M. Walter, Banerjee. S. Teller, and N. Roy. Understand- ing natural language commands for robotic navigation and mobile manipulation. In AAAI Conference on Artificial Intelligence (AAAI), 2011.
[65] T. Nguyen, N. Gopalan, R. Patel, M. Corsaro, E. Pavlick, and S. Tellex. Robot object retrieval with contextual natural language queries. arXiv preprint arXiv:2006. 13253, 2020.
[66] Y. Bisk, A. Holtzman, J. Thomason, J. Andreas, Y. Bengio, J. Chai, M. Lapata, A. Lazaridou, J. May, A. Nisnevich, N. Pinto. and J. Turian. Experience grounds language. In Empirical Methods in Natural Language Processing (EMNLP), 2020.
[67] S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn, et al. Learning language-conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning (CoRL), 2022.
[68] O. Mees, L. Hermann, and W. Burgard. What matters in language conditioned robotic imita- tion learning over unstructured data. IEEE Robotics and Automation Letters (RA-L), 2022.
[69] C. Lynch and P. Sermanet. Grounding language in play. arXiv preprint arXiv:2005.07648, 2020.
[70] O. Mees, L. Hermann, E. Rosete-Beas and W. Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. arXiv preprint 0rXiv:2112.03227. 2021.
12

---

[71] E. Johns. Coarse-to to-fie imitation learning: Robot manipulation from single demonstration.
In International Conference on Robotics and Automation (ICRA), 2021.
[72] S. James and A. Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Robotics and Automation Letters (RA-L), 7(2):1612-1619 2022.
[73] S. Liu, S. James, A. J. Davison, and E. Johns. Auto-lambda: Disentangling dynamic task relationships. Transactions on Machine Learning Research, 2022.
[74] H. Moravec. Robot spatial perceptionby stereoscopic vision and 3d evidence grids. Percep- tion, 1996.
[75] Y. Roth-Tabak and R. Jain. Building an environment model using depth information. Com- puter, 22(6):85-99 90, 1989.
[76] A. Radford, W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103. 0000000 2021.
[77] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical im- age segmentation. In International Conference Medical image computing and computer- assisted intervention, pages
Springer, 2015.
[78] Y. You, J. Li, S. Reddi, Hseu, S. Kumar, Bhojanapalli, X. Song. J. Demmel, K. Keutzer, and C.-. Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv: 00962, 2019.
[79] E. Rohmer, S. N. Singh, and M. Freese. V-rep: versatile and scalable robot simulation framework. In International Conference on Intelligent Robots and Systems (IROS), 2013.
[80] S. James, M. Freese, and A. J. Davison. Pyrep: Bringing v-rep to deep robot learning, arXiv preprint arXiv: // 176, 2019.
[81] E. Perez, Strub, H. De Vries, V. Dumoulin, and Courville. Film: Visual reasoning with a general conditioning layer. In AAAI Conference on Artificial Intelligence, 201 8.
[82] D. Misra, A. Bennett, V. Blukis, E. Niklasson, M. Shatkhin, and Y. Artzi. Mapping in- structions to actions in 3d environments with visual goal prediction. In 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
[83] Z. Mandi, P. Abbeel, and S. James. On the effectiveness of fine-tuning versus meta- reinforcement learning. arXiv preprint arXiv:2206.03271, 2022.
[84] S. Sodhani, A. Zhang, and Pineau. Multi-task reinforcement learning with context-based representations. In M. Meila and T. Zhang, editors, International Conference on Machine Learning (ICML), 2021.
[85] M. Shridhar, X. Yuan, M.-A. Côté, Bisk, A. Trischler, and M. Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In International Con- ference on Learning Representations (ICLR), 2021.
[86] A. Zeng, A. Wong, Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sind- hwani, J. Lee, V. Vanhoucke, et al. Socratic models: Composing zero-shot multimodal rea- soning with language. arXiv preprint urXiv:2204.00500 2022.
[87] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207 2022.
[88] L. P. Kaelbling and T. Lozano-Pérez. Integrated task and motion planning in belief space. The International Journal of Robotics Research, 2013.
[89] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano- Pérez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 2021.
13

---

[90] G. Konidaris, L. P. Kaelbling, and T. Lozano-Perez. From skills to symbols: Learning sym- bolic representations for abstract high-level planning. Journal of Artificial Intelligence Re- search, 2018.
[91] J. Mao, Y. Xue, M. Niu, H. Bai, J. Feng, X. Liang, H. Xu, and C. Xu. Voxel transformer for 3d object detection. In International Conference on Computer Vision (ICCV), 2021.
[92] C. He, R. Li, S. Li, and L. Zhang. Voxel set transformer: A set-to-set approach to 3d object detection from point clouds. In Computer Vision and Pattern Recognition (CVPR), pages 8417-8427, 2022.
[93] K. Zheng, R. Chitnis, Y. Sung, G. Konidaris, and S. Tellex. Towards optimal correlational object search. In International Conference on Robotics and Automation (ICRA), 2022.
[94] V. Blukis, C. Paxton, D. Fox, A. Garg, and Y. Artzi. A persistent spatial semantic representa- tion for high-level natural language instruction execution. In Conference on Robot Learning, pages 706-717. PMLR, 2022.
[95] R. Corona, S. Zhu, D. Klein, and T. Darrell. Voxel-informed language grounding. In Associ- ation for Computational Linguistics (ACI., 2022.
[96] V. Sitzmann, J. Thies, F. Heide, M. Nießner, G. Wetzstein, and M. Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[97] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 2022.
[98] Sara -ridovich-Keil and Alex Yu, M. Tancik, Q Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
[99] S. Lal, M. Prabhudesai, I. Mediratta, A. W. Harley, and K. Fragkiadaki. Coconets: Contin- uous contrastive 3d scene representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[100] H.-Y. F. Tung. Z. Xian, M. Prabhudesai, S. Lal, and K. Fragkiadaki. 3d-oes: Viewpoint- invariant object-factorized environment simulators. arXiv preprint arXiv:2011.06460 2020.
[101] M. Nicßner, M. Zollhöfer, S. Izadi, and M. Stamminger. Real-time 3d reconstruction at scale using voxel hashing. ACM Transactions on Graphics (ToG), 2013.
[102] Y. Tay, M. Dehghnnn, D. Bahri, and D. Metzler. Efficient transformers: A survey. ACM Computing Surveys (CSUR), 2020.
[103] A. Goyal, A. R. Didolkar, A. Lamb, K. Badola, N. R. Ke, N. Rahaman, J. Binas, C. Blundell, M. C. Mozer, and Y. Bengio. Coordination among neural modules through shared global workspace. In International Conference on Learning Representations (ICLR), 2021.
[104] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision (ECCV), 2020.
[105] F. Locatello, D. Weissenbom, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, Dosovitskiy, and Kipf. Object-centric learning with slot attention. Neural Informa- tion Processing Systems (NeuriPs), 2020.
[106] R. Ranftl, A. Bochkovskiy, and V. Koltun. Vision transformers for dense prediction. In
International Conference on Computer Vision (ICCV), pages 12179-12188, 2021.
[107] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
[108] S. James and P. Abbeel. Coarse-to-fine q-attention with learned path ranking. arXiv preprint SXXi::20040017// 2022.
14

---

[109] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions on Roboties (T-RO), 2020.
[110] A. Kamath, M. Singh, LeCun, Misra, G. Synnaeve, N. Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. arXiv preprint arX1v:2104.117633 2021.
[111] A. Birhane, V. U. Prabhu, and E. Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint 1rX0v:21/0001966, 2021.
[112] E. M. Bender, T. Gebru, A. MeMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In 2021 ACM Conference on Fairness, Account- ability, and Transparency, pages 610-623, 2021.
[113] Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. Huang. A tutorial on energy-based learning. Predicting structured data, 100), 2006.
[114] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. In Conference Robot Learning (CoRL), 2022.
15


Reviews:
[{"review_id": "gLpyBHZ-Dv", "reviewer": "Reviewer_YHWS", "summary": "This paper presents a transformer-based language-conditioned multi-task imitation learning method for 6-DoF manipulation. Unlike prior language-conditioned multi-task BC methods operated on 2D images, this paper encodes both the language instructions and the RGB-D voxel observations. The authors show that such a method improves over prior approaches in both simulated and real-world robotic manipulation tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. This paper presents a new language-conditioned multi-task BC method operated on RGB-D voxels. The authors utilize the Perceiver to handle the high-dimensional observation space, which seems novel.\n\n2. The empirical results appear rather convincing. The method is able to outperform prior methods such as Image-BC (similar to BC-Z) and C2FARM-BC in the multi-task simulated robotic manipulation tasks with limited demonstrations. The authors also show that the method can work well on the real robot.\n\n3. The paper is well-written and easy to understand.\n\n**Weakness:**\n\n1. The paper extends prior works such as SayCan and BC-Z to the setting with the extra RGB-D voxel in the observations but the main technique is not too different from prior works, which makes the contribution a bit less novel. \n\n2. The paper does not compare to SayCan in the experiments, which seems to be necessary. Moreover, there are no baseline results in the real-world setting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Based on the summary in the Strengths And Weaknesses section, I think while the paper lacks some additional comparisons and is a bit incremental, the good empirical results and novel usage of the Perceiver architecture make senses and I would give the paper a weak accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "1. Compare to SayCan.\n\n2. Add baseline results in the real-world experiments.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "_hS1n8jLZAAH", "reviewer": "Reviewer_vWwj", "summary": "This paper proposes to use a Transformer to perform multi-task manipulation via language-conditioned behavior-cloning. The key challenge is Transformer is known to be data inefficient. By casting behavior-cloning as an action detection task, PREACT achieves great efficiency on a range of simulated and real-world tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n* The paper is well-organized and easy to follow.\n* The proposed method is succinct and easy to understand.\n* The tasks are challenging, and the proposed method achieves good performance while maintaining good data efficiency.\n* The real robot experiments make the paper promising.\n\nSuggestions and Improvements：\n* As the author mentioned, both Transformers and 3D Convs are data inefficient, but I did not see why detecting voxel actions is extremely data efficient (i.e. achieving ~$60\\%$ trained with only 53 demos across 7 tasks.) Could you provide more analysis?\n* It is hard to evaluate the generalizability of PREACT. As mentioned in L213, all objects are seen during training. Could you please provide more detailed differences between training and testing? (e.g. the size differences as shown on the website; training: stack yellow block on blue block, testing: stack red block on yellow block; training: turn left tap, testing: turn right tap, etc)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I'd recommend \"weak accept\". The paper studies difficult problems and achieves good performance. It would be great if the author can provide more analysis about why action detection (classification) makes such a huge difference from regression (BC). ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Strength:\n* L45, please consider defining action-centric representation when it first shows up.\n* L125, please consider providing an intuitive understanding of keyframes, e.g. \"the pose of the end-effector at the point where object interaction should begin\"\n* L174, one task may require several keyframes to finish, are the language goals remain the same?\n"}}, {"review_id": "OkTx4K4rkXy", "reviewer": "Reviewer_hXJH", "summary": "This paper proposes a language-guided robot manipulation framework built on top of Perceiver IO. The proposed framework – Perceiver-Actor or PerAct in short, performs multi-stage manipulation tasks in a multi-task manner, showing superior performance in both simulated environments (RLBench) and in real-world settings trained with few demonstrations.   ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: This paper shows a promising direction to scale up vision-based manipulation tasks with transformers. It combines modern large-scale pre-training frameworks, using CLIP to encode language instructions, and using Perceiver IO for learning discretized actions. Using Perceiver IO is significantly helpful to reduce memory required for encoding high-resolution colored voxels inputs, (attended with a much smaller latent embedding) compared to previous methods like C2FARM alternatively require to multi-stage (coarse-to-fine) to encode fine-level voxels with limited memory. \n\nWeakness: Though the proposed framework is cleaner in design, it poses some other limitations.\n\n-- The training setting for keyframe discovery and learning Q-attention values is pretty much following ARM and C2F ARM. The authors did mention in keyframe discovery but missed the related references in action parametrization (L162 - 170).\n\n-- The training time requirement is significantly larger than the related works. In L185-186, the authors stated that the training needs 8 V100 for 16 days (ImageNet experiments with the same amount of compute takes only 2 days, but with 1M training data, here the number of training data would be definitely less than 100k), whilst C2F ARM only requires less than 1 hour for each task in a single GPU. Considering each task is only augmented with 10 demonstrations, I hope the authors could slightly explain why training takes such long time? Is it simply because transformers in general take longer time to train?\n\n-- Design details in baseline methods. Are two mentioned baseline methods: Image-BC / C2FARM-BC single-task or multi-task learning methods? L223-224 indicates all methods are multi-task agents, how the sense of “task” then is fed to these baselines? Or the authors just treat everything is an image-to-action type of learning tasks without introducing the idea of task in training? This is slightly unfair, since the sense of task is encoded in language in the proposed framework.\n\n-- Compared to other multi-task methods. Multi-task robot learning has also been heavily investigated in the multi-task learning community. For example, notable works like: PCGrad [1], Auto-Lambda [2] also evaluated multi-task robot learning which might be more suitable baselines compared to Image-BC / C2F ARM-BC which were designed as single-task learning methods. In particular, Auto-Lambda also performs multi-task learning on RLBench (though not language guided), but with the performance of same evaluated tasks like: slide block: 77 (72), money in safe: 64 (44), place wine: 19 (12), push buttons: 95 (48), trained with the same number of 100 demonstrations, achieved better performance than the proposed method. I would suggest to do a thorough literature review on multi-task learning on manipulation and should compare with at least 1 or 2 related baselines designed specifically for multi-task manipulation.\n\n[1] Yu et al.  Gradient Surgery for Multi-Task Learning. NeurIPS 2020\n\n[2] Liu et al. Auto-Lambda: Disentangling Dynamic Task Relationships.  TMLR 2022.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In general, I quite like the idea as it shows a promising direction to bring transformers to the manipulation community. I would further raise the score if the authors could resolve my concerns stated in the weaknesses section.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See weaknesses.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "Bb1ObntOOLO", "reviewer": "Reviewer_xg2V", "summary": "The paper introduces a language-conditioned behavior-cloning agent for multi-task 6-DoF visual manipulation. In Leyman terms, the paper is a variation of the Coarse-to-fine ARM paper, where the key differences are:  \n\n- (1) The addition of the Perceiver Transformer [1], rather an a 3D convolution.\n- (2) Using behavior cloning rather than reinforcement learning. \n- (3) Language conditioning to enable multi task/variation learning.  The paper evaluated in both simulation (on 18 RLBench task) and in 7 real world tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- An efficient BC algorithm that performs well on 53 demonstrations.  \n- A large number of simulation experiments for reproducibility, and also real world experiments to show that the method is suitable for real-word training.   \n- The authors did well to state clearly any modifications to the tasks in RLBench , ensuring reproducibility.\n- A detailed description of the limitations (in appendix).  \n- The addition of the perceiver seems to clearly help learning these complex tasks.   \n- Fig 2 gives a good overview of the system.   \n- Other than the introduction, which could be a little clearer, the rest of the paper is well written.   \n\n**Weaknesses**\n\n- The introduction could be improved to better frame the proposed work. There seems to be quite a lot of similarities with the C2F-ARM work through the paper (next-best pose action, 3D voxelization, motion planning, demo aug, keyframe discovery) but it would be good to perhaps front-load a summary of these similarities and differences, rather than having the reader have to go through the entire paper. Try to make it instantly clear what separates this work.  \n\n- There isn’t anything in the paper that is BC specific that has been introduced in this paper, and so it would have been nice to include results of the effect of adding the transformer for the RL agent, rather than just the BC one. \n\n- Ablation on patch size (other than the 5^3 used) would be very useful. \n\n\n**Other minor comments**\n\n- ln 69: “… has a limited receptive field that cannot look at the entire scene at the finest level”. This is not exactly true; worth noting that this can be eleviated with [b].  \n\n- ln 71:  “PERACT does BC instead of RL, which enables us to easily train a multi-task agent for several tasks”, but [a] has trained a multi-task policy with C2F-ARM. I’d suggest amending this comment and elaborate on how these two works differ. For one, the PERACT can do both multi task and multi-variation because it conditions on language, while [a] only does multi task because it does not use task conditioning.\n\n- [c] is relevant work on using a next-best action space for multi-task BC on RLBench tasks. \n\n- Fig 2: Could benefit from having the RGB views to make it clear what is in the voxel grid scene. \n\n- ln 136: [71] should be cited when discussing collide prediction.  \n\n- I wonder if PERACT training/memory overhead could be reduced by applying a coarse-to-fine structure as in C2F-ARM? Fig 4 evaluated multiple hierarchies for C2F-ARM, and it would be nice to see something similar also for PERACT. If we can get similar performance from having this hierarchy, but reduce the memory required, then that would be great.\n\n[a] Mandi et al.  \"On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning.\"\n\n[b] James at al. \"Coarse-to-fine Q-attention with Tree Expansion.\"\n\n[c] Liu et al. \"Auto-Lambda: Disentangling Dynamic Task Relationships.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I believe the work would be valuable to the community. I would consider increasing my recommendation if the majority of the issues are addressed in the rebuttal phase. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See weaknesses", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)