Instruction-driven history-aware policies for robotic manipulations
Authors: Pierre-Louis Guhur
Shizhe Chen
Ricardo Garcia Pinel
Makarand Tapaswi
Ivan Laptev
Cordelia Schmid
Venue: CoRL 2022 Oral (2022)
Keywords: Robotics Manipulation
Language Instruction
Transformer

Abstract: In human environments, robots are expected to accomplish a variety of manipulation tasks given simple natural language instructions. Yet, robotic manipulation is extremely challenging as it requires fine-grained motor control, long-term memory as well as generalization to previously unseen tasks and environments. To address these challenges, we propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations while (iii) keeping track of the full history of observations and actions. Such an approach enables learning dependencies between history and instructions and improves manipulation precision using multiple views. We evaluate our method on the challenging RLBench benchmark and on a real-world robot. Notably, our approach scales to 74 diverse RLBench tasks and outperforms the state of the art. We also address instruction-conditioned tasks and demonstrate excellent generalization to previously unseen variations. 

Instruction-driven history-aware policies for robotic manipulations
Ricardo Garcia', Makarand Tapaswi?, Ivan Laptev', Cordelia Schmid! Inria, École normale supérieure, CNRS, PSL Research University 'IIT Hyderabad gi titb..b.. i veie
Pierre-Louis Guhur!, Shizhe Chen', Ricardo
Abstract: In human environments, robots are expected to accomplish a variety of manipulation tasks given simple natural language instructions. Yct, robotic manipulation is extremely challenging it requires fine-grained motor control, long-term memory well generalization previously unseen tasks and envi- ronments. To address these challenges, we propose a unified transformer-basee approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations while (iii) keeping track of the full history of observations and ac- tions. Such an approach enables learning dependencies between history and structions and improves manipulation precision using multiple views. We evaluate our method on the challenging RLBench benchmark and on real-world robot. Notably, our approach scales to 74 diverse RLBench tasks and outperforms the state of the We also address corruction-ooddtionned tasks and demonstrate excellent generalization to previously unseen variations.
Keywords: Roboties Manipulation, Language Instruction, Transformer
1
Introduction
People can naturally follow language instructions and manipulate objects to accomplish wide range of tasks from cooking assembly and repair. It is also easy to generalize to new tasks by building upon skills learned from previously seen tasks. Hence, one the long-term goals for robotics create generic instruction instruction-oollowing agents that can generalize to multiple tasks and environments.
Thanks to significant advances in learning generic representations for vision and language [2, 3, 5], recent work has made great progress towards this goal [6, 7, 8, 9, 10]. For example, CLiPort [9] exploits CLIP models [5] to encode single-step visual observations and language instructions and to learn a single policy for 10 simulated tasks. BC-Z [10] uses pre-trained sentence encoder [11] generalize multiple manipulation tasks. However, several challenges remain underexplored. One important challenge is that sequential tasks require to track object states that may be hidden from current observations, or to remember previously executed actions. This behaviour is hard to model with recent methods that mainly rely on current observations [9, 10].
Another challenge concerns manipulation tasks that require precise control of the robot end- and-effector reach target locations. Such tasks be difficult to solve with single-vie -igle-view approaches [12], es- pecially in situations with visual occlusions and objects of different sizes, e.g. see put money in safe Figure (left). While several recent approaches combine views from multiple cameras by convert- ing multi-view images into unified 2D/3D space [13, 14] or through a late fusion of multi-view predictions [15], learning representations for multiple camera views is open research problem. Furthermore, cross-modal alignment between vision, action, and text is challenging, in particular when training and test tasks differ terms of objects
order of actions, see Figure I (right). Most of existing methods [9, 10, 16, 17] condense instructions into a global vector to condition policies [18] and are prone to lose fine-grained information about different objects.
To address the above challenges, we introduce Hiveformer- History-aware instraction-condiionne multi-view transformer. It converts instructions into language tokens given pre-trained language
6th Conference Robot Learning (CoRL 2022), Auckland, New Zealand.

---

74 RLBench Tasks
Variations of a Task
Training Episode (Seen Variation)
Testing Episode (Unseen Veriation)
Open
Press the white button, than push the green button, then push the grny one.
Press the do fer blue button, before tepping on the green button and then the lighter blue button.
Sweep to Dustpan
Put Money in Safe
Water Plants
Figure Left: Hiveformer can adapt perform tasks from RLBench [19] given language instructions. Right: Multiple variations of the push buttons task.
encoder [5], and combines visual tokens for both past and current visual observations and proprio- ception. These tokens are concatenated and fed into a multimodal transformer which jointly models dependencies between the current and past observations, spatial relations among views from multi- ple cameras, well fine-grained cross-modal alignment between vision and instruction. Based on the output representations from our multimodal transformer, we predict 7-DoF actions, i.e., position, rotation and state of the gripper. with a UNet [20] decoder.
carry out extensive experiments RLBench [19] in three setups: single-task learning, multi- lask learning, and multi-variation generalization! Our Hiveformer significantly outperforms state- of-the-art models for all three settings, demonstrating the effectiveness of encoding instruction, his- tory and views from multiple cameras with the proposed transformer. Moreover, evaluate model on 74 tasks of RLBench, which goes beyond the tasks used by Liu ally group all the tasks into categories according to their main challenges and analyze results per category for a better understanding. Hiveformer not only excels in the multiple task setting with seen instructions in training, but also enables generalization to instructions that represent dif- ferent variations of the task, even with human-written language instructions. Finally, evaluate our model deployed on real robot and show excellent performance. Interestingly, pretraining the model in the RLBench simulator results in significant performance gains when only small number of real robot demonstrations available.
[15]. manu-
To summarize, our contributions are three-fold:
We introduce new model, Hiveformer, solve various challenges in roboties tasks. jointly models an instruction, multiple views, and history via multimodal transformer for action prediction in robotic manipulation.
perform extensive ablations our model on RLBench with tasks grouped into distinct categorics. The history improves long-term tasks and multi-view setting is most helpful for tasks requiring high precision or in the presence of visual occlusions.
We demonstrate that Hiveformer outperforms the state the art in three RLBench setups, namely single-task, multi-lask and multi-variation. A single Hiveformer trained with synthetic instructions is able to solve multiple tasks and task variations, can generalize to unseen human- written instructions and shows excellent performance real robot after finetuning.
Our code, pre-trained models and additional results are available on the project webpage [1].
Related Work
Vision-based robotic manipulation. While earlier methods for solving robotics tasks such as visual servoing [21, 22] were designed manually, the need cope with large variations of objects and vironments led to the emergence of learning- based neural approaches [23, 24, 25, 26]. Deep neural
We follow definitions in RI Renhh [19] for tasks and variations. task composed of multiple variations that share the same skills but differ objects, attributes order shown in Figure (right).
2

---

networks [8] have achieved impressive results in manipulation for single tasks [29]. and recently led to more challenging setups such as multi-task learning [30, 31, 32, 33j. Different multi-task ap- proaches are explored by discovering which tasks should be trained together [15, 34J, determining shared features across tasks [35, 36], meta-learning [37, 38, 39J, coll-oondiiooed learning [40, 41], or inverse reinforcement learning [42]. These approaches can be generally split two categories according to the training algorithm: reinforcement learning (RL) methods [43, 44, 45, 46] which learn policies from rewards provided by environments and behavioral cloning methods [47, 48, 49] that learn from demonstrations using supervised learning. Demonstrations can be obtained from hu- mans [50], robots [23, 51] or play interactions [49]. The emergence of robotic simulators, such as Gym [52], manipulaTHOR [53], dm_control [54], Sapien [55], CausalWorld [56], and RI- Bench [19], also greatly accelcrated the development of manipulation methods. In this work, we use behavioral cloming to train policies given scripted demonstrations from RLBench [19] which covers many challenging manipulation tasks.
Instruction-driven vision-based robotic manipulation has received growing attention for manip- ulations in 2D planar [57, 58] or recent 3D environments [8, 59, 60), and has been transferred to the real world [9, 10]. As grounding the language in visual scenes is important, existing works have focused challenges in object grounding, such as localizing objects based referring ex- pressions [61. 62, 63] and grounding spatial relationships [7, 64, 65]. Since language describes high-level actions, several works [60, 66, 67] consider a hierarchical approach to decompose a task into sub-goals. Because natural language is rich and diverse, while training resources are limited, further works learn from collected offline data with instructions [10, 17] leverage pre-trained vision-language models (4, 5] for action prediction [9, 68). To further improve the precision of ma- nipulation skills, Mees et al. [8] align instructions with multiple cameras by fusing input images with known camera parameters. Most of these works [6, 8, 9, 10J are stateless, since they only employ current observations to predict next actions. Instead, our work proposes to jointly model language instructions, history, and multi-view observations.
Transformers [69] have led to significant gains in natural language processing [2], computer vi- sion [70] and related fields [4, 5, 71J. They have also been used in the context of supervised re- inforcement learning, such as Decision Transformer [72] Trajectory Transformer [73]. Recent works in Vision-and-Lann and-Language Navigation (VLN) [74, 75, 76] further demonstrate that the Trans- former allows to better leverage previous observations to improve multi-modal action prediction. Transformers are also used to build a multi-modal, multi-task, multi-embodiment generalist agent, GATO [77]. Inspired by the success of transformers, we explore the transformer architecture for instruction-driven and history-aware robotic manipulation.
3 Problem Definition
Our goal is to train policy π (4++1 (x)))//)))) [x))))) (o.)-i-(a;]]-.)) for robotic manipulation conditioned on a natural language instruction {xi visual observations (o:Y and previous actions {a: where is the number of words in the instruction and r is the current step. For robotic control, we use macro steps [12] - key turning points in action trajectories where the gripper changes its state (open/close) or velocities of joints are close to zero. We employ an inverse- -inemiisisss based controller to find trajectory between macro-steps. In this way, the sequence length of an episode is significantly reduced from hundreds of small steps to typically less than macro steps.
The observation o at step consists of RGB images I, and point clouds P. aligned with the RGB images. 4, is composed of { RGB images from K cameras, with each 1, being of size H W x 3 (height, width, 3 channels). Following [15], we use K= 3 with cameras on the wrist. left shoulder and right shoulder of the agent, and H = W = 128. Similarly, P. represents point clouds (Pf 1P from K = cameras. A point cloud P E I W 3x is obtained by projecting single channel depth image H x W from the k-th camera in world coordinates using known camera intrinsics and extrinsies. Each point in Pt has thus 3D coordinates and aligned with pixel in I/.
The action space consists of the gripper pose and its state following the standard setup in RL- Bench [12]. The gripper pose is composed of the Cartesian coordinates pt (x.)) and its rotation described by a quatermion 9, = ( .2,22 relative to the base frame. The gripper's 's state is hoolean and indicates whether the gripper is open or closed. An object is grasped when it is located in between the gripper's two fingers and the gripper is closing its grasp. The exccution of an action is achieved by motion planner in RLBench.
3

---

Inputs
Encoder
Transformer
Prediction
Textt
Use
IEGO
Maa
dirt into the dustpan
Language
LINE
CLIB UONI
Cross Attentitn
Seff-Attention
Feed FFrowardd
Step
UNet
Shap1
Position
RGED & Proproccppiion
Maa
Rotation Teel
UNet
Stop
CMN
State
Figure 2: Hiveformer jointly models instructions, views from multiple cameras, and past actions
and observations with multimodal transformer for robotic manipulation.
4 Our Model: Hiveformer
propose a unified architecture for robotic tasks called Hiveformer (History-aware instruction- conditioned multi-view transformer), see Figure 2 for an overview. It consists of three mod- ules: feature encoding, multimodal transformer and action prediction. The feature encoding mod- ule (Sec. 4.1) generates token embeddings for instructions {xi)/=1 visual observations {or and previous actions (4:)/=1- Then, multimodal transformer (Sec. 4.2) learns relationships between the instruction, current multi-camera observations and history. Finally, the action prediction module (Sec. 4.3) utilizes convolutional network (CNN) to predict the next rotation g++1 and gripper state c+1. and adopts a UNet decoder [20] predict the next position Pr+1-
4.1 Feature Encoding
encade the instruction, visual observations, and actions as sequence of tokens.
Instructions. We employ a pre-trained language encoder to tokenize and encode the sentence in- struction. Specifically, language pre-training, it better differentiating vision-related semantics such as colors compared to pure language-only pre-trained models like BERT [2), see Table in the supplementary material. We freeze the pre-trained language encoder and use a linear layer on top of it to obtain embeddings & Rd for each word token:
the language encoder in the CLIP model [5]. Thanks to its vision-and
(1) with & the I-th embedding output by the language encoder, LN layer normalization [78], W₂ pro- jection matrix, and F. type embedding which differentiates instructions from visual observations.
LN,W,)) Et:
Observations and Proprioception. We encode the RGB image I. point clouds pt. and proprio- ception A for each camera & separately. A (0. #x is a binary attention map used to encode the position of the gripper pt- takes value where. We concatenate 1 and A, in the channel dimension and use UNet encoder to obtain a feature map f* E #** where H" W" .d, the height, width, and number of channels of the feature map. More details about the CNN architecture are presented in Section of the supplemen- tary material. Next, we concatenate fü with point cloud representations in the channel dimension indicate the spatial location of each patch in the feature map. To match the size of Pt and f,A. we apply mean-pooling to pt. The final encoded feature map Ff e RR xdd,x+ x(d++)) is computed follows:
the location of the gripper center and
else-
F = [[NN((I).A))) MeanPool(PP))
(2)
We use patches P. E F. h [. [1, W"] as separate visual tokens. We further encode PA using embeddings of the camera id Et. of the step id E and the patch location E/."" as well as an embedding to indicate the visual nature of the tokens Ey as follows:
g LN(W
E + FA." Ey
The encoded visual tokens of the k-th camera at step are denoted as G² = (8) W" xw""""xx We concatenate the encoded tokens for all cameras G, (G..... ,,,)),
T E
4

---

4.2 Multimodal transformer
Given the encoded tokens at the current macro step the multimodal transformer aims to obtain contextualized representation for G, conditioned on the encoded instruction 11_ and history {G.l= This enables learning relationships among views from multiple cameras, the current ob- servations and instructions, and between the current observations and history for action prediction. We use the transformer's 's attention mechanism [79] to learn such relationships:
WoO(WEk)) vd
Attn (.V Softmax
WvV.
(4)
where Wo. Wx. Wy are learnable parameters. Unlike previous work [75] that uses self-altention layers to capture all relationships, we employ different attention layers to capture different types of relationships, in order to reinforce the importance of the context. First, we use a cross-attention layer to learn the inter-modal relationships between G, and its conditioned contexts C, consisting of tokens in the instruction {$ and history {Gi which is:
G CA(G,, G.) Attn CG).
(5)
Then learn the intra-modal relationships among patch tokens obtained from the views from multiple cameras through self-attention layer. i.e. SA(G,) Attn C, G.). Finally, a feed- forward network consisting of two linear layers Wi and W22 is applied as follows:
G IN (W2 GeLU (W, SA(G.)))
(6)
4.3 Action Prediction
We concatenate the output embeddings of the transformer G, in Eq (6) and the original encoded visual representations F, in Sec. 4.1 in the channel dimension and reshape the flattened sequence into a feature map H, E I x"""""""xxx=""" x
x(d+d)) to predict the next action a++1 = [p--119-110;4] As some RLBench tasks require accurate fine-grained positioning, different from the rotation 4++1 and gripper state 1,+11 the position Pr+1 is predicted through separate module that uses point clouds P.
Rotation and gripper's state. We transform H; into "f"" XX""""""(d'" and feed it into a CNN de- coder, described in Section A of the supplementary material. We then apply average pooling across spatial dimensions and employ linear layer to regress 5-dimension vector [q++10++1]-
Position. The prediction of the gripper position pr+1 is decomposed into an expected point on point clouds pf and an offset 1. i.e. Pt+1 Pf_1 + Pf-1- The offset allows us to predict a virtual point outside the convex hull of the point cloud, e.g. when robotic arm reaches first above the object and then touch the object. For each camera k, a CNN with an upsampling layer predicts an attention map B/ E R/ll over the point clouds Př. Each value By p. E B. corresponds to the probability
reaching the point the E pl. Therefore, we compute Pf+1 as the expected position over all cameras:
Pf+ = Σ (Bi P J.h.
The offset pf+1 is computed from the instruction and the current step id. Let Eo E V+x be a learnable embedding. where N₂ is the number of tasks and T is the maximum length of episodes. We predict the task id from the instruction: Pr(m) Softmax ( E & where Pr(m) e [0, 1]"r and we obtain the offset as: p Em Pr(m) Emm,t,.:).
4.4 Training and Inference
Losses. We use hehavioral cloning to train the models. In RIBench, we generate D. collection of N successful demonstrations for each task. Each demonstration 8 E D is composed of sequence of (maximum) T macro-steps with observations {o H actions {a; Y task m² and instruction (x))) We minimize loss function L over batch of demonstrations B { D. The loss function is the sum of two losses: mean-square error (MSE) on the gripper's 's action and a cross-entropy (CB) over the task classification:
L
Σ a)) BEB
Σ
(8)
5

---

Table 1: Success rate on the single-task setting, We report mean and variance for unseen episodes.
Inputs
Transformer
Training
Visual Tokens
Point
Gripper
Multi-
Mask Obs
History
Attn
Clouds Position
View
RI
x
x
x
X
x
72,9 4.1
R2
Channel
Self
73.1 4.5
R3
Channel
/
Self
5.8
Channel
Self
X
7.81
R5
Channel
Self
x
8..
R6
Channel
V
Self
82.3
R7
Patch
V
Self
84.4+6.4
Patch
Cross
88.4
Masking current observation. To ensure that the model uses past information (- [ instead of only relying the current observation o;, randomly mask the current observation with probability of The masking zeros out randomly selected patch features observation. Therefore, even if the unmasked current observations contain sufficient information, the model still requires to complete the masked observations from the history for action prediction.
current
5 Experiments
In this section we present experiments on RI RLBench [19] tasks to demonstrate the effectiveness of our Hiveformer model in three settings: single-task, multi-task, and multi-variatio In the single-task setup, separate model is trained and tested for each task with variations of the task. Multi-task refers to setting where one model trained for multiple tasks (but each task unique variation). In the multi-variation case we train a single model to solve multiple variations of a single task and test it on new variations of the task unseen during training.
5.1 Experimental Setup
Dataset setups. RLBench [19] a benchmark robotic tasks. compare our method with previous work [15], we use the same 10 tasks with 100 demonstrations for training unless stated otherwise. We further evaluate our model on 74 tasks for which RLBench provides successful demonstrations. We manually group these tasks into 9 categories according their challenges. More details on the split of the tasks are given in Section of the Supplementary material. We evaluate models by measuring the per task success rate for 500 unseen episodes.
implementation details. We use the Adam optimizer with a learning rate of 5 x 10-3, Each batch consists of 32 demonstrations. Models were trained for 100,000 iterations. We apply data augmen- tation training including jitter over RGB images 1 and random crop 1,, P. and A while keeping them aligned. Models are trained on one NVIDIA Tesla V100 SXM2 GPÚ using Singu- larity container with headless rendering. Auto-A [15] uses a UNet network and applics late fusion to predictions from multiple views.
5.2 Ablations
To demonstrate the effectiveness of the proposed model architecture, we ablate the impact of its components in Table I. The model in R1 (row 1) is UNet architecture similar to Auto-A [15] except that it conditioned observations at the current step and already achieves promising results with success rate 73.2%. On top of RI's architecture, a multimodal transformer with self-attention is added in R2 to improve the modeling of multi-view images. Visual tokens ) are different channels in the feature map instead of spatial patches used in our final model. In R3 and R4, we further add point clouds and gripper position A, in the feature encoding, which leads to 3% improvement total. The impact of history, Í.e. the of observations from previous steps, ((G))((T-))
instructions rather than task ids. This baseline only uses visual
studied in R5 and R6. The history information brings 4.5% absolute gains and the masking of observations during training further improves the performance by 0.5%. In R7, we replace the tokenization of fcature maps from channels ot patches st k and obtain another 2.2% gain. This improvement can be altributed
6

---

Table 2: Comparison with state-of-the methods on 10 tasks. We report success rate (%)
Pick & Pick-Up
Push Button
Put Knife
Put
Reach Target
Stide Block
Stack Wine
Take Money
Take Umbrella
Avg.
Lift
Cup
Money
Single-tase learning
ARM [12]
70
80
100
70
70
Auto-À [15]
82
72
95
36
31
100
36
23
38
37
55.0
Ours
92.2
77.1
99.6
69.7
96.2
100.0
95.4
81.9
82.1
90.1
88.4
Multi-task learning
Auto-A [15]
87
78
100
77
19
80
69.3
Ours (wlo inst)
83.8
13.9
97.0
41.9
54.3
98.9
36.2
68.5
74.1
73.0
64.2
Ours
88.9
92.9
100.0
75.3
58.2
100.0
78.7
71.2
79.1
89.2
83.3
Comparison with the state of the 74 RLBench tasks grouped into 9 categories. report success rate (%) for the single-task setting. "The performance of Auto-2 is obtained by running their code.
Table 3:
Long
Rot. Invar.
Motion Planning
Multi Modal
Visual Occlusion
Planning
Tools
Screw
Precision
Avg
Term
Num. of tasks
11
4
9
5
11
14
Auto-2 [15]*
58.9
20.0
2.3
73.1
66.7
48.2
47.6
34.6
40.6
44.0
Ours (w/o hist)
78.9
46.7
10.0
84.6
73.3
72.6
60.0
63.8
57.9
60.9
Ours (one view)
57.7
23.2
12.3
57.8
63.2
35.6
40.7
33.7
37.1
40.1
Ours
81.6
53.0
16.9
84.2
72.7
80.9
67.1
64.7
60.2
65.4
patch tokens that help encode fine-grained spatial information. Finally, cross-altention instead of self-attention (Eq. 5) to condition on instruction history context. further boosts the performance with a 3.8% gain.
5.3 Comparison with State of the Art
Single-task evaluation. The upper block in Table presents results of different models on 10 single tasks RLBench. We compare our model with ARM [12] and Auto-2 [15], two state-of-the-art methods on RLBench and observe consistent improvement for all tasks.
Extending tasks in a single-task evaluation setup. In Table 3, we further compare Auto-A and Hiveformer's variants across 74 RLBench tasks grouped into 9 categories. The variant without his- tory removes the history tokens in Hiveformer, while the variant with one view only uses at each step (we take the best among the cameras for each task). The full Hiveformer achieves consistently hetter performance compared to Auto-2 [15] on all types of tasks. Among them, the Long-term, Tools and Planning task groups assess the use of history. where our model brings im- proves significantly
the variant without history. Compared to the view variant, full model performs significantly better tasks requiring fine-grained control with large occlusions such as Screw, Precision and Visual Occlusion categories. Yet, our method performs relatively poorly for Long-term tasks with more than 10 steps, such as "take shoes out of box". As Long-tern tasks have average number of steps 2-4 times higher than others, they bution shift issues accumulated errors. Hierarchical modeling or better training algorithms such as reinforcement learning and dagger [80] could be helpful, but are left as future work.
more prone distri-
Multi-task evaluation. The lower half in Table 2 shows the results in a multi-task setting. Notably, Auto-2 training algorithm which dynamically adjusts the weights of different tasks, while model simply treats all tasks with equal weights. Nevertheless. our model outperforms Auto- by 14%, demonstrating the improvements due to our architecture. We further compare our model with a variant without instructions in the input sequence (since Ph_1 is predicted from instructions, we modify the model such as is predicted from H²)). The results show that instructions are important in the multi-task setting. Moreover, the performance of our single model trained for all tasks is only slightly worse than the performance of individual models for each task.
7

---

Table 4: Success rate (%) in the multi-variation setting for seen or unseen variations and synthetic or human-written instructions.
# Demos
Push Buttons
Tower
Per Variation
Seen Synthetic
Unseen
Scen Synthetic
Unseen
Synthetic
Human
Synthetic
Human
10
96.8
73.1
65.1
71.7
50.1
19.4
50
99.6
83.3
70.6
74.1
52.3
20.7
100
100
86.4
74.0
76.2
56.4
24.2
Generalization to multi-variations. Table 4 shows results Hiveformer trained on different vari- ations of the two tasks Tower and Push Buttons. The Tower (resp. Push Buttons) task requires the robot to sequentially stack colored cubes (resp. push colored buttons) using the order specified in the instruction, see Figure (right).
We use 100 variations in training and test models for both the 100 seen variations and 100 unseen variations. In this setting, instructions
necessary to generalize to unseen variations (e.g. it is impossible distinguish the order of pushing buttons red-green-blue looking at the scene). compare the models trained with different numbers demonstrations per variation. Even in the most challenging case where only 10 demonstrations are available per varia- tion, Hiveformer achieves success rate of 71.1% 1% for the push buttons task and 49 99.8% for the tower task in unseen variations. Furthermore, besides tests synthetic instructions (Synt), the generalization real instructions. Despite being only trained on synthetic instructions with lim- ited vocabulary and diversity, our model performs well on instructions generated by humans (Real), Finetuning Hiveformer on human instructions [75] is expected to result in further improvements. Details of gener-eeeerrtedd instructions are presented in Section C of the Supplementary material.
blue-red-green by only
also test
5,4 Real-robot Experiments
Setup. We conduct real-robot experiments for the push buttons task on 6-DoF URS robotic arm equipped with 2-finger Robotiq RG2 gripper and two cameras on each side of the scene. As there exists a large difference between simulated and real environments, we finetune the simulator- trained policy real-robot demonstrations. We use 10 variations of the task and 10 real-robot demonstrations per variation. More details are presented in Section E of the Supplementary material.
Results. We report success rates for real-robot experiments in Ta- ble 5 on the "push buttons" task using synthetic instructions. The models tested on 10 seen and 10 unseen variations. We compare two models: one trained from scratch using real-robot demonstra- tions; and the other pretrained on RLBench and then finetuned us- ing real-robot demonstrations. As shown in Table 5, the pretraining significantly improves the performance especially for unseen varia- tions. The model without pretraining prone to overfitting on seen variations. Although the domain gap between the real robot and RLBench environments is large, our model benefits from pretraining in the simulator. More analysis and examples are presented in Section E of the Supplementary material.
Table 5: Success rate of push buttons task on real robots.
Pretrain Seen Vars Unseen Vars
86.7
13.3
92.2
85.7
Conclusion
introduced Hiveformer, multimodal transformer that jointly models instructions, views from multiple cameras, and history for instruction-driven robotics manipulation. We evaluated the model on RLBBench in three settings: single-task learning, multi-task learning, and multi- variation general- ization and demonstrated its effectiveness outperforming state of the arts. We deployed model on a real robot which is able to generalize to unseen variations and human-written instructions.
Limitations. The computational cost quadratically increases with the input sequence length due to the transformer. Furthermore, exposure bias. Future works could improve efficiency for long term tasks with hierarchical mod- els and also incorporate reinforcement learning. Moreover, our model is trained on only synthetic instructions and performs worse on the human-writee instructions. Training on human-writter au- tomatically generated instructions could help improve the performance.
model is trained with behavioral cloning, which may suffer from
8

---

Acknowledgments
This work was granted access to the HPC resources of IDRIS under the allocation 101002 made by GENCL This work is funded in part by the French government under management of Agence Nationale de la Recherche P31A-0001 (PRAIRIE 31A Institute), the ANR project VidcoPredict (ANR-21-FAI1-0002-0) and by Louis Vuitton ENS Chair on Artificial Intelligence.
part of the "Investissements d' d'avenir" program, reference ANR19.
References
[1] Project webpage. https: / /gubur github.. 10/ www.hiiierrrrre
[2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[3] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[4] J. Lu, D. Batra, D. Parikh, and S. Lee. ViLBERT: Pretraining task-agnostic visiolinguistic representations for lision-and-languag tasks. In NeurlPS, 2019.
[5] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin. J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In ICMI,, pages 8748-8763. PMLR, 2021.
[6] C. Lynch and P. Sermanet. Language conditioned imitation learning over unstructured data. RSS, 2021.
[7] O. Mees and Burgard. Composing pick-and-place tasks by grounding language. In ISER, 2021.
[8] O. Mees, Hermann, E. busete-Beass and W. Burgard. CALVIN: benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. RA-I., 2022.
[9] M. Shridhar, L. Manuelli, and D. Fox. CLIPort: What and where pathways for robotic manip- ulation. In CoRI., pages 894-906. PMLR, 2022.
[10] E. Jang, A. Irpan, M. Khansari, D. Kappler, E. Ebert, C. Lynch, S. Levine, and C. Finn. BC- Z: Zerro-shot task generalization with robotic imitation learning. In CoRL, pages 991-1002. PMLR, 2022.
[11] Y. Yang. D. Cer, A. Ahmad, M. Guo, J. Law, N. Constant, G. H. Abrego, Yuan, C. Tar, Y.-H. Sung, et al. Multilingual universal sentence encoder for semantic retrieval. In ACL, pages 87-94, 2020.
[12) S. James and J. Davison. Q-Attention: Enabling efficient learning for vision-based robotic manipulation. RA-L, 2022.
[13] A. Zeng, Florence, J. Tompson, S. Welker, Chien, M. Attarian, T. Armstrong, Krasin, D. Duong, V. Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. In CoRI., pages 726-747. PMIR, 2021.
[14] S. James, K. Wada, Laidlow, and A. Davison. Coarse-to-fine Q-Attention: Efficient learning for visual robotic manipulation via discretisation. CVPR, 2022.
[15] S. Liu, S. James, A. J. Davison, and E. Johns. Auto-Lambda: Disentangling dynamic task relationships. TMLR, 2022.
[16] L. Shao, T. Migimatsu, Q Zhang, K. Yang. and J. Bobg. Concept2robot: Learning manip- ulation concepts from instructions and human demonstrations. The International Journal of Robotics Research, 40(12-14):141- 141-1444411444 2021.
[17] S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn, et Learning language robot behavior from offline data and crowd-sourced annotation. In CoRL, pages 1 1303-1315. PMLR, 2022.
9

---

[18] E. Perez, F. Strub, Vries, V. Dumoulin, and A. Courville. FILM: Visual reasoning with general conditioning layer. In AAAI, 2018.
[19] S. James, 7. Ma, D. R. Arrojo, and A. J. Davison. RLBench: The robot learning benchmark learning environment, RA-I, 5(2):3019-3026 2020.
[20] O. Ronneberger, Fischer, and T. Brox. UNet: Convolutional networks for biomedical im- age segmentation. In International Conference on Medical image computing and computer- assisted intervention, pages 234-241. Springer, 2015.
[21] J. Hill. Real time control of a robot with a mobile camera. In 9th Int. Symp. on Industrial Robots, 1979, pages 233-246 1979.
[22] F. Chaumette and S. Hutchinson. Visual servo control. i. basic approaches. IEEE Robotics & Automation Magazine, 13(4):82-90. 2006.
[23] R. Strudel, A. Pashevich, I. Kalevatykh, I. Laptev, and C. Schmid. Learning to combine prim- itive skills: A step towards versatile robotic manipulation. In ICRA, pages 4637-4643. IEEE, 2020.
[24] A. X. Lee, S. Levine, and P. Abbeel. Learning visual servoing with deep features and fitted q-iteration. In ICLR, 2017.
[25] Q Bateux, Marchand, Leitner, F. Chaumette, and Corke. Training deep neural networks for visual servoing. In ICRA, pages 1-8. IEEE, 2018
[26] P. Florence, C. Lynch, A. Zeng, O. A. Ramírez, A. Wahid, L. Downs, A. Wong, Lee, 1. Mor- datch, and Tompson. Implicit behavioral cloning. In CoRL, pages 158-168. PMLR, 2022.
[27] A. Károly, Galambos, Kuti, and I. Rudas. Deep learning in roboties: Survey on model structures and training strategies. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 51(1):266-23 277.2,,,, 2020.
[28] Y. Labbe, J. Carpentier, M. Aubry, and J. Sivic. Cosypose: Consistent multi-view multi-ohject 6d pose estimation. In ECCV, 2020.
[29] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik's 's cube with robot hand. arXiv preprint arXiv:1910.07113, 2019.
[30] R. Caruana. Learning many related tasks the same time with backpropagation. NeurlPS, 7, 1994.
[31] H. B. Ammar, F. Faton, P. Ruvolo, and M. Taylor. Online multi-task learning for policy gradient methods. In ICML, pages 1206-1214, PMLR, 2014.
[32] C. Devin, A. Gupta, T. Darrell, P. Abbeel, and S. Levine. Learning modular neural network policies for multi-task and multi-robot transfer. ICRA, pages 2169-2176. IEEE, 2017.
[33] S. Sodhani, A. Zhang, and J. Pineau. Multi-task reinforcement learning with context-based representations, International Conference on Machine Learning, pages 9767-9779. PMLR, 2021.
[34] T. Standley, A. Zamir, D. Chen, L. Guibas, Malik, and Savarese. Which tasks should be learned together in multi-task learning? In International Conference on Machine Learning, pages 1120-9132 PMLR, 2020.
[35] S. Thrun and J. O'Sullivan. Discovering structure in multiple learning tasks: The TC algorithm. In ICML, volume 96, pages 489-497, 1996.
[36] B. Chen, A. Sax, G. Lewis, I. Armeni, S. Savarese, A. Zamir, J. Malik, and L.. Pinto. Ro- bust policies via mid-level visual representations: An experimental study in manipulation and navigation. CoRL, 2020.
10

---

[37] C. Finn and Levine. Deep visual foresight for planning robot motion. ICRA, pages 2786-2793. IEEE, 2017.
[38] T. Yu, C. Finn, A. Xie, S. Dasari, T. Zhang. P. Abbeel, and S. Levine. One-shot imitation from observing humans via domain-adaptic adaptive meta-learning. RSS, 2018.
[39] T. Yu. D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRL, pages 1094-1100, PMLR, 2020.
[40] C. Finn, T. Yu, Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning. In CoRL, pages 357-368. PMLR, 2017.
[41] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. MT-Opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.0821 2021.
[42] A. S. Chen, S. Nair, and C. Finn. Learning generalizable robotic reward functions from "in- the-wild" human videos. RSS, 2021.
[43] R. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[44] T. Haarnoja, A. Zhou, K. Hartikainen, Tucker, Ha, Tan, Kumar, Zhu, Gupta, and P. Abbeel. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 1812.0590, 2018.
[45] J. Peters, S. Vijayakumar, and S. Schaal. Reinforcement learning for humanoid robotics. In Proceedings of the third TEFF-RAS international conference on humanoid robots, pages 1-20, 2003.
[46] A. Pashevich, D. Hafner, J. Davidson, R. Sukthankar, and Schmid. Modulated policy hier- archies. NeurlPS Deep RL. Workshop, 2018.
[47] F. Torabi, Warnell, and P. Stone. Behavioral cloning from observation. In Proceedings of the 27th International Joint Conference on Artificial Inteligence, pages 4950-4957, 2018.
[48] J. Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systemx, 29, 2016.
[49] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In CoRL, pages 1113-1132. PMLR, 2020.
[50] V. Petrík, M. Tapaswi, Laptev, and Sivie. Learning object manipulation skills via approxi- mate state estimation from real videos. In CoRL, 2020.
[51] Y. Hristov, D. Angelov, M. Burke, Lascarides, and Ramamoorthy, Disentangled relational representations for explaining and learning from demonstration. In CoRL, 2019.
[52] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAl Gym. arXiv preprint 1rXiv:1606.01544 2016.
[53] K. Ehsani, W. Han, A. Herrasti, E. VanderBilt, L. Weihs, E. Kolve, A. Kembhavi, and R. Mot- taghi. ManipulaTHOR: A framework for visual object manipulation. In CVPR, pages 4497- 4506, 2021.
[54] S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, Merel, T. Erez, T. Lillicrap, N. Heess, and Y. Tassa. dm_control: Software and tasks for contimuous control. Software Impacts, 6:100222, 2020.
[55] F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, et al. Sapien: A simulated bast-based interactive environment. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11097-11100 2020.
11

---

[56] O. Ahmed, F. Träuble, A. Goyal, Neitz, Y. Bengio, Schölkopf. M. Wüthrich, S. Bauer. CausalWorld: A robotic manipulation benchmark for causal structure and transfer learning. arXiv preprint arXi:20100.4299 2020.
[57] D. Misra, J. Langford, and Y. Artzi. Mapping instructions and visual observations to actions with reinforcement learning. ACL, 2017.
[58] E. Stengel-Fskkin, A. Hundt, 7. He, A. Murali, N. Gopalan, M. Gombolay, and G. Hager. Guiding multi-step rearrangement tasks with natural language instructions. In Conference on Robot Learning, pages 1586-1501. PMLR, 2021.
[59] K. M. Hermann, F. Hill, S. Green, E. Wang, R. Faulkner, H. Soyer, D. Szepesvari, W. M. Czamnekki M. Jaderberg. D. Teplyashin, M. Wainwright, C. Apps, D. Hassabis, and P. Blunso. Grounded language learning a simulated 3D world. NeurIPS Workshop. 2017.
[60] D. I. Team, Abramson, A. Ahuja, A. Brussce, F. Carnevale, M. Cassin, F. Fischer, P. Georgiev, A. Goldin, T. Harley, et Creating multimodal interactive agents with imita- tion and self-supervised learning. arXiv preprint arXiv:21 12.0362..7.... 2021.
[61] R. Paul, J. Arkin, N. Roy, and M Howard. Efficient grounding of abstract spatial concepts for natural language interaction with robot manipulators. In Robotics: Science and Systems Foundation, 2016.
[62] T. Nguyen, N. Gopalan, R. Patel, M. Corsaro, E. Pavlick, and S. Tellex. Robot object retrieval with contextual natural language queries. RSS, 2020.
[63] W. Goodwin, S. Vaze, I. Havoutis, and I. Posner. Semantically grounded object matching for robust robotic scene rearrangement. ICRA, 2021.
[64] S. Tellex. T. Kollar, S. Dickerson, M. R. Walter, A. G. Banerjee, S. Teller, and N. Roy. Un- derstanding natural language commands for robotic navigation and mobile manipulation. In Twenty- Fifth AAAI Conference Artificial Intelligence, 2011.
[65] W. Liu, C. Paxton, T. Hermans, and D. Fox. StructFormer: Learning spatial structure for language-guided semantic rearrangement of novel objects. ICRA, 2021.
[66] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models zero-shot planners: Fx- tracting actionable knowledge for embodied agents. arXiv preprint (tl::2001.07200 2022.
[67] D. Garg, S. Vaidyanath, K. Kim, J. Song, and S. Ermon. LISA: Learning interpretable skill abstractions from language. arXiv preprint arXiv:2203.0005.. 2022.
[68] A. Khandelwal, L. Weihs, R. Mottaghi. and A. Kembhavi. Simple but effective: Clip embed- dings for embodied ai. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages -4829, 2022.
[69] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, Jones, N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[70] A. Dosovitskiy, L. Beyer, A. Kolesníkov, D. Weissenbom, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transform- for image recognition scale. ICLR, 2020.
[71] H. Tan and M. Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. In EMNLP, 2019.
[72] L. Chen, K. Lu, A. Rajeswaran, K. Loe, A. Grover, M. Laskin, P. Abbecl, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In NeurIPS, 2021.
[73] A. L. Putterman, K. Lu, I. Mordatch, and P. Abbeel. Pretraining for language conditioned imitation with transformers. In NewrIPS, 2021.
12

---

[74] S. Chen, P.-L. Guhur, Schmid, and Laptev. History aware multimodal transformer for aaiio--andllanguaae navigation. Advances in Neural Information Processing Systems, 34, 2021.
[75] A. Pashevich, Schmid, and C. Sun. Episodic transformer for vision-and-languase naviga- tion. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 15942-15952, 2021.
[76] P.-L. Guhur, M. Tapaswi, S. Chen, L. Laptev, and C. Schmid. Airbert: In-domain pretraining for vision-and-language navigation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1634-1643, 2021.
[77] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez. Y. Sulsky, J. Kay, J. T. Springenberg, ct al. A generalist agent. arXiv preprint
arXin:2005.0//75. 2022.
[78] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:6607.06450 2016.
[79] Vaswani, N. Shazeer, N. Parmar, Uszkoreit, Jones, N. Gomez, Kaiser, and Polo- sukhin. Attention is all you need. NeurIPS, 2017.
[80] S. Ross, G. Gordon, and D. Bagnell, reduction of imitation learning and structured predic- tion no-regret online learning. Proceedings of fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 2011.
[81] S. James and P. Abbeel. Coarse-to-fin Q-Attention with learned path ranking, arXiv preprint (eXii:220400177) 2022.
[82] Silva, N. Moorman, W. Silva. Zaidi, Gopalan, and M. Gombolay. Lancon-leam: Learning with language to enable generalization in multi-task manipulation. JEEE Robotics and Automation Letters, 7(2): 1635-162, 2021.
[83] E. Olson, Apriltag: robust flexible visual fiducial system. ICRA, 2011.
13


Reviews:
[{"review_id": "xQLZfaLATbn", "reviewer": "Reviewer_yM9G", "summary": "This paper presents Hiveformer (History-aware instruction-condition multi-view transformer), which enables a robot to learn to perform novel manipulation tasks based upon natural language instructions and demonstrations for a set of training tasks. The approach uses Behavior Cloning to train the model, and the actions consist of gripper poses. A key aspect of the work is encoding multiple views of the scene and a history of information. Results in RLBench show that the approach outerpforms ARM [11] and Auto-\\lambda [14] as well as ablations of Hiveformer. A video is included and depicts a robot demonstration.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-The figures are clear and useful.\n-The right amount of space is allocated to the various parts of the paper.\n-The presentation of the algorithm is intelligible. \n-Code is provided.\n-The method shows positive results.\n\nWeaknesses:\n-It seems from Figure 1 that the paper addresses multi-step tasks; however, it isn't quite clear what examples of this task are and where the results are. I would guess those results are in Table 2 under \"Long Term.\" However, it would be helpful to provide more detail.\n-Some of the success rates are quite low (e.g., 16.9%). It would be helpful to discuss what it would take to reach > 70% on call tasks. On that note, it would be helpful if the table captions confirmed that what is show are the success rates (as opposed to rewards, etc.)\n-It would have been helpful to show an ablation study evaluating the quality of the algorithm's performance as a function of the number of demonstrations available. 100 demonstrations per task could be too much for a human demonstrator. Further, how much variation existed among the demonstrations, and how were these demonstrations collected? \n-Some tables (etc.) go over the margins for the paper and should be adjusted accordingly. \n-This paper might have missed related work in language-guided robot manipulation. In particular, please see the references below. The first two address only RL, whereas the last one addresses imitation learning-based manipulation with language. As such, it would have been helpful to benchmark against the Silva et al. work.\n\nSodhani, S., Zhang, A. and Pineau, J., 2021, July. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning (pp. 9767-9779). PMLR.\n\nStengel-Eskin, E., Hundt, A., He, Z., Murali, A., Gopalan, N., Gombolay, M. and Hager, G., 2022, January. Guiding Multi-Step Rearrangement Tasks with Natural Language Instructions. In Conference on Robot Learning (pp. 1486-1501). PMLR.\n\nSilva, A., Moorman, N., Silva, W., Zaidi, Z., Gopalan, N. and Gombolay, M., 2021. LanCon-Learn: Learning With Language to Enable Generalization in Multi-Task Manipulation. IEEE Robotics and Automation Letters, 7(2), pp.1635-1642.\n\n-CLIP, as noted by Abeba Birhane et al. (2021), is problematic. The authors leverage CLIP without discussing the potential harm that can come from its adoption. A recent paper exploring negative impacts of CLIP in robots was published by Hundt et al. (FAccT'22), but this was published after this paper's CoRL submission. It would be helpful if the paper addressed this issue in the limitations.\n\nBirhane, A., Prabhu, V.U. and Kahembwe, E., 2021. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963.\n\nHundt, A., Agnew, W., Zeng, V., Kacianka, S. and Gombolay, M., 2022, June. Robots Enact Malignant Stereotypes. In 2022 ACM Conference on Fairness, Accountability, and Transparency (pp. 743-756).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper is solid. The presentation is clear, the method is well-reasoned, and the results are positive. Ablations are provided, and the method outperforms baselines. It would have been helpful to include the work by Silva et al. (2022) as a direct baseline.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please address the notes on limitations, missing prior work, some minor issues of clarity, and the baseline of Silva et al. (2022).", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "V_y4_1H_QqV", "reviewer": "Reviewer_G4rm", "summary": "This paper proposes a model for learning to predict end-effector keypoints to solve manipulation tasks. The model consists of attention mechanisms modeling relationships between text tokens and visual patches, including across time. It also includes a U-Net that generates a spatial probability over gripper locations. The input to the model are RGBD observations from three different views: left shoulder, right shoulder and gripper, and also a tokenized instruction where tokens are encoded with the CLIP encoder.\nThe paper demonstrates SOTA performance on 10 RLBench manipulation tasks in simulation, includes extensive ablation studies of the different model parts, and also includes an experiment with two simple instruction-conditional tasks. One of the main points made by the paper is that it's important to model the history of observations. The paper provides empirical proof to this claim.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper doesn’t just take an off-the-shelf transformer and feed text and visual observations through it, but carefully considers what types of attention are required to model the task, and how it should fit together in an architecture. The result is a well-performing and lightweight model specifically targeted at fixed-base manipulation tasks.\n- Technical sections very nicely written, with clear definitions to the different components, indexing that makes sense and helps understand the data types involved, and a clear data flow through the model.\n- Impressive performance on 10 RLBench tasks.\n- Overall strong experimental setup, with a very good set of ablations to study the effects of the different modeling choices like instruction input, history input, cross- vs self-attention, etc.\n\nWeaknesses:\n- In terms of studying instruction-driven manipulation, the sort of instructions and generalization studied is relatively extremely simple and significantly behind prior work. E.g. the paper did not test interesting compositions of attributes or spatial relations [C, D, E, F], nor did they test generalization to unseen objects or relations [G, H], nor tasks requiring multi-step planning [J]. As such, calling it an instruction-driven manipulation in the title might be a disservice to the paper, since this aspect is relatively underlooked. There is a large body of work that does instruction-following including modeling of history [B, I, K]. On the other hand, it is a fairly strong simulated table-top manipulation paper in the multi-task behavior cloning setting. Perhaps focusing on this aspect might better highlight the positive attributes of the paper.\n\n- The method for computing position offsets (Lines 198-201) doesn’t seem like it could handle cases where the same task at the same time-step may require a different offset based on the environment configuration. This reduces the generality of the approach. Furthermore, the assumption of a fixed set of tasks (also present in the loss in Equation 8) cements this as a multi-task learning approach rather than an instruction-driven approach. In instruction-driven work, it's questionable to map to a fixed set of tasks, because the set of tasks is by definition extensible via specification of new instructions.\n\n- No experiments on real robots, so tolerance to noise, or sim2real generalization couldn’t be assessed.\n\n- Some design choices seem somewhat arbitrary and maybe strange (e.g. applying max-pooling to point-cloud patches, classifying instructions into a fixed set of task types).\n\n\nReview references:\n[A] Walter, Matthew R., et al. \"Learning semantic maps from natural language descriptions.\" Robotics: Science and Systems, 2013.\n[B] Blukis, Valts, et al. \"Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight.\" CoRL. 2019.\n[C] Arumugam, Dilip, et al. \"Grounding natural language instructions to semantic goal representations for abstraction and generalization.\" Autonomous Robots 43.2 (2019): 449-468.\n[D] Vaezipoor, Pashootan, et al. \"Ltl2action: Generalizing ltl instructions for multi-task rl.\" International Conference on Machine Learning. PMLR, 2021.\n[E] Wang, Christopher, et al. \"Learning a natural-language to LTL executable semantic parser for grounded robotics.\" Conference on Robot Learning. PMLR, 2021.\n[F] Corona, Rodolfo, et al. \"Modular Networks for Compositional Instruction Following.\" Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.\n[G] Shridhar, Mohit, Lucas Manuelli, and Dieter Fox. \"CLIPort: What and Where Pathways for Robotic Manipulation.\" 5th Annual Conference on Robot Learning. 2021.\n[H] Blukis, Valts, Ross Knepper, and Yoav Artzi. \"Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following.\" Conference on Robot Learning. PMLR, 2021.\n[I] Pashevich, Alexander, Cordelia Schmid, and Chen Sun. \"Episodic transformer for vision-and-language navigation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n[J] Paxton, Chris, et al. \"Prospection: Interpretable plans from language by predicting the future.\" 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.\n[K] Anderson, Peter, et al. \"Chasing ghosts: Instruction following as bayesian state tracking.\" Advances in neural information processing systems 32 (2019).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The main strength is superior performance on RLBench tasks compared to prior behavior-cloning work, along with great technical execution, good empirical work. However, it's not clear that the paper proposes anything paradigm-shifting. The method utilizes various techniques from prior work (such as the modelling keypoints, cross-modal attention, history modelling, U-Nets, multimodal transformers). None of these can be claimed as a contribution. However, the paper shows a good way of putting these pieces together in a robotics system.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Questions:\n- Line 159: why applying “max-pooling” to point clouds makes sense? Wouldn’t median-pooling (or even average-pooling) make more sense? Wouldn’t max-pooling select the highest coordinate along the x, y, and z axis individually, and take the resulting point to represent the location of the patch? Wouldn’t this representation be highly sensitive to an arbitrary choice of the origin? And e.g. for small objects, wouldn’t it be likely to select far away background points instead of a closer object point to represent the patch location?\n- Equation 6: what is LN? The inner layer was represented by explicitly writing down a matrix multiplication. Why does the outer layer use LN and what does it mean?\n- Equation 7: If the distribution is multi-modal, then averaging coordinates (e.g. points) may result in completely false predictions in-between the two modes, rather than better predictions on one of the two modes. Is this ever a problem in practice?\n\nSuggestions:\n- Line 149 - what type is the token E_{T}^{x}? At this point in reading I have not yet learned about how visual observations are represented or used downstream, so saying “to differentiate instructions from visual observations” doesn’t tell me much.\n- Line 151 - maybe call A_{t}^{k} a “binary mask” instead of “attention map”? The word “attention” can have connotations of it being a soft mask, a mask that sums to 1, and a mask computed by a learned model for downstream use in the same or different learned model.\n- Line 181 - would be nice to have specific section numbers and links when referring to supplementary material.\n- There are multiple prior works on instruction-following (e.g. [A, B, K, L]), that build maps as a means to provide a concise spatial memory to a robot. This is often seen as an advantage over keeping in memory a complete history of observations. Could the paper discuss why keeping history might be advantageous?\n\nComments:\n- Section 4.2: I’m not sure it is fair to call this method a “multi-modal transformer”. The attention mechanism was invented before the Transformer architecture for machine translation, and one important aspect of it is that the attention modules are layered in a deep network. The architecture in this paper consists of a single cross-attention and a single self-attention layer, followed by an MLP. Calling it something like “attention across modalities and time” model would more clearly pinpoint the contribution. (the whole time reading this paper I was trying to understand how it is fundamentally different from all the other transformer architectures, and this section finally got to it)."}}, {"review_id": "8ZDnq5KG2Dq", "reviewer": "Reviewer_R4uU", "summary": "The work introduces a transformer-based robot manipulation method to execute language instructions. The policy input contains the history of actions and observations, which enables the execution of sequential tasks. The method also integrates a multi-camera view, in order to deal with the need for precise control and issues like occlusions.\nThe method was evaluated in simulated RLBench in single-task, multi-task, and multi-variation setup. The method was also demonstrated on a real robot (the results are only in the supplementary material).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- good empirical results on RLBench, compared to ARM and Auto-$\\lambda$\n- good results on the real robot. In particular, it shows the significant benefit of pretaining in simulation (and transfer from 3 to 2 cameras)\n- a thorough analysis (Section 5.2) of the importance of different components of the perception and manipulation processing (not the language though).\n\nWeaknesses:\n- a significant part of the RLBench experiments is in the single-task setup, which if I understand it correctly, does not need the instructions at all (a separate model is trained for each task). Thus we get relatively little insight on the \"instruction-driven\" part. We can learn from Table 3 that it does help, but we don't know, how it compares to other ways of processing the language input. The \"average over word tokens\" presented in Table 4 seems to be very crude.\n- the description of the architecture (Sect. 4.1, 4.2, Fig 2) seems to be pretty difficult to comprehend.\n- The source code does not come with instructions, on how to run it.\n- The source code contains the name of the author in the file `pyproject.toml`. It should be anonymized.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The presented method has many strenghts. It is a complete solution for executing multi-task, potentially multi-step instructions. The additional benefit is that it can leverage multiple cameras and a history of actions and observations. The method was also demonstrated to work on a robot (in time for the supplementary material deadline), which is very important for the relevance to the conference. Finally, the RLBench results seem strong.\n\nOn the other hand, the novelty of the method is somehow limited. Moreover, the claims about handling natural-language instructions do not seem to be sufficiently supported by the experiments.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- Three setups for the RL bench experiments (single-task, multi-task, and multi-variation) should be defined or at least explained.\n- Please provide detail on how macro steps (line 115) are used.\n- Following the limited (in my opinion) evaluation of the language component, I'd like to learn answers to some of the following questions:\n    - how well the method behaves, if we use a large-language model embedding of the instructions\n    - lines 144-146 claim that CLIP embedding is better than BERT. Do you have any evidence to back this claim?\n    - is the $E_T^x$ in Eqn (1) useful/required?\n    - Does the model use task similarity expressed in language in the multi-task setup? How would the performance change, if we use a one-hot encoded representation for tasks, instead of the language input?\n- Could you please provide examples of the human-written instructions? The description in the supplementary does not give me an insight, into how diverse they were (especially for the push-button task). It would be great if the full set of human-written instructions is provided in the supplementary material.\n- I think that the limitation section should include a discussion on how well the method handles *natural* language / human-written instructions (as claimed in the abstract and conclusion line 309). Most of the experiments are using static descriptions of tasks. Using human-written instructions can reduce the performance up to 3x (from 77.4 to 21.3 in the tower task), which I'm not sure if it justifies the \"generalizes well\" claim (line 309).\n    - Alternatively, support evidence that the model can actually handle language instructions. I could imagine an experiment in the multi-task setup, with synonyms for the task description (e.g. \"Lamp on\" -> \"Turn the light on\"). A similarly interesting experiment would be to show a transfer from a model trained on 73 tasks to a different one, not seen in the training (e.g. one of the \"put/close/open\" tasks, which would have similar concepts in the training set)."}}]

Author rebuttal:

Decision: Accept (oral)