Instruction-Following Agents with Jointly Pre-Trained Vision-Language Models
Authors: Hao Liu
Lisa Lee
Kimin Lee
Pieter Abbeel
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
pre-training
multimodal representation
representation learning
transformer

Abstract: Humans are excellent at understanding language and vision to accomplish a wide range of tasks. In contrast, creating general instruction-following embodied agents remains a difficult challenge. Prior work that uses pure language-only models lack visual grounding, making it difficult to connect language instructions with visual observations. On the other hand, methods that use pre-trained vision-language models typically come with divided language and visual representations, requiring designing specialized network architecture to fuse them together. We propose a simple yet effective model for robots to solve instruction-following tasks in vision-based environments. Our InstructRL method consists of a multimodal transformer that encodes visual observations and language instructions, and a policy transformer that predicts actions based on encoded representations. The multimodal transformer is pre-trained on millions of image-text pairs and natural language text, thereby producing generic cross-modal representations of observations and instructions. The policy transformer keeps track of the full history of observations and actions, and predicts actions autoregressively. We show that this unified transformer model outperforms all state-of-the-art pre-trained or trained-from-scratch methods in both single-task and multi-task settings. Our model also shows better model scalability and generalization ability than prior work.

Under review as a conference paper at ICLR 2023
INSTRUCHION-FOLLOWIGG AGENTS WITH JOINTLY TRAINED VISION-LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Humans are excellent at understanding language and vision to accomplish a wide range of tasks. contrast, creating general instruction-followinng embodied agents remains difficult challenge. Prior work that pure language-only models lack visual grounding, making it difficult to connect language instructions with visual observations. On the other hand, methods that use pre-trained vision-language models typically come with divided language and visual representations, requiring designing specialized network architecture to fuse them together. We propose simple yet effective model for robots to solve instruction-followin based environments. Our InstrucIRL method consists of a multimodal transformer that encodes visual observations and language instructions, and policy trans- former that predicts actions based on encoded representations. The multimodal transformer is pre-trained
tasks in vision-
millions of image-text pairs and natural language text, thereby producing generic cross-modal representations of observations and instructions. The policy transformer keeps track of the full history of observations and actions, and predicts actions autoregressively. We show that this unified trans- former model outperforms all state-of-the-an pre-trained or trained-from-sceee methods in both single-task and multi-task settings. Our model also shows better model scalability and generalization ability than prior work
INTRODUCTION
Humans are able to understand language and vision accomplish wide range of tasks. Many tasks require language understanding and vision perception, from driving to whiteboard discussion and cooking. Humans can also generalize to new tasks by building upon knowledge acquired from previously-seen tasks. Meanwhile, creating generic instruction-folloiin agents that can generalize to multiple tasks and environments is one of the central challenges reinforcement learning (RL) and robotics.
Driven by sigmificant advances in learning generic pre-trained models for language understand- ing (Devlin et al. 2018 8 Brown al. 2020 Chowdhery et al.) (0022). recent work has made great
progress towards building instruction- cnatruccton-folllwwiii following agents (Lynch & Sermanet, 2020 Mandlekar et al.)
al. (022b)). For example, SayCan (Ahn et al.j 2022) exploits PaLM models (Chowdhery et al. 2022) to generate language descriptions of step-by-step plans from language instructions, then executes the plans by mapping the steps to predefined macro actions. HiveFormer (Guhur et al. 2022) uses a pre-trained language encoder to generalize multiple manipulation tasks. However, a remaining challenge is that pure language-only pre- pre-trained models are disconnected from visual representations, making it difficult to differentiate vision-related semantics such as colors. Therefore, visual semantics have be further leamed to connect language instructions and visual inputs.
2021 Ahn al. 2022 Jang al., 2022 Guhur et al., 2022|
Another category of methods use pre-trained vision- vision-language models, which have shown great suc- cess in joint visual and language understanding (Radford et al. 2021). This has made tremendous progress towards creating a general RL agent (Zeng al. 2022; Khandelwal et al., 2022 Nair al., Khandelwal et al. 2022: Shridhar et al.) 2022a) For example, CLiPort (Shridhar et al.
2022b
2022a
uses CLIP (Radford et al. 2021) vision encoder and language encoder to solve manipulation
'The code of InstructRI. is available httttts/sssseeeese
com www.///wwwwsw......///t.coo

---

Under review as a conference paper at ICLR 2023
Take shoes
Push butions
Push rubbishh in bin
Stack 2 blocks
Stuck 3 red blocks
Pul phone on base
Pick up cupe
Black blocks
Stack 4 red blocks
Stack mareon blocks
Figure 1: Examples of RLBench tasks considered in this work. Left: InstructRL can perform multiple tasks from RLBench given language instructions, by leveraging the representations of pre-trained vision- vision-language transformer model, and learning transformer policy. Right: Each task can be composed of multiple variations that share the same skills but differ objects. For example, in the block stacking task, Instruc/RI. can generalize to varying colors and ordering of the blocks.
tasks. However, drawback is that they come with limited language understanding compared to pure language-only pre-trained models like BERT (Devlin al. 2018). lacking the ability follow long and detailed instructions. In addition, the representations of visual input and textual input are often disjointly learned, so such methods typically require designing specialized network architectures on top of the pre-trained models to fuse them together.
To address the above challenges, we introduce InstructRL, simple yet effective method based on the multimodal transformer /Vaswani al. 2017 Tsai et al., 2019). It first encodes fine-grained cross-moda modal alignment between vision and language using a pre tramtrd multimodal encoder (Geng
et al. 2022), which is large transformer (Vaswani al. 2017 He al. (002)) jointly trained image-fext (Changpinyo et al. 2021 Thomee et al. 2016) and text-only data (Devlin et al.) 2018).
The generic representations of each camera and instructions form sequence, and are concatenated
with the embeddings of proprioception data and actions. These tokens are fed into a multimodal policy transformer, which jointly models dependencies between the current and past observations, and cross-modal alignment between instruction and views from multiple cameras. Based on the output representations from our multimodal transformer, we predict 7-DoF actions, i.e., position, rotation, and state of the gripper.
We evaluate InstructRL on RLBench (James et al.) 2020). measuring capabilities for single-task learning, multi-task learning, multi-variation generalization, long instructions following, and model scalability. On all 74 tasks which belong to 9 categories (see Figure [1] for example tasks), our InstructRL significantly outperforms state-of-the-art models (Shridhar al. 2022a Guhur et al., 2022 Liu et al. 2022), demonstrating the effectiveness of joint vision-language pre-traines repre- sentations. Moreover, InstructRL not only excels in following basic language instructions, but is also able to benefit from numan-writter long and detailed language instructions. We also demon- strate that InstructRL generalizes to new instructions that represent different variations of the task that are unseen during training, and shows excellent model scalability with performance continuing to increase with larger model size.
RELATED WORK
conduton- RL with pre-trained language models. Pre-trained language models have been shown to improve the generalization capabilities of language-conditiong
agents to new
structions and to new low-level tasks (Lynch & Sermanet 2020 Hill et al., 2020) Nair et al. 2202aa
Jang et al.) 2022 Ahn et al.) 2022, Huang et al. 2022). Some prior work use prompt engineering
with large language models (Brown et al. 2020
Chowdhery et al.] 2022) to extract temporally ex-
tended plans over predefined skills (Huang et al. 2022 Ahn et al. all.|..... [022 Jiang et al. 2022), similar to
|022]
work that decomposes high-level actions into sub-gou sub-goals (Team al. 202 These work rely purely
on language models to drive agents and require converting observations into language through pre- defined APIs. Others combine pre-trained language representations with visual inputs (Jang al.
2

---

Under review as a conference paper at ICLR 2023
pre-trained
from-seratch
Pre-trained vision model
11. Pre-trained language model
Place of the
Language encoder
Multi-soale festuure
Place of the
Language encoder
Multi-scale feahure
cubee
Vision
Multi-scale festuure
Vision
Multi-aesle feature
encoder
encoder
III. Disjointly pre-trained vision & language models
IV. Jointly pre-trained vision & language model
Place the
Language encoder
Multé-ccll feature
Place at the red erbees
Vision- language encoder
Multi-acale feature
Vision
Multi-scalo
encodor
feature
Figure 2: Different frameworks of leveraging pre-trained representations for instruction-followii agents. In prior work, additional maining-from-ssracch is needed to combine the representations of text and image from (I) pre-trained vision model, (II) pre-trained language model, (IIII) disjointly pre-trained language and vision models. In contrast, InstructRL extracts generic represen- tations from (IV) a jointly pre-trained vision-language model.
Team et al. 2021 K Khandelwal et al. 2022 using specialized archi- Tectures such UNet (Ronneberger al. 2015) and FiLM (Perez al. 2018). Such approaches have demonstrated success in solving challenging robotic manipulation benchmarks (Guhur et al. 2022 Shridhar et al. 20226). In this work, we argue that using jointly pre-trained viston-language representations with RL. can achieve superior performance in solving complex language-specifed tasks, and show that our proposed approach enjoys better scalability and simpler architecture. Our work is also complementary to prompt-based methods, e.g., SayCan (Ahn et al., (022). This work focuses on improving the mapping from language instructions to robot actions, and we expect that combining our approach with coompt-baset methods achieve greater success.
2022 Lynch & Sermanet, 2020
Language-connditione RL with pre-trained vision-language models. There have been strong interests in leveraging pre-trained vision-language models for language-conditioned RL (Shridhar al. 2022a Zeng et al., 2022 Khandelwal al. 2022), motivated by the effectiveness of vision- language models such as CLIP (Radford et al.) (021). However, these methods use disentangled pipelines for visual and language input, with the language primarily being used to guide perception. Our work uses jointly pre-trained vision-language models that comes with better grounding text-to- visual content (Geng et al.) 2022). The effectiveness of such jointly pre-trained models enables a simple final model, which is jointly pre-trained vision-language transformer followed by policy transformer. While using pretrained vision-language models has been explored in grounded navi-
gation (Guhur al. 202T Hao al. 2020 Majumdar et 2020 Shah al. 2022 our work fo- cuses on manipulation tasks which have combinatorial complexity (composition of objects/actions) Moreover, in contrast these methods, our method InstructRL is simple architecture that scales well to large-scale tasks and can directly merge long complex language instructions with visual input.
RL with Transformers. Transformers (Vawwani et al. 2017) have led to significant gains in natural
language processing (Devlin al. 2018 Brown al. 2020). computer vision (Dosovitskiy et al.
2020
He et al. 2022) and related hells (Lu et al. 2019 Radford et al. 2021 Geng et al. 2022)
They have also been used in the context of supervised reinforcement learning Chen al. 2021 la
navigation (Chnn et al. bb Shah et al. (022). robot learning and behavior cloning from noisy demonstrations (Shafiullah el al. 2022; Cui et al. 2022). and
Reed et al. 2022),, vision-langune
language-conden conditioned RL (Guhur al., 2022 Shridhar al.) (022a). Inspired by their success,
leverage the transformer architecture to extract pre-trained representations from language and vision and learn language-conditione policy.
PROBLEM DEFINITION
We consider the problem of robotic manipulation from visual observations and natural language instructions. We assume the agent receives a natural language instruction ( -. En con-
3

---

Under review as a conference paper at ICLR 2023
Frozen encoder
Policy
Action
Timestep 1
Text: Place 3 of the red oubes
quaternion
camera 3
Vision- language encoder
Cripper
camera.
camera
h1
Transformet Policy
h2
Gripper
position
Text: place
Timestep
Vision-
camera
camera
Catreera
language
hu
Gripper
state
encoder
Figure 3: InstructRL composed of vision-language transformer and policy transformer. First, the instruction (text) and multi-view image observations are jointly encoded using the pre-trained vision-language transformer. Next, the sequence of representations and a history of actions are encoded by the policy transformer to predict the next action.
sisting of n text tokens. At each timestep I, agent receives a visual observation or E and takes an action a, A in order to solve the task specified by the instruction.
We parameterize the policy π (a: | x, {= {a:)==)) as a transformer model, which is condi- tioned on the instruction x, observations {o.} and previous actions {a:)/=1 For robotic control, we use macro steps (James Davison 2022). which are key turning points the action trajectory where the gripper changes its state (open/close) or the joint velocities are set to near zero. Follow- ing Jaames Davison (2022), we employ an -inemaikinemsiics based controller to find trajectory between macro-steps. In this way, the sequence length of an episode is significantly reduced from hundreds small steps to typically less than 10 macro steps.
Observation space: Each observation on consists of images { taken from K different cam- era viewpoints, well as proprioception data of E R... Each image of is an RGB image of size 256 256 x 3. We use
camera viewpoints located on the robot's wrist, left shoulder, and right shoulder. The proprioception data of consists of 4 scalar values: gripper open, left fin- ger joint position, right finger joint position, and timestep of the action sequence. Note that we do not use point cloud data in order for method to be more flexibly applied to other domains. Since RLBench consists of sparse-reward reward and challenging tasks, using point cloud data can benefit performance (James & Davison 2022 Guhur et al. 2022). but leave this future work.
Action space: Following the standard setup in RLBench (James & Davison, 2022), each action a, (Pt, yt, 9t) consists of the desired gripper position p: = (Xi,, Ut. 2x) in Cartesian coordinates and quaternion 4 = (gh q:= qi q²') relative to the base frame, and the gripper state J: indicating whether the gripper is open or closed. An object is grasped when gripper's two fingers and the gripper is closing its grasp. The execution of an action is achieved by motion planner RLBench.
located in between the
4
INSTRUCTRL
We propose unified architecture for robotic tasks called InstructRL, which is shown in Figure
It consists of two modules: pre-trained multimodal masked autoencoder (He et al.) 2022 Geng
ct al. 2022) to encode instructions and visual observations, and a transformer-based (Vaswani al. 2017 policy that predicts actions. First, the feature encoding module (Sec. 4.1) generates token embeddings for language instructions (x, )リ= observations {or)_ and previous actions (a, H=1 Then, given the token embeddings, the multimodal transformer policy (See. 4.2 learn relationships between the instruction, image observations, and the history of past observations and actions, in order to predict the next action dt-
4

---

Under review as a conference paper at ICLR 2023
Action Languago-vision
Rt
Proprioception
Transformer
(Decoder)
Policy
ft
M
A
f++1
Figure 4: The architecture the transformer policy. The model is conditioned on history of language-vision representations and actions to predict the next action.
4.1 FEATURE ENCODING WITH PRE-TRAINED TRANSFORMER
We encode the instruction and visual observations using pre-trained vision-language encoder, as shown in Figure 3 Specifically, we use a pre-trained multimodal masked autoencoder (M3AE) (Geng et al. 2022 encoder, which is large transformer-based architecture based on VIT Dosovitskiy et al., 2020) and BERT (Devlin et al.) 2018). Specifically M3AE (Geng et al, -assed
2022 is transformer- based architecture that learns unified encoder for both vision and language data via masked token prediction. It is trained on a large-scale image-text dataset(CC12M (Chang- pinyo al.) 202T)) and text-only corpus (Devlin et al. 2018) and is able to learn generalizable representations that transfer well to downstream tasks.
Encoding Instructions and Observations. Following the practice of M3AE. we first tokenize the language instructions {-,) into embedding vectors and then apply 1D positional encodings. We denote the language instructions Ex E Raxd where n is the length of language tokens and d is embedding dimension.
We divide each image observation in { into image patches, and use linear projection to convert them to image embeddings that have the same dimension as the language embeddings. Then, apply 2D positional encodings. Each image is represented Eg E R where L. is the length of image patches tokens and de is the embedding dimension.
The image embeddings and text embeddings then concatenated along the sequence dimension: E = concat Fe, F.z) E R((c+))xdd The combined language and image embeddings are then processed by series of transformer blocks to obtain the final representation ot E R(c++)xxd xd. Following the practice of VIT and M3AE, we also apply average pooling on sequence length dimension of of to get of E Rd as the final representation of the k-th camera image of and the instruction. We use multi-scale features h E Rd which are a concatenation layer representations, where the feature dimension d L de equals the number of intermediate layers I. times embedding dimension d.. Finally, we can get the representations over all K camera viewpoints h, (ht,-. hK) E Rxd the representation of the vision-language input.
intermediate
Encoding Proprioceptions and Actions. The proprioception data of E R encoded with a linear layer to upsample the input dimension to d (i.e., each scalar in of is mapped to H&) to get representation 2; 2 xt) E RAxdd all each state in of. Similarly, the action is projected to
feature space E R
4.2 TRANSFORMER POLICY
We consider context-conditional policy, which takes all encoded instructions, observations and actions as input, i.e., { and [- By default, we use context length throughout the paper (i.e., 4(K + 5) embeddings are processed by the transformer policy). This enables learning relationships among views from multiple cameras, the current observations and instructions, and between the current observations and history for action prediction. The architecture of transformer policy is illustrated in F
We pass the output embeddings of the transformer into feature map to predict the next action a, = [pr qr gc]. We use behavioral cloning to train the models. In RLBench, we generate D, a collection of N' successful demonstrations for each task. Each demonstration sequence of (maximum) macro-steps with observations {o actions {aş [T] and instructions
D is composed of
{mi) We minimize loss function L over batch of demonstrations B (5,) | D. The

---

Under review as a conference paper at ICLR 2023
loss function is the mean -quar error (MSE) on the gripper's action:
1
C
MSE (at, di)
(I)
B
BEB
ST
EXPERIMENTAL SETUP
To evaluate the effectiveness of our method, we run experiments on RLBench (James et al. 2020). a benchmark of robotic manipulation task (see Figure We use the same setup as in Guhur et al. 2022), including the same set of 74 tasks with 100 demonstrations per task for training, and the same set of instructions for training and inference, unless stated otherwise. We group the 74 tasks into 9 categories according to their key challenges (see Appendix for each category's description and list of tasks).
For each task, there are a number of possible variations, such as the shapes, colors, and ordering of objects; the initial object positions; and the goal of the task. These variations are randomized at the start of each episode, during both training and evaluation. Based on the task and variation, the environment generates natural language task instruction using a language template (see Ap- pendix A.3)
We compare InstructRL with strong baseline methods from three categories:
RL with pre-trained language model: HiveFormer (Guhur et al. 2022) is state-of-the-art method for instruction instructionfollooinn agents that uses a multimodal transformer to encode multi- camera views, point clouds, proprioception data, and language representations from a pre-trained CLIIP language encoder (Radford et al. 2021). We report the published results of HiveFormer unless otherwise mentioned.
RL with pre-trained vision- lision-language model: CLIP-RL is inspired by CLIPort (Shridhar al. 2022a), which demonstrates the effectiveness of CLIP for robotic manipulation. CLIP-RL uses concatenated visual- and kinguage-representatiin from a pre-trained CLIP model. Similar to InstrucrRL,, CLIP-RL uses multi-scale features by concatenating intermediate layer representa- tions, and trained using the same hyperparameters.
RL trained from scratch: Auto-A (Liu et al. 2022) is model trained from scratch that uses the UNet network (Ronneberger et al., 2015) and applies late fusion to predictions from multiple views. We report the published results of Auto-A unless mentioned otherwise.
InstruciRL uses the official pre-trained multimodal masked autoencoder (M3AE) model (Geng al. which was jointly pre-trained on combination of image-text datasets (Changpinyo et al., 2021 Thomee al. 2016) and text-only corpus (Devlin al. 2018). CLIP-RL uses the official pre-trained CLIP models. See Appendix A.. for more details about the pre- pre-training datasets.
2022
All models are trained for 100K iterations. For evaluation, measure the per-task success rate for 500 episodes per task. Further implementation and training details can be found in Appendix A.I
EXPERIMENTAL RESULTS
We evaluate the methods on single-task performance (Figure 5) ures [). generalization to unseen instructions and variations (Figure Sap. and scalability to larger model size (Figure In all metrics, we find that InstructRL outperforms state-of-the-an baselines despite being simpler method.
multi-task performance (Fig-
Single-task performance. TigureBsshow that, across all categories of tasks, InstructRL performs on par better than all state-of-the-an baselines. On average, Instruc(RL significantly outperforms prior work despite being much simpler.
Multi-task performance. In the multi-task setting, each model is trained set of tasks and then evaluated on each of the training tasks. In Figure 6 we compare multi-task performance on 10 RIBench tasks considered in HiveFormer (Guhur al. 2022) and Auto-À (Liu al. 2022). with

---

Under review as a conference paper at ICLR 2023
100
11/2
66.7
7217
20.5
7628
74.9
65.4
00.7
72.6
Success Rate (%)
59
40
28.3
25
16
Planning
Tools
Rotation Invariant
Muttl Modal
Poroision
Visual Occiusion
Planning
Figure 5: Single-task performance on 74 RLBeecch tasks from 2022 grouped into 9 categories.
James et al.
2020 Guhur et al.
Auto-A
100
92.
35.6
95.5
96.7
TCD
100
41.
84 79.1
29.5
9113
HiveFormer
75.8
99-7
70
Success Rate (%)
69.1
InstructRL
60
53.1
20
10
Pickup
Money
Reach
Take
Average
Button
Umbrolla
Figure 6: Multi-task performance on RLBench tasks from Guhur et al. (2022 : Liu et al.
(0222) J.
100 training demonstrations per task. InstrucIRL exhibits strong multi-task performance compared to other methods. In particular, even though both InstructRL and HiveFormer use transformer policy with language instructions, InstructRL outperforms HiveFormer by a large margin. Fig- ure we further evaluate multi-task performance on all 74 RLBench tasks, and see that InstructRL outperforms CL.P-RI, in most categories. These results demonstrate the transferability of jointly pre-trained vision-language models to diverse multi-task settings.
Generalization unseen instructions and variations. In Figure we report the performance on the Push Buttons task, which requires the agent push colored buttons in specified sequen- tial order given the instruction. variations correctly (e.g., pushing the buttons in a blue-red-gre -rrenn vs. red-green-blue order cannot be inferred from only looking at the scene). We evaluate the models on instructions that are both seen and unseen during training; unseen instructions can contain new colors, colors. We see that InstructRL achieves higher performance on both seen and unseen instructions, in the most challenging case where only 10 demonstrations are available per variation.
this task, instructions are necessary to solve the unseen task
unseen ordering of
Scalability to larger model size. One of the key benefits of pre-trained models that we can use huge amount of pre-training data that is typically not affordable in RL and robot learning scenarios. In Figure 8b] we evaluate different model sizes on multi-task performance across selected tasks (listed in Appendix A.5). For fair comparison with meaninn--rom-scrattee policy transformer, and only vary the size of the transformer encoder. We compare four model sizes: B/32, B/16, 1/16, and H/16, where "B" denotes ViT-base; "I." denotes ViT-large; "H" denotes ViT-huge; and "16" and "32" denote patch sizes 16 and 32, respectively.
fix the size of the
Both CLIP-RL and InstrucIRL improve with larger model size, but InstructRL shows better scala- bility P fact, strong weight decay was needed to prevent this baseline from overfitting the limited data.
On the other hand, learning- bearning-from-scran -rro--scratcch is unable to benefit from larger model capacity; in
We only report the performances of CLIP-RIL RI. with B/32 and B/16, since the larger models (L/16 and H/16) provided in the open-source released models.

---

Under review as a conference paper at ICLR 2023
00.4
82.3
76.
CLIP-RL
70.2
70.1
70.8
$5.1
InstructRL
57.1
31.9
14.3 17.5
Planning
Toole
Term
Rotition Invariant
Mation Planning
Scrww
Mult Modal
Percision
Vasaal Occlusion
Augerage
Figure 7: Multi-task performance on all 74 RLBench tasks grouped into 9 categories.
70.
Demos per Variation
Performance (Success
10.0
From scartch
CLP-RR
InstrucRL
HiveFormer CLIP-RL
InstructRL
1000
400
10
96.8
95.4
97.8
362
Instr.
50
99.6
98.7
100.
100
100.
100.
100.
100
81.5
Unseen
Instr.
50
83.3
84.5
89.4
0232
9/16
100
86.4
85.4
88.9
(a) Success (%) for and unseen instructions
(b) Multi-task performance vs. model size
Figure 8: (a): Performance on the Push Buttons task, which has many variations on the ordering of colored buttons. InstruciRL achieves higher performance both seen and unseen instructions, even with only 10 training demonstrations per variation. (b): We compare four different model sizes of the transformer encoder. All methods are evaluated on multi-task performance across 14 selected tasks (listed in Appendix A.5 InstructRL scales well with larger model size.
7 ABLATION STUDY
effective with joint vision-language model. Table I shows models trained with and without instructions, on 10 RLBench tasks from Guhur et al. (2022); Liu et al. (2022). While using instructions improves the performance of all methods, they most effective with InstructRL. This is not surprising, as the CLIP language encoder in HiveFormer was not pre-trained on large text corpora. One can pair CLIP with a large language model such BERT, but doing so can often hurt performance (Guhur et al. (022) due to the language representations not being aligned with the visual representations. On the other hand, InstructRL shows the strongest performance due to using the M3AE encoder, which was jointly pre-trained on both image-text and text-only data.
Instructions
Detailed instructions require language understanding, In Table 2 also evaluate the methods on longer and more detailed instructions, which include step-by-stey instructions specific to the task (see Table [3] for examples). The detailed instructions are automatically generated using template that contains more details than the default instruction template from RLBench ("Default"). Each "Tuned Short Instruction" has maximum token length (to be compatible with CLIP's language encoder), while each "Tuned Long Instruction" can have up to 256 tokens (to be compatible with InstructRL).
We compare InstructRL with CLIP-RL, which uses concatenated visual- and language- representations from CLIP, as well as a variant of CLIP-RL called BERT-RL, which uses a BERT language encoder and trained-from-see from-scratch vision encoder. Generally across all methods, see that performance increases with longer and more detailed instructions, which implies that the pre-trained language encoders can extract task-relevant information from language instructions. InstructRI. achieves the best performance, especially with longer instructions (e.g., InstruciRI. achieves 63.7 vs. 46.1 from BERT-RL, on the Push Buttons task with Tuned Long Instructions).
Fusion strategy. How to fuse representations from multimodal data is one of the key design choices in InstructRI.. In Figure compare variants of InstructRI. with other fusion strategies: Concat
8

---

Under review as a conference paper at ICLR 2023
Table 1: Success rates (%) of all methods with and without instructions on 10 RLBench tasks.
Pick & Lin
Pick-Up Cup
Push Button
Put Knife
Put
Reach
Slide Block
Stack Wine
Take Money
Take Umbbrela
Avg.
Money
Target
Auto-A
100
38
HiveFormer
92.2
77.1
99.6
69.7
96.2
100
95.4
81.9
82.1
90.1
88.4
CLIP-R w/o inst
78.3
65.1
87.6
42
44.5
98.9
52.9
28.9
41.2
45.1
58.5
CLIIP-L RI.
80.4
72.4
95.2
72.1
62.1
100
56.8
56.5
69.1
78.8
74.3
InstructRL w/o inst
75.9
61.2
85.3
48.9
48.9
98.7
54.5
29.5
43.2
47.8
59.4
InstruciRL.
97.8
84,5
99.5
84.5
98.7
100
97.5
93.2
89.8
92,94
93.8
Table 2: Comparison between using default instructions vs. longer and more detailed instructions.
Push Butions
Light Bulb In
Default
Tuned Short Inst
Tuned Long Inst
Default
Tuned Short Inst
Tuned Long Inst
BERT-RL CLIP-RI RI.
41.3
46.1
13.6
12.2
15.9
45.4
$1.2
NIA
16.4
20.2
N/A
InstrwerRL
52.6
$3.2
22.6
26.1
31.8
Table Longer instructions for Push Buttons and Stack Blocks.
Task
Instructions type
Example instructions
Default
Push the red button, then push the green button, then push the yellow button
Move the gripper close red button, then push red bution, after that,
Shoet Tuned Inst
move the gripper close to the green button push the green button, finally,
move the gripper close to the yellow button to push the yellow button.
Push Buttons
Move the while griper closer red buiton, then push red bullon down, alter pusting red button.
pull the white gripper up and move the gripper closer to green button, then push
Long Tuned Inst
after pushing green button. pull white gripper up and move gripper closer to the yellow button, then push yellow bution down
green button
Default
Place 3 of the red cubes
of each other
Choose a red cube as the stack hase, then pick amother red cube and place onto the red cube slack,
Short Taned Inst
repeat until the stack has red ouhes.
Stuck Blocks
Choose a rod cube as the stack basc, then pick another red cube and place it onto the red cube stack,
Long
then move white gripper to pick another cube and place it onto the rod cube stack, repeat uotil the stack has red cubes.
43.4
Length
Success 1%
Representation selection
Success Rate
133
20.4
23.6
2
1
55.1
56.2
Last Layer
Second Lo-L Layer
47.1
46.3
400
4
58.1
Concat Last Eight Layers
48.4
8 60.3
Concat First Eight Layers
45.1
Concal Hidden Layers
51.4
(a) Fusion strategies
(b) History context lengths
(c) Feature selection
Figure 9: Ablations of InstructRL evaluated on the 14 selected tasks listed in Appendix A.5 We report the average success % over all tasks. (a): Performance of InstructRL with different strategies for fusing the language and vision features. (b): Effect of history context length on the performance of InstructRL. (c): InstruciRL with features selected from different layers.
runs the encoder twice to obtain vision and language representation separately, then concatenates them together. FiLM (Perez et al. (2018) fuses vision and language features layer- before concatenating them together. Default refers to InstructRL where language and vision are encoded once to obtain joint representations. We see that using joint representation of language and vision inputs is critical for performance.
History context encoding. The flexibility of the transformer architecture allows us to encode mul- tiple past steps. To investigate the benefit of utilizing historical information, we ablate the history context length. Figure 9b shows that there is steady increase in performance when we increase the context length. However, using a longer context requires more compute and memory (e.g., in- creasing the context length from 4 adds about 20% longer training time one task). Thus, choose a context length of 4 for computational efficiency.
Multi-scale features. In Figure 9c compare the performance of InstructRL with features lected from different layers. The results show that using a combination of intermediate representa- tions is most effective.

---

Under review as a conference paper at ICLR 2023
8 CONCLUSION
In this paper, we propose InstructRL, a novel transformer-base instruction following method. In- struc/RL has simple and scalable architecture which consists of pre-trained multimodal trans- former and a policy transformer. Experimental results show that InstructRL significantly outper- from--ccrach instruction following methods. As In-
forms the fate-of-ther of-thh--rtttttt pre-trained pre-trained and struc/RL achieves state-of-the-art results and scales well with model capacity, applying our approach larger scale of problems would be an interesting future work.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-11999 2020.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training recognize long-tail visual concepts. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 3558- 3568. Computer Vision Foundation IEEE, 2021. doi: 10. 109/CCPR44433. 202.33..22222222 2021.0036.. URL https::/openacceet
www.tceeceecc com httpw///eo......eeeoeeooo//////a
Conceptual_12M Pushing Web-Scale Lew.-e--eextPPe--raaaaae Pre- To
Recogmize_Long Tai [iccemm.e__Looggggaa1_sssssa____ 2ww.
html
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 50044-50977 2021a.
Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal
transformer for dasioonand-languaal navigation. Advances in Neural Information Processing Sys- tems, 34:5834-5847, 2021b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204. 02311, 2022.
Zichen Jeff Cui, Yibin Wang, Nur Muhammad, Lerrel Pinto, et al. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint 0rXiv:2210.10044 2022.
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Web-curated image- text data created by the people, for the people. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL
http::///dttaaetss-ennmmaaasssss ha21:/6000000553aooaaaaaaaaa2222 Absttacct- roundl.htm!
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprins arXiv:1810.04800. 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurams, Sergey Levine, and Pieter Abbeel. Multimodal masked autoencoders learn transferable representations, arXiv preprint arXiv: 14204, 2022.
Pierre-Louis Guhur, Makarand Tapaswi, Shizhe Chen, Ivan Laptev, and Cordelia Schmid. Airbert: In-domain pretraining for vision-and-lane lan-uag navigation. In Proceedings of the IEEE/CVF In- fernational Conference on Computer Vision, pp. 1634-1643, 2021.
10

---

Under review as a conference paper at ICLR 2023
Pierre-Louis Guhur, Shizhe Chen, Ricardo Garcia, Makarand Tapaswi, Ivan Laptev, and Cordelia Schmid. Instruction-driven history-aware policies for robotic manipulations. arXiv preprint arXiv:2209. 204.0999,9, 04899, 2022.
Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning generic agent for vision-and-language navigation via pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13137-13146, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au-
toencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human instruction-followinng with deep reinforcement learning via transfer-learning from text. arXiv preprint arXiv:2005.09388, 2020.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207 2022.
Stephen James and Andrew J Davison. Q-attention: Enabling efficient learning for basend robotic manipulation. IEEE Robotics and Automation Letters, 7(2):1612-169, 2022.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):0019 3026, 2020.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Confer- ence on Robot Learning, pp. 991-1002. PMLR, 2022.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yonggiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.03094, 2022.
Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effec- tive: Clip embeddings for embodied ai. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14829-14838, 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. 1tLhttp:
//arxxv.orq/ abs//////////// 1412. 6980
Shikun Liu, Stephen James, Andrew J. Davison, and Edward Johns. Auto-lambda: Disentangling dynamic task relationships. arXiv preprint arXiv: 2xxiv-2202.0009,, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17100 2017.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019.
Corey Lynch and Pieme Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint wxXiv22005.07648 2020.
Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruy Batra. Improving vision-and- languaae navigation with image-text pairs from the web. In European Conference on Computer Vision, pp. 259-274. Springer, 2020.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkamn,, Li Fei- Fei, Silvio Savarese. Yuke Zhu, and Roberto Martinn-aattn.... What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint 108 108.00080000000 2021.
11

---

Under review as a conference paper at ICLR 2023
Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, al. Learning language- conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning. pp. 1303-1315 1315. PMLR, 2022a.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A univer- sal visual representation for robot manipulation. arXiv preprint (oXiv:2203./260) 2022b.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.061775 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 241. Springer, 2015.
Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning k modes with one stone. arXiv preprint arXiv: Arxiv-2206. 11251, 2022.
Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. arXiv preprint arXiv:2207.0442: 2022.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, pp. 894-906. PMLR, 2022a.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. arXiv preprint arXiv:2209.05451, 2022b.
DeepMind Interactive Agents Team, Josh Abramson, Arun Ahuja, Arthur Brussee, Federico Carnevale, Mary Cassin, Felix Fischer, Petko Georgiev, Alex Goldin, Tim Harley, et al. Cre- ating multimodal interactive agents with imitation and self-supervised learning. arXiv preprint
a 2021.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfce100m: The new data in multimedia research. Communications of the ACM, 59(2):64-77 73, 2016.
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2019, pp. 6558. NIH Public Access, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Puro- hit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.0055 2022.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. The IEEE International Conference on Computer Vision (ICCV), December 2015.
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 IMPLEMENTATION AND COMPUTE RESOURCES
We use the Adam W (Kingma & Ba 2013 Loshchilov & Hutter 2017) optimizer with a learning rate of x 10-4 and weight decay be All models are trained on TPU v3- 128 using cloud TPU. Since TPU does not support headless rendering with PyRep simulators that RLBench is built upon, we evaluate the models on NVIDIA Tesla V100 SXM2 GPU using headless rendering. Each training batch per device consists of 32 demonstrations sequence with length 4, in total the batch size is 512. All models are trained for 100K iterations. We apply data augmentation in training, including jitter over the RGB images cx.
Our implementation of InstruciRL built upon the official JAX implementation of multimodal masked autoencoder (M3AE) Geng et al. 2022 4 and we the official pre-trained M3AE models that were pre-trained on CC12M (Changpinyo et al. (021) and text corpus (Devlin et al.) 2018).
Our implementation of CLIP-RL is built upon a JAX CLIP implementation and we use the official pre-trained CLIP model for the visual- and angguge-representation.
Both CLIP-RL and InstructRL ViT-B/16 in all experiments unless otherwise specified. As demonstrated in experiment section, while the larger models ViT-L/16 and ViT-H/16 can further boost InstructRL's 's results, comparison with baselines.
ViT-B/16 to reduce computation cost and to have apple apple
Our code is available httt://s///ttsss.oooo..o///////s
A.2 PRE-TRAINING DATASETS
A.2.1 M3AE
M3AE is trained on combination of image-text datasets and text-only datasets. The image-text data includes the publicly available Conceptual Caption 12M (CC12M) (Changpinyo et al. 2021), Redcaps (Desai et al. 2021), and 15M subset of the YFCC100M (Thomee al. 2016) selected
according to (Radford et al. 2021).
The text-only data includes the publicly available Wikipedia and the Toronto BookCorpus (Zhu et al., 2015). For language data, we use the BERT tokenizer from Huggingface to tokenize the text. Following Geng al. (2022), use 0.5 for text loss weight and 0.75 for text mask ratio. We use 0.1 for text-only loss.
A.2.2 CLIP
CLIP is trained on the publicly available YFCC100M (Thomee et al. 2016) which consists of 100M image-text pairs.
A.3 RLBENCH INSTRUCTIONS
We use the task instructions provided in RLBench. For each task, there are number of possible variations, such as the shapes and colors of objects, the initial object positions, and the goal of the task. Based the task and variation, the environment generates natural language task instruction. For example, in the task "Put groceries in cupboard", the generated instruction is of the form "pick up the OBJECT and place it in the cupboard" where OBJECT is one of possible grocery items. A full list of RLBench task instructions can be found at https: /ttpt//ttttt/ttt// com/ step jam/
baselines/
13

---

Under review as a conference paper at ICLR 2023
Task
stack blocks
slide block open drawer push buttons meat off grill]
pot money safe
|stack wine]
turn
Number of
demonsttations
100
10
100
100
10
100
100
10
100
10
100
10
100
10
HiveFormer
31%
19%
82%
63%
8866
52%
65%
34%
79%
45%
152%
15%
25%
7%
75%%
61% 83%
InstruetRL.
41%
32%
89%
78%
94%
81%
8996
61%
92%
65%
75%
42%
37%
12%
68%
Table 4: Performance (success rate %) of HiveFormer and InstrucrRL while varying the number of demonstrations. InstrucIRL use fewer demonstrations achieve high success rate, while HiveFormer requires more demonstrations.
A.4 EVALUATION DETAILS
In multi-task setting, models are trained on multiple tasks simultaneously and subsequently evalu- ated on each task independently. the generalization experiment setting, models are trained on subset of all variations, then evaluated on unseen task variations.
A.5 MULTI-TASK MULTI-VARII VULTI-VARIATION TASKS DETAILS
For experiments including model scaling (Figure 8b) and ablation studies (Figure 9 we selected 14 difficult tasks that have multiple variations: Stack wine, Open drawer, Meat off grill, Put item in drawer, Turn lap, Pul groceries in cupboard, Sort shape, Screw bulb in, Close jar, Stack blocks, Put item in safe, Insert peg, Slack cups, and Place cups.
Each task comes with multiple variations that include objects colors, objects shapes, and the ordering of objects to be interacted with. To make fair comparison with baselines, we use the slightly modified version of RLBench from HiveFormer (Guhur et al. (022). The changes include adding new tasks and improving the motion planner. The Tull details of each task can be found in RLBench Github repe and the HiveFormer Github repo
A.6 SAMPLE EFFICIENCY ABLATION
In the experiments, we used 100 demonstrations following the same setup as in prior work, and InstructRL outperforms prior state-of-the-arts significantly. However, getting 100 demonstrations can be expensive and difficult in many real world tasks. We hypothesize that InstrucrRL can achieve better sample-efficien learning thanks to the joint language-vision encoder pretrained on large-scale passive datasets.
To study this, randomly choose subset of tasks and evaluate the performance using only 10 demonstrations. The results from Table 4 show that InstructRL. outperforms baselines when using 100 demonstrations achieves competitive or higher success rates than baselines that use 100 demonstrations. For ex- ample, on the stack blocks task, InstructRL using 10 demos achieves 32% success rate while HiveFormer gets 31% using 100 demos. Similarly, on the open drawer task, InstructRL using 10 demos gets 81% % while HiveFormer gets 86% using 100 demos.
10 demonstrations. Moreover, using only 10 demonstrations, InstructRL
The results show that InstructRL is more sample efficient than prior state-of-the-arts.
A.7 TASK CATEGORIES DETAILS
To compare with baselines including Guhur et al. 2022). our experiments are conducted on the same 74 out of 106 tasks from R The 74 tasks can be categorized into 9 task groups according to their key challenges as shown in Table
Task diversity. The 74 tasks RLBench cover wide range of challenges that are essential for robot learning, including planning over multiple sub goals (e.g., picking a basket ball and then throwing the ball) and domains where there multiple possible trajectories to solve task due to large affordance area the target object (e.g., the edge of a cup).
14

---

Under review as a conference paper at ICLR 2023
RLBench employs 3 keys terms: Task, Variation, and Episode. For each task there are more variations, and from each variation, an infinite number of episodes can be drawn. Each variation of task comes with a list of textual descriptions that verbally summarise this variation of the task.
An example showing the distinction between task and variation is Figure and Figure 4 of James al. 2020
Task difficullyy. RLBench a manipulation benchmark that is significantly more difficult than loco- motion benchmarks. Some tasks involve precise object manipulation such as 'put knife on chopping board' and 'take usb out of computer', some tasks require preceise grasping such as 'close laptop lid' and "open and close drawer', some tasks require solving long horizon tasks that involve many composed sets of actions, for example, the 'empty dishwasher' task involves opening the washer door, sliding out the tray, grasping plate, and then lifting the plate out of the tray. In this work we consider tasks that can be grouped into 9 categories, as shown in Table 5 to comprehensively evaluate the effectiveness of InstructRL.
In addition to manipulation and perception challenges, RLBench comes with a suite of diverse task instructions that require natural language understanding. RLBench has large and diverse vocab- ulary, it has over 100 content words vocabulary size (e.g., table, cup, open, grasp, box, etc) with function words removed, combing with the diverse visual input and complex interactions leads to suite of challenging instruction following manipulation tasks.
B INSTRUCTION TEMPLATES AND GENERATIONS
Default instructions generation. The default instructions templates are shown in Table 0
Long instructions generation. The tuned instructions that contain step-by-step and detailed de- scriptions of objects are shown in Table
15

---

Under review as a conference paper at ICLR 2023
Group
Challenges
Tasks
Planning
multiple sub-goals (e.g., picking basket ball and then throwing the ball)
basketball in hoop, put rubbish in bin, meat off grill, meat on grill, change channel, tv on, tower3, push buttons, stack wine
Tools
a robot must grasp an object to in- teract with the target object
slide block to target, reach and drag. take frame off hanger, water plants, hang frame on hanger, scoop with spatula, place hanger on rack, move hanger, sweep to dustpan, take plate off colored dish rack, screw nail
Long term requires more than 10 macro-steps to completed
wipe desk, stack blocks, take shoes out of box, slide cabinet open and place cups
Rotation- invariant
can be solved without changes in the gripper rotation
reach target, push button, lamp on, lamp off, push buttons, pick and lift, take lid off saucepan
Motion planner
requires precise grasping
toilet seat down, close laptop lid, open box, open drawer, close drawer, close box, phone on base, toilet seat up, put books on book- shelf
Multimodal
can have multiple possible trajec- tories to solve task due large affordance area the target object (e.g., the edge of cup)
pick up cup. turn tap, lift numbered block, beat the buzz, stack cups
Precision
involves precise object manipula- tion
take out of computer, play jenga, in- sert onto square peg, take umbrella out of umbrella stand, insert usb in computer, straighten rope, pick and lift small, put knife on chopping board, place shape shape sorter, take toilet roll off stand, put umbrella umbrella stand, setup checkers
Screw
requires screwing an object
turn oven on, change clock, open window, open wine bottle
Visual-
involves tasks with large objects and thus there are occlusions from certain views
close microwave, close fridge, close grill, open grill, unplug charger, press switch, take money out safe, open microwave, put money in safe, open door, close door, open fridge, open oven, plug charger in power supply
Occlusion
Table 5: RLBench tasks used in our experiments grouped into 9 categories.
16

---

Under review as a conference paper at ICLR 2023
Task
Variation Type
# Variations
Language Tempalte
basketball in hoop
NA
the ball in the hoop put rubbish in bin take the (shicken, steak) off the grill 2
put rubbish in bin
NA
1
meat off grill
Name
meat on grill
Name
2
take the (chicken, steak) on the grill
change channel
Up/
turn channel up, down
tv on
NA
turn on the TV
push [color_name] button.
push buttons
Ordering & Colors
200
then push the {color .naee) } button, then stack wine bottle
stack wine
NA
elide block to Larget
NA
slide the block targe
reach and drag
Color
20
use the stick to drag the cube onto the [color_name) target hang frame on hanger
take frama off hanger
water plants
NA
pick up the watering can by its handle and water the plant hang frame on banger
hang frame on hanger
sccop with spstula
NA
take frame off hanger
place hanger on rack move hanger
NA
pick up the hunger and place in on the ruck move hanger
NA
other rack sweep dirt to dustpan put the plate between screw the nail in to the block
aweep to destpan
NA
1
take plate off colored dish rack
Colors
namler.name) pillars of rack
screw nall
NA
wipe desk
NA
wipe dirt off the desk
stack blocks
Number of blocks
90
stack blocks to stack (color name) }
blocks
Colors.
take shoes out of box
take shoes
slide cabinet open and place cups
Left/ / Right
2
cup in [aplion] cabinet
reach target
Colors
the (color name) target
puah button.
Colors
18
push the (button .color n
lamp on
NA
turn on light
lamp off
NA
1
tur off the light
pick and lift
Colors
20
pick up the bhlockscolor,.._
block and lift up to the target
take 1id off saucepan
NA
take lid off the saucepan
toilet seat down
NA
toilet seat down
close laptop lid
laptop
open box
NA
open box close {option] } drawer
open foptien) draver
Bottom Middle /
Top
close (option) drawer
Bottom Middle /
3
close option drawer
Top
close box
NA
close box
phone on base
put the phone on the base toilet seat
toilet seat up
NA
1
put books on bookshelf
Number
put (index) books on bookshelf
pick cup
Colors
20
pick up the [target_color, caallo_aamme___e cup turn [option] tap
turn tap
Left/ Right
2
1ift numbered block
Number
pick up the block with the number {block num} beat the buzz
beat the buzz
NA
sLack cups
Number
cups.
3x30
stack blocks.to_stack, [color_name] cups
Colors.
take usb out of computer play jenga
NA
take ush out computer play jenga
NA
1
insert onto aquaze peg
Colors
20
put the ring on the (color_name) spoke take umbrella
take umbrella out of unbrells stand
umbretla stand
insert usb in computer straighten repe
NA
insert usb computer straighten rope
NA
I
Shapes ('cuhe', 'cylinder', "triangular prism', 'star', 'moon')
pick and lift ama
pick the (shape_nams) and lift to the target
put knife chopping board place shape in shape sorver
NA
I Shapes ('cube', 'cylinder', "triangular prism', 'star', "moon')
the knife the chopping beard
the {shapc} in the shape sorter
Lake toilet roll stand
NA
take toilet roll off stand put umbrella in umbrella stand place the remaining {number] checker in initial position on the board turn on the oven change clock to time 12.15
put umbreils in umbrells stand
NA
1
setup checkers turn oven on change clock open window open wine bottle
Number
NA
NA
open left window open wine bottle
NA
close microwave
NA
close microwave close fridge the grill
close fridge
NA
closa grill
open gril!
NA
open the grill unplug charger press switch take the money out of the (index) shelf and place on the lable open microwave
unplug chazge:
press switch
NA
take money out safe
Number
open microweve
NA
put noney in sase
Number
3
put the money away in the safe on the (index) shell' open door close the door open fridge
open door
close door
NA
open fridge
open even
NA
open the
plug charger in power supply
NA
plug charger in power supply
17
Table 6: Language instructions templates in RL. (James et al.
2020

---

Under review as a conference paper at ICLR 2023
push bus- tons
stack blocks
Tisk
colors
Ordering bus
calors
Number blocks.
Variation Type
200
90
Num of Vartations
push color mume) hutton, then pash the (color fcolor Laann) button, ...
color nime) blocks
(blocks [blocks..o.ssuckk) stack))
stack
Template
Defaolt
Language
the
Chooses (color cnmme cube as stack bese, then pick another [color Ccolor-amn nume) cuhe and place it onto the (coler name stack, repeat until the stack has 3 [coloe name) cahes.
Move the grpper close to [lst color button, then push the 2nd color name) ballon. alter that muve the grip. close to the (3nl color name) button pash the (4th color nane) button, finally, ... move the griper close to (r-th color button.
Sheet Tuned Language Template
button push the (weth color color.name mnmee))))
Choose a (color chmee cube as the stack base. then pick ait- other (calar color..mn nume caht and place it anto the (colue polor.nama nume cahe stack, then move the white gripper to pick another (color name) cube and place it oeto the {solor (solor nanee) eahe stack, repeat the stack has
ton tondown down
Move the white gripper closer [1st color colornaane) bult- tom. then push (2nd color color nume bulonn down, alter push white gripper and move the gripper closer to (4th color nane) butoon, ... pull white gripper and move the gripper closer (nth color color nume) button, then push (rih color mame but-
ing [3rd color name) hutton,
Long Tuned Langinge Template
poraaammm cubos.
Table 7: Language instructions templates for tuned instructions used in Table and Table
18


Reviews:
[{"review_id": "nrw53h9-jN", "reviewer": "Reviewer_boPj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the instruction-following problem, prior work uses language-only models,which lacks the grounding between language and observations, this paper proposes a method (InstructRL), which contains a vision-and-language encoder, and a policy transformer for action prediction, the vision-and-language transformer encoder is pretrained on millions of image-text pairs and natural language text; empirical experiments on a robotics benchmark show the SoTA results.", "summary_of_the_review": "It is a borderline, empirical work to show the pretrained vision-and-language transformer as encoder for instruction-following agents in the robotics tasks/environments, and achieves the SoTA on a set of robotics benchmark; the main concern is the novelty of the proposed model/method, which was explored a lot in the vision-and-language navigation tasks.", "strength_and_weaknesses": "Strengths:\n1. This work leverages the pretrained vision-and-language transformer as encoder for instruction-following agents; \n2. This work shows promising (SoTA) results on a set of robotics benchmark.\n\nWeakness:\n1. Somehow the proposed model looks not novel in the vision-and-language navigation tasks, but may be a good attempt in the robotics tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CLY079SBGt", "reviewer": "Reviewer_uxTU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple method to solve the task of instruction-following in multimodal environments. While recent work makes use of pre-trained transformers, their\nperformance is limited by (i) lack of grounding (in the case where separate vision and\nlanguage models are used) and (ii) lack of ability to follow long instructions (in the case\nof CLIP). To overcome these limitations, this paper proposes using M3AE, a recently\nproposed multi-modal transformer, as the backbone to provide vision-language\nrepresentations. In experiments on RLBench, InstructRL, the proposed method, is able\nto outperform prior work such as HiveFormer (which makes use of separate vision and\nlanguage models) and a CLIPort-inspired CLIP-RL method across all categories of\ntasks. Further analyses reveal that the model is also capable of handling longer\ninstructions, novel instruction combinations, and scales well with model size.\n", "summary_of_the_review": "Current decision: Borderline Reject\nThe novelty and the lack of self-contained information in this paper imply that the paper\ncurrently stands as a reject. However, the system proposed is very simple and hence\nthe decision is at borderline reject. If the author(s) are able to address the issues wrt\nablations and writing, the score can be reconsidered.", "strength_and_weaknesses": "Pros:\nProposed method is simple and seems easy to implement with the provided details\nfor the task.\nStrong performance improvements across all categories on the RLBench\nbenchmark. Additional analyses also reveal the importance of multiple design\nchoices that have been used by the authors in InstructRL.\nCons:\nWhile the proposed system is simple, there is a slight novelty issue. Stronger pretraining models are expected to provide better representations and hence improve\nmodel performance. The architecture itself seems derivative, given the HiveFormer\nframework.\nIn the ablation studies, InstructRL performs with longer instructions than CLIP- or\nBERT-based models. However, there is no mention of what these instructions look\nlike and what aspect of the original instructions have been expanded to generate\nInstructRL 2\nthe longer instructions. Without such details, it is hard to judge the pure numbers\nprovided for this ablation.\nIn the ablation wrt context length, the author(s) mention that “improvement saturates\naround 4”. From the numbers in Table 9b, it is hard to arrive at this conclusion as\nthere is a steady increase in performance from 1 → 2 → 4 → 8.\nThe writing of the paper is also highly reliant on readers having prior knowledge\nabout the RLBench benchmark, and does not provide adequate details about the\nbenchmark for new readers to understand and evaluate the findings. Without\nappropriate discussion, it is hard to how hard the tasks are.\n\nMinor Edits:\nFigure 5b has some spelling mistakes in the x-axis.\nIn Section 4.1, under ‘Encoding instructions and observations’, the final representation\nof the transformer blocks is mentioned to have dimensionality ‘d’. If that’s the case,\nshouldn’t the dimensionality of be where L is the number of concatenated\nlayer representations?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "33VLAZMzswK", "reviewer": "Reviewer_uqHe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work introduces a new model for agents to perform vision and language task. The method, named InstructRL relies on a vision and language transformer (ViLT) trained on passive datasets with language and images aligned with description. This ViLT is used to convert observations and descriptions into tokens, which are consumed my a policy transformer, trained with expert demonstrations to generate gripper actions. The method is evaluated in RLBench, showing SOTA performance amongst selected baselines, generalizing to longer instructions and zero-shot task specifications.", "summary_of_the_review": "I really value the use of pre-trained models and simple architectures for visual-language embodied tasks. The fact that the performance scales well with larger model size is promising, and it would also be interesting to see if it scales with pre-training size. The proposed model also obtains better performance than the proposed baselines. Despite the above strengths, I think the paper needs to clarify a few-points, particularly the omission of certain baselines, the effect of model architecture vs pre-training data, the choice of the policy network, and effect of fine-tuning on the RLBench dataset. If authors can clarify why the baselines I mention in my weaknesses are not mentioned, as well as my questions regarding some of the differences in the baselines selected, and correct some of the mistakes I pointed I would be happy to raise my score.", "strength_and_weaknesses": "**Strengths**\n\n- Generally clear and well written paper. Both the contributions and the model are clearly explained, in some cases I would value a short description of the work that is cited to avoid switching to the refered paper for basic information. For instance for 4.1, it would help to explain M3AE is a transformer-based architecture via masked token prediction.\n\n- Very simple model, reuses components trained on passive datasets which allows the model to potentially improve as these representations improve, I would assume this makes the policy train much faster than if using a transformer from scratch, but it would be great to validate that. The performance is comparable or higher than baselines in all tasks (both single and multitask settings), though as mentioned in strengths, I have some questions and suggestions regarding baselines.\n\n- Results: \n\t- Nice results on unseen instructions, even though the fact that it is only in a button pussing task is slightly underwhelming. It would be amazing to see it for novel object categories as well.\n\n\t- Very good insight in Figure 4.b, in general I value the ablations performed in section 7, see weaknesses though for caveats.\n\n\n**Weaknesses**\n- I would include in related work methods that convert observations into language and rely purely on language models to drive agents. Note that these are different from methods such as (Huang et al. 2022), since the former group still consumes observations from the agent. Examples of those would be (Li et al. 2022, Shridhar et al. 2021)\n\n\n- Clarity and Figures\n\t- On the flip side on clarity, I would recommend adding some more detail in the VIT policy. For instance, h_{t}^{k} are intermediate layer representations after average pooling through the sequence length dimension? Or the average pooling is applied separately on each intermediate representation? More clarity here would be appreciated. Is the linear layer for proprioception a 4x(4d) layer or 4 (1xd) layers.\n\n\t- It would help if Figure 4 had a legend for colors representing what corresponds to instr/observation, proprioception and action.\n\n\t- Figures should be more carefully made. There are overlaps between numbers in Figure 5, and Figure 6, and I see no reason why the 2 figures should have different formats. Barplots should have standard error. \n\t- For Fusion strategy, a figure in the appendix of the strategies would be useful, together with a figure of how is done for CLIP-RL.\n\n\n- Model\n\t- The transformer policy is simple which is ok, but I would like a bit more motivation for it. Why not incorporating a reward at the input as done by the Decision Transformer?\n\n\n- Baselines: \n\t- Why not including Perceiver-Actor? \n\t- Could authors add more detail about what is the difference between CLIP-RL and the original CLIPORT? It does seem pretty different from CLIPORT so I am not sure how it is inspired, or where is the difference with the proposed method. Is it the fact that the transformer takes separately vision and language?  \n\t- Why do authors change the architecture in the different baselines? It is hard to know if the gains come from architectural changes or from the fact the models are trained from scratch.\n\t- Did authors try fine-tuning the vision-language model with the RLBench dataset/environment? Either in a MAE way to directly with the MSE objective?\n\n- Result:\n\t- Long instruction settings are a great idea but they are evaluated rather underwhelmingly here. From table 2 I get that InstructRL can support more tokens than Clip-RL, but the example of instructions do not add any extra info in the task. I think the right setting or evaluation would be to include instructions that add actually new information, and see if the task is done according to those.\n\n- It would be good to validate effect of #iter training for different baselines. I assume one advantage of the proposed method is that it learns faster.\n\n\n\nClarification Questions:\n- It seems like the policy here is deterministic, could authros comment on what is the source of randomness when running multiple seeds? Is the environment stochastic?\n\n- Hiveformer is not trained on large-text corpora. What is it trained on?\n\nMinor: \n- Typo: 4.2 Instruction --> instructions\n- Typo Figure 5 Avgerage\n\n\nLi, Shuang, et al. \"Pre-trained language models for interactive decision-making.\" arXiv preprint arXiv:2202.01771 (2022).\n\nShridhar, Mohit, et al. \"ALFWorld: Aligning Text and Embodied Environments for Interactive Learning.\" ICLR. 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject