VIMA: General Robot Manipulation with Multimodal Prompts
Authors: Yunfan Jiang
Agrim Gupta
Zichen Zhang
Guanzhi Wang
Yongqiang Dou
Yanjun Chen
Li Fei-Fei
Anima Anandkumar
Yuke Zhu
Linxi Fan
Venue: Submitted to ICLR 2023 (2023)
Keywords: Robot Learning
Foundation Model
Transformer
Language Model
Multi-Task Learning

Abstract: Prompt-based learning has emerged as a successful paradigm in natural language processing, where a single general-purpose language model can be instructed to perform any task specified by input prompts. Yet task specification in robotics comes in various forms, such as imitating one-shot demonstrations, following language instructions, and reaching visual goals. They are often considered different tasks and tackled by specialized models. This work shows that we can express a wide spectrum of robot manipulation tasks with *multimodal prompts*, interleaving textual and visual tokens. We design a transformer-based generalist robot agent, VIMA, that processes these prompts and outputs motor actions autoregressively. To train and evaluate VIMA, we develop a new simulation benchmark with thousands of procedurally-generated tabletop tasks with multimodal prompts, 600K+ expert trajectories for imitation learning, and four levels of evaluation protocol for systematic generalization. VIMA achieves strong scalability in both model capacity and data size. It outperforms prior SOTA methods in the hardest zero-shot generalization setting by up to 2.9x task success rate given the same training data. With 10x less training data, VIMA still performs 2.7x better than the top competing approach. Video demos are available at https://iclr3081.github.io/.

Under review as a conference paper at ICLR 2023
VIMA: GENERAL ROBOT MANIPULATION WITH MULTIMODAL PROMPTS
Anonymous authors Paper under double-blind review
ABSTRACT
Prompt-based learning has emerged successful paradigm in natural language processing, where a single general-purpose language model can be instructed to perform any task specified by input prompts. Yet task specification in robotics comes in various forms, such as imitating one-shot demonstrations, following lan- guage instructions, and reaching visual goals. They are often considered different tasks and tackled by specialized models. This work shows that we can express a wide spectrum of robot manipulation tasks with multimodal prompts, interleaving textual and visual tokens. We design a Transformr-bbssee robot agent, VIMA, that processes these prompts and outputs motor actions autoregressively. To train and evaluate VIMA, we develop new simulation benchmark with thousands of sereedurally-generrae tabletop tasks with multimodal prompts, 600K+ expert tra- jectories for imitation learning, and four levels of evaluation protocol for systematic generalization. VIMA achieves strong scalability in both model capacity and data size. It outperforms prior SOTA methods in the hardest zero-shot generalization setting by up to 2.9x task success rate given the same training data. With 10x less training data, VIMA still performs 2.7x better than the top competing approach. Video demos are available at ht https: /ett://icrrrrre iecrrrrr1111..1 wwe.:0111.ithia.e github. io/.
11
13
15
16
17
INTRODUCTION
19 Transformers have given rise to remarkable multi-task consolidation across many AI domains. For
20 example, users can describe a task using natural language prompt to GPT-3 (Brown et al., 2020), allowing the same model to perform question answering, machine translation, text summarization, etc. Prompt-basee based learning provides accessible and flexible interface to communicate natural language understanding task to a general-purpost model.
21
23
24 We envision that generalist robot agent should have similarly intuitive and expressive interface
25 for task specification. What does such an interface for robot learning look like? As a motivating example, consider personal robot tasked with household activities. We can ask the robot to bring us
27 a cup of water by a simple natural language instruction. If we require more specificity, we can instead
28 instruct the robot "bring < mage of the cup>". For tasks requiring new skills, the robot
29 should be able to adapt preferably from a few video demonstrations (Duan et al., 2017). Tasks that need interaction with unfamiliar objects can be easily explained via few image examples for novel concept grounding (Hermann et al., 2017). Finally, to ensure safe deployment, we can further specify visual constraints like "do not enter < <image> room".
30
31
32
33 To enable single agent with all these capabilities, we make three key contributions in this work: 1) novel multimodal prompting formulation that converts wide spectrum robot manipulation 35 tasks into one sequence modeling problem; 2) new robot agent model capable of multi-task
34
36 and zero-shot generalization; and 3) a large-seale benchmark with diverse tasks to systematically evaluate the scalability and generalization of our agents.
38 We start with the observation that many robot manipulation tasks can be formulated by multimodal prompts that interleave language and images or video frames (Fig. 1). For example, Rearrange- 40 ment (Batra et al., 2020), a type of Visual Goal, can be formulated as "Please rearrange objects to match this { (scene_imagee }"; Novel Concept Grounding looks like "This is dax {new_ob ject), } 42 and this is a blicket (new ob object], J-z. Put two metal dax on the marble blicket." " Few-shot Imita- tion embed video snippet in the prompt "Follow this motion trajectory for the wooden cube:

---

Under review as a conference paper at ICLR 2023
Visual Goal: Rearrangement
Rearrange
shot Demonstration
VIMA
Follww this motion
Novel Concept Grounding
Then biickal
This wug
Pur into blicket
Text
Visual Constraint
Pardding
Sweep
inco
Without exceoding
Figure 1: Multimodal prompts for task specification. We observe that many robot manipulation tasks can be expressed as multimodal prompts that interleave language and image/video frames. We propose VIMA, an embodied agent model capable of processing mulitimodal prompts (left) and controlling a robot arm to solve the task (right).
{frame1), (framez), (frames), (framee)"; and expressing Visual Constraint is as simple as adding the clause "without touching (safety boundary)".
45
Multimodal prompts not only have more expressive power than individual modalities, but also enable a uniform sequence IO interface for training generalist robot agents. Previously, different robot manipulation tasks require distinct policy architectures, objective functions, data pipelines, and training procedures (Aceituno et al., 2021; Stengel-Eskin et al., 2022; Lynch & Sermanet, 2021), leading siloed robot systems that cannot be easily combined for rich set of use cases. Instead, our multimodal prompt interface allows us to harness the latest advances in large transformer models (Lin al., 2021; Tay et al., 2020; Khan et al., 2021) for developing scalable multi-task robot learners.
47
49
50
51
52
53
To this end, we design a novel VituoMotor Attention model (VIMA). The architecture follows the encoder-decoder transformer design proven be effective and scalable in NLP (Raffel et al., 2020). VIMA encodes an input sequence of interleaving textual and visual prompt tokens with a pre-trained language model (Tsimpoukelli for each environment interaction step. The transformer decoder is conditioned cross-attention layers that alternate with the usual causal self-attention. Instead operating on raw pixels, VIMA adopts an object-centric approach. We parse all images in the prompt observation into objects by off-the- the--ff detectors (He et al., 2017), and flatten them into sequences of object tokens. All these design choices combined deliver conceptually simple architecture with strong model and data scaling properties.
54
55
56
al., 2021). ) and decodes robot control actions autoregressively
the prompt via
58
60
62
To systematically evaluate proposed algorithm, introduce new benchmark (VIMA- (BEAHB built on the Ravens simulator (Zeng et al., 2020; Shridhar et al., 2021). We provide 17 representative meta-tasks with multimodal prompt templates, which can be procedurally instantiated into thou- sands of individual tasks by various combinations of textures and tabletop objects. VIMA-BENCH establishes 4-level protocol evaluate progressively stronger generalization capabilities, from ran- domized object placement to novel tasks altogether (Fig. 2). To demonstrate the scalability of VIMA, we train spectrum of 7 models ranging from 2M to 200M parameters. Our approach outperforms strong prior SOTA methods such Gato (Reed et al., 2022), Decision Transformer (Chen et al., 2021), and Flamingo (Alayrac et al., 2022) across all levels zero-shot generalization and all model capacities, sometimes by large margin (up to 2.9x task success rate given the same amount of training data, and 2.7x better even with 10x less data). We plan to open- open-source the simulation envi- ronment, training dataset, algorithm code, and pre-trained model checkpoints to ensure reproducibility
64
85
66
87
88
69
70
71
73
75 and facilitate future works from the community. We attach supplementary materials as Appendix to
this PDF, and present video demos and anonymized code at https: htttpa://cee http///ll/000 1ae.0.....ititt github.. io/.
2

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
78
Multi-task Learning by Sequence Modeling. Transformers have enabled task unification across many Al domains (Raffel et al., 2020; Brown al., 2020; Chen et al., 2022a;b; Lu al., 2022; Wang et al., 2022e; Alayrac et al., 2022). For example, in NLP, T5 (Raffel et al., 2020) unifies all language problems into the same text-to-text format. GPT-3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022), and Megatron (Shoeybi et al., 2019) demonstrate emergent behaviours of intuitive task speci- fications by zero-shot prompting. In computer vision, Florence (Yuan et al., 202 I), BiT (Kolesnikov et al., 2020), and MuST (Ghiasi et al., 2021) pre-train a shared backbone model at scale for general visual representations and transfer it to downstream tasks. Pix2Seq (Chen al., 2022b) casts many vision problems into unified sequence format. In multimodal learning, Flamingo (Alayrac et al.,
79
80
81
82
83
84
85
86
87
2022) and Frozen (Tsimpoukelli et al., 2021) design a universal API that ingests an interleaving sequence of images and text and generates free-form text. Gato (Reed et al., 2022) is massively multi- task model across NLP, vision, and embodied agents. Our work is most similar in spirit to Gato, but we focus primarily enabling an intuitive, multimodal prompting interface for a generalist robot agent.
88
89
90
91
Foundation Models for Embodied Agents. Foundation models (Bommasani et al., 2021; Brown et al., 2020; Raffel et al., 2020; Ramesh et al., 2022; Wei et al., 2022) have demonstrated strong emer- gent properties like zero-shot prompting and complex reasoning. There are many ongoing efforts to replicate this success for embodied agents, focusing on 3 aspects: 1) Transformer agent architecture: Decision Transformer (Chen et al.. 2021; Janner et al., 2021; Zheng et al., 2022; Xu et al., 2022) and
92
99
94
85
96
Gato (Reed et al., 2022) leverage the powerful self-attention models for sequential decision making. CLiPort (Shridhar et al., 2021) and Perceiver-Actor (Shridhar et al., 2022) apply large transformers to robot manipulation tasks. Methods such as Dasari & Gupta (2020) and MOSAIC (Zhao al., 2022)
97
98
99
also leverage transformers to achieve superior performance in one-shot video imitation tasks. 2) Pre- training for better representations: Masked ViT (Gupta et al., 2022b), R3M (Nair et al., 2022), and Parisi al. (2022) pre-train general visual representations for robotic perception. Li et al. (2022); Reid et al. (2022) finetune from LLM checkpoints to accelerate policy learning. MineDojo (Fan et al., 2022) and Ego4D (Grauman et al., 2021) provide large-scale multimodal databases to facilitate scalable pol- icy training. 3) Large language models for robot learning: SayCan (Ahn et al., 2022) leverages the 500B PaLM (Chowdhery et al., 2022) for zero-shot concept grounding. Socratic Models (Zeng et al., 2022) composes multiple vision and language foundation models (VLMs) for multimodal reasoning in videos. Huang et al. (2022a), Inner Monologue (Huang et al., 2022b) and L.-Nav (Shah et al., 2022) successfully apply LLMS to long-horizon robot planning. VIMA differs from these works in our novel multimodal prompting formulation, which existing LLMs and VLMs do not easily support.
100
101
102
103
105
107
108
109
110
There are wide range of robot manipulation tasks that require different skills and task specification formats, such as instruction following (Stepputtis et al., 2020; Shridhar et al., 2021; Lynch & Sermanet, 2021), one-shot imitation (Finn et al., 2017: Dasari & Gupta, 2020; Duan et al., 2017), rearrangement (Batra et al., 2020; Weihs et al., 2021; Szot et al., 2021), constraint satisfaction (Brunke et al., 2021a; Srinivasan et al., 2020; Thananjeyan et al., 2021), and reasoning (Shridhar et al., 2020; Gupta et al., 2019; Ahmed et al., 2021; Toyer et al., 2020; Lim et al., 2021). 1). Multiple physics simulation benchmarks are introduced to study the above tasks. For example, iGibson (Shen et al., 2020; Li et al., 2021: Srivastava et al., 2021) simulates interactive household scenarios. Ravens (Zeng et al., 2020) and Robosuite (Zhu et al., 2020; Fan et al., 2021) design various tabletop manipulation tasks with realistic robot arms. MOSAIC (Zhao et al., 2022) features challenging benchmark built on top of Zhu et al. (2020) for one-shot imitation learning, Our VIMA-BENCE is the first robot learning benchmark to support multimodal pullimoddll-roompeed tasks. We also standardize the evaluation protocol to systematically measure an agent's generalization capabilities.
Robot Manipulation and Benchmarks.
111
112
113
114
116
118
121
122
123
A more extended literature review can be found in Appendix, Sec. F.
MULTIMODAL PROMPTS FOR TASK SPECIFICATION
124
125
central and open problem in robot learning is task specification (Agrawal, 2022). In prior liter- ature (Steppultis et al., 2020; Dasari & Gupta, 2020; Brunke et al., 2021b), different tasks often re- quire diverse and incompatible interfaces, resulting in siloed robot systems that do not generalize well across tasks, Our key insight is that various task specification paradigms (such as goal conditioning,
126
127
128
3

---

Under review as a conference paper at ICLR 2023
Stronger Generitzation
Training
Level
Level 2 Novil Cunbriation
Level 3 Novel Objject
Level 4 Novel
Object Placement
Put all ob with the tamte
Put
into the
itto
PaLU
inu the
in the
taoture
Intolt
Figure 2: Evaluation Protocol in VIMA-BENCH. We design 4 levels of evaluation settings to measure the zero-shot generalization capability of an agent systematically. Each level deviates more from the training distribution, and thus is strictly more challenging than the previous level.
129
video demonstration, natural language instruction) can all be instantiated as multimodal prompts (Fig. 1). Concretely, multimodal prompt P of length I is defined as an ordered sequence of arbitrarily interleaved texts and images P := [ro I1 #1,...;; 2i]], where each element x; E [text, image].
130
131
132
Task Suite. The flexibility afforded by multimodal prompts allows us to specify and build models for huge variety of task specification formats. Here we consider the following six task categories. Simple object manipulation: simple tasks like "put <ob ject> into <conta "oontainer>" where each image in the prompt corresponds to single object; 2. Visual goal reaching: manipulating objects to reach a goal configuration, e,g., Rearrangement (Batra et al., 2020); 3. Novel concept grounding: the prompt contains unfamiliar words like "dax" and "blicket", which are explained by in- prompt images and then immediately used in an instruction. This tests the agent's 's ability to rapidly intermalize new concepts; 4. One-shot video imitation: watching video demonstration and learning to reproduce the same motion trajectory for a particular object; 5. Visual constraint satisfaction: the robot must manipulate the objects carefully and avoid violating the (safety) constraints; 6. Visual reasoning: tasks that require reasoning skills, such as appearance matching "move all objects with same textures as <ob ject> into a container", and visual memory, "put <object> in container and then restore to their original position".
133
134
136
138
140
142
143
144
145
Note that these six categories are not mutually exclusive. For example, task may introduce a previously unseen verb (Novel Concepr) by showing a video demonstration, or combine goal reaching with visual reasoning. More details about the task suite are discussed in Appendix, Sec.
148
147
148
4 VIMA-BENCH: BENCHMARK FOR MULTIMODAL ROBOT LEARNING
149
Simulation Environment. Existing benchmarks are generally geared towards a particular task specification. To our knowledge, there is no benchmark that provides rich suite of multimodal tasks and a comprehensive testbed for targeted probing of agent capabilities. To this end, we introduce benchmark suite for multimodal robot learning that we call VIMA-BES -ENCH.. We built our benchmark by extending the Ravens robot simulator (Zeng et al., 2020). VIMA-BENCH supports extensible collections of objects and textures to compose multimodal prompts and procedurally
151
152
153
154
155
generate a large number of tasks. Specifically, we provide 17 meta-tasks with multimodal prompt templates, which can be instantiated into 1000s of individual tasks. Each meta-task belongs one or more of 6 task categories mentioned above. VIMA-BENCE can generate large quantities of imitation learning data via scripted oracle agents. More details are elaborated Appendix, Sec. A.
156
157
158
159
Observation and Actions. The observation space of our simulator includes RGB images rendered from both frontal view and top-down view. Groundtruth object segmentations and bounding boxes are also provided for training object-centric models (Sec. 5). We inherit the high-level action space from Zeng et al. (2020), which consists of primitive motor skills like "pick and place" and "wipe". These parameterized by poses of the end effector. Our simulator also features scripted oracle programs that can generate expert demonstrations by using privileged simulator state information, such as the precise location of all objects, and the groundtruth interpretation of the multimodal instruction.
160
162
164
165
186
Training Dataset. We leverage the pre- progrrmmme oracles to generate a large offline dataset of expert trajectories for imitation learning. Our dataset includes 50K trajectories per meta-task, and 650K successful trajectories in total. We hold out subset of object models and textures for evaluation, and designate 4 out of 17 meta-tasks testbed for zero-shot generalization.
188
169
4

---

Under review as a conference paper at ICLR 2023
Text Token
[1]
Object
Action token
SelfAAttetion
Cross-Attentin
Self-Attenton
Attantion
Prompt Tokana
History Tokena
Object Encoder
Object Encodier
Object Encodar
Sweap
eto
toucting
Muttimodal Prompt
Interaction
We encode the multimodal prompts with a pre-trained T5 model, and condition the robot controller on the prompt through cross-attention layers. The controller a causal transformer decoder consisting of alternating self and cross attention layers that predicts motor commands conditioned on prompts and interaction history.
Figure 3: VIMA.
Evaluating Zero-Shot Generalization. Each task in VIMA-BENCE has a binary success criterion and does not provide partial reward signals. During test time, execute the agent policies in the physics simulator for multiple episodes to compute a success rate in percentage. The average success rate over all evaluated meta-tasks will be the final reported metric.
170
172
173
174 We design a 4-level evaluation protocol (Fig. 2) to systematically probe the generalization capabilities of learned agents. Each level deviates more from the training distribution, and is thus strictly harder than the previous one - Level 1) placement generalization: all prompts are seen verbatim during training, but only the placement of objects on the tabletop randomized testing; Level 2) combinatorial generalization: all materials (adjectives) and 3D objects (nouns) are seen during training, but new combinations of them appear in testing; Level 3) novel object generalization: test prompts and the simulated workspace include novel adjectives and objects; Level novel task generalization: new meta-tasks with novel prompt templates at test time.
175
178
177
179
181
5 VIMA: VISUOMOTOR ATTENTION MODEL
183
Our goal is to build a robot agent capable of performing any task specified by multimodal prompts. learn an effective multi-task robot policy, we propose VIMA, a minimalistic multi- task encoder-decoder architecture with object-centric design (Fig. 3). Concretely, we learn a robot policy T(a,} |P PH), where : [01, d1. 02, 02 ... 0z] denotes the past interaction history, and of E O,a, az E A are observations and actions at each interaction steps. We encode multimodal prompts via a frozen pre-traines langauge model and decode robot motor commands conditioned on the en- coded prompts via cross-attention layers. Unlike prior works (Florence al., 2019; Sieb et al., 2019).
184
186
188
VIMA adopts object-centric token representation that computes features from bounding box coor- dinates and cropped RGB patches.
192
Tokenization. There are 3 formats raw input in the prompt - text, image of a single object, and image of full tabletop scene (e.g., . for Rearrangement inputs, we use pre-trained T5 tokenizer and word embedding to obtain word tokens. For images first extract individual objects using off-the off-the-shelf Mask R-CNN (He et al.. 2017). Each object is represented as a bounding box and cropped image. We then compute object tokens by encoding them with bounding box encoder and VIT, respectively. Since Mask-RCNI is imperfect,
imitation from video frames). For text
194
of full scenes,
196

---

Under review as a conference paper at ICLR 2023
Model Scalabiiiyy
Success Rate (5)
2D
60
10
20
20
20
100
Modet Stzo 140
Mode Siss (M]
Mode: Siee (V1
Modeel 5toe 190
Data Scalability
L3
50
80
80
Sucpess Rate (16)
40
627 10" 10" 102
10'
6 TD*
10¹
10
10>
0
10"
10
Ours Gato
Flamingo
Multimodal GPT Agent
Figure 4: Scaling model and data. Top: We compare performance different methods with model sizes ranging from 2M 200M parameters. Across all model sizes and generalization levels VIMA outperforms prior works. Bottom: For a fixed model size of 92M parameters we compare the effect of imitation learning dataset size of 0.1%, 1%. 10%, and full imitation data. VIMA is extremely sample efficient and can achieve performance comparable to other methods with 10x less data.
198
the bounding boxes can be noisy and the cropped image may have irrelevant pixels. For images of single objects, we obtain tokens in the same way except with dummy bounding box. Prompt tok- enization produces a sequence of interleaved textual and visual tokens. We then follow the practice in
199
200
201
simpoukelli et al. (2021) and encode the prompt via a pre-trained T5 encoder (Raffel et al., 2020).
202
Since T5 has been pre-trained on large text corpora, VIMA inherits the semantic understanding capability and robustness properties. To accommodate tokens from new modalities, we insert MLPs
203
204
between the non-textual tokens and T5. To prevent catastrophic forgetting, VIMA finetunes the last
205
two layers of the language encoder with layer-wise learning rate decay (He et al., 2021) but freezes
206
all other layers. Our positional embedding is learnable and absolute.
Robot Controller. A challenging aspect of designing multi-task policy is to select suitable 208
conditioning mechanism. In our schema (Fig. 3), the robot controller (decoder) is conditioned on the prompt sequence P by series of cross-attention layers between P and the trajectory history sequence H. We compute key Kp and value Vp sequences from the prompt and query QH from the trajectory history, following the encoder-decoder convention in T5 (Raffel et al., 2020), Each
210
211
aross-attnntion layer then generates an output sequence H' = softmax (uuk) vā
212
Vp, where d the embedding dimension. Residual connections (He et al., 2015) are added to connect higher layers
213
214
with the input rollout trajectory sequence, The cross-attention design enjoys three advantages: 1) strengthened connection to prompt; 2) intact and deep flow of the original prompt tokens; and 3) better computational efficiency, as demonstrated in VideoGPT (Yan et al., 2021) as well. VIMA decoder consists of I. alternating cross-attention and self-attention layers. Finally, we follow common practice (Baker et al., 2022) to map predicted action tokens to discretized coordinates the robot arm. See Appendix, Sec. C.2 for more details.
216
218
220
Training.
We follow behavioral cloning to train our models by minimizing the negative likeliood of predicted actions. Concretely, for trajectory with T steps, we minimize ming Σt= log nh [a+ H). The entire training is conducted on an offline dataset with no simula- tor access. To make VIMA robust to detection inaccuracies and failures, we apply object augmentation
221
222
223
224
by randomly injecting false-pessee possitive detection outputs. After training, we select model checkpoints for evaluation based on the aggregated accuracy a held-out validation set. The evaluation involves interacting with the physics simulator. We follow the best practices to train Transformer models
226
using the AdamW optimizer (Loshchilov & Hutter, 2019), learning rate warm-up, cosine annealing (Loshchilov & Hutter, 2016), etc. See Appendix Sec. D for comprehensive training hyperparameters.
228

---

Under review as a conference paper at ICLR 2023
6
EXPERIMENTS
231
In this section, we aim to answer three main questions: (1) How does VIMA compare with prior SOTA transformer- based agents diverse collection of cultimodal-promppten tasks? (2) What the scaling properties of our approach in model capacity and data size? (3) How do different visual tokenizers, prompt conditioning, and prompt encoding affect decision making?
233
234
6.1
BASELINES
236
Gato (Reed et al., 2022) introduces a decoder-only model that solves tasks from multiple domains where tasks are specified by prompting the model with the observation and action subsequence. For fair comparison, we provide the same conditioning as VIMA, ie., our multimodal embedded prompt. Input images are divided into patches and encoded by a ViT (Dosovitskiy et al., 2020) model to produce observation tokens.
237
239
241
Flamingo (Alayrac et al., 2022) vision-language model that learns to generate textual completion in response to multimodal prompts. It embeds variable number of prompt images into a fixed number of tokens via a Perceiver Resampler (Jaegle et al., 2021b), and conditions the language decoder on the encoded prompt by cross-attention. Flamingo does not work with embodied agents out of the box. We adapt to support gecision-masking by replacing the output layer with robot action heads.
243
245
247
Multimodal GPT agent a GPT-based behavior cloning agent conditioned on tokenized multimodal
248
prompts. autoregressively decodes next actions given instructions and interaction histories. Similar
249
to prior works casting RL problems as sequence modeling (Chen et al., 2021; Janner et al., 202 I), encodes an image into a single state token by a ViT encoder, and prepends the rollout trajectory with prompt tokens. This baseline does not involve cross-attention.
250
251
252
more detailed comparison between these methods can be found in Appendix, Sec. C.I.
253
6.2
EVALUATION RESULTS
254
We compare VIMA against other SOTA methods on the four levels of generalization provided in our benchmark for different model and training dataset sizes.
255
256
Model scaling. We train all methods for spectrum of model capacities from 2M to 200M parameters, evenly spaced on the log scale. The encoder size kept constant (pre-trained T5-Base) for all methods and excluded from the parameter count. Across all levels of zero-shot generalization, we find that VIMA strongly outperforms prior work. Although models like Gato and Flamingo show improved performance with bigger model sizes, VIMA consistently achieves superior performance over all model sizes. We note that this can only be achieved with both cross-attention and object token sequence representation without any downsampling performance significantly, especially in the low model capacity regime (ablations in Sec. 6.3).
257
259
261
altering any component will degrade the
263
Data scaling. Next we investigate how different methods scale with varying dataset sizes. We compare model performance at 0. 0.1%, 1%, 10% and full imitation learning dataset provided in VIMA-BENCH (Fig. 4). VIMA is extremely sample efficient and with just 1% of the data can achieve performance similar to baseline methods trained with 10x more data for LI and L2 levels of generalization. In fact, for L4 we find that with just 1% of training data, VIMA already outperforms prior work trained with entire dataset. Finally, across all levels with just 10% of the data, VIMA can outperform prior work trained with the full dataset by a significant margin. We hypothesize that the data efficiency can be attributed to VIMA's 's object-centric representation, which is less prone to overfitting than learning directly from pixels in the low-data regime. This is consistent with findings from Sax et al. (2018), which demonstrates that embodied agents conditioned on mid-level visual representations tend to be significantly more sample-efficient than end-to-end control from raw pixels.
265
266
287
268
269
270
272
274
Progressive Generalization. Finally, compare the relative performance degradation as we test the models on progressively challenging zero-shot evaluation levels without further finetuning (Fig. 5). 276
Our method exhibits minimal performance regression, especially between L1 + L2 and L1 + L3. In contrast, other methods can degrade as much as 20%, particularly in more difficult generalization
278
scenarios. Although all methods degrade significantly when evaluated on L4 (Novel Tasks), the drop
7

---

Under review as a conference paper at ICLR 2023
Dur (Drade)
Diject Porsvver
70
Irage Hacces
Irage Parcover Phannine
Success Rate (K0
60
Single Image
30
Ours
Musk
VIT
Patches
Image
Ours
(Oracee)
Object
Single
Perceiver
Image
Figure 6: Ablation on visual tokenizers. We compare the performance of VIMA-200M model across different visual tokenizers. Our proposed object tokens outperform all methods that learn directly from raw pixels, and Object Perceiver that downsamples the object sequence to a fixed number of tokens.
in performance for VIMA is only half has developed more generalizable policy and robust representations than the competing approaches.
all other baselines. This results suggest that VIMA
281
6.3
ABLATION STUDIES
Through extensive experiments, ablate different design choices in VIMA and study their impact on robot decision making. We focus the following 4 aspects: visual tok- enization, prompt encoding, prompt conditioning variants, and robustness against distractors and imperfect prompts.
284
-10
285
Performance Drop (%)
287
-40
288
Visu Visual tokenization. As explained in Sec. 5, VIMA pro- cesses the prompt and observation images into a variable number of object tokens with an off-the-shelf Mask R-
-50
289
Ours
Gato
-70
Ramingo
291
CNN implementation. How important this particular choice of visual tokenizer? We study 5 different variants and empirically evaluate their levels of generalization performance VIMA-BENCH. (1) Ours (Oracle): in- stead of using Mask R-CNN, we directly read out the groundtruth bounding box from the simulator. In other words, we use perfect object detector to estimate the upper bound on the performance of this study; (2) Object Perceiver: we apply a Perceiver module (Jaegle et al.,
CPT
L2
L1 +
293
Figure 5: VIMA incurs much less per- formance drop than baselines as we eval- uate on progressively harder zero-shot generalization.
295
287
298
289
300
202 Ib;a) convert the variable number of objects detected in each frame a fixed number of tokens.
Perceiver is more computationally efficient because it reduces the average sequence length; (3) Image Perceiver: the same architecture the Perceiver Resampler Flamingo, which converts an image to small, fixed number of tokens; (4) Image patches: following Gato, we divide an RGB frame into square patches, and extract ViT embedding tokens. The number of patches is more than the output of Decision Transformer's tokenizer, which encodes one image into
302
304
Image Perceiver; (5) Single image: a single token.
306
Fig. 6 shows the ablation results. We highlight a few findings. First, note that our Mask R-CNN
308
detection pipeline (Appendix, Sec. A.20) incurs a minimal performance loss compared to the oracle bounding boxes, thanks the object augmentation (Sec. 5) that boosts robustness during training. Second, tokenizing from raw pixels (Image Perceiver, patches, or single embedding) consistently underperforms object-centric format. We hypothesize that these tokenizers have allocate extra
310
8

---

Under review as a conference paper at ICLR 2023
L2
13
L4
20
80
Success Rate (9)
00
40
40
20
0
123
250
32 64
10
52 Sten (M)
125
266
250
Modil Sice 0vl
Model Size (M)
14ceet Sioe (M]
xattn
gpt-decoder
Figure 7: Ablation: Prompt conditioning. We compare our method (xaiii: cross-attention prompt conditioning) with a vanilla transformer decoder (gpt-decoder) across different model sizes. Cross- attention is especially helpful in low-parameter regime and for harder generalization tasks.
312
internal capacity to parse the objects from low-level pixels, which likely impedes learning. Sax al. (2018) echoes finding that using mid-level vision greatly improve agent generalization compared to an end-to-end pipeline. Third, even though Ours and Object Perceiver both use the same object bounding box inputs, the latter is significantly worse in decision making. We conclude that it is important to pass the variable sequence of objects directly to the robot controller rather than downsampling a fixed number of tokens.
314
315
316
317
318
Prompt Conditioning, VIMA conditions the robot controller (decoder) on the encoded prompt by cross -ross-attention. A simple alternative is to concatenate the prompt P and interaction history H. into one big sequence, and then apply decoder-only transformer like GPT (Radford et al., 2018) to predict actions. In this ablation, we keep the object tokenizer constant, and only switch the conditioning mechanism causal sequence modeling. Note that this variant is conceptually "Gato with object tokens" Fig. 7 shows the comparison of VIMA (xattn) and the gpt-decoder variant across 4 generalization levels. While the variant achieves comparable performance larger models, cross- attention still dominates in the small-capacity range and generalizes better in the most challenging L.4 (Novel Task) setting. Our hypothesis is that cross-attention helps the controller stay better focused on the prompt instruction at each interaction step. This bears resemblance to the empirical results in Sanh et al. (2021); 1); Wang et al. (2022b), which show that well-tuned mncoer--decoee architectures outperform GPT-3 in zero-shot generalization.
319
320
321
324
325
326
328
330
Prompt Encoding, We vary the size of the pre-trained T5 encoder to study the effect of prompt encoding. We experiment with three T5 capacities: small (30M), base (111M), la large (368M). For all T5 variants, we fine-tune the last two layers and freeze all other layers. We find no significant difference among the variants (Appendix, Sec. E.2), thus we set base as default for all our models.
332
333
334
Policy Robustness. We study the policy robustness against increased amounts of distractors and imperfect task specifications, See Appendix. Sec. E.3 for exact setup and results. VIMA exhibits minimal performance degradation with increased distractors and corrupted prompts. We attribute this robustness to the high-quality, pre-trained T5 language backbones.
335
336
337
CONCLUSION
338
339
Similar to GPT-3, a generalist robot agent should have an intuitive and expressive interface for human users to convey their intent. In this work, we introduce novel multimodal prompting formulation that converts diverse robot manipulation tasks into a uniform sequence modeling problem. We propose
341
VIMA, conceptually simple transformer- maasformer-baeedd agent capable of solving tasks like visual goal, ohet video imitation, and novel concept grounding with a single model. VIMA exhibits superior model and data scaling properties, and provides strong starting point for future work.
343
344
345
The current VIMA experiments are not without limitations. We identify the following weaknesses:
346
(1) limited action primitives (only pick-and -nkk-aed--eeee and wipe for now); (2) limited simulator realism;
347
33 reliance on finetuned Mask R-CNN to provide object tokens. However, VIMA's algorithm
348
design general-pe -prrpose and does not make assumptions about the particular observation and action formats. This opens the door to future works that may address many of these weaknesses with
349
350
more sophisticated environments (e.g. BEHAVIOR (Srivastava et al., 2021)), stronger vision pipeline (large-scale open-vocabular models like ViLD (Gu et al., 2021)), and temporally-extender robot controllers (such as MAPLE (Nasiriany al., 2021)). With these stronger modules, VIMA could potentially scale to challenging problems. We open-source all code to facilitate future research.
351
352
353

---

Under review as a conference paper at ICLR 2023
8
REPRODUCIBILITY STATEMENT
355
We provide comprehensive details to reproduce our work in the Appendix. Concretely, the specifi- cations of each meta-task in the benchmarking suite are explained in Sec. B. Model architectures are elaborated in Sec. C. Hyperparameter configurations are listed in Sec. D. Furthermore, we host anonymized code at httpp:///cclr300 htt.:/0011....tttt github. 1o/ for review.
356
358
360
REFERENCES
Josh Abramson, Arun Ahuja, lain Barr, Arthur Brussee, Federico Camevale, Mary Cassin, Rachita Chhaparia, Stephen Clark, Bogdan Damoc, Andrew Dudzik, Petko Georgiev, Aurelia Guy, Tim Harley. Felix Hill, Alden Hung, Zachary Kenton, Jessica Landon, Timothy Lillicrap. Kory Mathew- son, Soña Mokrá, Alistair Muldal, Adam Santoro, Nikolay Savinov, Vikrant Varma, Greg Wayne, Duncan Williams, Nathaniel Wong, Chen Yan, and Rui Zhu. Imitating interactive intelligence. arXiv preprint arXiv: Arxiv-2012.05672, 2020.
362
364
365
388
367
Bernardo Accituno, Alberto Rodriguez, Shubham Tulsiani, Abhinav Gupta, and Mustafa Mukadam. A differentiable recipe for learning visual non-prehensile planar manipulation. In 5th Annual Conference on Robot Learning, 2021. URL https:/ //wpe//eeeeeee.eeoo/ee/e net/ forum?? ?id=
369
370 E EoKAEE.
Pulkit Agrawal. The task specification problem. In Aleksandra Faust, David Hsu, and Gerhard Neumann (eds.), Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp. 1745-1751. PMLR, 08-11 Nov 2022. URL https:// proceedingg ml mts../.....s...s////ii.l /1/11144//g/aae 24/44grraaa22/
372
374
html.
375 Ossama Ahmed, Frederik Träuble, Anirudh Goyal, Alexander Neitz, Manuel Wuthrich, Yoshua Bengio, Bernhard Schölkopf, and Stefan Bauer. Causalworld: A robotic manipulation bench- mark for causal structure and transfer learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openrest
377
379
n forum? www.uoiioooteiiio. 1idiiddksskssgggo
380
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho. Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov. Yuheng Kuang. Kuang- Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan Yan. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv: Arxiv-2204.0169/ 2022.
382
384
386
387
388
389
Jean- Bant-Bttstte Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. arXiv preprint arXiv: Arxiv-2204. 220.222444999... 2022.
380
381
392
393
394
Bowen Baker, llge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. arXiv preprint arXiv: Arxiv-2206. 1 1755, 2022.
396
387
Dhruv Batra, Angel X. Chang, Sonia Chernova, Andrew J. Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, Manolis Savva, and Hao Su. Rearrangement: A challenge for embodied ai. arXiv preprint arXiv: Prii-2001.00197 2020.
399
400
10

---

Under review as a conference paper at ICLR 2023
Homanga Bharadhwaj. Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety critics for exploration. In 9th International Conference on Learning Representations. ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL httttps://openreeee http//vepeeeeeeeeeeeee ew net/ fo Forum? d 6DUuKi.
402
404
405
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson. Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie. Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia. Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportumities and risks of foundation models. arXiv preprint arXiv: 2ixiv-2108.07258 2021.
406
407
408
410
412
413
414
415
418
417
418
419
421
423
425
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Vosse Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz I. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin (eds.), Ad- vances in Neural Information Processing Systems 33: Annual Conference Neural Informa- tion Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, volume 33, pp. 1877-1901, 2020. URL https: / /ttps:://rrooe http///pooeeeennnnneeeee neurips wwtpppppr//2222. / 1457c0d6bfcb4
428
427
428
429
431
432
433
434
435
8bfbsacfbllboceec 64 4 html.
Abstractt
436
Lukas Brunke, Melissa Greeff, Adam W. Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P. Schoellig. Safe learning in robotics: From learning-basee control to safe reinforcement learning. arXiv preprini arXiv: Arxiv-2 100.0000800666 2021a.
437
Lukas Brunke, Melissa Greeff, Adam W. Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P. Schoellig. Safe Learning in Robotics: From Learning Based Control to Safe Re- inforcement Learning, December 2021b. URL http://arxiv .rgg /as///////// 22108.211000 66266. WrXiv:2108.06266 [cs, eess).
440
442
Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the "video" in video-language understanding. CVPR, 2022.
444
445
Arthur Bucker, Luis Figueredo, Sami Haddadin, Ashish Kapoor, Shuang Ma, Sai Vemprala, and Rogerio Bonatti. Latte: Language trajectory transformer. arXiv preprint arXiv: Arxiv-2208.02913 2022.
446
447
448
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Anmual Conference Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 15084-15097, 2021. URL https:tpp... http://ccccccccccccece neurips...... http://...neeee..eeeeeeeeeeeeeee cc/paper/ 2021 /ht.hhaahhaaa / 4 989 64 2a 0ddb1 027 0 Abst html.
449
451
453
11

---

Under review as a conference paper at ICLR 2023
Ting Chen, Saurabh Saxena, Lala Li, David J. Fleet, and Geoffrey E. Hinton. Pix2seq: A language modeling framework for object detection. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022a. URL https:: /ttp///ppeeeeeeeeeeeeeeeeeee i ew net/ / forum? i d= Iw
456
458
459
Ting Chen, Saurabh Saxena, Lala Li, Tsung- Yi Lin, David J. Fleet, and Geoffrey Hinton. A unified sequence interface for vision tasks. arXiv preprint arXiv: Arxiv-2206.0766 2022b.
460
461
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann. Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellsteer Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. arXiv preprint arXiv: Arxiv-2204.022 02211. 2022.
462
463
464
465
468
487
468
489
470
472
474
Jack Collins, Shelvin Chand, Anthony Vanderkop, and David Howard. A review of physics simulators for robotic applications. IEEE Access, 9:51416-51431, 2021.
475
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http:/ http:///pyblltt..o/e/ee org, 2016-2021.
477
478
Sudeep Dasari and Abhinav Gupta. Transformers for one-shot visual imitation. In Jens Kober, Fabio Ramos, and Claire J. Tomlin (eds.), 4th Conference on Robot Learning, CoRL 2020, 16-18 Novem- ber 2020, Virtual Event / Cambridge, MA, USA, volume 155 of Proceedings of Machine Learn- ing Research, pp. 2071-2084. PMLR, 2020. URL https: httpp:::ppocceed...ee.. /ttp///peedeng...c/ceee mlr, presss 2055/dasari211 html.
479
480
481
482
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large- embodied ai using procedural generation. arXiv preprint arXiv: 2ehv-22006.09994 2022.
494
486
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. XrXiv preprint arXiv: Arxiv-2010.1 1929, 2020.
487
488
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. arXiv preprini arXiv:2204. 1918, 2022.
491
482
493
Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied AI: from simulators to research tasks. IEEE Trans. Emerg. Top. Comput. Intell., 2(2):230-244, 2022. doi: 10. 1001199/ETTC1.222231111110
495
URL https:////// / http//iioooggg//////// wtt..//g//....././/.....
2022. 3141105.
487
Yan Duan, Marcin Andrychowicz, Bradly C. Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1087-1098, 2017. URL https: http::s:://///oceeeeeeeoeoeoeeee neuri ps wwcc/paper// 2201 /hash/ 3be Abst..ratbt.ttt........... html.
498
489
500
501
502
503
2..866600:3540667cccc
504
Kiana Ehsani, Winson Han, Alvaro Herrasti, Eli VanderBilt, Luca Weihs, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Manipulathor: A framework for visual object manipulation. In
505
12

---

Under review as a conference paper at ICLR 2023
Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR),
507
Pp. 1997-4506, June 2021.
508
Linxi Fan, Yuke Zhu, Jiren Zhu, Zihua Liu, Orien Zeng. Anchit Gupta, Joan Creus-Costa, Silvio Savarese, and Li Fei-Fei. Surreal: Open-source reinforcement learning framework and robot manipulation benchmark. In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto (eds.), Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pp. 767-782. PMLR, 29-31 Oct 2018. URL https: stttps://proceedingg
509
510
511
512
513
m1 alr.ppe w.r..ppes///e... / v87 fan1 1 8a html.
514
Linxi Fan, Yuke Zhu, Jiren Zhu, Zihua Liu, Orien Zeng, Anchit Gupta, Joan Creus-Costa, Silvio Savarese, and Li Fei-Fei. Surreal-system: Fully-integrated stack for distributed deep reinforcement learning. arXiv preprint arXiv: 1rxiv-1909.1298% 2019.
516
517
Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Animashree Anandkumar. SECANT: self-expert cloning for zero-shot generalization of visual policies. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 3088-3099 PMLR, 2021. URL http:/ pttp//eecccee.....///e
518
519
520
521
522
mlr.preesee www.presssvlll vv39/ fan21c html.
525
Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with snternet-scale knowledge. arXiv preprint arXiv: (rxxiv-2206.08855 2022.
524
525
626
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. arXiv preprint arXiv: Arxiv-1709.0490n, 2017.
527
528
Peter Florence, Lucas Manuelli, and Russ Tedrake. Self-supervised correspondence in visuomotor policy learning. arXiv preprint arXiv: Arxiv-190900690 2019.
529
530
Tuu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang. Lijuan Wang, and Zicheng Liu. Violet End-to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv: Arxiv-2111.1268/ 2021.
531
Chuang Gan, Siyuan Zhou, Jeremy Schwartz, Seth Alter, Abhishek Bhandwaldar, Dan Gutfre- und, Daniel L.. K. Yamins, James DiCarlo, Josh McDermott, Antonio Torralba, and Joshua B. Tenenbaum. The threedworld transport challenge: A visually guided task-and-motion planning benchmark for physically realistic embodied ai. arXiv preprint arXiv: Arxiv-2103, 14025, 2021.
534
536
537
Golnaz Ghiasi, Barret Zoph, Ekin D. Cubuk, Quoe V. Lc, and Tsung- Yi Lin. Multi-task self-training for learning general representations. In 202/ IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 8836 8845. IEEE, 2021. doi: 10.1009/CCV449222.001.0 000733. URL https: //d/////o/// /oi. org/ 10. 1109 110./1///911//1/29222 ICC449922. 2021. 00873.
538
539
540
512
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Car- tillier. Sean Crane, Tien Do. Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li. Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu. Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo. Yuchen Wang, Xindi Wu, Takuma Yagi. Ziwei Zhao, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani. Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3,000 hours of egocentric video. arXiv preprint arXiv: Arxiv-21 10.07058, 2021.
543
545
546
547
548
549
550
551
552
654
556
557
13

---

Under review as a conference paper at ICLR 2023
Xiuye Gu, Tsung- Yi Lin. Weicheng Kuo, and Yin Cui. Open- veenboocyyllary object detection via vision and language knowledge distillation. arXiv preprint arXiv: Arxiv-2104.13921, 2021.
559
560
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 November 2019, Proceedings, vol- ume 100 of Proceedings of Machine Learning Research, pp. 1025-1037. PMLR, 2019. URL http:/ httt::/pproceedingge mlr. press / vv10//100///////// http://pppaaaaaaaa html.
582
563
584
585
566
Agrim Gupta, Linxi Fan, Surya Ganguli, and Li Fei-Fei. Metamorph: Learning universal controllers with transformers. In International Conference on Leurning Representutions, 2022a. URL
568
https: /tpps////peeeeiieeee net/ wwt.:eerntotteeeeeeee forum?id=0pmmereeie
569
Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martin-Martin, and Li Fei-Fei. Maskvit: Masked visual pre-training for video prediction. arXiv preprint arXiv: Arxiv-2206.11 1189, 2022b.
571
572
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, Deep Residual Learning for Image Recog- nition, December 2015. URL http://arxiv.. org/ [cs].
573
wttpooogg//abbao.eee www.abs/.... 03385. arXiv:1512.0338,
574
575
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. arXiv preprint arXiv: 1rxiv-1703.06870 2017.
578
577
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv: Arxiv-2111.06377, 2021.
Tracy H Heibeck and Ellen M Markman. Word learning in children: An examination of fast mapping. Child development, pp. 1021-1034, 1987.
580
581
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang. Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czamecki, Max Jaderberg. Denis Teplyashin, Marcus Wainwright, Chris Apps. Demis Hassabis, and Phil Blunsom. Grounded language learning in a simulated 3d world. arXiv preprint arXiv: (ttiv-1706.0655// 2017.
583
584
585
Felix Hill, Olivier Tieleman, Tamara von Glehn, Nathaniel Wong, Hamza Merzic, and Stephen Clark. Grounded language learning fast and slow. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net 2021. URL https: wwwpreveeeeeeeeeeeeeeeee net/ forum Forummidduoo wwwwwsuu hyqa.
586
587
588
589
De-An Huang, Danfei Xu, Yuke Zhu, Animesh Garg, Silvio Savarese, Li Fei-Fei, and Juan Carlos Niebles. Continuous relaxation of symbolic planner for one-shot imitation learning. In 2019 IEEE/RSJ International Conference
590
Intelligent Robots and Systems. IROS 2019, Macau, SAR, China, November 3-8, 2019, pp. 2635-2642. IEEE, 2019. doi: 10. 1991000040888 8967761..
592
URL https: //pp:///oot doi org/ 10. 1109/ IROS40897 2019. 8967761
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models zero- shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Muchine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9118-9144 PMLR, 2022a. URL https: / pwwppeeiiggg...opooe/pppp.p...sp mlr. www.mrressssp.....mm..m..mmmm..e / v1 62 huang22e html.
595
596
587
596
599
600
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv: Arxiv- 0et.-2227.000000 2022b
601
603
604
Andrew Jaegle, Sebastian Borgeaud, Jean- Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula. Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver io: A general architec- ture for structured inputs& & outputs. arXiv preprint arXiv: Arxiy-2107.14790 2021a.
606
607
14

---

Under review as a conference paper at ICLR 2023
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General perception with iterative attention. arXiv preprint arXiv: Arxiv-2103.03200 2021b.
609
611
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew Davison, Rlbench: The robot learning benchmark & learning environment. arXiv preprint arXiv: Arxiv-1909. /2271, 2019.
613
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 1273-1286, 2021. URL saper/2021. hhw.///aah////eeee 099 6b 4 c2 4ee44422233033333000000303
614
615
616
617
neur
neur wwwurrpps.... ps. cc/ Abstract centrract.tmll... html.
619
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. arXiv preprint arXiv: Arxiv-2101.01 169, 2021.
620
Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: Clip embeddings for embodied ai. arXiv preprint arXiv: Arxiv-21 1.09888, 2021.
622
lasonas Kokkinos. Ubernet:
Training 'universal' convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. arXiv preprint arXiv: Arxiv- 1609.02 132, 2016.
624
625
626 A lexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (cds.). Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part V. volume 12350 of Lecture Notes in Computer Science, pp. 491-507. Springer, 2020. doi: 10. 1007/ 978-3-0300 1--030-55558-),-- URL https: /ttp///oiii/// org/ 0ttpg//111100/71--003333333 0-0-77788---0055555--3- 29.
828
630
632
Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby. Uvim: A unified modeling approach for vision with learned guiding codes. arXiv preprint arxiv: 1rxiv-2205.10331 2022.
634
636
Chengshu Li, Fei Xia, Roberto Martín- Martin, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Elliott Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, Andrey Kurenkov, C. Karen Liu, Hyowon Gweon, Jiajun Wu, Li Fei-Fei, and Silvio Savarese. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. In Aleksandra Faust, David Hsu, and Gerhard Neumann (eds.), Conference on Robot Learning, 8-11 November 2021, London, UK, volume 164 of Proceedings of Machine Learning Research, pp. 455-465. PMLR, 2021. URL https: / ptwceedeeedeeeeooooooooooo ngs ml w.r.ppessss...... / vv 64 64/1122b. html.
638
640
642
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Torralba, and Yuke Zhu. Pre-trained language models for interactive decision-making arXiv preprint arXiv: Arxiv Sixi=2200.01171,, 2022.
644
646
647
Michael H. Lim, Andy Zeng, Brian Ichter, Maryam Bandari, Erwin Coumans, Claire Tomlin, Stefan Schaal, and Aleksandra Faust. Multi-task learning with sequence-condttionee transporter networks. arXiv preprint arXiv: 200v=22009.0777, 2021.
648
650
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. arXiv preprint arXiv: Arxiv-2106.005555 2021.
652
Ziyuan Liu, Wei Liu, Yuzhe Qin, Fanbo Xiang, Minghao Gou, Songyan Xin, Maximo A Roa, Berk Calli, Hao Su. Yu Sun. et al. Ocrtoc: A cloud- based competition and benchmark for robotic grasping and manipulation. IEEE Robotics and Automation Letters, 7(1:486-493, 2021.
653
654
655
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: Arxiv-1608 03983, 2016.
656
15

---

Under review as a conference paper at ICLR 2023
Iyy Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations. ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL htttt:://opeeeeiieeee net/ /et.ffoo forum? i @ 9Scxx.
658
680
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL
662
664
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. CVPR 2020, Seattle. WA, USA, June 13-19, 2020, pp. 10434-10443. Computer Vision Foundation / IEEE, 2020. doi: (0.1100/CVPP222200000000000
685
686
667
URL
668
https: /ttp:////ppccccseeeeeeeee tweeef..eeeccc...ee com httt://oootetttteeeeeooeoeeeectt wtt.c///ooootetttnneeee 2020 httptttmmmmamimmmmaaaaaaa htw.rr/ew_.iiooooooooaoaaaaaaaea haw.aaaaa___aaggaaaeeeeeeeeettta Lew.oiigg_narrinngg___e_
and html.
670
671
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified- io: A unified model for vision, language, and multi-modal tasks, arXiv preprint arXiv: Arxiv- 2206.08916 2022.
672
673
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. In Dylan A. Shell, Mare Toussaint, and M. Ani Hsieh (eds.), Robotics: Science and Systems XVII, Virtual Event, July 12-16, 2021, 2021. doi: (155667/RS.2221.12. XVI1.07 URL
675
678
677
https: wtt.:/.....11.1...///111. /0/6607//S./// 2021 XVII. 047.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv: 1xxi--8806008700, 2018
679
681
Nathan Morrical, Jonathan Tremblay, Stan Birchfield, and Ingo Wald. NVISII: Nvidia scene imaging interface, 2020. https: / //t///g/ //ggggthbbbb hhtp//ww-aaa cccm///ww-roojjt//W......
682
683
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv: Arxiv-2203. 2022.
684
685
Soroush Nasiriany, Huihan Liu, and Yuke Zhu. Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks. International Conference On Robotics And Automation (icra), 2021. doi: 10.
686
688
OpenAL, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. arXiv preprint arviv: (xxiv-1912.06680 2019.
689
680
O. Pinto, Jonathan
692
694
Tom Le Paine, Sergio Gómez Colmenarejo, Ziyu Wang, Scott Reed, Yusuf Aytar, Tobias Pfaff, Matt W. Hoffman, Gabriel Barth-Maron, Serkan Cabi, David Budden, and Nando de Freitas. One-shot siddliiilityy imitation: Training large-scale deep nets with rl. arXiv preprint arXiv:
685
696
697
1rx10-7110.0001,, 2018.
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurprising effectiveness of pre-trained vision models for control. In Kamalika Chaudhuri. Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 17359 17371. PMLR, 2022. URL https: wtt.pns......................e.e
699
700
701
702
703
// /pppeediings mlr prwss / /vl//v22////// 20t.6//pariii parpsiaiiiie iil..iiia... html.
Adam Paszke, Sam Gross. Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner,
705
706
16

---

Under review as a conference paper at ICLR 2023
Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bir Buc. E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035 Curran Associates, Inc., 2019.
708
710
711
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang. Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In 2018 IEEE Conference on Computer
712
713
Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT. USA, June 18-22, 2018, PP. 8494-8502. Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10.1109/CVPR, 2018.00886. URL httpt::p///ppecccceseee thecvf..ec.cc com wttp://tttttttteeeeeeeeee wwtpeettccter___20000 201220011tttmt //html/ Puig Puig_Virt:n Home Htw.eeeessiiiiitttttoooooooooooo
714
715
716
2012 paper. paper.html.
717 Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. OpenAl, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763 PMLR, 2021.
720
722
723
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to- text transformer. J. Mach. Learn. Res., 21:140:1-140:61 2020. URL http://jmlr... orgg/ html.
724
725
728
papers/v21/20-
727
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv: 0rziv-2204.06125, 2022.
Harish Ravichandar, Athanasios S Polydoros, Sonia Chernova, and Aude Billard. Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous systems, 3: 297-330, 2020.
730
732
Scott
Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. arXiv preprint arXiv: Arxiv-2205.06175, 2022.
739
734
735
736
737
Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforcement learning? arXiv preprint arXiv: Arxiv-2201.12122, 2022.
738
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang A. Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Seao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Stella Rose Biderman, Leo Gao, T. Bers, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. Iclr, 2021.
740
742
744
745
746
747
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). October 2019.
748
749
750
751
Alexander Sax, Bradley Emi, Amir R. Zamir, Leonidas Guibas, Silvio Savarese, and Jitendra Malik. Mid-level visual representations improve generalization and sample efficiency for learning visuomotor policies. arXiv preprint arXiv: Arxiv-1812. 2018
753
754
Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. arXiv preprint arXiv: Arxiv-2207.0444 2022.
756
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv: Aexiv-2002.05202, 2020.
17

---

Under review as a conference paper at ICLR 2023
Bokui Shen, Fei Xia, Chengshu Li, Roberto Martín- Linxi Fan, Guanzhi Wang, Claudia Pérez-D' Arpino, Shyamal Buch, Sanjana Srivastava, Lyne P. Tchapmi, Micael E. Tchapmi, Kent Vainio, Josiah Wong, Li Fei-Fei, and Silvio Savarese. igibson 1.0: simulation environment for interactive tasks in large realistic scenes. arXiv preprint arXiv: Arxiv-2012.029244 2020.
758
780
761
Tianlin Tim Shi, Andrej Karpathy, Linxi Jim Fan, Jonathan Hernandez, and Percy Liang. World of bits: an open-domain platform for web-based agents. ICML, 2017. URL https: //dl acm. org /tg//oo11.... 5555 3 3308890. 330 6005.
783
764
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro Mmgatron-Im: Training multi-billion parameter language models using model parallelism. XrXiv preprint arXiv: 080500009000,,,.... 2019.
786
767
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In 2020 IEEE/CVF Conference on Computer Vision and
768
769
Puttern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 10737-10746. Computer Vision Foundation / IEEE, 2020. doi: 00110900CVR422000020000000
771
URL
thw.efeeeeee.. com/ wwt./o/t//cccccc/////cc/octcecte wwwttotttttttcccccccceoeooeeeeet _cc____________ Inwttnntiiss__seeee..... httpr//Aeennnnnrncfooooneeeeeeee httpp//eeee....ooooeeeeeoonoonnn
773
774
wweryday_Taskss_cRR__00 2twp:P__222__ppprr..ee html.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic
776
manipulation. arXiv preprint arXiv: Mxxiv-2109.20090 2021.
777
Mohit Shridhar, Lueas Manuelli, and Dieter Fox. Perceiver-actor: multi-task transformer for robotic manipulation. arXiv preprint arXiv: Arxiv-2209.05451, 2022.
779
Maximilian Sieb, Zhou Xian, Audrey Huang, Oliver Kroemer, and Katerina Fragkiadaki. Graph- structured visual imitation. arXiv preprint arXiv: Arxiv-1907.05511 2019.
780
781
Krishnan Srinivasan, Benjamin Eysenbach, Schoon Ha, Jie Tan, and Chelsea Finn. Learning to be safe: Deep rl with a safety critic. arXiv preprint arXiv: 1ixi-20010114600 2020.
782
783
Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martin-Martín, Fei Xia, Kent Elliott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, C. Karen Liu, Silvio Savarese, Hyowon Gweon, Jiajun Wu, and Li Fei-Fei. BEHAVIOR: benchmark for everyday household activities in virtual, interactive, and ecological environments. In Alcksandra Faust, David Hsu, and Gerhard Neumann (eds.), Conference on Robot Learning, 8-11 November 2021, London, UK, volume 164 of Proceedings of Machine Learning Research, pp. 477-49044 PMLR, 2021. URL https: / /ppocediings..... mlr. press / wtl.///rrivsttvveee.........aaaa
785
786
787
788
html.
790
Elias Stengel-Eskin. Andrew Hundt, Zhuohong Hc, Aditya Murali, Nakul Gopalan, Matthew Gombo- lay, and Gregory Hager. Guiding multi-step rearrangement tasks with natural language instructions. In Aleksandra Faust, David Hsu, and Gerhard Neumann (eds.), Proceedings of the 5th Con- ference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp. 1486-1501. PMLR, 08-11 Nov 2022. URL htttps://ppocceednngg/t// ml mlr.press/v164/ html.
791
792
793
795
stengel-eskingg
796
Simon Stepputtis, Joseph Campbell, Mariano Phielipp, Stefan Lee, Chitta Baral, and Heni Ben Amor. Language-Conditioone Imitation Learning for Robot Manipulation Tasks, October 2020. URL http://arxia org/ abs 2010 12083. arXiv:2010. 122800000..208331 [cs].
797
798
799
Andrew Szot. Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam. Devendra Singh Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojeiech Galuba, Angel X. Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habi- tat 2.0: Training home assistants to rearrange their habitat. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 251-266, 2021. URL httpps:/prooceeinnn nwwrips www.wpp..c/pppeeee..eeeee wwtpcpppppp./c0ccc /2te//ra/////h/////// 1/hash/ Abst html. httm.
800
801
802
803
805
807
808
021
18

---

Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. Visual room rearrangement. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 5922-599 59311. Computer Vision Foundation / IEEE, 2021. doi: 10. 10.1144744 Under review as a conference paper at ICLR 2023
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. arXiv preprint arXiv: Axxiv-2009.06732, 2020.
810
Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, Nat McAleese, Nathalie Bradley-Schmieg, Nathaniel Wong, Nicolas Porcel, Roberta Raileanu, Steph Hughes-Fitr, Valentin Dalibard, and Wojciech Marian Czarnekii. Open-ended learning leads to generally capable agents. arXiv preprint arviv: Arxiv-2107.12805 2021.
812
813
814
815
818
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishman Srinivasan, Minho Hwang, Joseph E. Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery RL: safe reinforcement learning with learned recovery zones. IEEE Robotics Autom. Lett. 6(3): 4922, 2021, doi: 10.1 10901099/.... 2021 13070252. URL https: http:///oo....gg//. 1111111.111111111 2021. 3070252.
817
818
819
820
821
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android. arXiv preprint arXiv: 200i--2105.13331, 2021.
823
Sam Toyer, Rohin Shah, Andrew Critch, and Stuart Russell. The MAGICAL benchmark for robust imitation. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrlPS 2020, December 6-12, 2020, virtual, 2020. URL http:: / 44 64b5ac9 9e 7 4 6222112222112 www.cccccccccc.... Abst html.
825
827
828
/ http://ooocoddeeoooooooo ngs - neurips wwtpppppp///200000 /
829
830
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In Mare' Aurelio Ran- zato, Alina Beygelzimer. Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, neurips. Abstract html.
831
832
833
834
pp. 200-212, 2021. URL 31135775c3cac4223ecfF
2ttp//a22//hhhhh/ /hash/
836
Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wojeiech M Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev Richard Powell, et al. Alphastar: Mastering the real-time strategy game starcraft ii. DeepMind blog, 2, 2019.
838
840
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through simple sequence-ta-sequee
learning framework. arXiv preprint arXiv: 1rxiv-2202.03055 2022a.
842
843
Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy Julien Launay, and Colin Raffel, What language model architecture and pretraining objective work best for zero-shot generalization? Icml, 2022b. doi: 10445550k.7X1.222040532...
844
845
848
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: Beit pretraining for all vision and vision- lision-aagguage tasks. arXiv preprint arXiv: Arxiv- 2208. 10442, 2022c.
847
848
849
850
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. arXiv preprint arXiv: Arxiv-2206.00 276662222077 2022.
851
853
855
857
2021.00586. URL http:::::t/psccceecccc /owp//cccs.cccccccpoo//s// thecvf. com/
858
We heihs_ visual 1 Room_ Rea Rerrrmneerraggeeentt....
wttprieeee_____e________ CVPR_ 2011 _wwp__221_paper_____ pepperhhml.
19

---

Under review as a conference paper at ICLR 2023
Andrea Weikert, Andy Goralezyk, Basse Salmela, Ben Dansie, Campbell Barton, Enrico Valenza, Gleb Alexandrov, Ian Hubert, Kjartan Tysdal, Lech Sokolowski, Manu Järvinen, Massimiliana Pulieso, Matt Ebb. Pablo Vazquez, Rob Tuytel, Roland Hess, Sarah Feldlaufer, Sebastian König, Sebastian Platen, and Sönke Mäter. Blender online libraries for textures, 2022. URL https: //cloud. bww.der..d.ooo www.eeorrrroeeeeeeeeoooooeo /t/p///ee/ /tettreee/
860
862
863
884
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Amthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's 's transformers: State-of-the-an
885
866
867
868
natural language processing. arXiv preprint arXiv: Avxiv-9110.0377/ 2019.
869
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
https: / //gitttttbb wwt.//ooooooooooaeee
871 Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua B. Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. arXiv preprint arXiv: Arxiv-2206. 13499, 2022.
873
874
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv: Arxiv-2104. 10157, 2021.
875
878
Ziyi Yang, Yuwei Fang. Chenguang Zhu, Reid Pryzant, Dongdong Chen, Yu Shi, Yichong Xu, Yao Qian, Mei Gao, Yi-Ling Chen, Liyang Lu, Yujia Xie, Robert Gmyr, Noel Codella, Naoyuki Kanda, Bin Xiao, Lu Yuan, Takuya Yoshioka, Michael Zeng, and Xuedong Huang. i-code: An integrative and composable multimodal learning framework. arXiv preprini arXiv: Arxiv-2205.018/1 2022.
877
878
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Avnish Narayan, Hayden Shively, Adithya Bellathur, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. arXiv preprint arXiv: Eexiv-191011e 2019.
880
882
883
884
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang. Boxin Li, Chunyuan Li, Ce Liu, Mengehen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengehuan Zhang. Florence: A new foundation model for computer vision. arXiv preprint arxiv: Arxiv- 2tfi-22111.114333 2021.
885
886
887
888
889
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. arXiv preprint arXiv: Arxiv- 106.000 02636, 2021.
881
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati. Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. CVPR, 2022.
893
885
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Ayzaan Wahid, Vikas Sindhwani, and Johnny Lee. Transporter networks: Rearranging the visual world for robotic manipulation. arXiv preprint arXiv: Arxiv-2010.144000 2020.
896
887
898
889
Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveck Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratie models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv: Arxiv- 2204.005 00598, 2022.
900
902
Mandi Zhao, Fangchen Liu, Kimin Lee, and Pieter Abbeel. Towards more generalizable one- shot visual imitation learning. In 2022 International Conference on Robotics and Automation, ICRA 2022, Philadelphia. PA, USA, May 23-27, 2022, pp. 2434-24444 2444. IEEE, 2022. doi: 10. 1199/CCRA4669..222288888
904
906
URL https: / /ooi www.doorggoooo org/10 1109 (11////6R246.222222222 2022.
907
9812450
20

---

Under review as a conference paper at ICLR 2023
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. arXiv preprint arXiv: Arxiv-2202.05607 2022.
909
910
Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martin-Martin. robosuite: A modular
simulation framework and benchmark for robot learning, arXiv preprint arxiv: Arxiv-2009.122 12293, 2020.
912
913
21

---

Under review as a conference paper at ICLR 2023
A
SIMULATOR DETAILS
915
We build our VIMA- BEAAHBENCH simulation suite upon the Ravens physics simulator (Zeng ct al., 2020; Shridhar al., 2021). Specifically, it is supported by PyBullet (Coumans & Bai, 2016-2021) with Universal Robot UR5 arm. The size of the tabletop workspace 0.5 x 1m. Our benchmark contains extensible sets of object geometries and textures. Instantiated from an object-texture combination, all object instances can be rendered as RGB images appeared in multimodal prompts. Figure A.1
916
917
918
919
920
displays all object geometries. Figure A.2 displays all textures.
921
The observation space of VIMA-BENCH includes RGB images from both frontal and top- down views. It also includes a one-hot vector E {0,1)² to indicate type of the end-effector E {suction cup, spatula). While a suction cup is equipped in most manipulation tasks, a spat- ula is used in particular for visual constraint tasks, where an agent is asked to "wipe" objects.
922
923
924
925
VIMA BENCH inherits the same action space from Zeng et al. (2020) and Shridhar et al. (2021),
926
which consists of primitive actions of "pick and place" for tasks with suction cup as the end effector,
or "push" for tasks with a spatula. Both primitive actions contain two poses E SE(() specifying target poses of the end effector. For the "pick and place" primitive, they represent the pick pose and the place pose. Fir the "push" primitive, they represent the push starting pose and push ending pose.
928
930
Similar to prior work (Zeng et al., 2020; Shridhar et al., 2021). VIMA-BENCE provides scripted oracles to generate successful demonstrations for all tasks. We leverage them to construct an offline imitation dataset for behavioral cloning. Given a prompt, these pre-programmed bots can access privileged information such as the correct object to pick and target location to place.
932
933
B TASK SUITE
934
925
We develop 17 meta tasks that belong to 6 diverse categories. Thousands of individual tasks and their corresponding multimodal prompts can be procedually generated from these meta-task templates. We use PyBullet (Coumans & Bai, 201 6-2221) as our backend and the default renderer to produce the RGB frames for training data and interactive test environments. For demonstration purpose, we apply the NVISII (Morrical et al., 2020) raytracing renderer to enhance the visual quality. We elaborate each meta task in the following subsections.
936
938
940
941
B.
SIMPLE OBJECT MANIPULATION
942
This task category asks agents follow basic instructions specified by multimodal prompts.
943
Task 01:
Pick the specified object(s) and place it into the specified object.
Prompt: Put the {object), into the {object]2-
945
Description: The image placeholder {object], is the object to be picked and the {object), is the container object. The agent requires to recognize the ob- jects with the correct color-shape combinations. To extend the difficulties, it sup- ports more than one object to be picked or placed. For example, the prompt Put the jobject, and {object], into the {object) asks to pick two different objects and place into a target container. We uniformly sample different color-shape combos for objects to be picked and containers.
946
947
948
950
952
Success Criteria: All specified object(s) to pick are within the bounds of the container object(s), with specified shapes and textures provided in the prompt.
953
Oracle Trajectory:
Shown in Fig, A.3 with its multimodal prompt.
955
Task 02:
the workspace, put the objects with a specified texture shown in the scene image the prompt into container object(s) with specified color. This task requires the agent to find the correct object to manipulate by grounding the textural attributes from both natural language descriptions and the visual scene images.
957
958
Prompt: Put the (texture), object in {scene) into the (texture), object.
959
960
22

---

Under review as a conference paper at ICLR 2023
L-shaped block
block
bowl
container
cross
diamond
flower
frame
heart
hexagon
letter A
letter E
M
R
V
letter
letter
letter
letter
letter V
line
pallet
pan
pentagon
ring
round
shorter block
small
square
three-sided rectangle
triangle
Figure A.I: Object Gallery in VIMA-BENCH textured with random textures. Bowl and pan are
from Google Scanned Objects (Downs al., 2022) while others are from Ravens (Zeng et al., 2020)
23

---

Under review as a conference paper at ICLR 2023
RGB Colors
6yan
gronn
orange
pvrple
Vellw
dert bium
dent open
Clrbk green
dert crangM
dess pink
ders purshe
derk red
deels yollow
Image
Images
brik
Flos
waoden
grornl
polea dut
checkerbourd
Sger
ragee
relrccm
Light Stripes
red yellow strpe
red and green pe
red and
and purple strpe
yellow and green
yellow and cae stripe
yelow and purple
green and blve stripe
giten and purple striee
blie and purple
Derk Stripes
clrkk red and
dank rnd and
darl red and
durk red and
dark yalloe and
dark yalloe and
dalk yallow and
clrkk groOT
dark SrOUT and
Gark DAM and
Light
ned and yellow polka
rnd and green
red and brue prlki
and parple polka
yelinw and grewn pelka
yellow blde
polka dot
polica dot
polka cot
polic dol
Dark Dots
durk poliaa
dark purprepeñk
yellow polia dur
green pulka der
BRUE polko polkatooo
patple polka dut
polko
parple polka dot
truegolia
purplepoko
Ligh Swirts
Paisleys
Figure A.2: Texture Gallery in VIMA-BENCE -BNCH.. The first row of image-based textures are from Blender Cloud Libraries (Weikert al., 2022), while others are hard-coded.
24

---

Under review as a conference paper at ICLR 2023
Put the
R
into the
Figure A.3: Simple Object Manipulation: Task 01
The text placeholder (texture), and (texture], are sampled textures for objects to be picked and the container objects, respectively. The number of dragged objects with the same texture can be varied. {scene) is the workspace-like image placeholder. There is a designated number distractors with different textures (and potentially different shapes) in the scene. For each distractor in the workspace, it has 50% chance to be either dragged or container distractor object with different textures from those specified in the prompt.
Description:
962
964
986
Success Criteria: All objects in the workspace with (texture), are within the bounds of the container object with (texture)
968
Oracle Trajectory: Shown in Fig. A.4 with its multimodal prompt.
the green and blue stripe object
yellow paisley object.
Figure A.4: Simple Object Manipulation: Task 02
Rotate objects clockwise by certain degrees along z-axis. Only rotationally asymmetric objects are considered in this task.
Task 03:
972
973
Prompt: Rotate the {object], {angles} degrees.
974
The agent required rotate all objects the workspace specified by the image placeholder {object),. There are also objects with different color-shape combinations in the workspace as distractors. bangles) is the sampled degree that the dragged object needs to be rotated. A target angle is sampled from 30", 60%, 90% 120", and 150"
Description:
976
978
Success Criteria: The position of the specified object matches its original position, and the orientation matches the orientation after rotating specific angles.
980
981
Oracle Trajectory: Shown Fig. A.5 with its multimodal prompt.
B.2
VISUAL GOAL REACHING
983
This task category requires agents to manipulate objects in the workspace to reach goal states represented images shown in prompts.
25

---

Under review as a conference paper at ICLR 2023
Rotate degrees.
Figure A.5: Simple Object Manipulation: Task 03
985
Task 04: Rearrange target objects in the workspace to match goal configuration shown in prompts.
986
Note that achieve the goal configuration, distractors may need be moved away first.
Prompt: Rearrange
this {scene).
988
Description: Objects in the scene placeholder (scene) are target objects to manipulated and rearranged. In the workspace, the same target objects are spawned randomly, potentially with distractors randomly spawned as well. With defined distractor conflict rate, the position of each distractor has this probability to occupy the position of any target object such that the rearrangement can only succeed if moving away that distractor first.
980
992
983
Success Criteria: The contiguration of target objects in the workspace matches that specified in the prompt.
995
Oracle Trajectory: Shown in Fig. A.6 with its multimodal prompt..
Rearrange to this
Figure A.6: Visual Goal Reaching: Task 04
996
Task 05:
Extend the task 04 by requiring the agent to restore rearranged objects to the initial setup after the "rearranging" phase.
Prompt: Rearrange objects to this setup {scene} and then restore.
Description: Same as the task 04, except introducing the instruction "restore".
1000
Success Criteria: Meet the success criteria the task 04, and then within the allowed max steps restore all target objects to their initial configurations.
1001
Oracle Trajectory: Shown in Fig. A.7 with its multimodal prompt.
26

---

Under review as a conference paper at ICLR 2023
Rearrange objects
and
Figure A.7: Visual Goal Reaching: Task 05
1003
B.3
NOVEL CONCEPT GROUNDING
1004
This task category requires agents to ground new concepts of adjectives, nouns, verbs via visual per- ception and language understanding. Similar task design can be found in prior work (Hill et al., 2021). Completing these tasks are challenging, because the model should a) first understand prompts with
1005
1006
1007
interleaved texts, images, and even video frames; b) quickly internalize new concepts that are different across task instances, which even tests the ability to meta learn; and e) do complicated reasoning such as comparing between "taller" "less taller" vs "shorter" and then ground this reasoning into the robot action space.
1008
1009
1010
1011
Prompts consist of two parts: a definition part followed by an instruction part. In the definition part, novel conceptions are defined by multimodal illustrations with multiple support examples. In the instruction part, agents are asked to achieve the goal by properly applying concepts from the definition part. The assignment of unique nonsense words is varied and independent for each task instance such that tasks can only be solved if the agent applies the reasoning correctly. This ability is also referred to as fast-mapping (Heibeck & Markman, 1987).
1012
1013
1014
1015
1016
1017
Task 06:
Ground comparative adjectives by comparing the size or the textural saturation of objects and manipulating the correct object(s) instructed in the prompt.
1018
(denooocddmo. is [ndvel_adj} than (demo_object]pp Put the {adv} (novel.adj} {object), into the {object]2.
1019
1020
1021
Description: The sampled adjective [novel.adj} is a dummy adjective placeholder for agent to ground. By default, the novel adjective set is (daxer, blicker, modier, kobar). The real meaning can be related to size (smaller/larger) or textu- ral saturation (lighter/darker texture). The image placeholders [demo_ob ject), ) and [demo_object)] }22 illustrate how the novel adjective defined. For example, if the real comparison is "taller", then the sampled object in fdemo ject The choices of the novel adjective and the real meaning are indepen- dently sampled for different task instances. For the instruction part, this task is similar to task 01, where the agent required to pick the specified dragged object(s) with the novel adjective attribute and then place it into the specified container object. To avoid revealing the correct object to manipulate, we use neutral texture for objects appeared in the instruction part.
1022
1023
1024
1025
1026
is taller than
1027
1028
1029
1030
1031
1032
1033
Success Criteria: All target objects with the specified adjective attribute are within the bounds of the specified container object.
1034
1035
Oracle Trajectory: Shown in Fig. A.8 with its multimodal prompt.
1036
Task 07:
Orthogonal to task 06 by requiring to learn mappings of novel nouns.
27

---

Under review as a conference paper at ICLR 2023
kobar than
kobar than
is kobar than
Put the kobar
Into the
Figure A.8: Novel Concept Grounding: Task 06
1037
Prompt: This is a {object)2. Put (novel_name), into [nevel_name),,
fobject). This is a (novel.name),
1038
1039
Description: Novel noun words are defined with the text placeholders (novel and (novel_name]p following their image placeholders {object]; and fobject 2 for the target object and container object, respectively. Novel nouns are sampled from {dax, bl icket, wug, zup]. In the instruction part, objects are expressed as novel nouns defined in the previous definition part. Distractors are defined the same task 0/.
1040
1041
1042
1043
104
Success Criteria: All target object(s) are within the bounds of the container object(s).
1045
Oracle Trajectory: Shown in Fig. A.8 with its multimodal prompt.
This is a blicket
This is zup
Put a zup into a blicket.
Figure A.9: Novel Concept Grounding: Task 07
1046
Task 08:
Combination of tasks 06 and 07.
Prompt: This is [novel_name), fobject) This is a (novel_name), {object)2. {demo-objecth, is {adi} than {demo.object/z. Put the (adv) (novel_adj} (novel_name) into the (novel_name)2-
1048
1049
Description: see task description for task 06 and task 07.
1051
Success Criteria: Similar as tasks 06 and 07.
1052
Oracle Trajectory: Shown Fig. A.10 with its multimodal prompt.
1053
novel verb "twist" is defined as rotating a specific angle conveyed by several examples. This task is similar to task 03, but it requires the agent to infer what is the exact angle to rotate from the prompt and to ground novel verbs that are semantically similar but different exact definitions.
Task 09:
1054
1055
1056
Prompt:
"Twist" is defined as rotating object specific angle. For examples: From (before.twist), to /after.twist), Now twist all (texture) objects.
1057
1058
Description: Both (before.twist), and (after.twist), scene placehold- ers where (before twist), shows a randomly sampled object before "twist" and (after twist) shows the same object pose after "twist". All examples illustrate the
1080
1061
28

---

Under review as a conference paper at ICLR 2023
This is wug
This zup
is blicker than
blicker than
is blicker than
Put the blicker zup into the wug.
Figure A.10: Novel Concept Grounding: Task 08
sampled angle of the rotation. In the workspace, the target objects have the texture specified by (texture) and randomly sampled shapes.
1063
1084
Success Criteria: Same as the task 03.
1085
Oracle Trajectory: Shown Fig. A.11 with its multimodal prompt.
"Twist' is defined as rotating object specific angle. For examples:
From
From
From
Now twist all yellow and purple stripe objects.
Figure A.11: Novel Concept Grounding: Task 09
1086
B.4
ONE-SHOT VIDEO IMITATION
1067
This task category requires agents to imitate motions demonstrated through videos shown in prompts. We follow prior works (Finn et al., 2017; Dasari & Gupta, 2020; Duan et al., 2017) to formulate the problem by giving one video demonstration (represented consecutive frames in prompts), then test the learned imitator's 8 ability to produce target trajectories. This setup is challenging because only one demonstration is available to the agent; b) the model needs to understand video frames interleaved with textual instructions; and c) missing correspondences between demonstrations and
1068
1070
1071
1072
1073
target trajectories since demonstrations only show partial key frames.
1074
Task 10:
Follow motions for specific objects.
Prompt: Follow this motion for {object): (frame) ..rrame)..(fframme (frame), [frame),,
1075
Description: Image placeholder jobject) is the target object to be manipulated and {[frame); is set workspace-like scene placeholders to represent a video trajectory, where is the trajectory length. There an object spawned at the center in both the
1078
1079
29

---

Under review as a conference paper at ICLR 2023
workspace and the prompt video but with different textures distractor. The initial position the target object matches that {frame)]-
1081
1082
Success Criteria: In each step, the pose of the target object matches the pose in the
1083
corresponding video frame. Incorrect manipulation sequences are considered as failures.
1084
Oracle Trajectory: Shown in Fig. A.12 with its multimodal prompt.
Follow this motion
Figure A.12: One-shot video imitation: Task 10
1085
Task 11:
Stack objects with the order illustrated in the prompt video.
1086
Prompt: Stack ob ebjetss in this order
},
1087
Description: There are multiple objects with the same shape but different textures spawned in the workspace without any stacking initially. Distractor objects with different shapes spawned in the workspace but not in the prompt video. At each step of the prompt video, of the top objects stacked over another object or put at empty position.
1089
1080
1091
Success Criteria: Similar as task 10.
1092
Oracle Trajectory: Shown Fig. A.13 with its multimodal prompt.
objects
Figure A.13: One-shot video imitation: Task
1093
B.5
VISUAL CONSTRAINT SATISFACTION
1094
This task category requires agents to wipe specific number of objects in the workspace to goal region while also satisfy the given visual constraint.
1095
Task 12:
Sweep the designated number of objects into specified region without exceeding the boundary.
1097
1098
Prompt:
Sweep (quantifier) {object} into (bounds} without exceeding (constraint).
1089
1100
Description: {object) is the image placeholder the target object to be swept spawned with a random amount in the workspace. Distractors have the same amount, same shape, but different color from target objects. (quantifier) is the text placeholder to determine the target quantity of objects to be wiped, sampled from any, one, two, three, and al1. (bounds) is the image placeholder for a three-sided rectangle as the goal region. constraint] is the constraint line.
1102
1104
1105
30

---

Under review as a conference paper at ICLR 2023
Success Criteria:
The exact number of target objects be swept all inside the specified
1107
region. Failure reasons include 1) any distractor being wiped into the region, 2) target object exceeding the constraint, or 3) incorrect number of target objects being swept into the goal region.
1109
1110
Oracle Trajectory: Shown Fig. A.14 with its multimodal prompt.
Sweep any
into
without exceeding
Figure A. 1414: Visual Constraint Satisfaction: Task 12
1111
Task 13:
Sweep the designated number of objects into a specified region without touching the constraint.
1112
1113
Prompt:
Sweep (quantifier) {object) into (bounds} without touchimg ng (constraint).
1114
1115
Description: Similar as task /2 but requiring different way to satisfy the constraint. The agent has to learn avoid "touching" the constraint line this case.
1116
Success Criteria: Similar as task 12 except that the constraint is to not touch the red line.
1118
Oracle Trajectory: Shown in Fig. A.15 with its multimodal prompt.
Sweep
into
without touching
Figure A.15: Visual Constraint Satisfaction: Task 13
1119
B.6
VISUAL REASONING
1120
This task category requires agents make decisions by reasoning over or memorizing information conveyed through multimodal prompts.
1121
1122
By reasoning the "same texture", the agent is required to pick all objects in the workspace with the same texture as the container objects specified in the prompt and place them into it.
Task 14:
1123
1124
Prompt: Put all objects with the same texture as {object] into
it.
31

---

Under review as a conference paper at ICLR 2023
Description: {ob jobt) is the sampled goal container object. In the workspace, there objects with the same texture as the container but potentially different shapes. Distractors with different textures are spawned.
1127
Success Criteria: All objects with the same texture as the goal container are within the bounds of the container.
1130
1131
Oracle Trajectory: Shown in Fig. A.16 with its multimodal prompt.
objects
Figure A.16: Visual Reasoning: Task 14
Task 15: By reasoning the "same shape", the agent is required to pick all objects in the workspace with the same top-down shape as the goal container specified the prompt and place them into it. For example, blocks and boxes have the same rectangular shape.
1139
1134
1135
Prompt: Put all ob objects with the same profile as fobject) into
1136
it.
1137
Description: Similar the task 14 except the objects be picked and placed with the same shape. There are three different shapes: rectangular-like (e.g. block and pallet), circle-like (e.g. ring and bowl), and undetermined for the rest.
1138
1139
Success Criteria: All objects with the same shape the container are within the container.
1141
Oracle Trajectory: Shown in Fig. A.17 with its multimodal prompt.
objects profile
Figure A.17: Visual Reasoning: Task 15
1142
Task 16:
Put the target object into the container, and then put one of its old neighbors into the same
1143
container.
Prompt:
First put (object) into {object], then put the ob ject that was previously at its (direction] into the same {object);
1145
1146
Deseription: Objects image placeholders (object), and {object), are the target object to be picked and the container, respectively. We then ask the agent put of old neighbors of the previous target object into the same container. The old neighboring object is specified through cardinal directions (north, south, west, east).
1148
1150
Success Criteria:
The target object and the correct neighboring object are inside the
1151
container.
1152
Oracle Trajectory: Shown in Fig. A.18 with its multimodal prompt.
32

---

Under review as a conference paper at ICLR 2023
First put
into
then put the object that was previously at its west into the same
Figure A.18: Visual Reasoning: Task 16
1153
Task 17:
Pick and place the target object specified in the prompt into different containers order then restore to the initial container.
1154
Prompt: Put {object), into {object) Finally restore it into its original container.
1158
1157
Description: The object in the image placeholder {objecth is the target ob- ject to be manipulated across the task. There are more than one target containers (e.g. Put fobject), into {object], then fobject Finalll restore it into its original container. for two target base objects to be placed order). The rest of spawned containers naturally becomes distractors.
1158
1180
1162
Success Criteria: The target object are first put into multiple containers following the specific order. Finally it should be restored into its original container.
1163
Oracle Trajectory: Shown in Fig.A.19 1.19 with its multimodal prompt.
into
then
Finally restore original container.
Figure A.19: Visual Reasoning: Task 17
C MODEL ARCHITECTURE
1185
In this section, provide comprehensive details about VIMA model architecture well other adapted baseline methods. We implement all models in Py PyTorch (Paszke et al., 2019) and adapt Transtormer-rllteed implementation from Wolf al. (2019).
1187
1189
C.1
SUMMARY OF DIFFERENT METHODS
1170
We summarizes differences between VIMA and other baseline methods Table 1. the column "Prompt Conditioning", an alternative of cross-attention first concatenate prompt and interaction
1171
1172
into big sequence, then repetitively apply transformer decoders predict actions. It is referred "direct modeling". The relative computation is quadratically proportional number of
1173
observation tokens.
33

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison of different methods.
Tokenizer
Prompt Conditioning
Observation per Step
Object tokens consisting of cropped images bounding boxes.
Ours
Cross-atiention
Equal number of objects, typically
(Roed ct al., 2022)
Image pulch tokems encoded by a VIT
Direct modeling
Equal to number of image pulches,
Flamingo Agent (Alayrac al, 2022)
Image patch tokens encoded by a VIT. further downsampled Perceiver module
Cross-atiennton
Equal number of learned query voctors,
Multimodal GPT Agent (Brown et al., 2020)
Single image
by a
modeling
Single feature,
Table Model hyperparameters for multimodal prompt tokenization.
Hyperparameter
Value
Text Tokenization
Tokenizer Embedding Dimension
t5-base tokenizer
768
Image Tokenization
ViT Input Image Size
32 x 32
ViT Patch Size
16
ViT Width
768
VIT Layer
4
ViT Number of Heads
24
Bounding Box MLP
Hidden Dimension
768
Hidden Depth
2
Prompt Encoding
Pre-trained LM
t t-bass
Unfreeze Last Layers
2
Positional Embedding
Absolute
Token Adapter MIP Depth 2
1175
C.2
VIMA ARCHITECTURE
1178
C.2.1
MULTIMODAL PROMPT TOKENIZATIONS
1177 As introduced in Section 5, there are 3 types of input formats multimodal prompts, namely (1) text inputs, images of full scenes, and (3) images of single objects.
1178
1179
For text inputs, we follow the standard pipeline in NLP to first tokenize raw languages to discrete indices through pre-trained t base tokenizer. We then obtain corresponding word tokens from the embedding look-up of the pre-trained + base model. For images of full scenes, we first parse the scene through fine-tuned mask R-CNN detection model (He et al., 2017: Wu al., 2019) to extract individual objects. Each object representation contains a bounding box and cropped image. The bounding box is in the format of [Feenter Veenters height, width). We normalize to be within (0, 1] by dividing each dimension with corresponding upper-bound value. We then pass it through a bounding box encoder MLP and obtain a feature vector. To process the cropped image, we first pad non-square image to square by padding along the shorter dimension. We then resize it pre-configured size and pass it through a VIT (trained from scratch) to obtain the image feature. Finally, an object token is obtained by concatenating the bounding box feature and the image feature and mapping to the embedding dimension. For images of single objects, we obtain tokens in the same way except with a
1180
1181
1182
1183
1184
1185
1186
1188
1190
dummy bounding box. Detailed model hyperparameters about tokenizations listed in Table 2.
34

---

Under review as a conference paper at ICLR 2023
After obtaining sequence of prompt tokens, follow Tsimpoukelli et al. (2021) pass it through 1198 a pre-trained t5-base encoder to obtain encoded prompt. Note that we add adapter MLP be- tween object tokens and the T5 encoder. We adopt learned absolute positional embedding. Model hyperparameters are listed in Table 2 as well.
1185
Table Model hyperparameters for observation encoding.
Hyperparameter
Value
Observation Token Dimension
768
End Effector Embedding Dimension
Positional Embedding
Absolute
Table 4: Model hyperparameters for action decoders.
Hyperparameter
Value
Hidden Dimension
512
Hidden Depth
Activation
ReLU
X-axis Discrete Bins
50
Y-axis Discrete Bins
100
Rotation Discrete Bins
50
1196
C.2.2
OBSERVATION ENCODING
1197
Since all RGB observations are images full scenes, we follow the same procedure discussed above to obtain flattened object tokens. Because we provide RGBs from two views (frontal and top-down), we order object tokens by following the order of [frontal, top-down]. We one-hot encode the state of the end effector. We then concatenate object tokens with the end-effector state and transform to observation tokens. We adopt learned absolute positional embedding. Detailed model hyperparameters about observation encoding is provided in Table
1199
1201
1202
1203
C.2.3
ACTION ENCODING
1204
Since our model is conditioned on observation-action interleaved history, we also tokenize past actions. We follow common practice in Chen et al. (2021), I); Zheng et al. (2022) to encode past actions with two-layer MLP. It has hidden dimension of 256. We then map outputs to token dimension and obtain action tokens.
1205
1206
1207
C.2.4
SEQUENCE MODELING
1209
The robot controller in VIMA is causal decoder that autoregressively predicts actions. To condition the decoder prompt tokens, we perform cross-attention between history tokens and prompt tokens (Figure 3). Concretely, we pass history tokens as the query sequence and prompt tokens as the
1210
1211
1212
key-value sequence into cross-attention blocks. The output prompt-aware trajectory tokens then go through causal self-attention blocks. We alternate cross-attention and self-attention times. This procedure is technically described in Pseudocode 1.
1213
Table 5: Model hyperparameters for ViT used in baseline methods.
Hyperparameter
Value
Image Size
64 128
Patch Size
32
ViT Width
768
ViT Layers
ViT Heads
24
35

---

Under review as a conference paper at ICLR 2023
Table 6: Model hyperparameters for Perceiver Resampler used in Flamingo method.
Hyperparameter
Value
Number of Latent Queries
Number of Blocks
4
Self-attn per Block
Self-attn Heads
24
Cross-attn Heads
24
def xati__eqquecee_oddding
prompt_tokens, the [L, d] prompt tokens (L=prompt length)
obs_tckens, $ the (T, d] obs tokens (T=time step)
act_tokens, traj_pos_embd,
# the (T-1, d] action tokens
learned positional embedding for trajectory
prompt_pos_endd,
learned pesitional embedding for prompt
):
interleave abs and action tckens traj_tckens - interleave (cbs_tckens, act_tokens) [21-1, d] add positional embedding to tra trajectory tokens traj_tokens traj_pos_embo
add positionsl embedding to prompt tokens prompt_tokens = prompt_tokent prompt_ pos
_os_ee_____dddd embod
apply xattn and causal self-attn for range (num_layers):
# cross-attention
x = x attn_i (q=x, kv<prompt_tokens)
# feed forward
ffw_xattn_i
selt-attention
x = x causal attn (q=x, kv=x)
# feed forward
ffw_i(x)
the last token is the predicted action token seett = x[-1]
return predicted_act_tt act_token
Pseudocode 1: Cross-attention operation that conditions the trajectory history on prompt. We
repetitively alternate cross-attention and self-attention to model the trajectory given specific task.
1216
C.2.5
ACTION DECODING
After obtaining the predicted action token, map it to the action space A and obtain the predicted action. This is achieved though group of action heads. Since the action space consists of two SE(2) poses, for each pose use six independent heads decode discrete actions (two for xy coordinate and four for rotation represented in quaternion). These discrete actions are then de- discretized and mapped continuous actions through affine transformation. The poses modeled independently. Early ablations show that this independent modeling is equally good as alternatives techniques like autoregressive decoding (Vinyals et al., 2019; OpenAl et al., 2019).
1219
1220
1221
1229
1224
1225
Detailed model hyperparameters are listed in Table 4.
1226
C.3
BASELINES ARCHITECTURES
1227
In this section, elaborate model architectures for baseline methods. Some components such the action decoder are same across all baseline methods and ours. Therefore, we only discuss unique model components.
1228
36

---

Under review as a conference paper at ICLR 2023
C.3.1 GATO
Gato (Reed et al., 2022) introduces a decoder-only model that solves tasks from multiple domains including robotics, video game, image captioning, language modeling, etc. Different tasks speci- fied by supplying the model with an initial sequence of corresponding tokens. For example, in tasks involving decision making, these tokens include observation and action tokens. For fair comparison.
1233
1234
1235
we provide the same conditioning as VIMA, i.e., our multimodal tokenized prompts. Similar to our method, Gato also predicts actions in an autoregressive manner. Gato and our method share the same training philosophy to only optimize the causal behavior cloning objective. However, unlike our method that adopts an object-centric representation to treat individual objects as observation tokens,
1236
1237
1238
1239
Gato divides input images into patches and encodes them by ViT (Dosovitskiy et al., 2020) model
1240
to produce observation tokens. Furthermore, Gato relies on causal self-attention to model entire
1241
trajectory sequences starting with prompt tokens. Hyperparameters of Gato's ViT is listed Table
1242
The transformeer-eccoder style sequence modeling is technically illustrated Pseudocode 2.
def causal sequence mode ling prompt_tckens, I the [b, d] prompt tokens (L-prompt length) sep_token, F the [I, d] learned token to separate prompt and trajectory history obs_tckens,
the IT, d] cbs tokens (T=time step)
act_tckens: # the (T-1, d] action tokens
pos_embd, # learned positional embedding
): interlcave obs and action tckens
traj_tokens interleave (ebs_tckens, act_tokens) [2I-1, assemble input tokens
x concat ( (prompt_tokens, sep_token, traj_tokens]) pos_embd
1243
apply GPT layers with causal mask for i in range (num (oum__ayers):
$ self-attention causal_attn_i (q=x, kv=x) feed forward
x x ffw_i(x)
the last taken is the predicted action token ppedicted_act_tote return predietedLact__tkee
x[-1]
Pseudocode 2: Plain sequence modeling that temporally concatenates prompt and trajectory history and repetitively perform causal self-attention operation.
1245
C.3.2
FLAMINGO
1246
Flamingo (Alayrac et al., 2022) is vision-language model that learns to generate textual completion in response to multimodal prompts. embeds a variable number of prompt images into fixed number of tokens via the Perceiver Resampler module (Jaegle et al., 2021b), and conditions the language decoder on encoded prompts by cross-atted attss-attntion. Flamingo does not work with embodied agents out
1247
1248
1249
1250
of the box. We adapt by replacing the output layer with robot action heads (hyperparameters listed
1251
in Table 4) and using tokenized rollout histories as inputs. We train end-to-end with causal behavior cloning loss. The modified Flamingo agent differs from ours since it processes image observations
1252
1253
into a fixed number visual tokens through learned Perceiver Resampler. Model hyperparameters our reimplementation of the Perceiver Resampler is listed in Table
1254
1255
C.3.3
MULTIMODAL GPT AGENT
37

---

Under review as a conference paper at ICLR 2023
Multimodal GPT agent (Brown al., 2020) is behavior cloning agent conditioned tok- enized multimodal prompts with the GPT architecture. It autoregressively decodes next actions given multimodal prompts and interaction histories. We optimize this method end-to-end with causal behavior cloning loss. Similar to prior works of casting RL problems as sequence mod- cling (Chen et al., 2021; Janner et al., 2021; Zheng et al., 2022), it encodes an image into a single state" token through a learned ViT encoder. It also directly models entire trajectory sequences prepended with prompt tokens. Therefore, it differs from our method in the representation of ob-
1257
1258
1259
1280
1281
1262
1263
servation tokens and prompt conditioning, For visual tokenizer, we employ learned ViT with
1264
hyperparameters listed Table 5.
1265
C.4
MASK R-CNN DETECTION MODEL.
1286
Finally, we elaborate the mask R- R-CNN model (He et al., 2017) for scene parsing and object extraction.
1267
from Wu et al. (2019) to adapt to scenes and images in our tabletop environment. visualization of its output is 1268
We fine-tuned a pre-trained lightweight mask R-CNN
provided in Figure A.20. We do not use the predicted object names in our models.
bowl 99 %
bowl
cross 99 frame 100 %
cross
frame
Figure A.20: Visualization of fine-tuned mask R-CNN. Left: Prediction from the detection model.
Right: Ground-truth scene parsing. The detection model agrees well with ground-tre tounh objects.
1270
D
VIMA TRAINING DETAILS
We follow the best practices to train Transformer models using the AdamW optimizer (Loshchilov Hutter, 2019), learning rate warm-up, cosine annealing (Loshchilov & Hutter, 2017), etc. Train-
1272
ing hyperparameters provided in Table 7. We use GEGLU activation (Shazeer, 2020) inside Transformer models across all methods.
1274
Table 7: Hyperparameters used during training.
Hyperparameter
Value
Learning Rate
0.0001
Warmup Steps
7K
LR Cosine Annealing Steps
17K
Weight Decay
0
Dropout
0.1
Gradient Clip Threshold
1.0
1275
To make trained models robust to detection inaceuracies and failures, we apply object augmentation by randomly injecting false-positive detection outputs. Concretely, for observation at each time step, we sample number of augmented objects i.i.d. R'uugmented objects ~ Cat( K, p), where Cat(-) denotes a multi-categorical distribution with K supports parameterized by p. For each augmented object, then randomly sample bounding box and corresponding cropped image to add to object tokens. In experiments, p {0: 0.95, 1 0.05) with K 2.
1277
1279
1280
38

---

Under review as a conference paper at ICLR 2023
D.1
VARY MODEL CAPACITY
1282
We train spectrum of 7 models ranging from 2M to 200M parameters. To vary the model capacity,
1283
we follow prior work (Chowdhery al., 2022) to change embedding dimension and number of layers.
We list configurations for methods with cross-attention prompt conditioning (i.e., ours and Flamingo) Table and configurations for methods only with causal self-attention (i.e., Gato and DT) in
1284
1285
Table 9.
Table 8: Configurations for different sized models with cross-attention prompt conditioning.
Model Size (M)
Embedding Dimension
Num Blocks
X-attn Heads
Self-attn Heads
2
256
8
256
8
9
320
3
10
10
20
384
12
12
43
512
5
16
16
92
640
20
200
768
11
24
24
Table 9: Contigurations for different sized models with causal self-attention prompt conditioning.
Model Size (M)
Embedding Dimension
Num Blocks
Self-attn Heads
2
64
2
96
2
3
9
192
3
6
20
320
4
10
43
512
5
16
92
768
7
24
200
768
18
24
E
MORE EXPERIMENT RESULTS
E.I
BREAKDOWN RESULTS
1289
We show breakdown results for Figure in Tables 10, 11, 12, and 13, respectively.
1290
E.2
VARY T5 ENCODER SIZES
We vary the size of the pre-trained T5 encoder (Raffel al., 2020) study the effect of prompt encoding. We experiment with three T5 model capacities: t:5-sma 11 (30M), t5-base (11 (11M), to fine-tune the last two layers and freeze all other layers. We fix the parameter count of the decision-making part to be 200M. As shown in Table 14, we find no significant difference among the variants. Thus we the standard t 5-base default for all our models.
1292
1293
5 large (368M). For all T5 variants,
1294
1285
1296
1297
E.3
POLICY ROBUSTNESS
1298
Increased Amounts of Distractors. We study the policy robustness against increased amounts of
1299
distractors in scenes, For all tasks being evaluated, we add one more distractor object. We ran our
1300
largest VIMA model with 200M parameters. The result presented in Table 15.
1301
turns out that the performance of VIMA degrades minimally with more distractors than the training distribution. This indicates that our agent has learned reasonably robust policy against objects that
1302
1303
irrelevant the task.
39

---

Under review as a conference paper at ICLR 2023
Table 10: LI level generalization results. Model indicates robot controller parameter count.
13.5
Fluningo
37.5
45.0
53.5
28.5
Multinedal
43.5
40.5
61.5
76.5
27.5
0.0
Durs
100.0
100.0
0
100
100
15.5
74.0
72.5
47.5
20M
Faamiggo
Gatn
63.0
61.5
55.0
32.5
30.0
49.0
18.0
42.5
41.5
58.0
62.0
68.0
$3.0
64.0
38.5
2.0
1.0
Multindial GPT
60.5
64.00
50.5
44.0
41.0
45.0
61.5
7.0
86.0
84.0
44.5
39.0
2.5
Ours
100.0
100.0
99.5
100
56.5
100.0
100.0
18.0
77.0
93.0
97.0
76.5
43.0
200M
Flamingo
Gato
56.0
79.0
66.0
58.5
63.0
91.5
57.0
48.5
18.0
44.5
54.0
43.5
62.5
74.0
18.0
3.5
61.0
66.5
58.5
$6.0
83.5
40.0
33.5
43.5
2.5
2.5
Multiedal GPT
60.0
57.5
41.0
55.5
45.5
47.5
54.5
8.5
77.0
$1.5
41.0
38.0
0.5
Table L2 level generalization results. Model indicates robot controller parameter count.
Mathod
Ti
Durs
37.5
100.0
17.5
87.5
97.55
Gatn
17.5
47.5
82.5
49.0
Tamingo
43.5
56.0
56.0
29.5
35.5
49.0
67.0
53.0
30.0
43.0
29.5
1,0
Multiodal GPT
$1.01
45.5
9.5
7.0
0.5
45.5
45.0
0.0
65.0
$1.5
32.0
5.0
0100
Ours
100.0
100.0
100.0
100
61.0
100.0
100.0
16.5
75.5
75.0
96.0
37.5
47.5
20M
Flamingo
Gato
48.5
4411)
51.5
49.0
55.5
39.0
$1.0
48.0
42.5
38.5
46.5
43.5
5255
$2.0
6.0
6.0
65.5
66.0
84.0
80.0
52.5
47.5
4015
37.0
0.5
1.0
Multimdaal GPT
50.5
49.5
$3.0
44.5
43.5
47.0
46.0
8.0
83.5
30.0
46.5
41.0
2.5
Ours
100.0
100.0
99.5
100
54.5
100.0
100.0
17.5
77.0
93.0
98.5
75.0
45.0
200M
Flaningo
Gato
56.5
$1.0
52.5
53.5
88.0
61.5
55.5
19.5
18.5
43.5
47.5
55.5
53.0
55.5
14.0
5.5
63.0
70.5
$2.0
90.5
$1.5
42.0
33.0
39.0
4,0
3.0
Multinodial GPT
52.0
52.0
49.5
54.5
45.5
52.5
51.0
11.0
76.5
64.0
43.0
38.0
01.5
Table 12: L3 level generalization results. Model indicates robot controller parameter count.
Model
Method
Task D1
Tick 02
Task 03
Tisk 04
Tisk 05
Task 06
Tisk 07
Tisk (9)
Tisk 11
Tisk 15
Task 16
Tisk 17
Ours
100.0
100
100
98.0
34.5
100.0
99.55
17.0
97.5
94.00
48.5
39.0
48
Multimodal GPT
48.5
50.0
3.0
7.0
45.5
2.0
69.5
22.5
3.0
Ours
98.0
100.
100
98.5
55
100.0
99.5
15.0
88.5
99
44.0
29.5
20M
Flamingo
Gato
47.0
46.5
51.5
55
53.0
44.5
55,0
$7.0
31.5
36
42.5
47.5
51.5
48.0
2.5
6.5
70.0
72.5
33.0
30.5
44.0
41.5
Multiodal
Ours
100
100
97.0
54.5
99.0
17.5
80.5
97.5
0
43.5
200M
Flamingo
Gato
89.0
51.0
50.0
58
66.5
$4.5
47.0
56.5
35.5
35
53.5
47.5
49.0
50.0
15.0
4.0
65.0
66.0
52.0
30.5
43.5
0.5
Multimodal GPT
52.0
51.0
55.0
49.5
10.0
50.5
5.0
37.0
38.0
Table 13: L4 level generalization results. Model indicates robot controller parameter count.
Model
Method
Task 08
Task
Task
Task 14
Ours
6.5
0
916.
Gato
21.0
0.5
32
2M
Flamingo
22.0
27.5
Multimodal GPT
22.5
0.0
22.0
95.5
20M
Flamingo
Gato
21.0
20.5
0.0
27.5
29
Multimodal
20.5
0.5
36.0
Ours
100.0
94.5
200M
30.5
Flamingo
24.5
0
24.0
Multimodal GPT
20.0
0.0
28.5
1304
Imperfect Prompts. We then study the policy robustness against imperfect prompts, including incom-
1305
plete prompts (randomly masking out words with <UNK> token) and corrupted prompts (randomly swapping words, which could have changed the task meaning altogether). We our largest VIMA model with 200M parameters, results shown in Table 16.
1306
1307
40

---

Under review as a conference paper at ICLR 2023
Table 14: Performances of our method with different sized pre-trained T5 prompt encoder, We fix the parameter count of the decision-making part to be 200M.
10-small (30M)
t5-base (11IM)
t5 25-large (368M)
L1
78.8
81.5
80.8
I.2
79.0
81.5
81.0
L3
80.3
78.7
81.0
L4
49.1
48.6
49.3
Table 15:
Evaluation results on tasks with increased amounts of distractors. We fix the parameter count of the decision-making part to be 200M.
L1
L2
L3
L4
Original
81.5
81.5
78.7
48.6
More Distractors
78.5
78.6
72.9
47.8
Relevant Performance Decrease (%)
3.6
3.5
7.3
1.6
1308
Our well-trained model exhibits minimal performance decrease when evaluated on masked prompts and minor decrease on corrupted prompts. We attribute this robustness to the high-quality, pre-trained
1309
1310
T5 language backbones.
Table 16: Evaluation results with incomplete and corrupted prompts. We fix the parameter count of the decision decision-making making part be 200M
L1
L.2
L3
L4
Original
81.5
81.5
78.7
48.6
Incomplete Prompts
80.8
81.1
77.0
48.0
Corrupted Prompts
78.2
78.1
73.8
45.3
Relevant Performance Decrease w/ Incomplete Prompts (%)
0.8
0.4
2.1
1.2
Relevant Performance Decrease w/ Corrupted Prompts (%
4.3
7.2
F EXTENDED RELATED WORK
:311
1312
In this section, we provide an extended review of related work as complementary to Section
1313
Multi-task Learning by Sequence Modeling. In NLP domain, the Natural Language De- format for a suite of 10 NLP tasks. In computer vision, Mask R-CNN (He et al., 2017). UberNet (Kokkinos, 2016), and 12- in-1 (Lu et al., 2020) leverage single backbone model with multiple independent heads for different tasks. UVim (Kolesnikov et al., 2022) another unified approach for vision that uses a language model to generate the guiding code for second model to predict raw vision outputs. In multimodal learning, numerous works (Lu et al., 2022; Wang et al., 2022a; Zellers et al., 2021; 2022; Buch et al.,
1314
cathlon (McCann et al., 2018) adopts a consistent
1315
1316
1317
1318
1319
1320
2022; Fu et al., 2021; Yang et al., 2022) investigate the unification of image, video, audio. and/or lan-
1321
guage modalities to deliver multi-purpose foundation models, though most of which are not equipped with decision-makee making facilities. Perceivers (Jaegle al., 2021 propose efficient architecture to handle general-purposs inputs and outputs. BEÎT-3 (Wang et al., 2022c) performs masked data modeling images, texts and image-text pairs pre-train backbone for various downstream tasks.
1323
1324
1325
MetaMorph (Gupta et al., 2022a) learns a universal controller over modular robot design space.
1326
Foundation Models for Embodied Agents. Embodied agent research (Duan et al., 2022; Batra et al., 2020; Ravichandar al., 2020; Collins et al.. 2021) is adopting the large-scale pre-training paradigm, powered by a collection of learning environments (Abramson et al., 2020; Shridhar al,, 2020; Savva al., 2019; Puig al., 2018; Team et al., 2021; Toyama al., 2021; Shi et al.,
1327
1328
1329
41

---

Under review as a conference paper at ICLR 2023
1330
2017). From the aspect of pre-training for better representations, LaTTe (Bucker et al., 2022) and
1331
Mabodied-CLIIP (Khandelwal et al., 2021) 1) leverage the frozen visual and textual representations of CLIP (Radford et al., 2021) for robotic manipulation. From the perspective of leveraging transformer
1333
as agent architecture, methods such Dasari & Gupta (2020) and MOSAIC (Zhao et al., 2022)
1334
achieve superior performance in one-shot video imitation tasks, They both use the self-attention
1335
mechanism with auxiliary losses such as inverse dynamics loss (Dasari & Gupta, 2020) and
1336
contrastive loss (Zhao et al., 2022) to learn robot controllers, Our work differs from them mainly in three aspects: a) our method employs a transformer backbone to autoregressively predict actions: b) we utilize pre-trained language models (Raffel et al., 2020) and best practices from simpoukelli et al. (2021) to learn policies conditioned on prompts with interleaved texts, images, and even videos; and e) while these works mainly focus on solving the single task of one-shot video imitation with highly customized objectives, conceptually simple but effective, our model is learned in multi-task way with only the behavior cloning objective to solve a strict superset of tasks,
1337
1338
1339
1340
1341
1342
1343
mentioned in the main paper that study different robotic manipulation tasks, such as constraint satisfaction (Bharadhwaj al., 2021), one-shot imitation (Paine et al., 2018; Huang et al., 2019; Aceituno al., 2021;
Robot Manipulation and Benchmarks. There are many prior works that
134
1345
1346
Zhao et al., 2022). and rearrangement (Liu et al., 2021; Ehsani et al., 2021; Gan et al., 2021;
1347
Stengel-Eskin et al., 2022). Multiple simulation benchmarks are introduced to study the above tasks: 1) Indoor simulation environments: Habitat (Savva et al., 2019; Szot et al., 2021) is
1348
1349
equipped with a perforaeform 3D simulator for fast rendering and proposes a suite of common tasks for assistive robots. A12-THOR (Ehsani et al., 2021; Deitke et al., 2022) is framework that supports visual object manipulation and procedural generation of environments. 2) Tabletop environments: Meta-Worid (Yu et al., 2019), RLBench (James et al., 2019), and SURREAL (Fan et al., 2018; 2019) are widely used simulator benchmarks studying robotics manipulation with tabletop settings. CausalWorld (Ahmed et al., 2021) is benchmark for causal structure and transfer learning in manipulation, requiring long-horizon planning and precise low-level motor control.
1350
1351
1352
1359
1354
1355
1356
MOSAIC (Zhao et al., 2022) features a challenging benchmark built on top of Zhu et al. (2020)
1357
to evaluate one-shot imitation learning. It proposes three-step test setting to evaluate the representational and generalization capability. Compared to it, ours supports wide spectrum of manipulation tasks, including one-shot imitation learning. All these aforementioned simulators and
1358
1359
1380
benchmarks do not natively support task specification and prompting with multiple modalities.
42


Reviews:
[{"review_id": "xeZOheLRFf", "reviewer": "Reviewer_xPBt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a technique to create a robot control agent that can control a manipulator to execute a variety of tasks specified in the form of a multi-model prompt, combining text and visual tokens. \n\nThe paper also introduces a large scale benchmark for problems focusing on the properties of the system VIMA-Bench.  \n", "summary_of_the_review": "The paper describes an impressive project, with a large number of demonstrations. However, it does not clearly focuses on what the novelty of the project is. \n\n----\n\nI have read the responses of the authors. They do not change my rating of the paper. ", "strength_and_weaknesses": "+ The paper reports on an impressive amount of work on the development of the agent and the benchmark. The 20+ pages of appendices are an impressive quantity of work. \n- The paper proper only spends about 2/3 of a page (section 5) describing the model. This is unsufficient for understanding what exactly the contribution is. \n- As a robot manipulation problem, the tasks are very simple. These are pick and place tasks, performed on a flat surface, in simulation, with objects that are clearly distinguishable. \n- The paper does not explain why it considers that the multi-model prompt is superior. Are the prompts supposed to be generated by humans? Humans don't insert visual snippets or texture samples in the speech. \n- The tasks in the tasks suite are specified in ways that are much simpler than what is the state of the art in this field. For instance, \"novel councept grounding\" appears to be just attaching a label to a definition. \"one shot video imitiation\" appears to be the exact reproducing of an exact trajectory - a much weaker definition that what is normally meant by imitation learning. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VbI54-Uzzv", "reviewer": "Reviewer_nbxK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents VIMA, a robot agent that can act conditioned on multimodal prompts consisting of natural language and image objects. The architecture consists of self-attention and cross-attention layers and train the model to minimize behavior cloning loss. The main novelty of the paper is to propose a scheme for multimodal prompting, and the paper shows that the proposed approach can work well for various tasks with a single model.", "summary_of_the_review": "The paper is written well, provides a thorough investigation of the proposed architecture in a very wide setups. I don't see major concerns with the paper but I'm reluctant to further increase my score as all the results are based on a newly introduced benchmarks and baselines which are not exactly clear how they are implemented. I'm willing to change my score up or down after the discussion during the rebuttal phase.", "strength_and_weaknesses": "### Strengths\n- Clear writing with nicely looking illustrations\n- Exhaustive experiments\n- Introducing an interesting benchmark\n\n### Weaknesses\n- Dependency on a separate object detection module. While it's shown to be effective as using oracle information in considered setups, it's not clear whether this can work in real-world scenarios or non-tabletop tasks as the paper promises to be a generalist robot agent. To add on this, I don't see this as a big weakness but it could be an interesting future direction to consider.\n- Interesting results and analysis on a newly introduced benchmark. But comparison with the existing approaches [1,2] on other benchmarks [3] could further strengthen the claims made in the paper.\n- MIssing details on baselines. From only reading the paper, it's a bit difficult to understand what's the main difference of the approach to considered baselines without reading the baseline papers in a detail.\n- on Decision Transformer -- Trajectory Transformer [Janner et al., 2021] significantly differs from Decision Transformer [Chen et al., 2021] but the paper writes this as Decision Transformer (DT) (Chen et al., 2021; Janner et al., 2021). This should be corrected. Moreover, the main component of DT is to condition on desired returns and make the agent select the desired actions -- without this, the method that conditions on multimodal prompt cannot be called DT. It's just a behavior cloning agent conditioned on multimodal prompt with the GPT architecture. \n- on cross-attention -- it's worthwile to cite relevant video prediction literature [4] that also utilizes cross-attention to reduce the compute cost when conditioning on previous frames (it seems that the architecture is almost the same, or very similar).\n\n[1] Dasari, Sudeep, and Abhinav Gupta. \"Transformers for one-shot visual imitation.\" In Conference on Robot Learning, pp. 2071-2084. PMLR, 2021.\n\n[2] Mandi, Zhao, Fangchen Liu, Kimin Lee, and Pieter Abbeel. \"Towards more generalizable one-shot visual imitation learning.\" In 2022 International Conference on Robotics and Automation (ICRA), pp. 2434-2444. IEEE, 2022.\n\n[3] Zhu, Yuke, Josiah Wong, Ajay Mandlekar, and Roberto Martín-Martín. \"robosuite: A modular simulation framework and benchmark for robot learning.\" arXiv preprint arXiv:2009.12293 (2020).\n\n[4] Yan, Wilson, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. \"Videogpt: Video generation using vq-vae and transformers.\" arXiv preprint arXiv:2104.10157 (2021).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GA1nCk4H8b", "reviewer": "Reviewer_L5UP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an agent architecture called VIMA (for **Vi**suo**M**otor **A**ttention model), which is a standard transformer encoder-decoder pair. Multimodal inputs are encoded as a single sequence of interleaved text tokens and visual object tokens. Transformer outputs are mapped to actions, as in the Decision Transformer (DT). VIMA is evaluated on a new multi-modal benchmark, VIMA-Bench, consisting of 17 meta-tasks, and thousands of tabletop task instantiations arranged in a 4-level protocol of increasing difficulty. Baseline models (including Gato and DT) are also evaluated on VIMA-Bench. The results show that VIMA outperforms the baselines by consistently large margins.", "summary_of_the_review": "VIMA-Bench is potentially useful. But because of the flaws in VIMA's comparisons to baseline agents, little can be concluded about VIMA's capabilities.\n\n***** POST-DISCUSSION UPDATE *****\n\nIn their last response to me, the authors said:\n\n**\"Among these variants, we find that the recipe of object token + cross-attention is the best for multimodal-prompted robot agents through extensive experiments.\"**\n\n\nThis is a nice summary of the work’s key experimental finding. My concern is that both parts of this recipe are unremarkable:\n- The object tokens come from a model (Mask R-CNN) trained on this task’s ground-truth state information, making it unlikely to scale well beyond VIMA-Bench. This limitation makes the object tokens a crutch, and readers already know that such crutches boost performance. \n- A long line of prior work, starting with the original transformer paper of 2017, has shown that cross-attention is a powerful method for processing complex input spaces. Since VIMA-Bench’s input space is complex by construction, readers shouldn’t be surprised to see that cross-attention performs well here.\n\nI still see value in VIMA-Bench itself, with its diversity of tasks. In fact, I would argue for acceptance of a *VIMA-Bench* paper submitted to a datasets and benchmarks track, as long as the models evaluated (including VIMA itself) were presented simply as baselines that the community could compare against in further work. In such a light, VIMA’s reliance on an unscalable vision processing pipeline would not pose a problem, and its usage of cross-attention would simply provide experimental verification of performance one might expect. \n\nBut instead of focusing on VIMA-Bench, this paper focuses on VIMA as a model for general robot manipulation, starting with the title itself:  **”VIMA: General Robot Manipulation…”** This is why I have to leave my scored unchanged, despite all the impressive work that has gone into the paper.", "strength_and_weaknesses": "**Strengths**\n\nVIMA-Bench is a flexible and useful environment for evaluation of multi-modal prompting, scalability, and generalization in a robotic setting. Interleaving language and image tokens in a single input sequence has shown recent promise, as with the Gato agent. \n\n**Weaknesses**\n\nVIMA’s performance is reliant on three critical design features:\n\n- Visual **object tokens** from an off-the-shelf object detector.\n- **Cross-attention** to the prompt sequence.\n- A pre-defined set of **high-level actions**.\n\nThe **object tokens** are a crutch, giving VIMA effective access to the underlying state space, while baseline models are not given similar access. This disparity is more than enough to explain VIMA’s improved results over the baselines, and makes the comparisons unfair. The paper says “We hypothesize that the data efficiency can be attributed to VIMA’s object-centric representation, which is less prone to overfitting than learning directly from pixels in the low-data regime.” But it’s common knowledge that state representations allow greater sample efficiency than raw image observations. So there is nothing novel or surprising about taking advantage of object-level representations when available.\n\n**Cross-attention** is one standard method of inputting a prompt sequence. Another standard method is to prepend the prompt to the single sequence, and rely on self-attention. Cross-attention is most appropriate when the prompt sequence (as in machine translation) contains a wealth of potentially variable information that strongly determines the correct output sequence. Recent models like Imagen have leveraged cross-attention in this way. Self-attention (over a single sequence) is most appropriate when the prompt sequence is less informative, and serves simply to identify the current task from among a trained set of tasks. Recent models like Gato have leveraged self-attention in this way. By construction, prompts in VIMA-Benchmark are highly variable, and strongly determine the correct outputs, and therefore call for cross-attention. Gato and DT were originally designed for the other type of prompt, and cannot be expected to perform as well on VIMA-Benchmark. But cross-attention could easily be added to Gato or DT. In fact, VIMA’s transformer architecture is equivalent to DT with cross-attention. There is nothing novel or surprising about taking advantage of cross-attention when it is called for.\n\nVIMA’s **high-level actions** are severely limited to a pre-defined set including “pick-and-place” and “wipe”. The baseline agents are given access to the same high-level actions, so fair comparisions are at least possible. But the results are not indicative of how the agents would perform on more common low-level actions, like robotic joint torques. This makes the term “generalist robot agent” in the paper’s title especially inappropriate.\n\nVIMA’s performance depends on these three components. The paper itself notes that good results “can only be achieved with both cross-attention and object token sequence representation”. So given the strong caveats explained above, the experimental results presented in the paper provide no reliable evidence for how VIMA compares to the baseline models. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "1bCwnlOhLhR", "reviewer": "Reviewer_DxBB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This research develops a novel multimodal , multitask architecture for robot manipulation. Also introduces a multimodal and multitask benchmark for robot manipulation. Finally introduce a carefully thought methodology in 4 steps to quantify the capability of the developed architecture.  The results section provide relevant ablation studies that bring insights on this line of research, providing ideas to further improve this research domain.", "summary_of_the_review": "A large effort on developing a novel human robot interaction interface on robotics manipulation tasks that generalize. The work provide a relevant datase, new methodology to measure generalization, and novel models to solved this multitask and multimodal tasks.", "strength_and_weaknesses": "Strengths:\n1) Novel transformer architecture for robot manipulation.\n2) Novel multimodal interface for robot human interaction.\n3) Novel insight on improved performance on using object centric approaches for this tasks.\n4) New dataset for multimodal and multitask robot manipulation.\n5) Novel methodology to measure generalization in robot manipulation.\n6) Novel human robot interaction modality.\nWeaknesses:\n1) Unclear how this method will generalize to temporal extended actions : i.e, working the dough, beating eggs, writing, folding clothes, etc.\n2) Finteunning Mask-RCNN seems that might not be practical if we want to extend this method to work in-the-wild. What's the accuracy for a non-finetune Mask-RCNN?\n3) What will happen if the user change his mind after saying the command? \n4) How is uncertainty deal in this model? How could be resolved?\n5) What would happen if you have a cluttered scene and the number of object grow and there some that irrelevant to task?\n6) What would happened if there is something incomplete, or wrong in the text command?\n7) It could be interesting if also the model can output text to resolved some of the aforementioned items.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject