VIMA: General Robot Manipulation with Multimodal Prompts

Yunfan Jiang 1 Agrim Gupta 1 † Zichen Zhang 2 † Guanzhi Wang 3 4 † Yongqiang Dou 5 Yanjun Chen 1
Li Fei-Fei 1 Anima Anandkumar 3 4 Yuke Zhu 3 6 ‡ Linxi Fan 3 ‡Abstract

the same model to perform question answering, machinetranslation, text summarization, etc. Prompt-based learning
provides an accessible and flexible interface to communicate
a natural language understanding task to a general-purpose
model.

Prompt-based learning has emerged as a successful paradigm in natural language processing,where a single general-purpose language model
can be instructed to perform any task specified by
input prompts. Yet task specification in robotics
comes in various forms, such as imitating oneshot demonstrations, following language instructions, and reaching visual goals. They are often
considered different tasks and tackled by specialized models. We show that a wide spectrum ofrobot manipulation tasks can be expressed withmultimodal prompts, interleaving textual and visual tokens. Accordingly, we develop a new simulation benchmark that consists of thousands ofprocedurally-generated tabletop tasks with multimodal prompts, 600K+ expert trajectories forimitation learning, and a four-level evaluation
protocol for systematic generalization. We design a transformer-based robot agent, VIMA, thatprocesses these prompts and outputs motor actions autoregressively. VIMA features a recipe
that achieves strong model scalability and data
efficiency. It outperforms alternative designs in
the hardest zero-shot generalization setting by up
to 2.9× task success rate given the same training data. With 10× less training data, VIMA
still performs 2.7× better than the best competing
variant. Code and video demos are available atvimalabs.github.io.

We envision that a generalist robot should have a similarly intuitive and expressive interface for task specification. What
does such an interface for robot learning look like? As amotivating example, consider a personal robot tasked withhousehold activities. We can ask the robot to bring us a cup
of water by a simple natural language instruction. If werequire more specificity, we can instead instruct the robot to“bring me <image of the cup>”. For tasks requiring
new skills, the robot should be able to adapt, preferablyfrom a few video demonstrations (Duan et al., 2017). Tasks
that need interaction with unfamiliar objects can be easily explained via a few image examples for novel concept
grounding (Hermann et al., 2017). Finally, to ensure safe
deployment, we can further specify visual constraints like“do not enter <image> room”.
To enable a single agent with all these capabilities, we makethree key contributions in this work: 1) a novel multimodal
prompting formulation that converts a wide spectrum ofrobot manipulation tasks into one sequence modeling problem; 2) a large-scale benchmark with diverse tasks tosystematically evaluate an agent’s scalability and generalization; and 3) a multimodal-prompted robot agent capableof multi-task and zero-shot generalization.
We start with the observation that many robot manipulation tasks can be formulated by multimodal prompts
that interleave language and images or video frames(Fig. 1). For example, Rearrangement (Batra et al., 2020),
a type of Visual Goal, can be formulated as “Please rearrange objects to match this {scene image}”; Fewshot Imitation can embed video snippet in the prompt“Follow this motion trajectory for the wooden cube:{frame1 }, {frame2 }, {frame3 }, {frame4 }”. Multimodal prompts not only have more expressive power than
individual modalities but also enable a uniform sequence
IO interface for training generalist robots. Previously, different robot manipulation tasks required distinct policy architectures, objective functions, data pipelines, and training

1. Introduction
Transformer models (Vaswani et al., 2017) have given rise
to remarkable multi-task consolidation across many AI domains. For example, users can describe a task using natural
language prompt to GPT-3 (Brown et al., 2020), allowing
1
Stanford University; 2 Macalester College, now at Allen Institute for AI; 3 NVIDIA; 4 Caltech; 5 Tsinghua; 6 UT Austin. Workdone during the first author’s internship at NVIDIA. †: Equal
contribution. ‡: Equal advising.

1

VIMA: General Robot Manipulation with Multimodal Prompts
Visual Goal: Rearrangement

Rearrange objects to match this scene:One-shot Demonstration

VIMA
Follow this motion for:Novel Concept Grounding

This is a blicket

This is a wug

Put a wug into a blicket

Text token
Object token

Visual Constraint

Sweep allinto

Padding

without exceeding

Figure 1: Multimodal prompts for task specification. We observe that many robot manipulation tasks can be expressed as
multimodal prompts that interleave language and image/video frames. We introduce VIMA, an embodied agent capable ofprocessing mulitimodal prompts (left) and controlling a robot arm to solve the task (right).
procedures (Aceituno et al., 2021; Stengel-Eskin et al., 2022;
Lynch & Sermanet, 2021), leading to siloed robot systems
that cannot be easily combined for a rich set of use cases. Instead, our multimodal prompt interface allows us to harness
the latest advances in large transformer models (Lin et al.,2021; Tay et al., 2020; Khan et al., 2021) for developing
scalable multi-task robot learners.

alternate with the usual causal self-attention. Instead ofoperating on raw images, VIMA adopts an object-centric
approach. We parse all images in the prompt or observation into objects by off-the-shelf then domain fine-tuned
detectors (He et al., 2017) and flatten them into sequencesof object tokens. To demonstrate the scalability of VIMA,
we train a spectrum of 7 models ranging from 2M to 200M
parameters. Our approach outperforms other design alternatives, such as image patch tokens (Reed et al., 2022), image
Perceiver (Jaegle et al., 2021b; Alayrac et al., 2022), anddecoder-only conditioning (Radford et al., 2018). VIMA
obtains consistent performance gains across all four levelsof zero-shot generalization and all model capacities, in somecases by a large margin (up to 2.9× task success rate given
the same amount of training data, and 2.7× better evenwith 10× less data). We open-source the simulation environment, training dataset, algorithm code, and pre-trainedmodel checkpoints to ensure reproducibility and facilitate future work from the community. These materials along withvideo demos are available at vimalabs.github.io.

To systematically evaluate agents with multimodal prompts,
we develop a new benchmark, named VIMA-B ENCH, built
on the Ravens simulator (Zeng et al., 2020; Shridhar et al.,2021). We provide 17 representative tasks with multimodal
prompt templates. Each task can be procedurally instantiated into thousands of instances by various combinations of
textures and tabletop objects. VIMA-B ENCH establishes afour-level protocol to evaluate progressively stronger generalization capabilities, from randomized object placement tonovel tasks (Fig. 2).
To this end, we introduce the VisuoMotor Attention agent(VIMA) to learn robot manipulation from multimodal
prompts. The model architecture follows the encoderdecoder transformer design proven to be effective and scalable in NLP (Raffel et al., 2020). VIMA encodes an input sequence of interleaving textual and visual prompt tokens witha pre-trained language model (Tsimpoukelli et al., 2021)
and decodes robot control actions autoregressively for eachenvironment interaction step. The transformer decoder isconditioned on the prompt via cross-attention layers that

2. Multimodal Prompts for Task Specification
A central and open problem in robot learning is task specification (Agrawal, 2022). In prior literature (Stepputtis et al.,2020; Dasari & Gupta, 2020; Brunke et al., 2021b), different tasks often require diverse and incompatible interfaces,
resulting in siloed robot systems that do not generalize well2

VIMA: General Robot Manipulation with Multimodal Prompts
Stronger Generalization

Training
Put theinto theLevel 1

Object Placement
Put theinto theLevel 2

Level 3Novel Combination
Put theinto theNovel Object
Put theinto theLevel 4Novel Task
Put all objects with the same texture as

into itFigure 2: Evaluation Protocol in VIMA-B ENCH. We design 4 levels of evaluation settings to systematically measure thezero-shot generalization capability of an agent. Each level deviates more from the training distribution, and thus is strictly
more challenging than the previous level.
across tasks. Our key insight is that various task specification paradigms (such as goal conditioning, video demonstration, natural language instruction) can all be instantiatedas multimodal prompts (Fig. 1). Concretely, a multimodal
prompt P of length l is defined as an ordered sequence of ar
bitrarily interleaved texts and images P := x1 , x2 , . . . , xl ,where each element xi ∈ {text, image}.

Note that these six categories are not mutually exclusive.
For example, a task may introduce a previously unseenverb (Novel Concept) by showing a video demonstration, or
combine goal reaching with visual reasoning. More details
about the task suite are discussed in Appendix, Sec. B.

3. VIMA-B ENCH: Benchmark forMultimodal Robot Learning

Task Suite. The flexibility afforded by multimodal
prompts allows us to specify and build models for avariety of task specification formats. Here we consider thefollowing six categories.

Simulation Environment. Existing benchmarks are
generally geared towards a particular task specification.
To our knowledge, there is no benchmark that provides arich suite of multimodal tasks and a comprehensive testbed
for targeted probing of agent capabilities. To this end, we
introduce a new benchmark suite for multimodal robot
learning called VIMA-B ENCH. We build our benchmark
by extending the Ravens robot simulator (Zeng et al.,2020). VIMA-B ENCH supports extensible collections ofobjects and textures to compose multimodal prompts and toprocedurally generate a large number of tasks. Specifically,
we provide 17 tasks with multimodal prompt templates,
which can be instantiated into thousands of task instances.
Each task belongs to one or more of the 6 task categories
mentioned above. VIMA-B ENCH can generate large
quantities of imitation learning data via scripted oracle
agents. More details are elaborated in Appendix, Sec. A.

1. Simple object manipulation. Simple tasks like “put<object> into <container>”, where each image
in the prompt corresponds to a single object;
2. Visual goal reaching. Manipulating objects to reach agoal configuration, e.g., Rearrangement (Batra et al.,2020);
3. Novel concept grounding. The prompt contains unfamiliar words like “dax” and “blicket”, which are
explained by in-prompt images and then immediately
used in an instruction. This tests the agent’s ability to
rapidly internalize new concepts;
4. One-shot video imitation. Watching a video demonstration and learning to reproduce the same motion
trajectory for a particular object;

Observation and Actions. The observation space of oursimulator includes RGB images rendered from both frontal
view and top-down view. Ground-truth object segmentation
and bounding boxes are also provided for training objectcentric models (Sec. 4). We inherit the high-level action
space from Zeng et al. (2020), which consists of primitive
motor skills like “pick and place” and “wipe”. These areparameterized by poses of the end effector. Our simulator
also features scripted oracle programs that can generate
expert demonstrations by using privileged simulator state information, such as the precise location of all objects, and theground-truth interpretation of the multimodal instruction.

5. Visual constraint satisfaction. The robot must manipulate the objects carefully and avoid violating the(safety) constraints;
6. Visual reasoning.
Tasks that require reasoning skills, such as appearance matching “moveall objects with same textures as <object>into <container>”, and visual memory “put<object> in <container> and then restore to
their original position”.
3VIMA: General Robot Manipulation with Multimodal Prompts
Text Token
Object token
Action token

Self-Attention
Cross-Attention

Self-Attention
Cross-Attention
History Tokens

Prompt Tokens

T5

Sweep all

Object Encoder

Object Encoder

Object Encoder

into

without touching

Multimodal Prompt

Interaction

Figure 3: VIMA Architecture. We encode the multimodal prompts with a pre-trained T5 model, and condition the
robot controller on the prompt through cross-attention layers. The controller is a causal transformer decoder consisting ofalternating self and cross attention layers that predicts motor commands conditioned on prompts and interaction history.
Training Dataset. We leverage oracles to generate a large
offline dataset of expert trajectories for imitation learning.
Our dataset includes 50K trajectories per task, and 650K
successful trajectories in total. We hold out a subset ofobjects and textures for evaluation and designate 4 out of
17 tasks as a testbed for zero-shot generalization.

3. Novel object generalization. Test prompts and thesimulated workspace include novel textures and objects;
4. Novel task generalization. New tasks with novel
prompt templates at test time.

Evaluating Zero-Shot Generalization. Each task in
VIMA-B ENCH has a binary success criterion and does notprovide partial reward. During test time, we execute agent
policies in the simulator for multiple episodes to computea percentage success rate. The average success rate over
all evaluated tasks will be the final reported metric.

4. VIMA: Visuomotor Attention AgentOur goal is to build a robot agent capable of performing
any task specified by multimodal prompts. There is no
prior method that works out of the box with multimodal
prompts. To learn an effective multi-task robot policy, we
propose VIMA, a robot agent with a multi-task encoderdecoder architecture and object-centric design (Fig. 3).
Concretely,
we learn a robot policy π(at |P, H), whereH := o1 , a1 , o2 , a2 , . . . , ot denotes the past interaction
history, and ot ∈ O, at ∈ A are observations and actions
at each interaction steps. We encode multimodal prompts
via a frozen pre-trained language model and decode robot
waypoint commands conditioned on the encoded prompts
via cross-attention layers. Unlike prior work (Florence et al.,2019; Sieb et al., 2019; Zhu et al., 2022), VIMA adopts
an object-centric representation that computes tokens frombounding box coordinates and cropped RGB patches.

We design a four-level evaluation protocol (Fig. 2) to systematically probe the generalization capabilities of learned
agents. Each level deviates more from the training distribution, and is thus strictly harder than the previous one.
1. Placement generalization. All prompts are seen verbatim during training, but only the placement of objects
on the tabletop is randomized at testing;
2. Combinatorial generalization. All textures and objects are seen during training, but new combinations ofthem appear in testing;4

VIMA: General Robot Manipulation with Multimodal Prompts
Model Scalability

L2

L1

L3

L4

80

80

80

80

60

60

60

60

40

40

40

40

20

20

20

20

0

0
2

4

8

32

16

64 128 256

02Model Size (M)

4

8

16

32

0

64 128 256

2Model Size (M)

8

32

16

64 128 256

2Model Size (M)Data Scalability

L2

L1

4

80

80

60

60

60

60

40

40

40

40

20

20

20

20

10

Data Size

100
6 x 10²

Ours

10³

10

Data Size0
6 x 10²

10

VIMA-Gato

16

10³

10

Data Size

VIMA-Flamingo

32

64 128 256

L4

80

10³

8

Model Size (M)

L3

80

0
6 x 10²

4

10

0
6 x 10²

10³

10Data Size

10

VIMA-GPT

Figure 4: Scaling model and data. Top: We compare performance of different methods with model sizes ranging from 2M
to 200M parameters. Across all model sizes and generalization levels, VIMA outperforms baseline variants. Bottom: For a
fixed model size of 92M parameters we compare the effect of imitation learning dataset size with 0.1%, 1%, 10%, and full
data. VIMA is extremely sample efficient and can achieve performance comparable to other methods with 10× less data.
history sequence H. We compute key KP and value VP
sequences from the prompt and query QH from the trajectory history, following the encoder-decoder convention
in Raffel et al. (2020). Each cross-attentionthen layer
⊺ 
QH KP
′
√
generates an output sequence H = softmax
VP ,dwhere d is the embedding dimension. Residual connections
are added to connect higher layers with the input rollout
trajectory sequence. The cross-attention design enjoys threeadvantages: 1) strengthened connection to prompt; 2) intact
and deep flow of the original prompt tokens; and 3) better
computational efficiency. VIMA decoder consists of L
alternating cross-attention and self-attention layers. Finally,
we follow common practice (Baker et al., 2022) to map
predicted action tokens to discretized poses of the robot
arm. See Appendix, Sec. C.2 for more details.

Tokenization. There are 3 formats of raw input in theprompt — text, image of a single object, and image of afull tabletop scene (e.g., for Rearrangement or imitationfrom video frames). For text inputs, we use pre-trained T5
tokenizer and word embedding to obtain word tokens. For
images of full scenes, we first extract individual objects
using domain fine-tuned Mask R-CNN (He et al., 2017)(Appendix, Sec. C.4). Each object is represented as a bounding box and a cropped image. We then compute object
tokens by encoding them with a bounding box encoder anda ViT (Dosovitskiy et al., 2020), respectively. Since Mask
R-CNN is imperfect, the bounding boxes can be noisy andthe cropped images may have irrelevant pixels. For imagesof single objects, we obtain tokens in the same way except
with a dummy bounding box. Prompt tokenization produces
a sequence of interleaved textual and visual tokens. We thenfollow the practice in Tsimpoukelli et al. (2021) and encode
the prompt via a pre-trained T5 encoder (Raffel et al., 2020).
Since T5 has been pre-trained on large text corpora, VIMA
inherits the semantic understanding capability and robustness properties. To accommodate tokens from new modalities, we insert MLPs between non-textual tokens and T5.

Training. We follow behavioral cloning to train ourmodels by minimizing the negative log-likelihood ofpredicted actions. Concretely,
for a trajectory with T steps,PT
we optimize minθ t=1 − log πθ (at |P, H). The entire
training is conducted on an offline dataset with no simulator
access. To make VIMA robust to detection inaccuracies andfailures, we apply object augmentation by randomly injecting false-positive detection outputs. After training, we selectmodel checkpoints for evaluation based on the aggregatedaccuracy on a held-out validation set. The evaluation involves interacting with the physics simulator. We follow the

Robot Controller. A challenging aspect of designinga multi-task policy is to select a suitable conditioning
mechanism. In our schema (Fig. 3), the robot controller(decoder) is conditioned on the prompt sequence P by aseries of cross-attention layers between P and the trajectory
5

VIMA: General Robot Manipulation with Multimodal Prompts
0best practices to train Transformer models. See Appendix,
Sec. D for comprehensive training hyperparameters.
Performance Drop (%)

10

5. Experiments
In this section, we aim to answer three main questions:
1. What is the best recipe for building multi-task
transformer-based robot agents with multimodal
prompts?

20
30
40
50
60
70

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT
L1 → L2

L1 → L3

L1 → L4

2. What are the scaling properties of our approach inmodel capacity and data size?

Figure 5: VIMA incurs much less performance drop thanbaselines as we evaluate on progressively harder settings.

3. How do different components, such as visual tokenizers, prompt conditioning, and prompt encoding, affectrobot performance?

5.2. Evaluation Results
We compare VIMA against the baseline variants on fourlevels of generalization provided in our benchmark for different model and training dataset sizes. Our empirical results
demonstrate that VIMA’s choice of object tokens combinedwith cross-attention conditioning is the most effective recipe
among the model designs we consider.

5.1. Baselines
Because there is no prior method that works out of the box
with our multimodal prompting setup, we make our best
effort to select a number of representative transformer-based
agent architectures as baselines, and re-interpret them to becompatible with VIMA-B ENCH:

Model Scaling. We train all methods for a spectrum ofmodel capacities from 2M to 200M parameters, evenly
spaced on the log scale (Fig. 4). The encoder size is kept
constant (T5-Base, 111M) for all methods and excluded
from the parameter count. Across all levels of zero-shot
generalization, we find that VIMA strongly outperforms
other alternatives. Although models like VIMA-Gato
and VIMA-Flamingo show improved performance withbigger model sizes, VIMA consistently achieves superior
performance over all model sizes. We note that this canonly be achieved with both cross-attention and object token
sequence representations — altering any component will
significantly degrade the performance, especially in the low
model capacity regime (ablations in Sec. 5.3).

Gato (Reed et al., 2022) introduces a decoder-only modelthat solves tasks from multiple domains where tasks are
specified by prompting the model with the observation andaction subsequence. For a fair comparison, we provide thesame conditioning as VIMA, i.e., our multimodal encoded
prompts. Input images are divided into patches and encoded
by a ViT model to produce observation tokens. This variant
is referred to as “VIMA-Gato”.
Flamingo (Alayrac et al., 2022) is a vision-language modelthat learns to generate textual completion in response tomultimodal prompts. It embeds a variable number of prompt
images into a fixed number of tokens via Perceiver (Jaegle
et al., 2021b), and conditions the language decoder on theencoded prompt by cross-attention. Flamingo does not workwith embodied agents out of the box. We adapt it to support
decision-making by replacing the output layer with robot
action heads. We denote the method as “VIMA-Flamingo”.

Data Scaling. Next we investigate how different methods
scale with varying dataset sizes. We compare model performance at 0.1%, 1%, 10% and full imitation learning dataset
provided in VIMA-B ENCH (Fig. 4). Note that to ensure
all methods are fairly pre-trained on the same amount of
data, we initialize baseline variants that directly learn fromraw pixels with MVP pre-trained ViT (Xiao et al., 2022;
Radosavovic et al., 2022). It is further MAE fine-tuned (Heet al., 2021), using the same in-domain data as for the
Mask R-CNN object detector. See Appendix, Sec. E.3 fordetailed setup. VIMA is extremely sample efficient and,with just 1% of the data, can achieve performance similar
to baseline methods trained with 10× more data on L1 andL2 levels of generalization. In fact, for L4 we find that with

VIMA-GPT is a decoder-only architecture conditioned on
tokenized multimodal prompts. It autoregressively decodes
the next actions given instructions and interaction histories.
Similar to prior work (Chen et al., 2021; Janner et al., 2021),
it encodes an image into a single state token by a ViT encoder and prepends the rollout trajectory with prompt tokens.
This baseline does not use cross-attention.
A more detailed comparison between these variants can befound in Appendix, Sec. C.1.
6

VIMA: General Robot Manipulation with Multimodal Prompts
Ours
Ours (Oracle)
Object Perceiver
Image Patches
Image Perceiver (VIMA-Flamingo)
Single Image

80
70

Success Rate (%)

60
50
40
30
20
10
0

L1

L2

L3

L4

Variable #tokens
Mask R-CNN

Ours

ViT

Image

Patches

ViT

Ours(Oracle)

ViT

Image

Perceiver

PerceiverObjectPerceiver

Mask R-CNN

Fixed #tokens

SingleImage

Perceiver

ViT

Figure 6: Ablation on visual tokenizers. We compare the performance of VIMA-200M model across different visual
tokenizers. Our proposed object tokens outperform all methods that learn directly from raw pixels, and Object Perceiver that
downsamples the object sequence to a fixed number of tokens.
just 1% of training data, VIMA already surpasses othervariants trained with entire dataset. Finally, across all levels
with just 10% of the data, VIMA can outperform otherarchitectures trained with the full dataset by a significant
margin. We hypothesize that the data efficiency can be
attributed to the object-centric representation employedin the VIMA recipe, which is less prone to overfitting thanlearning directly from pixels in the low-data regime. Thisis consistent with findings from Sax et al. (2018), which
demonstrates that embodied agents conditioned on midlevel visual representations tend to be significantly more
sample-efficient than end-to-end control from raw pixels.

5.3. Ablation Studies
Through extensive experiments, we ablate different design
choices in VIMA and study their impact on robot decision
making. We focus on four aspects: visual tokenization,
prompt conditioning, prompt-encoding language models,and policy robustness against distractions and corruptions.
Visual Tokenization. As explained in Sec. 4, VIMA
processes the prompt and observation images into a variable
number of object tokens with a domain fine-tuned Mask
R-CNN implementation. How important is this particular
choice of visual tokenizer? We study 5 different variants
and empirically evaluate their 4 levels of generalization performance on VIMA-B ENCH. 1) Ours (Oracle): instead ofusing Mask R-CNN, we directly read out the ground-truth
bounding box from the simulator. In other words, we use
a perfect object detector to estimate the upper bound on theperformance of this study; 2) Object Perceiver: we apply a
Perceiver module to convert the variable number of objects
detected in each frame to a fixed number of tokens. Perceiver is more computationally efficient because it reduces
the average sequence length; 3) Image Perceiver: the samearchitecture as the Perceiver Resampler in VIMA-Flamingo,
which converts an image to a small, fixed number of tokens;4) Image patches: following VIMA-Gato, we divide an
RGB frame into square patches, and extract ViT embedding

Progressive Generalization. Finally, we compare the relative performance degradation as we test the models on progressively challenging zero-shot evaluation levels without
further fine-tuning (Fig. 5). Our method exhibits a minimal
performance regression, especially between L1 → L2 andL1 → L3. In contrast, the baselines can degrade as much as20%, particularly in more difficult generalization scenarios.
Although all methods degrade significantly when evaluatedon L4 (Novel Tasks), the performance drop for VIMA isonly half as severe as all other baselines. These results suggest that VIMA has developed a more generalizable policy
and robust representations than the alternative approaches.
7

VIMA: General Robot Manipulation with Multimodal Prompts
L1

Success Rate (%)

80

L2

80

L3

80

60

60

60

60

40

40

40

40

20

20

20

20

0

2

4

8 16 32 64 128 256
Model Size (M)

0

0
2

4

8 16 32 64 128 256
Model Size (M)24

8 16 32 64 128 256
Model Size (M)

L4

80

0

2

4

8 16 32 64 128 256
Model Size (M)Figure 7: Ablation on prompt conditioning. We compare our method (xattn: cross-attention prompt conditioning) with avanilla transformer decoder (gpt-decoder) across different model sizes. Cross-attention is especially helpful in low-parameter
regime and for harder generalization tasks.
tokens. The number of patches is greater than the output of
Image Perceiver; 5) Single image: VIMA-GPT’s tokenizer,
which encodes one image into a single token.

empirical results in Sanh et al. (2021); Wang et al. (2022b),
which show that well-tuned encoder-decoder architectures
can outperform GPT-3 in zero-shot generalization.

Fig. 6 shows the ablation results. We highlight a few findings. First, we note that our Mask R-CNN detection pipeline(Appendix, Sec. C.4) incurs a minimal performance loss
compared to the oracle bounding boxes, thanks to the object
augmentation (Sec. 4) that boosts robustness during training. Second, tokenizing from raw pixels (Image Perceiver,patches, or single embedding) consistently underperforms
our object-centric format. We hypothesize that these tokenizers have to allocate extra internal capacity to parse theobjects from low-level pixels, which likely impedes learning.
Sax et al. (2018) echoes our finding that using mid-levelvision can greatly improve agent generalization compared toan end-to-end pipeline. Third, even though Ours and Object
Perceiver both use the same object bounding box inputs,
the latter is significantly worse in decision making. We
conclude that it is important to directly pass the variablelength object sequence to the robot controller rather thandownsampling to a fixed number of tokens.

Prompt Encoding. We vary the size of the pre-trained
T5 encoder to study the effect of prompt encoding. We
experiment with three T5 capacities: small (30M), base
(111M), and large (368M). We further fix the parametercount of the decision-making part to be 200M. For all T5variants, we fine-tune the last two layers and freeze all otherlayers. We find no significant difference among the variants(Appendix, Sec. E.4), thus we set base as default for all
our models.
Policy Robustness. We study the policy robustness
against increasing number of distractors and corrupted task
specifications, including incomplete prompts (randomly
masking out words with <UNK> token) and corrupted
prompts (randomly swapping words, which could havechanged the task meaning altogether). See Appendix,
Sec. E.5 for exact setup and results. VIMA exhibits minimal
performance degradation with increased distractors andminor decrease with corrupted prompts. We attribute thisrobustness to the high-quality pre-trained T5 backbone.

Prompt Conditioning. VIMA conditions the robot controller (decoder) on the encoded prompt by cross-attention.
A simple alternative is to concatenate the prompt P and interaction history H into one big sequence, and then apply a
decoder-only transformer like GPT (Radford et al., 2018) topredict actions. In this ablation, we keep the object tokenizer
constant and only switch the conditioning mechanism tocausal sequence modeling. Note that this variant is conceptually “VIMA-Gato with object tokens”. Fig. 7 shows thecomparison of VIMA (xattn) and the gpt-decoder
variant across 4 generalization levels. While the variant
achieves comparable performance in larger models,cross-attention still dominates in the small-capacity range
and generalizes better in the most challenging L4 (Novel
Task) setting. Our hypothesis is that cross-attention helps
the controller stay better focused on the prompt instruction
at each interaction step. This bears a resemblance to the

6. Related WorkMulti-Task Learning by Sequence Modeling. Transformers (Vaswani et al., 2017) have enabled task unification
across many AI domains (Brown et al., 2020; Chen et al.,2022a;b; Lu et al., 2022; Wang et al., 2022c). For example,
in NLP, the Natural Language Decathlon (McCann et al.,2018) adopts a consistent question-answering format fora suite of 10 NLP tasks. T5 (Raffel et al., 2020) unifies
all language problems into the same text-to-text format.
GPT-3 (Brown et al., 2020) and Megatron (Shoeybi et al.,2019) demonstrate emergent behaviours of intuitive task
specifications by zero-shot prompting. In computer vision,
Pix2Seq (Chen et al., 2022b) casts many vision problems
into a unified sequence format. Florence (Yuan et al., 2021),
8

VIMA: General Robot Manipulation with Multimodal Prompts

BiT (Kolesnikov et al., 2020), and MuST (Ghiasi et al.,2021) pre-train shared backbone models at scale for generalvisual representations and transfer them to downstreamtasks. In multimodal learning, Perceiver (Jaegle et al.,2021b;a) proposes an efficient architecture to handle
structured inputs and outputs. Flamingo (Alayrac et al.,2022) and Frozen (Tsimpoukelli et al., 2021) design auniversal API that ingests interleaving sequences of images
and text and generates free-form text. Gato (Reed et al.,2022) is a massively multi-task model across NLP, vision,and embodied agents. Our work is most similar in spirit
to Gato, but we focus primarily on enabling an intuitive
multimodal prompting interface for a generalist robot agent.

tasks. For example, iGibson (Shen et al., 2020; Li et al.,2021; Srivastava et al., 2021; Li et al., 2022a) simulatesinteractive household scenarios. Ravens (Zeng et al., 2020)
and Robosuite (Zhu et al., 2020; Fan et al., 2021) design
various tabletop manipulation tasks with realistic robot
arms. CALVIN (Mees et al., 2021) develops long-horizon
language-conditioned tasks. Meta-World (Yu et al., 2019)is a widely used simulator benchmark studying robotics
manipulation with tabletop settings. CausalWorld (Ahmed
et al., 2021) is a benchmark for causal structure and transfer
learning in manipulation, requiring long-horizon planningand precise low-level motor control. AI2-THOR (Ehsani
et al., 2021; Deitke et al., 2022) is a framework that supports
visual object manipulation and procedural generation ofenvironments. Our VIMA-B ENCH is the first robot learningbenchmark to support multimodal-prompted tasks. We
also standardize the evaluation protocol to systematicallymeasure an agent’s generalization capabilities.

Foundation Models for Embodied Agents. Foundation
models (Bommasani et al., 2021) have demonstrated
strong emergent properties. There are many ongoing
efforts to replicate this success for embodied agents (Yanget al., 2023), focusing on 3 aspects. 1) Transformer
agent architecture: Decision Transformer and Trajectory
Transformer (Chen et al., 2021; Janner et al., 2021; Zheng
et al., 2022; Xu et al., 2022; 2023) leverage the powerfulself-attention models for sequential decision making. CLIPort (Shridhar et al., 2021), Perceiver-Actor (Shridhar et al.,2022), and RT-1 (Brohan et al., 2022) apply large transformers to robot manipulation tasks. BeT (Shafiullah et al.,2022) and C-BeT (Cui et al., 2022) design novel techniques
to learn from demonstrations with multiple modes withtransformers. 2) Pre-training for better representations:
MaskViT (Gupta et al., 2022b), R3M (Nair et al., 2022),
VIP (Ma et al., 2022), and VC-1 (Majumdar et al., 2023)
pre-train general visual representations for robotic perception. Li et al. (2022b) fine-tunes from LLM checkpoints
to accelerate policy learning. MineDojo (Fan et al., 2022)
and Ego4D (Grauman et al., 2021) provide large-scale multimodal databases to facilitate scalable policy training. 3)
LLMs for robot learning: SayCan (Ahn et al., 2022) leverages PaLM (Chowdhery et al., 2022) for zero-shot concept
grounding. Huang et al. (2022a), Inner Monologue (Huang
et al., 2022b) and LM-Nav (Shah et al., 2022) apply LLMs tolong-horizon robot planning. PaLM-E (Driess et al., 2023)
is instead a multimodal language model that can be repurposed for sequential robotic manipulation planning. Ours
differs from these works in our novel multimodal prompting
formulation, which existing LLMs do not easily support.

An extended review can be found in Appendix, Sec. F.

7. Conclusion
In this work, we introduce a novel multimodal prompting
formulation that converts diverse robot manipulation tasks
into a uniform sequence modeling problem. We instantiate
this formulation in VIMA-B ENCH, a diverse benchmark
with multimodal tasks and systematic evaluation protocols
for generalization. We propose VIMA, a conceptually simple transformer-based agent capable of solving tasks suchas visual goal reaching, one-shot video imitation, and novel
concept grounding with a single model. Through comprehensive experiments, we show that VIMA exhibits strongmodel scalability and zero-shot generalization. Therefore,
we recommend our agent design as a solid starting point for
future work.

Acknowledgement
We are extremely grateful to Shyamal Buch, Jonathan Tremblay, Ajay Mandlekar, Chris Choy, De-An Huang, Silvio
Savarese, Fei Xia, Josiah Wong, Abhishek Joshi, Soroush
Nasiriany, and many other colleagues and friends for theirhelpful feedback and insightful discussions. We also thankthe anonymous reviewers for offering us highly constructive
advice and kind encouragement during the review period.
NVIDIA provides the necessary computing resource andinfrastructure for this project. This work is done during
Yunfan Jiang and Guanzhi Wang’s internships at NVIDIA.
Guanzhi Wang is supported by the Kortschak fellowship in
Computing and Mathematical Sciences at Caltech.

Robot Manipulation and Benchmarks. A wide rangeof robot manipulation tasks require different skills andtask specification formats, such as instruction following (Stepputtis et al., 2020), one-shot imitation (Finn
et al., 2017; Duan et al., 2017), rearrangement (Batraet al., 2020), constraint satisfaction (Brunke et al., 2021a),and reasoning (Shridhar et al., 2020). Multiple physics
simulation benchmarks are introduced to study the above
9

VIMA: General Robot Manipulation with Multimodal Prompts

References

Baker, B., Akkaya, I., Zhokhov, P., Huizinga, J., Tang, J.,
Ecoffet, A., Houghton, B., Sampedro, R., and Clune,
J. Video pretraining (vpt): Learning to act by watching
unlabeled online videos. arXiv preprint arXiv: Arxiv2206.11795, 2022.

Abramson, J., Ahuja, A., Barr, I., Brussee, A., Carnevale, F.,
Cassin, M., Chhaparia, R., Clark, S., Damoc, B., Dudzik,A., Georgiev, P., Guy, A., Harley, T., Hill, F., Hung, A.,
Kenton, Z., Landon, J., Lillicrap, T., Mathewson, K.,
Mokrá, S., Muldal, A., Santoro, A., Savinov, N., Varma,
V., Wayne, G., Williams, D., Wong, N., Yan, C., and Zhu,
R. Imitating interactive intelligence. arXiv preprint arXiv:
Arxiv-2012.05672, 2020.

Batra, D., Chang, A. X., Chernova, S., Davison, A. J., Deng,
J., Koltun, V., Levine, S., Malik, J., Mordatch, I., Mottaghi, R., Savva, M., and Su, H. Rearrangement: A
challenge for embodied ai. arXiv preprint arXiv: Arxiv2011.01975, 2020.

Aceituno, B., Rodriguez, A., Tulsiani, S., Gupta, A., andMukadam, M. A differentiable recipe for learning visual non-prehensile planar manipulation. In 5th Annual
Conference on Robot Learning, 2021. URL https://openreview.net/forum?id=f7KaqYLO3iE.

Berscheid, L., Meißner, P., and Kröger, T. Self-supervised
learning for precise pick-and-place without object model.
arXiv preprint arXiv: Arxiv-2006.08373, 2020.
Bharadhwaj, H., Kumar, A., Rhinehart, N., Levine, S.,
Shkurti, F., and Garg, A. Conservative safety critics for
exploration. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria,
May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=iaO86DUuKi.

Agrawal, P.
The task specification problem.
In
Faust, A., Hsu, D., and Neumann, G. (eds.), Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp. 1745–1751. PMLR, 08-11 Nov
2022. URL https://proceedings.mlr.press/
v164/agrawal22a.html.

Bommasani, R., Hudson, D. A., Adeli, E., Altman, R.,
Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card,
D., Castellon, R., Chatterji, N., Chen, A., Creel, K.,
Davis, J. Q., Demszky, D., Donahue, C., Doumbouya,M., Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh,K., Fei-Fei, L., Finn, C., Gale, T., Gillespie, L., Goel,
K., Goodman, N., Grossman, S., Guha, N., Hashimoto,T., Henderson, P., Hewitt, J., Ho, D. E., Hong, J., Hsu,
K., Huang, J., Icard, T., Jain, S., Jurafsky, D., Kalluri, P.,
Karamcheti, S., Keeling, G., Khani, F., Khattab, O., Koh,P. W., Krass, M., Krishna, R., Kuditipudi, R., Kumar, A.,
Ladhak, F., Lee, M., Lee, T., Leskovec, J., Levent, I., Li,
X. L., Li, X., Ma, T., Malik, A., Manning, C. D., Mirchandani, S., Mitchell, E., Munyikwa, Z., Nair, S., Narayan,A., Narayanan, D., Newman, B., Nie, A., Niebles, J. C.,
Nilforoshan, H., Nyarko, J., Ogut, G., Orr, L., Papadimitriou, I., Park, J. S., Piech, C., Portelance, E., Potts, C.,
Raghunathan, A., Reich, R., Ren, H., Rong, F., Roohani,Y., Ruiz, C., Ryan, J., Ré, C., Sadigh, D., Sagawa, S., Santhanam, K., Shih, A., Srinivasan, K., Tamkin, A., Taori,R., Thomas, A. W., Tramèr, F., Wang, R. E., Wang, W.,
Wu, B., Wu, J., Wu, Y., Xie, S. M., Yasunaga, M., You, J.,
Zaharia, M., Zhang, M., Zhang, T., Zhang, X., Zhang, Y.,
Zheng, L., Zhou, K., and Liang, P. On the opportunities
and risks of foundation models. arXiv preprint arXiv:
Arxiv-2108.07258, 2021.

Ahmed, O., Träuble, F., Goyal, A., Neitz, A., Wuthrich,M., Bengio, Y., Schölkopf, B., and Bauer, S. Causalworld: A robotic manipulation benchmark for causal
structure and transfer learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net,
2021. URL https://openreview.net/forum?
id=SK7A5pdrgov.
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes,O., David, B., Finn, C., Gopalakrishnan, K., Hausman,K., Herzog, A., Ho, D., Hsu, J., Ibarz, J., Ichter, B.,
Irpan, A., Jang, E., Ruano, R. J., Jeffrey, K., Jesmonth,S., Joshi, N. J., Julian, R., Kalashnikov, D., Kuang, Y.,
Lee, K.-H., Levine, S., Lu, Y., Luu, L., Parada, C., Pastor,P., Quiambao, J., Rao, K., Rettinghouse, J., Reyes, D.,
Sermanet, P., Sievers, N., Tan, C., Toshev, A., Vanhoucke,
V., Xia, F., Xiao, T., Xu, P., Xu, S., and Yan, M. Do
as i can, not as i say: Grounding language in robotic
affordances. arXiv preprint arXiv: Arxiv-2204.01691,
2022.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds,
M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z.,
Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S.,
Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski,
M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. Flamingo: a visual language model for fewshot learning. arXiv preprint arXiv: Arxiv-2204.14198,
2022.

Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis,
J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog,
A., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jackson, T.,
Jesmonth, S., Joshi, N. J., Julian, R., Kalashnikov, D.,
Kuang, Y., Leal, I., Lee, K.-H., Levine, S., Lu, Y., Malla,
10

VIMA: General Robot Manipulation with Multimodal Prompts

U., Manjunath, D., Mordatch, I., Nachum, O., Parada, C.,
Peralta, J., Perez, E., Pertsch, K., Quiambao, J., Rao, K.,
Ryoo, M., Salazar, G., Sanketi, P., Sayed, K., Singh, J.,
Sontakke, S., Stone, A., Tan, C., Tran, H., Vanhoucke,
V., Vega, S., Vuong, Q., Xia, F., Xiao, T., Xu, P., Xu, S.,
Yu, T., and Zitkovich, B. Rt-1: Robotics transformer for
real-world control at scale. arXiv preprint arXiv: Arxiv2212.06817, 2022.

neurips.cc/paper/2021/hash/
7f489f642a0ddb10272b5c31057f0663-Abstract.
html.
Chen, T., Saxena, S., Li, L., Fleet, D. J., and Hinton, G. E.
Pix2seq: A language modeling framework for object detection. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April
25-29, 2022. OpenReview.net, 2022a. URL https://openreview.net/forum?id=e42KbIw6Wb.

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan,J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Chen, T., Saxena, S., Li, L., Lin, T.-Y., Fleet, D. J., andAskell, A., Agarwal, S., Herbert-Voss, A., Krueger, G.,
Hinton, G. A unified sequence interface for vision tasks.
Henighan, T., Child, R., Ramesh, A., Ziegler, D. M.,
arXiv preprint arXiv: Arxiv-2206.07669, 2022b.
Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E.,
Litwin, M., Gray, S., Chess, B., Clark, J., Berner,
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra,C., McCandlish, S., Radford, A., Sutskever, I., and
G., Roberts, A., Barham, P., Chung, H. W., Sutton,
Amodei, D. Language models are few-shot learners.
C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko,
In Larochelle, H., Ranzato, M., Hadsell, R., Balcan,
S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer,M., and Lin, H. (eds.), Advances in Neural Information
N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B.,
Processing Systems 33: Annual Conference on NeuralPope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari,
Information Processing Systems 2020, NeurIPS 2020,
G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev,
December 6-12, 2020, virtual, volume 33, pp. 1877–
S., Michalewski, H., Garcia, X., Misra, V., Robinson,
1901, 2020.
URL https://proceedings.
K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim,neurips.cc/paper/2020/hash/
H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D.,1457c0d6bfcb4967418bfb8ac142f64a-Abstract.
Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pelhtml.
lat, M., Lewkowycz, A., Moreira, E., Child, R., Polozov,
O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Diaz, M.,
Brunke, L., Greeff, M., Hall, A. W., Yuan, Z., Zhou, S.,
Firat, O., Catasta, M., Wei, J., Meier-Hellstern, K., Eck,
Panerati, J., and Schoellig, A. P. Safe learning in robotics:
D., Dean, J., Petrov, S., and Fiedel, N. Palm: Scaling
From learning-based control to safe reinforcement learnlanguage modeling with pathways. arXiv preprint arXiv:
ing. arXiv preprint arXiv: Arxiv-2108.06266, 2021a.
Arxiv-2204.02311, 2022.
Brunke, L., Greeff, M., Hall, A. W., Yuan, Z., Zhou, S.,
Panerati, J., and Schoellig, A. P. Safe Learning in
Collins, J., Chand, S., Vanderkop, A., and Howard, D. A reRobotics: From Learning-Based Control to Safe Reinview of physics simulators for robotic applications. IEEE
forcement Learning, December 2021b. URL http://
Access, 9:51416–51431, 2021.
arxiv.org/abs/2108.06266. arXiv:2108.06266
Coumans, E. and Bai, Y. Pybullet, a python module for[cs, eess].
physics simulation for games, robotics and machine learnBuch, S., Eyzaguirre, C., Gaidon, A., Wu, J., Fei-Fei, L., anding. http://pybullet.org, 2016–2021.
Niebles, J. C. Revisiting the ”video” in video-language
Cui, Z. J., Wang, Y., Shafiullah, N. M. M., and Pinto, L.
understanding. CVPR, 2022.
From play to policy: Conditional behavior generation
Bucker, A., Figueredo, L., Haddadin, S., Kapoor, A., Ma, S.,from uncurated robot data. arXiv preprint arXiv: ArxivVemprala, S., and Bonatti, R. Latte: Language trajectory
2210.10047, 2022.
transformer. arXiv preprint arXiv: Arxiv-2208.02918,
Dasari, S. and Gupta, A. Transformers for one-shot vi2022.
sual imitation. In Kober, J., Ramos, F., and TomChen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A.,lin, C. J. (eds.), 4th Conference on Robot Learning,
Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. DeCoRL 2020, 16-18 November 2020, Virtual Event /cision transformer: Reinforcement learning via sequence
Cambridge, MA, USA, volume 155 of Proceedings ofmodeling. In Ranzato, M., Beygelzimer, A., Dauphin,
Machine Learning Research, pp. 2071–2084. PMLR,
Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in
2020. URL https://proceedings.mlr.press/
Neural Information Processing Systems 34: Annual Conv155/dasari21a.html.
ference on Neural Information Processing Systems 2021,
NeurIPS 2021, December 6-14, 2021, virtual, pp. 15084–
Dasari, S., Ebert, F., Tian, S., Nair, S., Bucher, B., Schmeck15097, 2021.
URL https://proceedings.
peper, K., Singh, S., Levine, S., and Finn, C. Robonet:
11

VIMA: General Robot Manipulation with Multimodal Prompts

Large-scale multi-robot learning. In Kaelbling, L. P.,
Kragic, D., and Sugiura, K. (eds.), 3rd Annual Conferenceon Robot Learning, CoRL 2019, Osaka, Japan, October
30 - November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pp. 885–897.
PMLR, 2019. URL http://proceedings.mlr.
press/v100/dasari20a.html.
Deitke, M., VanderBilt, E., Herrasti, A., Weihs, L., Salvador,J., Ehsani, K., Han, W., Kolve, E., Farhadi, A., Kembhavi,A., and Mottaghi, R. Procthor: Large-scale embodied
ai using procedural generation. arXiv preprint arXiv:
Arxiv-2206.06994, 2022.
Devin, C., Rowghanian, P., Vigorito, C., Richards, W., andRohanimanesh, K. Self-supervised goal-conditioned pick
and place. arXiv preprint arXiv: Arxiv-2008.11466, 2020.

Ehsani, K., Han, W., Herrasti, A., VanderBilt, E., Weihs, L.,
Kolve, E., Kembhavi, A., and Mottaghi, R. Manipulathor:
A framework for visual object manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR), pp. 4497–4506, June
2021.
Fan, L., Zhu, Y., Zhu, J., Liu, Z., Zeng, O., Gupta, A., CreusCosta, J., Savarese, S., and Fei-Fei, L. Surreal: Opensource reinforcement learning framework and robot manipulation benchmark. In Billard, A., Dragan, A., Peters,J., and Morimoto, J. (eds.), Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedingsof Machine Learning Research, pp. 767–782. PMLR, 2931 Oct 2018. URL https://proceedings.mlr.
press/v87/fan18a.html.
Fan, L., Zhu, Y., Zhu, J., Liu, Z., Zeng, O., Gupta, A., CreusCosta, J., Savarese, S., and Fei-Fei, L. Surreal-system:
Fully-integrated stack for distributed deep reinforcement
learning. arXiv preprint arXiv: Arxiv-1909.12989, 2019.

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn,D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer,
M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby,
N. An image is worth 16x16 words: Transformers forimage recognition at scale. arXiv preprint arXiv: Arxiv2010.11929, 2020.

Fan, L., Wang, G., Huang, D., Yu, Z., Fei-Fei, L., Zhu, Y.,
and Anandkumar, A. SECANT: self-expert cloning forzero-shot generalization of visual policies. In Meila, M.
and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 1824 July 2021, Virtual Event, volume 139 of Proceedingsof Machine Learning Research, pp. 3088–3099. PMLR,
2021. URL http://proceedings.mlr.press/v139/fan21c.html.

Downs, L., Francis, A., Koenig, N., Kinman, B., Hickman, R., Reymann, K., McHugh, T. B., and Vanhoucke, V. Google scanned objects: A high-quality
dataset of 3d scanned household items. arXiv preprint
arXiv:2204.11918, 2022.
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery,A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu,
T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth,D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint,M., Greff, K., Zeng, A., Mordatch, I., and Florence, P.
Palm-e: An embodied multimodal language model. arXiv
preprint arXiv: Arxiv-2303.03378, 2023.

Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu,
H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A.
Minedojo: Building open-ended embodied agents withinternet-scale knowledge. arXiv preprint arXiv: Arxiv2206.08853, 2022.
Finn, C., Yu, T., Zhang, T., Abbeel, P., and Levine, S. Oneshot visual imitation learning via meta-learning. arXiv
preprint arXiv: Arxiv-1709.04905, 2017.

Duan, J., Yu, S., Tan, H. L., Zhu, H., and Tan, C. A surveyof embodied AI: from simulators to research tasks. IEEE
Trans. Emerg. Top. Comput. Intell., 6(2):230–244, 2022.
doi: 10.1109/TETCI.2022.3141105. URL https://doi.org/10.1109/TETCI.2022.3141105.

Florence, P., Manuelli, L., and Tedrake, R. Self-supervised
correspondence in visuomotor policy learning. arXiv
preprint arXiv: Arxiv-1909.06933, 2019.

Duan, Y., Andrychowicz, M., Stadie, B. C., Ho, J.,
Fu, T.-J., Li, L., Gan, Z., Lin, K., Wang, W. Y., Wang, L.,
Schneider, J., Sutskever, I., Abbeel, P., and Zaremba,and Liu, Z. Violet : End-to-end video-language transformW. One-shot imitation learning. In Guyon, I., voners with masked visual-token modeling. arXiv preprint
Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R.,
arXiv: Arxiv-2111.12681, 2021.
Vishwanathan, S. V. N., and Garnett, R. (eds.), Advancesin Neural Information Processing Systems 30: Annual
Gan, C., Zhou, S., Schwartz, J., Alter, S., Bhandwaldar,
Conference on Neural Information Processing Systems
A., Gutfreund, D., Yamins, D. L. K., DiCarlo, J. J., Mc2017, December 4-9, 2017, Long Beach, CA, USA, pp.
Dermott, J., Torralba, A., and Tenenbaum, J. B. The
1087–1098, 2017. URL https://proceedings.
threedworld transport challenge: A visually guided taskneurips.cc/paper/2017/hash/and-motion planning benchmark for physically realisticba3866600c3540f67c1e9575e213be0a-Abstract. embodied ai. arXiv preprint arXiv: Arxiv-2103.14025,
html.
2021.
12

VIMA: General Robot Manipulation with Multimodal Prompts

Ghiasi, G., Zoph, B., Cubuk, E. D., Le, Q. V., and Lin, T.
Multi-task self-training for learning general representations. In 2021 IEEE/CVF International Conference on
Computer Vision, ICCV 2021, Montreal, QC, Canada,
October 10-17, 2021, pp. 8836–8845. IEEE, 2021. doi:
10.1109/ICCV48922.2021.00873. URL https://doi.
org/10.1109/ICCV48922.2021.00873.

He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R.
Masked autoencoders are scalable vision learners. arXiv
preprint arXiv: Arxiv-2111.06377, 2021.

Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari,A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X.,
Martin, M., Nagarajan, T., Radosavovic, I., Ramakrishnan, S. K., Ryan, F., Sharma, J., Wray, M., Xu, M., Xu,
E. Z., Zhao, C., Bansal, S., Batra, D., Cartillier, V., Crane,S., Do, T., Doulaty, M., Erapalli, A., Feichtenhofer, C.,
Fragomeni, A., Fu, Q., Gebreselasie, A., Gonzalez, C.,
Hillis, J., Huang, X., Huang, Y., Jia, W., Khoo, W., Kolar,
J., Kottur, S., Kumar, A., Landini, F., Li, C., Li, Y., Li,
Z., Mangalam, K., Modhugu, R., Munro, J., Murrell, T.,
Nishiyasu, T., Price, W., Puentes, P. R., Ramazanova, M.,
Sari, L., Somasundaram, K., Southerland, A., Sugano, Y.,
Tao, R., Vo, M., Wang, Y., Wu, X., Yagi, T., Zhao, Z.,
Zhu, Y., Arbelaez, P., Crandall, D., Damen, D., Farinella,
G. M., Fuegen, C., Ghanem, B., Ithapu, V. K., Jawahar,
C. V., Joo, H., Kitani, K., Li, H., Newcombe, R., Oliva,
A., Park, H. S., Rehg, J. M., Sato, Y., Shi, J., Shou, M. Z.,
Torralba, A., Torresani, L., Yan, M., and Malik, J. Ego4d:
Around the world in 3,000 hours of egocentric video.
arXiv preprint arXiv: Arxiv-2110.07058, 2021.

Hermann, K. M., Hill, F., Green, S., Wang, F., Faulkner, R.,
Soyer, H., Szepesvari, D., Czarnecki, W. M., Jaderberg,
M., Teplyashin, D., Wainwright, M., Apps, C., Hassabis,D., and Blunsom, P. Grounded language learning in a simulated 3d world. arXiv preprint arXiv: Arxiv-1706.06551,
2017.

Heibeck, T. H. and Markman, E. M. Word learning inchildren: An examination of fast mapping. Child development, pp. 1021–1034, 1987.

Hill, F., Tieleman, O., von Glehn, T., Wong, N., Merzic,
H., and Clark, S. Grounded language learning fastand slow. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria,
May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=wpSWuz_hyqA.

Gupta, A., Kumar, V., Lynch, C., Levine, S., and Hausman,
K. Relay policy learning: Solving long-horizon tasks via
imitation and reinforcement learning. In Kaelbling, L. P.,
Kragic, D., and Sugiura, K. (eds.), 3rd Annual Conferenceon Robot Learning, CoRL 2019, Osaka, Japan, October
30 - November 1, 2019, Proceedings, volume 100 ofProceedings of Machine Learning Research, pp. 1025–
1037. PMLR, 2019. URL http://proceedings.
mlr.press/v100/gupta20a.html.
Gupta, A., Fan, L., Ganguli, S., and Fei-Fei, L. Metamorph: Learning universal controllers with transformers.
In International Conference on Learning Representations,2022a. URL https://openreview.net/forum?
id=Opmqtk_GvYL.

Huang, D., Xu, D., Zhu, Y., Garg, A., Savarese, S., Fei-Fei,
L., and Niebles, J. C. Continuous relaxation of symbolic planner for one-shot imitation learning. In 2019
IEEE/RSJ International Conference on Intelligent Robots
and Systems, IROS 2019, Macau, SAR, China, November 3-8, 2019, pp. 2635–2642. IEEE, 2019. doi: 10.
1109/IROS40897.2019.8967761. URL https://doi.
org/10.1109/IROS40897.2019.8967761.
Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C., Niu,G., and Sabato, S. (eds.), International Conference
on Machine Learning, ICML 2022, 17-23 July 2022,
Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9118–9147.
PMLR, 2022a. URL https://proceedings.mlr.
press/v162/huang22a.html.
Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence,
P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y.,
Sermanet, P., Brown, N., Jackson, T., Luu, L., Levine, S.,
Hausman, K., and Ichter, B. Inner monologue: Embodied
reasoning through planning with language models. arXiv
preprint arXiv: Arxiv-2207.05608, 2022b.

Gupta, A., Tian, S., Zhang, Y., Wu, J., Martı́n-Martı́n, R.,
and Fei-Fei, L. Maskvit: Masked visual pre-training
for video prediction. arXiv preprint arXiv: Arxiv2206.11894, 2022b.
Hansen, N., Yuan, Z., Ze, Y., Mu, T., Rajeswaran, A., Su,
H., Xu, H., and Wang, X. On pre-training for visuomotor control: Revisiting a learning-from-scratch baseline. arXiv preprint arXiv: Arxiv-2212.05749, 2022.

Jaegle, A., Borgeaud, S., Alayrac, J.-B., Doersch, C.,
Ionescu, C., Ding, D., Koppula, S., Zoran, D., Brock,A., Shelhamer, E., Hénaff, O., Botvinick, M. M., Zisserman, A., Vinyals, O., and Carreira, J. Perceiver io:
A general architecture for structured inputs & outputs.
arXiv preprint arXiv: Arxiv-2107.14795, 2021a.

He, K., Gkioxari, G., Dollár, P., and Girshick, R. Mask
r-cnn. arXiv preprint arXiv: Arxiv-1703.06870, 2017.
13

VIMA: General Robot Manipulation with Multimodal Prompts

Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals,O., and Carreira, J. Perceiver: General perception with iterative attention. arXiv preprint arXiv: Arxiv-2103.03206,
2021b.

G., Jain, T., Kurenkov, A., Liu, C. K., Gweon, H., Wu, J.,
Fei-Fei, L., and Savarese, S. igibson 2.0: Object-centric
simulation for robot learning of everyday household tasks.
In Faust, A., Hsu, D., and Neumann, G. (eds.), Conference on Robot Learning, 8-11 November 2021, London,
UK, volume 164 of Proceedings of Machine Learning
Research, pp. 455–465. PMLR, 2021. URL https://
proceedings.mlr.press/v164/li22b.html.

James, S., Ma, Z., Arrojo, D. R., and Davison, A. J. Rlbench:
The robot learning benchmark & learning environment.
arXiv preprint arXiv: Arxiv-1909.12271, 2019.

Janner, M., Li, Q., and Levine, S. Offline reinforcement
Li, C., Zhang, R., Wong, J., Gokmen, C., Srivastava, S.,learning as one big sequence modeling problem. In
Martı́n-Martı́n, R., Wang, C., Levine, G., Lingelbach,Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang,
M., Sun, J., Anvari, M., Hwang, M., Sharma, M., AyP., and Vaughan, J. W. (eds.), Advances in Neuraldin, A., Bansal, D., Hunter, S., Kim, K.-Y., Lou, A.,
Information Processing Systems 34: Annual Conference
Matthews, C. R., Villa-Renteria, I., Tang, J. H., Tang,on Neural Information Processing Systems 2021,
C., Xia, F., Savarese, S., Gweon, H., Liu, K., Wu, J.,
NeurIPS 2021, December 6-14, 2021, virtual, pp. 1273–
and Fei-Fei, L. BEHAVIOR-1k: A benchmark for em1286, 2021.
URL https://proceedings.
bodied AI with 1,000 everyday activities and realistic
neurips.cc/paper/2021/hash/simulation. In 6th Annual Conference on Robot Learning,
099fe6b0b444c23836c4a5d07346082b-Abstract. 2022a. URL https://openreview.net/forum?
html.
id=_8DoIe8G3t.
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S.,
and Shah, M. Transformers in vision: A survey. arXiv
preprint arXiv: Arxiv-2101.01169, 2021.

Li, S., Puig, X., Paxton, C., Du, Y., Wang, C., Fan, L., Chen,
T., Huang, D.-A., Akyürek, E., Anandkumar, A., Andreas,J., Mordatch, I., Torralba, A., and Zhu, Y. Pre-trained
language models for interactive decision-making. arXiv
preprint arXiv: Arxiv-2202.01771, 2022b.

Khandelwal, A., Weihs, L., Mottaghi, R., and Kembhavi, A.
Simple but effective: Clip embeddings for embodied ai.
arXiv preprint arXiv: Arxiv-2111.09888, 2021.

Lim, M. H., Zeng, A., Ichter, B., Bandari, M., Coumans, E.,
Tomlin, C., Schaal, S., and Faust, A. Multi-task learningwith sequence-conditioned transporter networks. arXiv
preprint arXiv: Arxiv-2109.07578, 2021.

Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C.,
Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo,
W.-Y., Dollár, P., and Girshick, R. Segment anything.
arXiv preprint arXiv: Arxiv-2304.02643, 2023.

Lin, T., Wang, Y., Liu, X., and Qiu, X. A survey of transformers. arXiv preprint arXiv: Arxiv-2106.04554, 2021.

Kokkinos, I. Ubernet: Training a ‘universal’ convolutional
neural network for low-, mid-, and high-level vision using
diverse datasets and limited memory. arXiv preprint
arXiv: Arxiv-1609.02132, 2016.

Liu, F., Liu, H., Grover, A., and Abbeel, P. Masked autoencoding for scalable and generalizable decision making.
arXiv preprint arXiv: Arxiv-2211.12740, 2022a.

Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung,J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning. In Vedaldi, A.,
Bischof, H., Brox, T., and Frahm, J. (eds.), Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings,
Part V, volume 12350 of Lecture Notes in Computer
Science, pp. 491–507. Springer, 2020. doi: 10.1007/
978-3-030-58558-7\ 29. URL https://doi.org/
10.1007/978-3-030-58558-7_29.

Liu, H., Lee, L., Lee, K., and Abbeel, P. Instructrl: Instruction-following agents with jointly pre-trained
vision-language models. arXiv preprint arXiv: Arxiv2210.13431, 2022b.
Liu, Z., Liu, W., Qin, Y., Xiang, F., Gou, M., Xin, S., Roa,
M. A., Calli, B., Su, H., Sun, Y., et al. Ocrtoc: A cloudbased competition and benchmark for robotic grasping
and manipulation. IEEE Robotics and Automation Letters,
7(1):486–493, 2021.

Kolesnikov, A., Pinto, A. S., Beyer, L., Zhai, X., Harmsen,J., and Houlsby, N. Uvim: A unified modeling approachfor vision with learned guiding codes. arXiv preprint
arXiv: Arxiv-2205.10337, 2022.

Loshchilov, I. and Hutter, F. SGDR: stochastic gradient descent with warm restarts. In 5th International
Conference on Learning Representations, ICLR 2017,
Toulon, France, April 24-26, 2017, Conference Track
Proceedings. OpenReview.net, 2017. URL https://openreview.net/forum?id=Skq89Scxx.

Li, C., Xia, F., Martı́n-Martı́n, R., Lingelbach, M., Srivastava, S., Shen, B., Vainio, K. E., Gokmen, C., Dharan,
14

VIMA: General Robot Manipulation with Multimodal Prompts

Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In 7th International Conference on Learning
Representations, ICLR 2019, New Orleans, LA, USA,May 6-9, 2019. OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.

Minderer, M., Gritsenko, A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A.,
Dehghani, M., Shen, Z., Wang, X., Zhai, X., Kipf, T., andHoulsby, N. Simple open-vocabulary object detection
with vision transformers. arXiv preprint arXiv: Arxiv2205.06230, 2022.

Lu, J., Goswami, V., Rohrbach, M., Parikh, D., andLee, S. 12-in-1: Multi-task vision and language
representation learning. In 2020 IEEE/CVF Conference
on Computer Vision and Pattern Recognition, CVPR
2020, Seattle, WA, USA, June 13-19, 2020, pp. 10434–
10443. Computer Vision Foundation / IEEE, 2020.
doi: 10.1109/CVPR42600.2020.01045. URL https://openaccess.thecvf.com/content_CVPR_
2020/html/Lu_12-in-1_Multi-Task_
Vision_and_Language_Representation_
Learning_CVPR_2020_paper.html.

Mittal, M., Yu, C., Yu, Q., Liu, J., Rudin, N., Hoeller, D.,
Yuan, J. L., Tehrani, P. P., Singh, R., Guo, Y., Mazhar,
H., Mandlekar, A., Babich, B., State, G., Hutter, M., andGarg, A. Orbit: A unified simulation framework forinteractive robot learning environments. arXiv preprint
arXiv: Arxiv-2301.04195, 2023.
Morrical, N., Tremblay, J., Birchfield, S., and Wald, I.
NVISII: Nvidia scene imaging interface, 2020. https://github.com/owl-project/NVISII/.
Nair, S., Rajeswaran, A., Kumar, V., Finn, C., and Gupta,
A. R3m: A universal visual representation for robot
manipulation. arXiv preprint arXiv: Arxiv-2203.12601,
2022.

Lu, J., Clark, C., Zellers, R., Mottaghi, R., and Kembhavi,
A. Unified-io: A unified model for vision, language,
and multi-modal tasks. arXiv preprint arXiv: Arxiv2206.08916, 2022.

OpenAI, Berner, C., Brockman, G., Chan, B., Cheung, V.,
Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme,
S., Hesse, C., Józefowicz, R., Gray, S., Olsson, C., Pachocki, J., Petrov, M., d. O. Pinto, H. P., Raiman, J., Salimans, T., Schlatter, J., Schneider, J., Sidor, S., Sutskever,I., Tang, J., Wolski, F., and Zhang, S. Dota 2 with large
scale deep reinforcement learning. arXiv preprint arXiv:
Arxiv-1912.06680, 2019.

Lynch, C. and Sermanet, P. Language conditioned imitation learning over unstructured data. In Shell, D. A.,
Toussaint, M., and Hsieh, M. A. (eds.), Robotics: Science and Systems XVII, Virtual Event, July 12-16, 2021,
2021. doi: 10.15607/RSS.2021.XVII.047. URL https://doi.org/10.15607/RSS.2021.XVII.047.
Ma, Y. J., Sodhani, S., Jayaraman, D., Bastani, O., Kumar,V., and Zhang, A. Vip: Towards universal visual reward
and representation via value-implicit pre-training. arXiv
preprint arXiv: Arxiv-2210.00030, 2022.

Paine, T. L., Colmenarejo, S. G., Wang, Z., Reed, S., Aytar,
Y., Pfaff, T., Hoffman, M. W., Barth-Maron, G., Cabi, S.,
Budden, D., and de Freitas, N. One-shot high-fidelity
imitation: Training large-scale deep nets with rl. arXiv
preprint arXiv: Arxiv-1810.05017, 2018.

Majumdar, A., Yadav, K., Arnaud, S., Ma, Y. J., Chen, C.,
Silwal, S., Jain, A., Berges, V.-P., Abbeel, P., Malik, J.,
Batra, D., Lin, Y., Maksymets, O., Rajeswaran, A., andMeier, F. Where are we in the search for an artificial
visual cortex for embodied intelligence? arXiv preprint
arXiv: Arxiv-2303.18240, 2023.

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J.,
Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga,
L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang,
L., Bai, J., and Chintala, S. Pytorch: An imperative
style, high-performance deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc,
F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran
Associates, Inc., 2019.

Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang,
C., Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y., andMartı́n-Martı́n, R. What matters in learning from offline
human demonstrations for robot manipulation. arXiv
preprint arXiv: Arxiv-2108.03298, 2021.
McCann, B., Keskar, N. S., Xiong, C., and Socher, R. The
natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv: Arxiv-1806.08730,
2018.

Puig, X., Ra, K., Boben, M., Li, J., Wang, T., Fidler, S.,
and Torralba, A. Virtualhome: Simulating household
activities via programs. In 2018 IEEE Conference on
Computer Vision and Pattern Recognition, CVPR 2018,
Salt Lake City, UT, USA, June 18-22, 2018, pp. 8494–
8502. Computer Vision Foundation / IEEE Computer
Society, 2018. doi: 10.1109/CVPR.2018.00886. URL
http://openaccess.thecvf.com/content_

Mees, O., Hermann, L., Rosete-Beas, E., and Burgard, W.
Calvin: A benchmark for language-conditioned policy
learning for long-horizon robot manipulation tasks. arXiv
preprint arXiv: Arxiv-2112.03227, 2021.
15

VIMA: General Robot Manipulation with Multimodal Prompts

Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans,E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J.,
Parikh, D., and Batra, D. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF
International Conference on Computer Vision (ICCV),
October 2019.

cvpr_2018/html/Puig_VirtualHome_
Simulating_Household_CVPR_2018_paper.
html.
Radford, A., Narasimhan, K., Salimans, T., and Sutskever,I. Improving language understanding by generative pretraining. OpenAI, 2018.

Sax, A., Emi, B., Zamir, A. R., Guibas, L., Savarese, S., andMalik, J. Mid-level visual representations improve generalization and sample efficiency for learning visuomotor
policies. arXiv preprint arXiv: Arxiv-1812.11971, 2018.

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G.,
Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J.,et al. Learning transferable visual models from natural
language supervision. In International Conference on
Machine Learning, pp. 8748–8763. PMLR, 2021.

Shafiullah, N. M. M., Cui, Z. J., Altanzaya, A., and Pinto, L.
Behavior transformers: Cloning k modes with one stone.
arXiv preprint arXiv: Arxiv-2206.11251, 2022.

Radosavovic, I., Xiao, T., James, S., Abbeel, P., Malik, J.,
and Darrell, T. Real-world robot learning with masked
visual pre-training. CoRL, 2022.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S.,
Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring thelimits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.
URL http://jmlr.org/papers/v21/20-074.
html.
Ravichandar, H., Polydoros, A. S., Chernova, S., and Billard,
A. Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous
systems, 3:297–330, 2020.

Shah, D., Osinski, B., Ichter, B., and Levine, S. Lm-nav:
Robotic navigation with large pre-trained models of language, vision, and action. arXiv preprint arXiv: Arxiv2207.04429, 2022.
Shazeer, N. Glu variants improve transformer. arXiv
preprint arXiv: Arxiv-2002.05202, 2020.
Shen, B., Xia, F., Li, C., Martı́n-Martı́n, R., Fan, L.,
Wang, G., Pérez-D’Arpino, C., Buch, S., Srivastava, S.,
Tchapmi, L. P., Tchapmi, M. E., Vainio, K., Wong, J.,
Fei-Fei, L., and Savarese, S. igibson 1.0: a simulation
environment for interactive tasks in large realistic scenes.
arXiv preprint arXiv: Arxiv-2012.02924, 2020.
Shi, T. T., Karpathy, A., Fan, L. J., Hernandez, J., and Liang,
P. World of bits: an open-domain platform for web-based
agents. ICML, 2017. URL https://dl.acm.org/
doi/10.5555/3305890.3306005.

Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G.,
Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky,
Y., Kay, J., Springenberg, J. T., Eccles, T., Bruce, J.,
Razavi, A., Edwards, A., Heess, N., Chen, Y., Hadsell, R.,
Vinyals, O., Bordbar, M., and de Freitas, N. A generalist
agent. arXiv preprint arXiv: Arxiv-2205.06175, 2022.

Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper,J., and Catanzaro, B. Megatron-lm: Training multibillion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.

Reid, M., Yamada, Y., and Gu, S. S. Can wikipedia help
offline reinforcement learning? arXiv preprint arXiv:
Arxiv-2201.12122, 2022.

Shridhar, M., Thomason, J., Gordon, D., Bisk, Y.,
Han, W., Mottaghi, R., Zettlemoyer, L., and Fox, D.
ALFRED: A benchmark for interpreting grounded
instructions for everyday tasks. In 2020 IEEE/CVF
Conference on Computer Vision and Pattern Recognition,
CVPR 2020, Seattle, WA, USA, June 13-19, 2020,
pp. 10737–10746. Computer Vision Foundation /IEEE, 2020. doi: 10.1109/CVPR42600.2020.01075.
URL
https://openaccess.thecvf.com/
content_CVPR_2020/html/Shridhar_
ALFRED_A_Benchmark_for_Interpreting_Grounded_Instructions_for_Everyday_
Tasks_CVPR_2020_paper.html.

Ren, S., He, K., Girshick, R., and Sun, J. Faster r-cnn:
Towards real-time object detection with region proposal
networks. arXiv preprint arXiv: Arxiv-1506.01497, 2015.
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika,
L. A., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L.,
Raja, A., Dey, M., Bari, M. S., Xu, C., Thakker, U.,
Sharma, S., Szczechla, E., Kim, T., Chhablani, G., Nayak,
N. V., Datta, D., Chang, J., Jiang, M. T.-J., Wang, H.,
Manica, M., Shen, S., Yong, Z. X., Pandey, H., Bawden,
R., Wang, T., Neeraj, T., Rozen, J., Sharma, A., Santilli,A., Févry, T., Fries, J. A., Teehan, R., Biderman, S. R.,
Gao, L., Bers, T., Wolf, T., and Rush, A. M. Multitask
prompted training enables zero-shot task generalization.
Iclr, 2021.

Shridhar, M., Manuelli, L., and Fox, D. Cliport: What andwhere pathways for robotic manipulation. arXiv preprint
arXiv: Arxiv-2109.12098, 2021.
16

VIMA: General Robot Manipulation with Multimodal Prompts

Shridhar, M., Manuelli, L., and Fox, D. Perceiver-actor: A
multi-task transformer for robotic manipulation. arXiv
preprint arXiv: Arxiv-2209.05451, 2022.

habitat. In Ranzato, M., Beygelzimer, A., Dauphin,
Y. N., Liang, P., and Vaughan, J. W. (eds.), Advancesin Neural Information Processing Systems 34: Annual
Conference on Neural Information Processing Systems
2021, NeurIPS 2021, December 6-14, 2021, virtual,
pp. 251–266, 2021. URL https://proceedings.
neurips.cc/paper/2021/hash/021bbc7ee20b71134d53e20206bd6feb-Abstract.
html.

Sieb, M., Xian, Z., Huang, A., Kroemer, O., and Fragkiadaki, K. Graph-structured visual imitation. arXiv
preprint arXiv: Arxiv-1907.05518, 2019.
Song, S., Zeng, A., Lee, J., and Funkhouser, T. Grasping in the wild:learning 6dof closed-loop grasping fromlow-cost demonstrations. arXiv preprint arXiv: Arxiv1912.04344, 2019.

Tay, Y., Dehghani, M., Bahri, D., and Metzler, D. Efficienttransformers: A survey. arXiv preprint arXiv: Arxiv2009.06732, 2020.

Srinivasan, K., Eysenbach, B., Ha, S., Tan, J., and Finn, C.
Learning to be safe: Deep rl with a safety critic. arXiv
preprint arXiv: Arxiv-2010.14603, 2020.

Team, O. E. L., Stooke, A., Mahajan, A., Barros, C., Deck,
C., Bauer, J., Sygnowski, J., Trebacz, M., Jaderberg, M.,
Mathieu, M., McAleese, N., Bradley-Schmieg, N., Wong,N., Porcel, N., Raileanu, R., Hughes-Fitt, S., Dalibard,V., and Czarnecki, W. M. Open-ended learning leads togenerally capable agents. arXiv preprint arXiv: Arxiv2107.12808, 2021.

Srivastava, S., Li, C., Lingelbach, M., Martı́n-Martı́n, R.,
Xia, F., Vainio, K. E., Lian, Z., Gokmen, C., Buch, S.,
Liu, C. K., Savarese, S., Gweon, H., Wu, J., and FeiFei, L. BEHAVIOR: benchmark for everyday household activities in virtual, interactive, and ecological environments. In Faust, A., Hsu, D., and Neumann, G.
(eds.), Conference on Robot Learning, 8-11 November 2021, London, UK, volume 164 of Proceedingsof Machine Learning Research, pp. 477–490. PMLR,
2021. URL https://proceedings.mlr.press/
v164/srivastava22a.html.

Thananjeyan, B., Balakrishna, A., Nair, S., Luo, M., Srinivasan, K., Hwang, M., Gonzalez, J. E., Ibarz, J., Finn, C.,
and Goldberg, K. Recovery RL: safe reinforcement learning with learned recovery zones. IEEE Robotics Autom.
Lett., 6(3):4915–4922, 2021. doi: 10.1109/LRA.2021.
3070252. URL https://doi.org/10.1109/LRA.
2021.3070252.

Stengel-Eskin, E., Hundt, A., He, Z., Murali, A., Gopalan,N., Gombolay, M., and Hager, G. Guiding multistep rearrangement tasks with natural language instructions. In Faust, A., Hsu, D., and Neumann, G. (eds.),
Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp. 1486–1501. PMLR, 08–11 Nov
2022. URL https://proceedings.mlr.press/
v164/stengel-eskin22a.html.

Toyama, D., Hamel, P., Gergely, A., Comanici, G., Glaese,A., Ahmed, Z., Jackson, T., Mourad, S., and Precup,
D. Androidenv: A reinforcement learning platform forandroid. arXiv preprint arXiv: Arxiv-2105.13231, 2021.

Stepputtis, S., Campbell, J., Phielipp, M., Lee, S., Baral,
C., and Amor, H. B. Language-Conditioned Imitation Learning for Robot Manipulation Tasks, October 2020. URL http://arxiv.org/abs/2010.
12083. arXiv:2010.12083 [cs].
Stone, A., Xiao, T., Lu, Y., Gopalakrishnan, K., Lee, K.-H.,
Vuong, Q., Wohlhart, P., Zitkovich, B., Xia, F., Finn, C.,
and Hausman, K. Open-world object manipulation using
pre-trained vision-language models. arXiv preprint arXiv:
Arxiv-2303.00905, 2023.

Toyer, S., Shah, R., Critch, A., and Russell, S. The MAGICAL benchmark for robust imitation. In Larochelle, H.,
Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.),
Advances in Neural Information Processing Systems 33:
Annual Conference on Neural Information Processing
Systems 2020, NeurIPS 2020, December 6-12, 2020,
virtual, 2020.
URL https://proceedings.
neurips.cc/paper/2020/hash/d464b5ac99e74462f321c06ccacc4bff-Abstract.
html.
Tsimpoukelli, M., Menick, J., Cabi, S., Eslami, S.
M. A., Vinyals, O., and Hill, F. Multimodal few-shotlearning with frozen language models. In Ranzato,
M., Beygelzimer, A., Dauphin, Y. N., Liang, P., andVaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference
on Neural Information Processing Systems 2021,
NeurIPS 2021, December 6-14, 2021, virtual, pp.
200–212, 2021. URL https://proceedings.
neurips.cc/paper/2021/hash/Szot, A., Clegg, A., Undersander, E., Wijmans, E., Zhao,
Y., Turner, J., Maestre, N., Mukadam, M., Chaplot,
D. S., Maksymets, O., Gokaslan, A., Vondrus, V.,
Dharur, S., Meier, F., Galuba, W., Chang, A. X., Kira,
Z., Koltun, V., Malik, J., Savva, M., and Batra, D.
Habitat 2.0: Training home assistants to rearrange their
17

VIMA: General Robot Manipulation with Multimodal Prompts

01b7575c38dac42f3cfb7d500438b875-Abstract. Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y.,
html.
Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest,
Q., and Rush, A. M. Huggingface’s transformers: StateVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones,of-the-art natural language processing. arXiv preprintL., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention
arXiv: Arxiv-1910.03771, 2019.
is all you need. arXiv preprint arXiv: Arxiv-1706.03762,
Wu, Y., Kirillov, A., Massa, F., Lo, W.-Y., and Gir2017.
shick, R. Detectron2. https://github.com/
Vinyals, O., Babuschkin, I., Chung, J., Mathieu, M., Jaderfacebookresearch/detectron2, 2019.
berg, M., Czarnecki, W. M., Dudzik, A., Huang, A.,
Xiao, T., Radosavovic, I., Darrell, T., and Malik, J. Masked
Georgiev, P., Powell, R., et al. Alphastar: Masteringvisual pre-training for motor control. arXiv preprint
the real-time strategy game starcraft ii. DeepMind blog,
arXiv:2203.06173, 2022.
2, 2019.
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu,
Y., Fan, L., and Anandkumar, A. Voyager: An openended embodied agent with large language models. arXiv
preprint arXiv: Arxiv-2305.16291, 2023.

Xu, M., Shen, Y., Zhang, S., Lu, Y., Zhao, D., Tenenbaum,
J. B., and Gan, C. Prompting decision transformer for
few-shot policy generalization. arXiv preprint arXiv:
Arxiv-2206.13499, 2022.

Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J.,
Zhou, C., Zhou, J., and Yang, H. Ofa: Unifying architectures, tasks, and modalities through a simple sequenceto-sequence learning framework. arXiv preprint arXiv:
Arxiv-2202.03052, 2022a.

Xu, M., Lu, Y., Shen, Y., Zhang, S., Zhao, D., and Gan,
C. Hyper-decision transformer for efficient online policy adaptation. arXiv preprint arXiv: Arxiv-2304.08487,
2023.
Yang, S., Nachum, O., Du, Y., Wei, J., Abbeel, P., andSchuurmans, D. Foundation models for decision making:Problems, methods, and opportunities. arXiv preprint
arXiv: Arxiv-2303.04129, 2023.

Wang, T., Roberts, A., Hesslow, D., Scao, T. L., Chung,
H. W., Beltagy, I., Launay, J., and Raffel, C. What language model architecture and pretraining objective work
best for zero-shot generalization? Icml, 2022b. doi:
10.48550/arXiv.2204.05832.

Yang, Z., Fang, Y., Zhu, C., Pryzant, R., Chen, D., Shi, Y.,
Xu, Y., Qian, Y., Gao, M., Chen, Y.-L., Lu, L., Xie, Y.,
Gmyr, R., Codella, N., Kanda, N., Xiao, B., Yuan, L.,
Yoshioka, T., Zeng, M., and Huang, X. i-code: An integrative and composable multimodal learning framework.
arXiv preprint arXiv: Arxiv-2205.01818, 2022.

Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q.,
Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S.,
and Wei, F. Image as a foreign language: Beit pretraining
for all vision and vision-language tasks. arXiv preprint
arXiv: Arxiv-2208.10442, 2022c.

Yoon, Y., DeSouza, G., and Kak, A. Real-time tracking
and pose estimation for industrial objects using geometric features. In 2003 IEEE International Conferenceon Robotics and Automation (Cat. No.03CH37422), volume 3, pp. 3473–3478 vol.3, 2003. doi: 10.1109/ROBOT.
2003.1242127.

Weihs, L., Deitke, M., Kembhavi, A., and Mottaghi,
R. Visual room rearrangement. In IEEE Conference
on Computer Vision and Pattern Recognition, CVPR
2021, virtual, June 19-25, 2021, pp. 5922–5931.
Computer Vision Foundation / IEEE, 2021.
doi:
10.1109/CVPR46437.2021.00586.
URL https://openaccess.thecvf.com/content/
CVPR2021/html/Weihs_Visual_Room_
Rearrangement_CVPR_2021_paper.html.
Weikert, A., Goralczyk, A., Salmela, B., Dansie, B., Barton, C., Valenza, E., Alexandrov, G., Hubert, I., Tysdal,K., Sokolowski, L., Järvinen, M., Pulieso, M., Ebb, M.,
Vazquez, P., Tuytel, R., Hess, R., Feldlaufer, S., König,
S., Platen, S., and Mäter., S. Blender online libraries
for textures, 2022. URL https://cloud.blender.
org/p/textures/.

Yu, T., Quillen, D., He, Z., Julian, R., Narayan, A., Shively,
H., Bellathur, A., Hausman, K., Finn, C., and Levine, S.
Meta-world: A benchmark and evaluation for multi-task
and meta reinforcement learning. arXiv preprint arXiv:
Arxiv-1910.10897, 2019.
Yu, T., Xiao, T., Stone, A., Tompson, J., Brohan, A., Wang,
S., Singh, J., Tan, C., M, D., Peralta, J., Ichter, B., Hausman, K., and Xia, F. Scaling robot learning with semantically imagined experience. arXiv preprint arXiv:
Arxiv-2302.11550, 2023.
Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao,J., Hu, H., Huang, X., Li, B., Li, C., Liu, C., Liu, M.,
Liu, Z., Lu, Y., Shi, Y., Wang, L., Wang, J., Xiao, B.,

Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C.,
Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M.,
18

VIMA: General Robot Manipulation with Multimodal Prompts

Xiao, Z., Yang, J., Zeng, M., Zhou, L., and Zhang, P.
Florence: A new foundation model for computer vision.
arXiv preprint arXiv: Arxiv-2111.11432, 2021.
Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J. S., Cao, J.,
Farhadi, A., and Choi, Y. Merlot: Multimodal neuralscript knowledge models. arXiv preprint arXiv: Arxiv2106.02636, 2021.
Zellers, R., Lu, J., Lu, X., Yu, Y., Zhao, Y., Salehi, M.,
Kusupati, A., Hessel, J., Farhadi, A., and Choi, Y. Merlot reserve: Neural script knowledge through vision andlanguage and sound. CVPR, 2022.
Zeng, A., Florence, P., Tompson, J., Welker, S., Chien, J.,
Attarian, M., Armstrong, T., Krasin, I., Duong, D., Wahid,A., Sindhwani, V., and Lee, J. Transporter networks:
Rearranging the visual world for robotic manipulation.
arXiv preprint arXiv: Arxiv-2010.14406, 2020.
Zeng, A., Wong, A., Welker, S., Choromanski, K., Tombari,F., Purohit, A., Ryoo, M., Sindhwani, V., Lee, J., Vanhoucke, V., and Florence, P. Socratic models: Composing
zero-shot multimodal reasoning with language. arXiv
preprint arXiv: Arxiv-2204.00598, 2022.
Zhang, Z. and Weihs, L. When learning is out of reach, reset:
Generalization in autonomous visuomotor reinforcement
learning. arXiv preprint arXiv: Arxiv-2303.17600, 2023.
Zhao, M., Liu, F., Lee, K., and Abbeel, P. Towards
more generalizable one-shot visual imitation learning.
In 2022 International Conference on Robotics and Automation, ICRA 2022, Philadelphia, PA, USA, May
23-27, 2022, pp. 2434–2444. IEEE, 2022. doi: 10.
1109/ICRA46639.2022.9812450. URL https://doi.
org/10.1109/ICRA46639.2022.9812450.
Zheng, Q., Zhang, A., and Grover, A. Online decisiontransformer. arXiv preprint arXiv: Arxiv-2202.05607,
2022.
Zhu, Y., Wong, J., Mandlekar, A., and Martı́n-Martı́n, R.
robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv: Arxiv2009.12293, 2020.
Zhu, Y., Joshi, A., Stone, P., and Zhu, Y. Viola: Imitation
learning for vision-based manipulation with object proposal priors. arXiv preprint arXiv: Arxiv-2210.11339,
2022.

19

VIMA: General Robot Manipulation with Multimodal Prompts

A. Simulator Details
We build our VIMA-B ENCH simulation suite upon the Ravens physics simulator (Zeng et al., 2020; Shridhar et al., 2021).
Specifically, it is supported by PyBullet (Coumans & Bai, 2016–2021) with a Universal Robot UR5 arm. The size of thetabletop workspace is 0.5 × 1m. Our benchmark contains extensible sets of 3D objects and textures. Instantiated from an
object-texture combination, all object instances can be rendered as RGB images appeared in multimodal prompts. Figure A.1
displays all 3D objects. Figure A.2 displays all textures.

L-shaped block

block

bowl

container

crossdiamondflower

frame

heart

hexagonletter A

letter E

letter G

letter M

letter R

letter T

letter V

linepalletpanpentagon

ring

round

shorter block

small blocksquarestar

three-sided
rectangle

triangle

Figure A.1: Object Gallery in VIMA-B ENCH textured with random textures. Bowl and pan are from Google Scanned
Objects (Downs et al., 2022), while others are from Ravens (Zeng et al., 2020).

The observation space of VIMA-B ENCH includes RGB images from both frontal and top-down views. It also includes
a one-hot vector ∈ {0, 1}2 to indicate type of the end-effector ∈ {suction cup, spatula}. While a suction cup is equippedin most manipulation tasks, a spatula is used in particular for visual constraint tasks, where an agent is asked to “wipe”
objects. VIMA-B ENCH inherits the same action space from Zeng et al. (2020) and Shridhar et al. (2021), which consists ofprimitive actions of “pick and place” for tasks with a suction cup as the end effector, or “push” for tasks with a spatula. Bothprimitive actions contain two poses ∈ SE(2) specifying target poses of the end effector. For the “pick and place” primitive,
they represent the pick pose and the place pose. For the “push” primitive, they represent the push starting pose and push20

VIMA: General Robot Manipulation with Multimodal Prompts

ending pose.
Similar to prior work (Zeng et al., 2020; Shridhar et al., 2021), VIMA-B ENCH provides scripted oracles to generate
successful demonstrations for all tasks. We leverage them to construct an offline imitation dataset for behavioral cloning.
Given a prompt, these programmed bots can access privileged information, such as the correct object to pick and target
location to place.
>=<(:

O /!




 





 



 

(

(

(  (

( (

( (  

S

J5 (E B! 

J5 
 

! 

  !

! (!

/  !

 (( (!  ((  (!  (((!  ((
(!  ((  (!  (((!  ((
(!  (((! 

5 5

  ((
(! 

 ((O /Z
X!  

(! 

Sc
X!  ( (( (! ( ((  (! ( (( (! ( (( (! ( ((  (! ( (( (! ( (( (! (  (( (! (  (( (! ( (( (! 

 (( ((!

 ((  ((!

 ((((!

 ((
((!

 ((  ((!

 ((((!

 ((
((!

 ((((!

 ((
((!

 ((
((!

( (( ((!

( ((  ((!

( (( ((!

( (( ((!

( ((  ((!

( (( ((!

( (( ((!

(  (( ((!

(  (( ((!

( (( ((!

 (  ( 

 ( 

 (  ( ( ( ( ( (  ( ( ( ( ( 

 (   (  

 (  

 (   (  

O /Zp(S!

Scp(S!

O /((ScX p  

Figure A.2: Texture Gallery in VIMA-B ENCH. The first row of image-based textures is from Blender Cloud Libraries (Weikert et al., 2022), while others are hard-coded.
21

VIMA: General Robot Manipulation with Multimodal Prompts

B. Task Suite
We develop 17 task templates that belong to 6 diverse categories. Thousands of individual task instances and theircorresponding multimodal prompts can be procedurally generated from these task templates. We use PyBullet (Coumans &Bai, 2016–2021) as our backend and the default renderer to produce the RGB frames for training data and interactive test
environments. For demonstration purpose, we apply the NVISII (Morrical et al., 2020) ray tracing to enhance the visualquality. We elaborate on each task in the following subsections.
B.1. Simple Object Manipulation
This task category asks agents to follow basic instructions specified by multimodal prompts.
Task 01:

Pick the specified object(s) and place it (them) into the specified container.

• Prompt: Put the {object}1 into the {object}2 .
• Description: The image placeholder {object}1 is the object to be picked and the {object}2 is the container
object. The agent requires to recognize the objects with the correct color-shape combinations. To extend the difficulties,
it supports more than one object to be picked or placed. For example, the prompt “Put the {object}1 and{object}2 into the {object}3 ” asks to pick two different objects and place into a target container. We
uniformly sample different color-shape combos for objects to be picked and containers.
• Success Criteria: All specified object(s) to pick are within the bounds of the container object(s), with specified shapes
and textures provided in the prompt.
• Oracle Trajectory: Shown in Fig. A.3 with its multimodal prompt.

Put theinto the.

Figure A.3: Simple Object Manipulation: Task 01
Task 02: In the workspace, put the objects with a specified texture shown in the scene image in the prompt into container
object(s) with a specified color. This task requires the agent to find the correct object to manipulate by grounding the textural
attributes from both natural language descriptions and the visual scene images.
• Prompt: Put the {texture}1 object in {scene} into the {texture}2 object.
• Description: The text placeholder {texture}1 and {texture}2 are sampled textures for objects to be picked andthe container objects, respectively. The number of dragged objects with the same texture can be varied. {scene} is theworkspace-like image placeholder. There is a designated number of distractors with different textures (and potentiallydifferent shapes) in the scene. For each distractor in the workspace, it has 50% chance to be either dragged or container
distractor object with different textures from those specified in the prompt.
• Success Criteria: All objects in the workspace with {texture}1 are within the bounds of the container object with{texture}2 .
22

VIMA: General Robot Manipulation with Multimodal Prompts• Oracle Trajectory: Shown in Fig. A.4 with its multimodal prompt.

Put the green and blue stripe object ininto the yellow paisley object.

Figure A.4: Simple Object Manipulation: Task 02
Task 03:
this task.

Rotate objects clockwise by certain degrees along z-axis. Only rotationally asymmetric objects are considered in• Prompt: Rotate the {object}1 {angles} degrees.
• Description: The agent is required to rotate all objects in the workspace specified by the image placeholder {object}1 .
There are also objects with different color-shape combinations in the workspace as distractors. {angles} is the
sampled degree that needs to be rotated. A target angle is sampled from 30◦ , 60◦ , 90◦ , 120◦ , and 150◦ .
• Success Criteria: The position of the specified object matches its original position, and the orientation matches theorientation after rotating specific angles.
• Oracle Trajectory: Shown in Fig. A.5 with its multimodal prompt.

Rotate the120 degrees.

Figure A.5: Simple Object Manipulation: Task 03

23

VIMA: General Robot Manipulation with Multimodal Prompts

B.2. Visual Goal ReachingThis task category requires agents to manipulate objects in the workspace to reach goal states represented as images shownin prompts.
Task 04: Rearrange target objects in the workspace to match goal configuration shown in prompts. Note that to achieve
the goal configuration, distractors may need to be moved away first.
• Prompt: Rearrange to this {scene}.
• Description: Objects in the scene placeholder {scene} are target objects to be manipulated and rearranged. In theworkspace, the same target objects are spawned randomly, potentially with distractors randomly spawned as well. Witha pre-defined distractor conflict rate, the position of each distractor has this probability to occupy the position of any
target object such that the rearrangement can only succeed if moving away that distractor first.
• Success Criteria: The configuration of target objects in the workspace matches that specified in the prompt.
• Oracle Trajectory: Shown in Fig. A.6 with its multimodal prompt.

Rearrange to this.

Figure A.6: Visual Goal Reaching: Task 04

Task 05:
phase.

Extend the task 04 by requiring the agent to restore rearranged objects to the initial setup after the “rearranging”• Prompt: Rearrange objects to this setup {scene} and then restore.
• Description: Same as the task 04, except introducing the instruction “restore”.
• Success Criteria: Meet the success criteria of the task 04, and then within the allowed max steps restore all target
objects to their initial configurations.
• Oracle Trajectory: Shown in Fig. A.7 with its multimodal prompt.

24

VIMA: General Robot Manipulation with Multimodal Prompts

Rearrange objects to this setup

and then restore.

Figure A.7: Visual Goal Reaching: Task 05

B.3. Novel Concept Grounding
This task category requires agents to ground new concepts of adjectives, nouns, or verbs via visual perception and language
understanding. Similar task design can be found in prior work (Hill et al., 2021). Completing these tasks are challenging,
because the model should a) first understand prompts with interleaved texts, images, and even video frames; b) quickly
internalize new concepts that are different across task instances, which even tests the ability to meta-learn; and c) docomplicated reasoning such as comparing between “taller” vs “less taller” vs “shorter” and then ground this reasoning into
the robot action space.
Prompts consist of two parts: a definition part followed by an instruction part. In the definition part, novel concepts aredefined by multimodal illustrations with multiple support examples. In the instruction part, agents are asked to achieve
the goal by properly applying concepts from the definition part. The assignment of dummy object names is varied andindependent for each task instance such that tasks can only be solved if the agent applies the reasoning correctly. This ability
is also referred to as fast-mapping (Heibeck & Markman, 1987).
Task 06: Ground comparative adjectives by comparing the size or the textural saturation of objects and manipulating thecorrect object(s) instructed in the prompt.
• Prompt:{demo object}1 is {novel adj} than {demo object}2 .
{novel adj} {object}1 into the {object}2 .

Put the {adv}• Description: The sampled adjective {novel adj} is a dummy adjective placeholder for agent to ground. By
default, the novel adjective set is {daxer, blicker, modier, kobar}. The real meaning can be related tosize (smaller/larger) or textural saturation (lighter/darker texture). The image placeholders {demo object}1 and{demo object}2 illustrate how the novel adjective is defined. For example, if the real comparison is ”taller”, thenthe sampled object in {demo object}1 is taller than {demo object}2 . The choices of the novel adjective and thereal meaning are independently sampled for different task instances. For the instruction part, this task is similar to task
01, where the agent is required to pick the specified object(s) with the novel adjective attribute and then place it into thespecified container object. To avoid revealing the correct object to manipulate, we use a neutral texture for objects
appeared in the instruction part.
• Success Criteria: All target objects with the specified adjective attribute are within the bounds of the specified
container object.
• Oracle Trajectory: Shown in Fig. A.8 with its multimodal prompt.

25

VIMA: General Robot Manipulation with Multimodal Prompts

is kobar than.

is kobar than.

is kobar than

. Put the kobar

into the.

Figure A.8: Novel Concept Grounding: Task 06
Task 07:

Orthogonal to task 06 by requiring to learn mappings of novel nouns.

• Prompt:{object}2 .

This is a {novel name}1 {object}1 .
Put {novel name}1 into a {novel name}2 .

This is a {novel name}2• Description: Novel noun words are defined with the text placeholders {novel name}1 and {novel name}2 ,following their image placeholders {object}1 and {object}2 , for the target object and container object, respectively.
Novel nouns are sampled from {dax, blicket, wug, zup}. In the instruction part, objects are expressed as
novel nouns defined in the previous definition part. Distractors are defined the same as task 01.
• Success Criteria: All target object(s) are within the bounds of the container object(s).
• Oracle Trajectory: Shown in Fig. A.9 with its multimodal prompt.

This is a blicket. This is a zup. Put a zup into a blicket.

Figure A.9: Novel Concept Grounding: Task 07
Task 08:

Combination of tasks 06 and 07.

• Prompt:
This is a {novel name}1 {object}1 . This is a {novel name}2{object}2 . {demo object}1 is {adj} than {demo object}2 . Put the{adv} {novel adj} {novel name}1 into the {novel name}2 .
• Description: See task description for task 06 and task 07.
• Success Criteria: Similar as tasks 06 and 07.
• Oracle Trajectory: Shown in Fig. A.10 with its multimodal prompt.

26

VIMA: General Robot Manipulation with Multimodal Prompts

This is a wug
is blicker than. This is a zup.

is blicker than.

is blicker than.

. Put the blicker zup into the wug.
Figure A.10: Novel Concept Grounding: Task 08

Task 09: A novel verb “twist” is defined as rotating a specific angle illustrated by several examples. This task is similar totask 03, but it requires the agent to infer what is the exact angle to rotate from the prompt and to ground novel verbs that aresemantically similar but different in exact definitions.
• Prompt:"Twist" is defined as rotating object a specific angle.
For examples: From {before twist}ito{after twist}i .
Now twist
all {texture} objects.
• Description: Both {before twist}i and {after twist}i are scene placeholders where {before twist}i
shows a randomly sampled object before “twisting” and {after twist}i shows the same object pose after “twisting”.
All examples illustrate the same sampled angle to rotate. In the workspace, the target objects have the texture specifiedby {texture} and randomly sampled shapes.
• Success Criteria: Same as the task 03.
• Oracle Trajectory: Shown in Fig. A.11 with its multimodal prompt.

  

   





-  

Figure A.11: Novel Concept Grounding: Task 0927 

VIMA: General Robot Manipulation with Multimodal Prompts

B.4. One-Shot Video Imitation
This task category requires agents to imitate motions demonstrated through videos shown in prompts. We follow prior
works (Finn et al., 2017; Dasari & Gupta, 2020; Duan et al., 2017) to formulate the problem by giving one video
demonstration (represented as key frames in prompts), then test the learned imitator’s ability to produce target trajectories.
This setup is challenging because a) only one demonstration is available to the agent; b) the model needs to understand video
frames interleaved with textual instructions; and c) missing correspondences between demonstrations and target trajectories
since demonstrations only show partial key frames.
Task 10:

Follow motions for specific objects.

• Prompt: Follow this motion for {object}:{frame}1 ...{frame}i ... {frame}n .

• Description: Image placeholder {object} is the target object to be manipulated and {{frame}i } is set of workspacelike scene placeholders to represent a video trajectory, where n is the trajectory length. There is an object spawned at
the center in both the workspace and the prompt video but with different textures as a distractor. The initial position of
the target object matches that in {frame}1 .
• Success Criteria: In each step, the pose of the target object matches the pose in the corresponding video frame.
Incorrect manipulation sequences are considered as failures.
• Oracle Trajectory: Shown in Fig. A.12 with its multimodal prompt.

Follow this motion for:.

Figure A.12: One-shot video imitation: Task 10

Task 11:

Stack objects with the order illustrated in the prompt video.

• Prompt: Stack objects in this order {frame}1 ...{frame}i ...{frame}n .
• Description: There are multiple objects with the same shape but different textures spawned in the workspace without
any stacking initially. Distractor objects with different shapes are spawned in the workspace but not in the prompt
video. At each step of the prompt video, one object is stacked over another or put at an empty position.
• Success Criteria: Similar as task 10.
• Oracle Trajectory: Shown in Fig. A.13 with its multimodal prompt.

B.5. Visual Constraint Satisfaction
This task category requires agents to wipe a specific number of objects in the workspace to a goal region while also satisfy
the given visual constraint.
28

VIMA: General Robot Manipulation with Multimodal Prompts

Stack objects in this order.

.

Figure A.13: One-shot video imitation: Task 11
Task 12:

Sweep the designated number of objects into a specified region without exceeding the boundary.

• Prompt:{constraint}.

Sweep {quantifier} {object} into {bounds} without exceeding• Description: {object} is the image placeholder of the target object to be swept spawned with a random amount in
the workspace. Distractors have the same amount, same shape, but different color from target objects. {quantifier}is the text placeholder to determine the target quantity of objects to be wiped, sampled from any, one, two, three,
and all. {bounds} is the image placeholder for a three-sided rectangle as the goal region. {constraint} is theconstraint line.
• Success Criteria: The exact number of target objects to be swept are all inside the specified region. Potential failurecases include 1) any distractor being wiped into the region, 2) target object exceeding the constraint, or 3) incorrect
number of target objects being swept into the goal region.
• Oracle Trajectory: Shown in Fig. A.14 with its multimodal prompt.

Sweep any

into

without exceeding.

Figure A.14: Visual Constraint Satisfaction: Task 12
Task 13:

Sweep the designated number of objects into a specified region without touching the constraint.

• Prompt: Sweep {quantifier} {object} into {bounds} without touching {constraint}.
• Description: Similar as task 12 but requiring a different way to satisfy the constraint. The agent has to learn to avoid
contacting the constraint line in this case.
• Success Criteria: Similar as task 12 except that the constraint is to not touch the red line.
• Oracle Trajectory: Shown in Fig. A.15 with its multimodal prompt.
29

VIMA: General Robot Manipulation with Multimodal Prompts

Sweep two

into

without touching.

Figure A.15: Visual Constraint Satisfaction: Task 13
B.6. Visual Reasoning
This task category requires agents to make decisions by reasoning over or memorizing information conveyed throughmultimodal prompts.
Task 14: By reasoning the “same texture”, the agent is required to pick all objects in the workspace with the same texture
as the container objects specified in the prompt and place them into it.
• Prompt: Put all objects with the same texture as {object} into it.
• Description: {object} is the sampled goal container object. In the workspace, there are objects with the sametexture as the container but potentially different shapes. Distractors with different textures are spawned.
• Success Criteria: All objects with the same texture as the goal container are within the bounds of the container.
• Oracle Trajectory: Shown in Fig. A.16 with its multimodal prompt.

Put all objects with the same texture as

into it.

Figure A.16: Visual Reasoning: Task 14
Task 15: By reasoning the “same shape”, the agent is required to pick all objects in the workspace with the same top-down
profile as the goal container specified in the prompt and place them into it. For example, blocks and boxes have the same
rectangular profile.
• Prompt: Put all objects with the same profile as {object} into it.
• Description: Similar to the task 14 except the objects to be picked and placed have the same shape. There are threedifferent shapes: rectangular-like (e.g. block and pallet), circle-like (e.g. ring and bowl), and undetermined for the rest.
30

VIMA: General Robot Manipulation with Multimodal Prompts• Success Criteria: All objects with the same shape as the container are within the container.
• Oracle Trajectory: Shown in Fig. A.17 with its multimodal prompt.

Put all objects with the same profile as

into it.

Figure A.17: Visual Reasoning: Task 15
Task 16:

Put the target object into the container, and then put one of its old neighbors into the same container.

• Prompt:First put {object}1
into {object}2then put the object
previously at its {direction} into the same {object}2 .

that was• Description: Objects in image placeholders {object}1 and {object}2 are the target object to be picked and thecontainer, respectively. We then ask the agent to put one of old neighbors of the previous target object into the same
container. The old neighboring object is specified through cardinal directions {north, south, west, east}.
• Success Criteria: The target object and the correct neighboring object are inside the container.
• Oracle Trajectory: Shown in Fig. A.18 with its multimodal prompt.

First putintothen put the object that was previously at its west into the same.

Figure A.18: Visual Reasoning: Task 16
Task 17: Pick and place the target object specified in the prompt into different containers in order then restore to the initial
container.
• Prompt:
Put {object}1
container.

into {object}2.

Finally restore it into its original• Description: The object in the image placeholder {object}1 is the target object to be manipulated across thetask. There are more than one target containers (e.g. “Put {object}1 into {object}2 then {object}3 .
Finally restore it into its original container” for two target containers to be placed in order).
The rest of spawned containers naturally becomes distractors.
31

VIMA: General Robot Manipulation with Multimodal Prompts• Success Criteria: The target object is first put into multiple containers following the specific order. Finally it should
be restored into its original container.
• Oracle Trajectory: Shown in Fig.A.19 with its multimodal prompt.

Put

into

then

. Finally restore it into its original container.

Figure A.19: Visual Reasoning: Task 17

32

VIMA: General Robot Manipulation with Multimodal Prompts

C. Model Architecture
In this section, we provide comprehensive details about VIMA model architecture as well as other adapted baseline methods.
We implement all models in PyTorch (Paszke et al., 2019) and adapt Transformer-related implementation from Wolf et al.
(2019).
C.1. Summary of Different Methods
We summarize differences between VIMA and other baseline variants in Table 1. In the column “Prompt Conditioning”,
an alternative to cross-attention is to first concatenate prompt and interaction into a big sequence, then repetitively apply
transformer decoders to predict actions. It is referred to as “Direct modeling”. The relative computation cost is quadratically
proportional to the number of observation tokens.
Table 1: Comparison of different methods.
Visual Tokenizer

Prompt Conditioning

Number of Observation Tokens per Step

Ours

Object tokens consisting ofcropped images and bounding boxes

Cross-attention

Equal to number of objects, typically 3 to 8

VIMA-Gato(Reed et al., 2022)

Image patch tokens encoded by a ViT

Direct modeling

Equal to number of image patches, 16

VIMA-Flamingo(Alayrac et al., 2022)

Image patch tokens encoded by a ViT,
further downsampled by a Perceiver module

Cross-attention

Equal to number of learned query vectors, 4

VIMA-GPT(Brown et al., 2020)

Single image token encoded by a ViT

Direct modeling

Single visual feature, 1

C.2. VIMA Architecture
C.2.1. M ULTIMODAL P ROMPT T OKENIZATIONAs introduced in Section 4, there are 3 types of input formats in multimodal prompts, namely (1) text inputs, (2) images offull scenes, and (3) images of single objects.
For text inputs, we follow the standard pipeline in NLP to first tokenize raw language to discrete indices through pre-trained
t5-base tokenizer. We then obtain corresponding word tokens from the embedding look-up of the pre-trained t5-basemodel. For images of full scenes, we first parse the scene through a fine-tuned Mask R-CNN detection model (He et al.,2017; Wu et al., 2019) to extract individual objects.
Each object representation
contains a bounding box and a cropped

image. The bounding box is in the format of xcenter , ycenter , height, width . We normalize it to be within [0, 1] by dividing
each dimension with corresponding upper-bound value. We then pass it through a bounding box encoder MLP and obtaina feature vector. To process the cropped image, we first pad non-square image to a square by padding along the shorter
dimension. We then resize it to a pre-configured size and pass it through a ViT (trained from scratch) to obtain the image
feature. Finally, an object token is obtained by concatenating the bounding box feature and the image feature and mapping to
the embedding dimension. For images of single objects, we obtain tokens in the same way except with a dummy boundingbox. Detailed model hyperparameters about tokenization are listed in Table 2.
After obtaining a sequence of prompt tokens, we follow Tsimpoukelli et al. (2021) to pass it through a pre-trained t5-baseencoder to obtain encoded prompt. Note that we add adapter MLP between object tokens and the T5 encoder. To prevent
catastrophic forgetting, VIMA only fine-tunes the last two layers of the language encoder with layer-wise learning rate
decay (He et al., 2021) but freezes all other layers. We adopt learned absolute positional embedding. Model hyperparameters
are listed in Table 2 as well.
C.2.2. O BSERVATION E NCODING
Since all RGB observations are images of full scenes, we follow the same procedure discussed above to obtain flattened
object tokens. Because we provide RGBs from two views (frontal and top-down), we order object tokens by following the33

VIMA: General Robot Manipulation with Multimodal Prompts

Table 2: Model hyperparameters for multimodal prompt tokenization.
Hyperparameter

Value

Text Tokenization
Tokenizer
Embedding Dimension

t5-base tokenizer
768

Image Tokenization
32 × 32
16
768
4
24

ViT Input Image Size
ViT Patch Size
ViT Width
ViT Layer
ViT Number of Heads

Bounding Box MLP
Hidden Dimension
Hidden Depth

7682

Prompt Encoding
Pre-Trained LM
Unfrozen Last N Layers
Positional Embedding
Token Adapter MLP Deptht5-base2
Absolute2order of frontal, top-down . We one-hot encode the state of the end effector. We then concatenate object tokens with theend-effector state and transform to observation tokens. We adopt learned absolute positional embedding. Detailed model
hyperparameters about observation encoding is provided in Table 3.

Table 3: Model hyperparameters for observation encoding.
Hyperparameter

Value

Observation Token Dimension
End Effector Embedding Dimension
Positional Embedding

768
2
Absolute

C.2.3. ACTION E NCODING
Since our model is conditioned on observation-action interleaved history, we also tokenize past actions. We follow common
practice in Chen et al. (2021); Zheng et al. (2022) to encode past actions with a two-layer MLP. It has a hidden dimension of
256. We then map outputs to token dimension and obtain action tokens.
C.2.4. S EQUENCE M ODELINGThe robot controller in VIMA is a causal decoder that autoregressively predicts actions. To condition the decoder on prompt
tokens, we perform cross-attention between history tokens and prompt tokens (Figure 3). Concretely, we pass history tokens
as the query sequence and prompt tokens as the key-value sequence into cross-attention blocks. The output prompt-aware
trajectory tokens then go through causal self-attention blocks. We alternate cross-attention and self-attention L times. This
procedure is technically described in Pseudocode 1.
34

VIMA: General Robot Manipulation with Multimodal Prompts
def xattn_sequence_modeling(
prompt_tokens,# the [L, d] prompt tokens (L=prompt length)
obs_tokens,# the [T, d] obs tokens (T=time step)
act_tokens,# the [T-1, d] action tokens
traj_pos_embd,# learned positional embedding for trajectory
prompt_pos_embd,# learned positional embedding for prompt):# interleave obs and action tokens
traj_tokens = interleave(obs_tokens, act_tokens) # [2T-1, d]# add positional embedding to trajectory tokens
x = traj_tokens + traj_pos_embd# add positional embedding to prompt tokens
prompt_tokens = prompt_tokens + prompt_pos_embd# apply xattn and causal self-attn
for i in range(num_layers):# cross-attentionx = x + attn_i(q=x, kv=prompt_tokens)# feed forwardx = x + ffw_xattn_i(x)# self-attentionx = x + causal_attn_i(q=x, kv=x)# feed forwardx = x + ffw_i(x)# the last token is the predicted action token
predicted_act_token = x[-1]return predicted_act_token

Pseudocode 1: Cross-attention operation that conditions the trajectory history on prompt. We repetitively alternate crossattention and self-attention to model the trajectory given a specific task.

C.2.5. ACTION D ECODING
After obtaining the predicted action token, we map it to the action space A and obtain the predicted action. This is achieved
though a group of action heads. Since the action space consists of two SE(2) poses, for each pose we use six independentheads to decode discrete actions (two for xy coordinate and four for rotation represented in quaternion). These discrete
actions are then integrated and mapped to continuous actions through affine transformation. The two poses are modeled
independently. Early ablations show that this independent modeling is equally good as alternative techniques, such as
autoregressive decoding (Vinyals et al., 2019; OpenAI et al., 2019). Detailed model hyperparameters are listed in Table 4.

Table 4: Model hyperparameters for action decoders.
Hyperparameter

Value

Hidden Dimension
Hidden Depth
Activation
X-Axis Discrete Bins
Y-Axis Discrete Bins
Rotation Discrete Bins

512
2
ReLU
50
100
50

C.3. Baselines Architectures
In this section, we elaborate model architectures for adapted baseline methods. Some components such as the action decoder
are same across all models. Therefore, we only discuss unique model components.
35

VIMA: General Robot Manipulation with Multimodal Prompts

C.3.1. VIMA-G ATO
Gato (Reed et al., 2022) introduces a decoder-only model that solves tasks from multiple domains including robotics,
video game, image captioning, language modeling, etc. Different tasks are specified by supplying the model with an initial
sequence of corresponding tokens. For example, in tasks involving decision making, these tokens include observation andaction tokens. For fair comparison, we provide the same conditioning as VIMA, i.e., our multimodal tokenized prompts.
This adapted baseline variant is referred to as “VIMA-Gato”. Similar to our method, VIMA-Gato also predicts actionsin an autoregressive manner. VIMA-Gato and our method share the same training philosophy to only optimize the causal
behavior cloning objective. However, unlike our method that adopts an object-centric representation to treat individual
objects as observation tokens, VIMA-Gato divides input images into patches and encodes them by a ViT (Dosovitskiy et al.,2020) to produce observation tokens. Furthermore, VIMA-Gato relies on causal self-attention to model entire trajectory
sequences starting with prompt tokens. Hyperparameters of VIMA-Gato’s ViT is listed in Table 5. The transformer-decoder
style sequence modeling is technically illustrated in Pseudocode 2.
Table 5: Model hyperparameters for ViT used in baseline methods.
Hyperparameter

Value

Image Size
Patch Size
ViT Width
ViT Layers
ViT Heads

64 × 128
32
768
4
24

def causal_sequence_modeling(
prompt_tokens, # the [L, d] prompt tokens (L=prompt length)
sep_token,# the [1, d] learned token to separate prompt and trajectory history
obs_tokens,# the [T, d] obs tokens (T=time step)
act_tokens,# the [T-1, d] action tokens
pos_embd,# learned positional embedding):# interleave obs and action tokens
traj_tokens = interleave(obs_tokens, act_tokens) # [2T-1, d]# assemble input tokens
x = concat([prompt_tokens, sep_token, traj_tokens])x = x + pos_embd# apply GPT layers with causal maskfor i in range(num_layers):# self-attentionx = x + causal_attn_i(q=x, kv=x)# feed forwardx = x + ffw_i(x)# the last token is the predicted action token
predicted_act_token = x[-1]return predicted_act_token

Pseudocode 2: Plain sequence modeling that directly concatenates prompt and trajectory history and repetitively performcausal self-attention operation.

C.3.2. VIMA-F LAMINGO
Flamingo (Alayrac et al., 2022) is a vision-language model that learns to generate textual completion in response tomultimodal prompts. It embeds a variable number of prompt images into a fixed number of tokens via the PerceiverResampler module (Jaegle et al., 2021b), and conditions the language decoder on encoded prompts by cross-attention.
Flamingo does not work with embodied agents out of the box. We adapt it by replacing the output layer with robot action
heads (hyperparameters listed in Table 4) and using tokenized rollout histories as inputs. We thus call it “VIMA-Flamingo”.
36

VIMA: General Robot Manipulation with Multimodal Prompts

We train it end-to-end with causal behavior cloning loss. VIMA-Flamingo differs from ours since it processes image
observations into a fixed number of visual tokens through a learned Perceiver Resampler. Model hyperparameters for our
reimplementation of the Perceiver Resampler is listed in Table 6.
Table 6: Model hyperparameters for Perceiver Resampler used in VIMA-Flamingo method.
Hyperparameter

Value

Number of Latent Queries
Number of Blocks
Self-Attn per Block
Self-Attn Heads
Cross-Attn Heads

4
4
4
24
24

C.3.3. VIMA-GPT
VIMA-GPT is a GPT-based behavior cloning agent conditioned on tokenized multimodal prompts with the GPT architecture.
It autoregressively decodes next actions given multimodal prompts and interaction histories. We optimize this method
end-to-end with causal behavior cloning loss. Similar to prior works of casting RL problems as sequence modeling (Chenet al., 2021; Janner et al., 2021; Zheng et al., 2022), it encodes an image into a single “state” token through a learned ViT
encoder. It also directly models entire trajectory sequences prepended with prompt tokens. Therefore, it differs from our
method in the representation of observation tokens and prompt conditioning. For visual tokenizer, we employ a learned ViT
with hyperparameters listed in Table 5.
C.4. Mask R-CNN Detection Model
Finally, we elaborate on the mask R-CNN model (He et al., 2017) for scene parsing and object extraction. We fine-tune apre-trained lightweight mask R-CNN (mask rcnn R 50 FPN 3x) from Wu et al. (2019) to adapt to scenes and images in
our tabletop environment. We fine-tune it on a subset of agent training dataset. It contains 100 trajectories for each task,
resulting in 22,741 images and 61,822 annotations in total. We use learning rate 5 × 10−4 and train for 10 epochs. Duringmodel selection, we particularly favor models with high recall to reduce the number of missed objects. To compensate forresulting false-positives, we adopt object augmentation during agent training (Appendix, Sec. D).
A visualization of its output is provided in Figure A.20. We do not use the predicted object names in our models.

Figure A.20: Visualization of fine-tuned mask R-CNN. Left: Prediction from the detection model. Right: Ground-truth
scene parsing. The detection model agrees well with ground-truth objects.

37

VIMA: General Robot Manipulation with Multimodal Prompts

D. VIMA Training Details
We follow the best practice to train Transformer models using the AdamW optimizer (Loshchilov & Hutter, 2019), learning
rate warm-up, cosine annealing (Loshchilov & Hutter, 2017), etc. Training hyperparameters are provided in Table 7. We use
GEGLU activation (Shazeer, 2020) inside Transformer models across all methods.
Table 7: Hyperparameters used during training.
Hyperparameter

Value

Learning Rate
Warmup StepsLR Cosine Annealing Steps
Weight Decay
Dropout
Gradient Clip Threshold

0.0001
7K
17K
0
0.1
1.0

To make trained models robust to detection inaccuracies and failures, we apply object augmentation by randomly injecting
false-positive detection outputs. Concretely, for observation at each time step, we sample number of augmented objects i.i.d.
naugmented objects ∼ Cat(K, p), where Cat(·) denotes a categorical distribution with K supports parameterized by p. For eachaugmented object, we then randomly sample a bounding box and corresponding cropped image to add to object tokens. In
our experiments, we set p = {0 : 0.95, 1 : 0.05} with K = 2.
D.1. Vary Model Capacity
We train a spectrum of 7 models ranging from 2M to 200M parameters. To vary the model capacity, we follow priorwork (Chowdhery et al., 2022) to change embedding dimension and number of layers. We list configurations for methods
with cross-attention prompt conditioning (i.e., ours and VIMA-Flamingo) in Table 8, and configurations for methods onlywith causal self-attention (i.e., VIMA-Gato and VIMA-GPT) in Table 9.
Table 8: Configurations for differently sized models with cross-attention prompt conditioning.
Model Size (M)Embedding Dimension

Num Blocks

X-Attn Heads

Self-Attn Heads

2
4
9
20
43
92
200

256
256
320
384
512
640
768

1
2
3
4
5
7
11

8
8
10
12
16
20
24

8
8
10
12
16
20
24

Table 9: Configurations for differently sized models with causal self-attention prompt conditioning.
Model Size (M)Embedding Dimension

Num Blocks

Self-Attn Heads

2
4
9
20
43
92
200

64
96
192
320
512
768
768

1
2
3
4
5
7
18

2
3
6
10
16
24
24

38

VIMA: General Robot Manipulation with Multimodal Prompts

E. Extended Experiment Results
E.1. Training Time and Compute
All experiments are conducted on cluster nodes, each with 8 NVIDIA V100 GPUs. The largest experiment takes approximately one day. We utilize DDP (distributed data parallel) to accelerate the training.
E.2. Model Scaling
E.2.1. N UMERICAL R ESULTS
We present numerical results that constitute Fig. 4 in Table 10. The claim of “up to 2.9× improvement” made in Abstract
and Sec. 1 is calculated as follows. The best competing variant is VIMA-Gato. On the hardest L4, our method shows themost significant relative improvement with a model size of 20M. We compute the performance gap, divide by VIMA-Gato’s
performance, and only keep the first digit after decimal to obtain the result.
Table 10: Model scaling numerical results that constitute Fig. 4. Numbers in the first row indicate robot controller parametercount.
Level

Method

2M

4M

9M

20M

43M

92M

200M

L1

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

76.5
37.6
42.4
30.0

79.2
42.6
48.9
37.0

77.4
44.2
45.6
44.9

77.1
46.1
46.6
48.5

78.2
49.5
47.0
48.0

79.3
57.0
47.2
47.9

81.5
58.0
47.4
46.9

L2

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

77.1
35.9
41.0
29.8

79.2
39.3
46.5
35.0

78.2
41.3
44.6
43.3

77.6
44.1
44.6
45.8

77.6
46.6
45.4
45.9

80.1
53.9
47.1
47.4

81.5
53.1
46.0
46.9

L3

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

77.3
29.0
35.0
25.3

77.8
33.2
41.9
29.3

78.5
37.5
39.2
39.0

77.3
40.2
40.5
43.5

81.8
42.5
40.3
43.0

81.9
45.6
42.1
42.6

78.7
46.0
40.7
42.2

L4

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

25.7
13.3
12.3
11.1

49.0
13.2
11.6
10.3

47.1
12.2
10.7
12.7

48.8
12.3
12.1
14.2

49.0
12.8
10.7
11.8

49.6
13.5
11.1
12.1

48.6
16.8
12.1
12.1

E.3. Data Scaling
E.3.1. D ETAILED S ETUPTo ensure all methods are fairly pre-trained on the same amount of data (i.e., they have roughly the same amount of built-in
information, thus the x-axis in Fig. 4 faithfully corresponds to the extra bits of information seen during further training), we
initialize variants that directly learn from raw pixels with MVP pre-trained ViT (Xiao et al., 2022; Radosavovic et al., 2022).
It is further MAE fine-tuned (He et al., 2021), using the same in-domain data as for the Mask R-CNN object detector. Notethat the MVP pre-trained then domain fine-tuned ViT also updates weights jointly with robot controllers later on. We use the
ViT-B backbone from MVP. The in-domain data for fine-tuning include 100 trajectories for each task.
E.3.2. N UMERICAL R ESULTS
We present numerical results that constitute Fig. 4 in Table 11. The claim of “2.7× improvement” made in Abstract andSec. 1 is calculated as follows. The best competing variant is VIMA-Gato that achieves 12.2% average success rate trainedwith full data on L4. Our method trained with 10% data achieves 46% average success rate on the same level. We computethe performance gap, divide by VIMA-Gato’s performance, and only keep the first digit after decimal to obtain the result.
39

VIMA: General Robot Manipulation with Multimodal Prompts

Table 11: Data scaling numerical results that constitute Fig. 4. Numbers in the first row indicate the size of training dataset.
Level

Method

0.1%

1%

10%

Full (100%)

L1

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

0.0
0.0
0.0
0.0

36.3
11.5
2.0
6.0

76.3
41.5
37.7
30.9

79.3
57.5
52.3
52.8

L2

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

0.0
0.0
0.0
0.0

34.3
10.1
2.0
6.0

75.8
37.9
33.8
29.7

80.1
41.2
32.6
40.3

L3

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

0.0
0.0
0.0
0.0

15.4
10.2
1.0
5.5

73.2
34.8
33.1
28.6

81.9
40.9
33.6
39.2

L4

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

0.0
0.0
0.0
0.0

17.0
2.7
0.5
1.1

46.0
10.8
11.2
7.1

49.6
12.2
12.0
14.3

E.3.3. W HAT I F BASELINE VARIANTS ’ V I T I S T RAINED FROM S CRATCH ?
We further investigate what if baseline variants’ ViT is trained from scratch and end-to-end with the robot controllers. We
visualize the results in Fig. A.21 and numerically present them in Table 12. We annotate with arrows to indicate performanceincrease (↑) and decrease (↓). We highlight two findings.
First, MVP pre-trained ViT is most beneficial in the setting with sufficient in-domain training data (i.e., the 10% data
scenario). It boosts the performance for the most competing baseline variant VIMA-Gato. However, in other settings
with abundant in-domain data (i.e., the full data scenario) or insufficient in-domain data (i.e., 1% and 0.1% scenarios), theadvantage of MVP pre-trained ViT diminishes and it even becomes detrimental. This aligns with the finding in previous
empirical studies (Hansen et al., 2022). Second, in settings with reasonable amounts of in-domain data (i.e., the 1%, 10%,and 100% scenarios), our recommended recipe always outperforms other variants. We notice that such a data demand
generally can be satisfied by both simulated robotics data (Mandlekar et al., 2021) and real robotics data (Dasari et al.,2019; Brohan et al., 2022). Therefore, it demonstrates that our recommended recipe is highly sample-efficient compared toalternative designs, especially under practical settings.
L1

L2

L4

L3

80

80

80

60

60

60

60

40

40

40

40

20

20

20

20

Success Rate (%)

80

0
6 x 10²

0
10

10³

10

6 x 10²

Data Size

0
10

10³

10

6 x 10²

Data Size

Ours

VIMA-Gato

0
10

10³

10

6 x 10²

Data Size

VIMA-Flamingo

10

10³

10

Data Size

VIMA-GPT

Figure A.21: Data scaling when baseline variants’ ViT is trained from scratch. In settings with reasonable amounts of
in-domain data (i.e., the 1%, 10%, and 100% scenarios), our recommended recipe always outperforms other variants.
E.4. Vary T5 Encoder Sizes
We vary the size of the pre-trained T5 encoder (Raffel et al., 2020) to study the effect of prompt encoding. We experimentwith three T5 model capacities: t5-small (30M), t5-base (111M), and t5-large (368M). For all T5 variants, we
40VIMA: General Robot Manipulation with Multimodal Prompts

Table 12: Data scaling when baseline variants’ ViT is trained from scratch, indicated inside parentheses. ↑ and ↓ denote
performance increase and decrease. Numbers in the first row represent the size of training dataset.
Level

Method

0.1%

1%

10%

Full (100%)

L1

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

0.0
0.0 (2.2 ↑)
0.0 (0.0)
0.0 (0.0)

36.3
11.5 (11.9 ↑)
2.0 (6.2 ↑)6.0 (17.0 ↑)76.3
41.5 (26.5 ↓)37.7 (33.9 ↓)30.9 (38.9 ↑)79.3
57.5 (57.0 ↓)52.3 (47.2 ↓)52.8 (47.9 ↓)

L2

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

0.0
0.0 (2.0 ↑)
0.0 (0.0)
0.0 (0.0)

34.3
10.1 (11.4 ↑)
2.0 (6.0 ↑)6.0 (15.2 ↑)75.8
37.9 (24.6 ↓)33.8 (34.1 ↑)29.7 (36.7 ↑)80.1
41.2 (53.9 ↑)32.6 (47.1 ↑)40.3 (47.4 ↑)

L3

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

0.0
0.0 (1.1 ↑)
0.0 (0.0)
0.0 (0.0)

15.4
10.2 (10.1 ↓)
1.0 (5.4 ↑)
5.5 (15.0 ↑)

73.2
34.8 (22.6 ↓)33.1 (31.0 ↓)28.6 (35.7 ↑ )81.9
40.9 (45.6 ↑)33.6 (42.1 ↑)39.2 (42.2 ↑)

L4

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

0.0
0.0 (0.0)
0.0 (0.0)
0.0 (0.0)

17.0
2.7 (2.5 ↓)
0.5 (0.0 ↓)
1.1 (4.1 ↑)46.0
10.8 (5.8 ↓)11.2 (8.3 ↓)
7.1 (9.0 ↑)49.6
12.2 (13.5 ↑)12.0 (11.1 ↓)14.3 (12.1 ↓)fine-tune the last two layers and freeze all other layers. We fix the parameter count of the decision-making part to be 200M.
As shown in Table 13, we find no significant difference among the variants. Thus we set the standard t5-base as default
for all our models.
Table 13: Performances of our method with differently sized pre-trained T5 prompt encoder. We fix the parameter count of
the decision-making part to be 200M.

L1
L2
L3
L4

t5-small (30M)

t5-base (111M)

t5-large (368M)

78.8
79.0
80.3
49.1

81.5
81.5
78.7
48.6

80.8
81.0
81.0
49.3

E.5. Policy Robustness
Increasing Amounts of Distractors. We study the policy robustness against increasing amounts of distractors in scenes.
For all tasks being evaluated, we add one more distractor object. We run our largest VIMA model with 200M parameters.
The result is presented in Table 14.
It turns out that the performance of VIMA degrades minimally with more distractors than the training distribution. This
indicates that our agent has learned a reasonably robust policy against objects that are irrelevant to the task.
Table 14: Evaluation results on tasks with increased amounts of distractors. We fix the parameter count of the decisionmaking part to be 200M.

OriginalMore Distractors
Relevant Performance Decrease (%)

41

L1

L2

L3

L4

81.5
78.5
3.6

81.5
78.6
3.5

78.7
72.9
7.3

48.6
47.8
1.6

VIMA: General Robot Manipulation with Multimodal Prompts

Imperfect Prompts. We then study the policy robustness against imperfect prompts, including incomplete prompts(randomly masking out words with <UNK> token) and corrupted prompts (randomly swapping words, which could havechanged the task meaning altogether). We run our largest VIMA model with 200M parameters, results are shown in Table 15.
Our well-trained model exhibits minimal performance decrease when evaluated on masked prompts and minor decrease on
corrupted prompts. We attribute this robustness to the high-quality pre-trained T5 language backbone.
Table 15: Evaluation results with incomplete and corrupted prompts. We fix the parameter count of the decision-making partto be 200M.

Original
Incomplete Prompts
Corrupted Prompts
Relevant Performance Decrease w/ Incomplete Prompts (%)
Relevant Performance Decrease w/ Corrupted Prompts (%)

42

L1

L2

L3

L4

81.5
80.8
78.2
0.8
4.2

81.5
81.1
78.1
0.4
4.3

78.7
77.0
73.8
2.1
6.6

48.6
48.0
45.3
1.2
7.2

VIMA: General Robot Manipulation with Multimodal Prompts

F. Extended Related WorkIn this section, we provide an extended review of related work as complementary to Section 6.
Multi-Task Learning by Sequence Modeling. In computer vision, Mask R-CNN (He et al., 2017), UberNet (Kokkinos,
2016), and 12-in-1 (Lu et al., 2020) leverage a single backbone model with multiple independent heads for different tasks.
UVim (Kolesnikov et al., 2022) is another unified approach for vision that uses a language model to generate the guiding
code for a second model to predict raw vision outputs. In multimodal learning, numerous works (Lu et al., 2022; Wanget al., 2022a; Zellers et al., 2021; 2022; Buch et al., 2022; Fu et al., 2021; Yang et al., 2022) investigate the unificationof image, video, audio, and/or language modalities to deliver multi-purpose foundation models, although most of which
are not equipped with decision-making capabilities. BEiT-3 (Wang et al., 2022c) performs masked data modeling on images,
texts and image-text pairs to pre-train a backbone for various downstream tasks. MetaMorph (Gupta et al., 2022a) learns
a universal controller over a modular robot design space.
Foundation Models for Embodied Agents. Embodied agent research (Duan et al., 2022; Batra et al., 2020; Ravichandar
et al., 2020; Collins et al., 2021) is adopting the large-scale pre-training paradigm (Yang et al., 2023), powered by a collectionof learning environments (Abramson et al., 2020; Shridhar et al., 2020; Savva et al., 2019; Puig et al., 2018; Team et al., 2021;
Toyama et al., 2021; Shi et al., 2017). From the aspect of pre-training for better representations, Reid et al. (2022) finetunes from LLM checkpoints to accelerate policy learning. LaTTe (Bucker et al., 2022) and Embodied-CLIP (Khandelwal
et al., 2021) leverage the frozen visual and textual representations of CLIP (Radford et al., 2021) for robotic manipulation.
MaskDP (Liu et al., 2022a) pre-trains bidirectional transformers for various downstream embodied tasks. From the perspective of leveraging transformer as agent architecture, methods such as Dasari & Gupta (2020) and MOSAIC (Zhao et al.,2022) achieve superior performance in one-shot video imitation tasks. They both use the self-attention mechanism with
auxiliary losses such as inverse dynamics loss (Dasari & Gupta, 2020) and contrastive loss (Zhao et al., 2022) to learn robot
controllers. InstructRL (Liu et al., 2022b) leverages jointly pre-trained vision-language models as robot agents to perform
manipulation tasks. From the perspective of large language models for robot learning, Socratic Models (Zeng et al., 2022)
composes multiple vision and language foundation models for multimodal reasoning in videos. ROSIE (Yu et al., 2023)
leverages text-to-image diffusion models to augment existing robotic dataset (Brohan et al., 2022) via inpainting. MOO (Minderer et al., 2022) adopts a similar object-centric representation as ours for open-world object manipulation. Furthermore,
Voyager (Wang et al., 2023) develops a LLM-powered agent operating in an open-ended virtual world (Fan et al., 2022).
Robot Manipulation and Benchmarks. There are many prior works that are not mentioned in the main paper thatstudy different robotic manipulation tasks, such as instruction following (Shridhar et al., 2021; Lynch & Sermanet, 2021),
constraint satisfaction (Bharadhwaj et al., 2021; Srinivasan et al., 2020; Thananjeyan et al., 2021), one-shot imitation (Paineet al., 2018; Huang et al., 2019; Dasari & Gupta, 2020; Aceituno et al., 2021; Zhao et al., 2022), rearrangement (Weihs
et al., 2021; Szot et al., 2021; Liu et al., 2021; Ehsani et al., 2021; Gan et al., 2021; Stengel-Eskin et al., 2022), andreasoning (Gupta et al., 2019; Ahmed et al., 2021; Toyer et al., 2020; Lim et al., 2021). Multiple simulation benchmarks
are introduced to study the above tasks: 1) Indoor simulation environments: Habitat (Savva et al., 2019; Szot et al., 2021)
is equipped with a high-performance 3D simulator for fast rendering and proposes a suite of common tasks for assistive
robots. 2) Tabletop environments: RLBench (James et al., 2019) and SURREAL (Fan et al., 2018; 2019) are other widelyused simulator benchmarks studying robotics manipulation with tabletop settings. STRETCH-P&P (Zhang & Weihs,2023) studies generalization across goals for reset-free reinforcement learning. All these aforementioned simulators andbenchmarks do not natively support task specification and prompting with multiple modalities.

43

VIMA: General Robot Manipulation with Multimodal Prompts

G. Limitations and Further Discussions
Reliance on a separate object detector. VIMA inherits the errors from the standalone object detector, which may struggle
in the cases of occlusion or out-of-distribution object forms. However, using object detectors is not entirely without merits.
First, it allows us to seamlessly switch to stronger detection models when they become available. For example, we can switch
to object detectors that are more robust and open-vocabulary, such as OWL-ViT (Minderer et al., 2022). This would enable
VIMA to transfer to real-world scenarios with minimal modifications. Second, by leveraging pre-trained vision pipelines,
several concurrent works have demonstrated the superiority of object-centric representation in robot manipulation. For example, VIOLA (Zhu et al., 2022) achieves better performance with a pre-trained Region Proposal Network (Ren et al., 2015).
MOO (Stone et al., 2023) also shows that a robot agent with OWL-ViT (Minderer et al., 2022) as the object detector significantly outperforms RT-1 (Brohan et al., 2022), which directly learns from raw pixels, on various real-world manipulation
tasks. In fact, MOO (Stone et al., 2023) includes a baseline called “VIMA-like” that already demonstrates strong performance on real robots under real-world scenarios. As we witness image segmentation is becoming more robust and generalpurpose (Kirillov et al., 2023), we envision such design choice will become more effective and further gain more popularity.
Limited simulator realism and task complexity. Our goal with VIMA-B ENCH is to explore the multi-task ability,
generalization, and understanding of multi-modality. Therefore, these aspects are not the primary focus of this work.
However, we envision future works can combine this formulation with more physically realistic simulators such as Zhu
et al. (2020), Srivastava et al. (2021), and Mittal et al. (2023).
Limited action primitives. We inherit the same high-level action space from well-established prior works, such as
Transporter (Zeng et al., 2020). While “pick-and-place” and “wipe” seem simple, they do cover a wide range of tabletop
manipulation tasks and are crucial to industrial use cases like warehouse robots (Yoon et al., 2003; Berscheid et al.,2020; Devin et al., 2020; Song et al., 2019). While VIMA is currently using these two actions, the algorithm design is
general-purpose and does not make assumptions about the particular action choices. For example, VIMA would require
only minimal modifications to support more low-level action spaces like joint-torque control.

44

VIMA: General Robot Manipulation with Multimodal Prompts

H. Full Tables
This section contains more detailed tables that correspond to the results in Figure 4. Specifically, we show breakdown resultson each task that constitute the model scaling results in Tables 16, 17, 18, and 19.
Table 16: L1 level generalization results. Model indicates robot controller parameter count. Integers in the first row refer toindices of tasks described in Appendix, Sec. B.
Model

Method

01

02

03

04

05

06

07

09

11

12

15

16

17

2M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

100.0
62.0
56.0
59.5

100.0
61.0
56.0
50.5

100.0
22.5
53.5
7.5

96.0
13.5
36.5
7.0

37.0
7.0
37.5
0.5

100.0
44.5
45.0
43.5

100.0
54.0
55.5
49.5

9.5
4.0
3.5
2.0

87.0
48.0
54.0
61.5

64.0
85.0
83.5
76.5

93.5
44.5
40.5
27.5

45.0
43.0
28.5
5.0

63.0
0.0
2.0
0.0

4M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

100.0
61.0
61.0
58.0

100.0
61.5
62.0
55.0

100.0
8.0
57.5
17.5

99.5
46.0
47.5
25.0

45.5
32.5
45.0
12.0

100.0
45.5
49.5
47.5

100.0
57.0
59.5
54.5

10.5
1.0
5.5
3.0

90.5
64.5
80.0
59.5

90.0
86.0
83.5
80.5

96.5
46.5
40.5
27.0

46.5
42.5
43.0
41.5

51.0
2.0
2.0
0.5

9M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

100.0
59.0
58.5
58.5

100.0
61.0
60.0
54.5

100.0
41.0
46.0
40.5

99.5
50.5
49.0
47.5

51.5
38.5
42.5
37.5

100.0
47.5
45.5
47.5

100.0
59.5
60.5
58.5

13.0
9.5
4.0
9.0

82.5
58.0
66.5
72.0

58.5
80.5
81.5
85.0

96.0
44.0
36.5
38.5

42.0
24.0
41.5
34.0

63.5
2.5
1.0
1.0

20M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

100.0
61.5
63.0
60.5

100.0
62.0
61.5
64.0

100.0
32.5
55.0
50.5

100.0
49.0
50.0
44.0

59.5
38.0
42.5
41.0

100.0
46.0
41.5
48.0

100.0
60.0
58.0
61.5

13.5
5.0
6.0
7.0

74.0
68.0
62.0
85.0

72.5
83.0
83.0
84.0

96.5
47.0
44.0
44.5

39.5
46.5
38.5
39.0

47.5
2.0
1.0
2.5

43M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

100.0
57.0
54.5
58.0

100.0
65.5
57.0
60.5

100.0
59.0
54.5
69.5

100.0
57.5
54.0
53.5

57.0
43.5
45.0
41.5

99.5
50.0
43.5
47.0

100.0
56.0
55.5
55.5

15.0
5.0
6.0
4.0

86.0
67.0
67.5
66.5

69.5
83.5
82.5
81.5

99.0
63.0
49.0
45.0

40.0
37.0
40.5
40.5

51.5
0.0
1.5
1.5

92M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

100.0
76.5
56.0
57.0

100.0
59.5
56.0
57.5

99.5
90.0
65.5
58.5

100.0
56.5
50.5
53.0

58.0
44.5
41.0
45.0

100.0
48.5
48.0
51.0

100.0
68.5
56.0
61.0

14.0
14.0
3.0
8.0

80.5
64.5
70.0
65.5

92.0
89.5
87.0
87.0

98.5
85.0
41.5
46.0

40.5
43.0
38.0
33.0

48.5
1.5
2.0
1.0

200M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

100.0
79.0
56.0
62.0

100.0
68.0
58.5
57.5

99.5
91.5
63.0
41.0

100.0
57.0
48.5
55.5

56.5
44.5
38.0
45.5

100.0
54.0
48.5
47.5

100.0
74.0
62.5
54.5

18.0
18.0
3.5
8.5

77.0
61.0
66.5
77.0

93.0
88.5
86.0
81.5

97.0
83.5
40.0
41.0

76.5
33.5
43.5
38.0

43.0
2.5
2.5
0.5

45

VIMA: General Robot Manipulation with Multimodal Prompts

Table 17: L2 level generalization results. Model indicates robot controller parameter count. Integers in the first row refer toindices of tasks described in Appendix, Sec. B.
Model

Method

01

02

03

04

05

06

07

09

11

12

15

16

17

2M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

100.0
49.5
45.5
51.0

100.0
49.0
46.0
45.5

100.0
23.0
56.0
9.5

95.5
17.5
39.5
7.0

37.5
5.0
35.5
0.5

100.0
47.5
49.0
45.5

100.0
46.5
47.0
45.0

17.5
5.5
9.0
0.0

87.5
50.0
53.0
65.0

67.0
82.5
80.0
81.5

97.5
49.0
43.0
32.0

46.0
42.0
29.5
5.0

54.5
0.5
1.0
0.0

4M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

100.0
44.5
49.5
50.5

100.0
52.0
50.5
49.5

100.0
9.0
51.0
16.5

99.5
39.0
48.0
25.5

44.5
28.0
43.0
12.0

99.5
49.5
50.5
41.0

100.0
48.5
53.5
47.0

14.5
2.0
5.5
4.0

89.5
64.0
81.5
63.0

91.5
86.5
82.5
79.0

95.5
44.5
48.5
28.5

43.0
42.5
39.5
39.0

52.5
2.0
1.0
0.0

9M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

100.0
47.0
48.0
48.5

100.0
44.5
47.5
47.0

100.0
39.5
49.0
43.5

100.0
46.5
52.5
47.0

49.5
37.5
42.0
37.0

100.0
48.5
47.5
47.5

100.0
51.0
48.5
45.5

19.0
5.5
8.5
10.5

80.5
59.0
66.0
74.5

65.0
83.0
81.5
85.0

95.5
51.5
45.5
43.5

42.0
23.5
42.0
33.0

66.0
1.0
2.0
1.0

20M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

100.0
44.0
48.5
50.5

100.0
51.5
49.0
49.5

100.0
39.0
55.5
53.0

100.0
51.0
48.0
44.5

61.0
38.5
42.5
43.5

100.0
47.5
46.5
47.0

100.0
52.5
52.0
46.0

16.5
6.0
6.0
8.0

75.5
65.5
66.0
83.5

75.0
84.0
82.0
80.0

96.0
52.5
47.5
46.5

37.5
40.5
37.0
41.0

47.5
1.0
0.5
2.5

43M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

100.0
50.0
48.0
45.0

100.0
51.5
46.5
49.0

100.0
53.0
52.0
64.5

100.0
57.5
51.5
53.5

54.5
42.5
43.5
40.0

100.0
47.0
45.0
46.5

100.0
51.0
51.5
48.5

14.5
8.5
5.0
8.5

83.5
67.0
68.0
68.0

69.0
83.0
81.5
82.0

98.0
63.5
52.5
50.0

38.5
32.0
44.0
40.0

51.5
0.5
1.5
1.5

92M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

100.0
64.5
53.0
50.5

100.0
50.0
48.5
55.0

99.0
83.0
67.5
55.5

100.0
56.5
53.0
54.5

57.5
46.0
43.0
43.0

99.5
55.5
49.0
51.5

100.0
54.5
53.0
54.5

19.5
10.5
4.5
10.5

81.5
64.5
67.0
68.5

92.0
92.5
84.0
87.0

97.5
81.0
50.0
49.5

42.0
42.0
40.0
34.0

53.5
1.0
1.0
3.0

200M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

100.0
56.5
51.0
52.0

100.0
53.5
52.5
52.0

99.5
88.0
61.5
49.5

100.0
55.5
49.5
54.5

54.5
43.5
38.5
45.5

100.0
55.5
47.5
52.5

100.0
53.0
55.5
51.0

17.5
14.0
5.5
11.0

77.0
63.0
70.5
76.5

93.0
90.5
82.0
84.0

98.5
81.5
42.0
43.0

75.0
33.0
39.0
38.0

45.0
4.0
3.0
0.5

46

VIMA: General Robot Manipulation with Multimodal Prompts

Table 18: L3 level generalization results. Model indicates robot controller parameter count. Integers in the first row refer toindices of tasks described in Appendix, Sec. B.
Model

Method

01

02

03

04

05

06

07

09

11

15

16

17

2M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

100.0
45.5
41.5
48.5

100.0
48.0
54.5
50.0

100.0
28.0
50.5
5.0

98.0
23.0
39.5
7.0

34.5
3.0
29.0
2.5

100.0
45.5
45.0
47.0

99.5
45.0
49.5
45.5

17.0
2.5
5.5
2.0

97.5
40.5
57.5
69.5

94.0
29.5
22.5
22.5

48.5
37.0
25.0
5.0

39.0
1.0
0.0
0.0

4M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

99.5
44.5
46.0
44.0

100.0
55.0
53.5
47.0

100.0
9.5
59.0
14.5

98.0
37.5
49.5
22.0

44.0
24.5
35.5
9.0

99.5
47.0
47.5
39.5

99.5
50.0
48.0
40.0

12.0
3.5
7.0
2.0

92.5
60.0
87.5
62.0

98.5
30.5
30.5
28.5

47.0
37.5
39.5
43.0

43.5
0.0
0.0
1.0

9M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

99.5
44.5
44.5
49.0

100.0
53.5
53.0
50.5

100.0
42.5
53.0
39.0

98.5
52.0
48.5
46.5

44.5
28.0
33.0
30.5

99.5
46.5
41.0
43.0

99.5
51.5
45.5
52.0

18.5
6.0
8.0
6.5

88.5
67.0
72.5
84.0

98.5
35.0
27.0
31.5

48.5
23.0
44.5
35.0

46.5
0.5
0.5
0.5

20M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

98.0
46.5
47.0
50.0

100.0
55.0
54.5
60.5

100.0
44.5
53.0
56.5

98.5
57.0
55.0
48.0

55.5
31.5
36.0
33.5

100.0
47.5
42.5
51.0

99.5
51.5
48.0
46.0

15.0
2.5
6.5
6.5

88.5
72.5
70.0
92.5

99.5
30.5
33.0
32.5

44.0
44.0
41.5
43.5

29.5
0.0
0.0
1.5

43M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

99.0
44.0
47.0
47.5

100.0
55.0
54.0
57.0

100.0
59.5
56.5
61.0

98.0
58.0
52.5
50.0

47.5
34.0
37.0
34.5

100.0
49.0
46.5
48.0

99.5
54.0
44.5
53.5

18.5
7.0
6.5
8.0

93.0
74.0
69.5
74.0

98.0
40.0
27.0
40.5

45.0
35.0
43.0
41.5

84.0
0.5
0.0
0.5

92M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

99.0
61.5
51.0
50.0

99.5
54.0
51.5
56.5

99.5
73.0
68.0
63.0

97.0
56.0
51.5
52.5

58.0
36.0
36.5
32.0

100.0
50.0
50.5
49.5

99.0
48.0
47.0
53.0

13.0
17.0
6.0
5.0

94.5
66.5
69.5
78.0

99.0
44.0
28.0
34.5

42.0
41.5
45.5
37.5

82.5
0.0
0.5
0.0

200M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

99.0
51.0
49.0
52.0

100.0
58.0
50.0
51.0

100.0
84.5
66.5
55.0

97.0
56.5
47.0
49.5

54.5
35.5
35.0
40.0

100.0
53.5
47.5
46.0

99.0
49.0
50.0
50.5

17.5
15.0
4.0
5.0

90.5
65.0
66.0
82.0

97.5
52.0
30.5
37.0

46.0
33.0
43.5
38.0

43.5
0.0
0.5
1.5

47

VIMA: General Robot Manipulation with Multimodal Prompts

Table 19: L4 level generalization results. Model indicates robot controller parameter count. Integers in the first row refer toindices of tasks described in Appendix, Sec. B.
Model

Method

08

10

13

14

2M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

6.5
21.0
22.0
22.5

0.0
0.5
0.0
0.0

0.0
0.0
0.0
0.0

96.5
32.0
27.5
22.0

4M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

97.0
17.0
17.0
19.0

0.0
2.0
0.5
0.0

0.0
0.0
0.0
0.0

99.0
34.0
29.0
22.5

9M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

92.0
18.0
21.5
20.5

0.0
0.0
0.0
0.0

0.0
0.0
0.0
0.0

96.5
31.0
21.5
30.5

20M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

100.0
20.5
21.0
20.5

0.0
0.0
0.0
0.5

0.0
0.0
0.0
0.0

95.5
29.0
27.5
36.0

43M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

99.0
21.0
18.5
17.5

0.0
0.0
0.0
0.0

0.0
0.0
0.0
0.0

97.0
30.5
24.5
30.0

92M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

100.0
22.0
19.5
18.5

0.0
0.0
0.0
0.5

0.0
0.0
0.0
0.0

98.5
32.0
25.0
29.5

200M

Ours
VIMA-Gato
VIMA-Flamingo
VIMA-GPT

100.0
30.5
24.5
20.0

0.0
0.0
0.0
0.0

0.0
0.0
0.0
0.0

94.5
37.0
24.0
28.5

48