Vision-Language Foundation Models as Effective Robot Imitators
Authors: Xinghang Li
Minghuan Liu
Hanbo Zhang
Cunjun Yu
Jie Xu
Hongtao Wu
Chilam Cheang
Ya Jing
Weinan Zhang
Huaping Liu
Hang Li
Tao Kong
Venue: ICLR 2024 spotlight (2024)
Keywords: Large Visual Language Model
Robotics
Imitation Learning
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Recent progress in vision language foundation models has shown their ability to understand multimodal data and resolve complicated vision language tasks, including robotics manipulation. We seek a straightforward way of making use of existing vision-language models (VLMs) with simple fine-tuning on robotics data.
To this end, we derive a simple and novel vision-language manipulation framework, dubbed RoboFlamingo, built upon the open-source VLMs, OpenFlamingo. Unlike prior works, RoboFlamingo utilizes pre-trained VLMs for single-step vision-language comprehension, models sequential history information with an explicit policy head, and is slightly fine-tuned by imitation learning only on language-conditioned manipulation datasets. Such a decomposition provides RoboFlamingo the flexibility for open-loop control and deployment on low-performance platforms. By exceeding the state-of-the-art performance with a large margin on the tested benchmark, we show RoboFlamingo can be an effective and competitive alternative to adapt VLMs to robot control.
Our extensive experimental results also reveal several interesting conclusions regarding the behavior of different pre-trained VLMs on manipulation tasks. We believe RoboFlamingo has the potential to be a cost-effective and easy-to-use solution for robotics manipulation, empowering everyone with the ability to fine-tune their own robotics policy. Our code will be made public upon acceptance.

Published as a conference paper at ICLR 2024
VISION-LANGUAGE FOUNDATION MODELS AS EFFEC- TIVE ROBOT IMITATORS
Xinghang Li Li Minghuan Liu Hanbo Zhang', Cunjun Yu', Jie Xu², Hongtao Wu?, Chilam Cheang?, Ya Jing", Weinan Zhang', Huaping Liu 1ss Hang Li², Tao Kong "Tsinghua University, "ByteDance Research,
Shanghai Jíao Tong University, "National University of Singapore
Hixingha23@mail pos.sssiion tinggua. edu. hpliu@ts siniuutuinnguuu.cmm edu. cn, (minghuanliu, wnzhang @ eet..ee..ucooom en, kongtao@bytedance, com
ABSTRACT
Recent progress in vision language foundation models has shown their ability to derstand multimodal data and resolve complicated vision language tasks, including robotics manipulation. We seek way of making of existing -sionnlanguage models (VLMs) with fine-tuning on robotics data. To this end, we derive a simple and novel vision-language manipulation framework, dubbed RoboFlamingo, built upon the open-source VLMs, OpenFlamingo. Unlike prior works, RoboFlamingo utilizes pre-trained VLMs for single-step languae comprehension. models sequential history information with an explicit policy head, and is slightly fine- tuned by imitation learning only on Ianguage-condtioone Such a decomposition provides RoboFlaminge the flexibility for open-loop control and deployment on low-performance platforms. By surpassing the state-of-the- art performance the benchmark by significant margin, we demonstrate that RoboFlamingo presents itself as an effective and competitive alternative for adapt- ing VLMs to robot control. Our extensive experimental results also reveal several interesting conclusions regarding the behavior of different pre-trained VLMs on manipulation tasks. RoboFlamingg server, and we believe it has the potential to be a cost-effective and easy-to-use so- lution for robotics manipulation, empowering everyone with the ability fine-tune their own robotics policy. Codes and models will be public.
manipulation datasets.
be trained evaluated a single GPU
INTRODUCTION
Recent progress in vision-language foundation models (VLM) has presented their exhilarating ability in modeling and aligning the representation of images and words, and the unlimited potential to resolve wide range of downstream tasks with multi-modality data, for instance, visual question- answering (Li et al., 2023; Zhou et al., 2022), image captioning (Zeng et al., 2022; Wang et al., 2022; Li et al., 2021), human-agent interactions (Liu et al., 2022b; Oertel al., 2020; Seaborn al., 2021). These successes, undeniably, encourage people to imagine generalist robot equipped with such lision-language comprehension ability interact naturally with humans and perform complex manipulation tasks.
Therefore, aim to explore integrating vision-language foundation models to serve as robot manipulation policies. While there have been some previous studies that incorporated large language models (LLMs)) and vision-language models (VLMs) robot systems high-level planners (Ahn et al., 2022; Driess et al., 2023), making use of them directly for low-level control still poses challenges. Most VLMs are trained on static image-language pairs, whereas roboties tasks require video comprehension for closed-loop control. Additionally, VLM outputs primarily consist of language tokens, which significantly differ in representation compared to robot actions. A recent work (Brohan et al., 2023), namely Robotics Transformer 2 (RT-2), has demonstrated a possible solution for adapting VLMs to low-level robot control. However, democratizing such an expensive framework for all robotics practitioners proves difficult it utilizes private models and necessitates
1 Equal contribution. Works done during the first authors' internship ByteDance. ""Corresponding authors.

---

Published as a conference paper at ICLR 2024
From scratch
Cofretaree
cancaa thi
Adion
Viasa Longuage Model
Freeet: 1. Sreso the bottle
Largege
Goode
Testractten:
heat
planii Lollie
Vrige ciptio
Viea DSA
waat 10
Medaddaee
Inatraccionn Pus
teal.
AnkeFlvsingg
eteauet
Acfice Hood
Adian
Vireil Langaage
Vaacl langange Model
Weigt PO20
Adice
Socatioral Charge
ptenttc tato 296).
teage?
Figure 1: Comparison among RoboFlamingo and existing vision-language manipulation solutions.
co-fine-tuning extensive vision-language data to fully showcase its effectiveness. Consequently, there is an urgent need for robot communities to have a low-cost alternative solution that effectively enables robot manipulation policy with VLMs.
To this end, we introduce RoboFlamingo, a novel vision- -asiun-languag manipulation framework that lever- ages publicly accessible pre-trained VLMs to effectively construct manipulation policies for robotics. Specifically, RoboF lamingo is grounded upon the open-source VL.M, OpenFlamingo (Awadalla et al., 2023), and resolves the challenge by decoupling visual-language understanding and decision-making. Unlike previous works, RoboFlamingo takes advantage of pre-trained VLMs mainly for understand- ing vision observations and language instructions at every decision step, models the historical features with an explicit policy head, and is fine-tuned solely on language- condiiiiino manipulation datasets using imitation learning. With such decomposition, we only need combine small amount of robotics demonstration to adapt the model to downstream manipulation tasks, and RoboFlaminge also offers flexibility for open-looy control and deployment on low- oo-peeffrmaance platforms. Moreover, tasks, Robo
benefiting from the pre-training on extensive rision-langual of-the-art performance with a large margin over previous works, and generalizes well to zero-shot be trained or evaluated
achieves state-
settings and environments. It is worth noting that Robo Paaoilam gle GPU server. As a result, we believe RoboFlamingo can be a cost-effective yet high- sigh-performance solution for robot manipulation, empowering everyone with the ability to fine-tune their own robots with VLMs.
sin-
Through extensive experiments, we demonstrate that RoboFlamingo outperforms existing methods by a clear margin. Specifically, we evaluate its performance using the Composing Actions from Language and Vision benchmark (CALVIN) (Mees al., 2022b), a widely-recognized simulation benchmark for long-horizon language-conditioned tasks. Our findings indicate that RoboFlaming is an effective and competitive alternative for adapting VLMs to robot control, achieving performance improvement that is two times greater compared to the previous state-of of-the-art method. Our comprehensive results also yield valuable insights into the use of pre-trained VLMs for robot manipulation tasks, offering potential directions for further research and development.
2 RELATED WORK
Language can be the most intuitive and pivotal interface for human-robot interaction, enabling non-expert humans to seamlessly convey their instructions to robots for achieving diverse tasks. Consequently, the realm of language-conditioned multi-task manipulation has garnered substantial attention in recent years. Intuitively, such tasks require robots to have good understanding of not only the visual captures of the outside world, but also the instructions represented by words. With the strong representation ability of pre-trained vision and language models, lot of previous works have incorporated me--trained models into the learning framework. Among them, we roughly classify them into the following three categories, which is also illustratively compared in Fig. 1.
2

---

Published as a conference paper at ICLR 2024
Fine-tuning. While some early works such as Jang et al. (2022); Lynch & Sermanet (2020) trained a vision encoder and a language encoder to learn representations for the imput language and vision data from manipulation tasks, some recent work directly takes pre-trained models obtain great representations, then trains the policy model beyond them from scratch or fine-tuning the whole model. For instance, Jiang et al. (2023) utilizes pre-trained T5 (Raffel et al., 2020) model encode the multi-modal prompts, and learn the actions by fine-tuning the T5 model and additionally training an object encoder and attention layers. HULC (Mees et al., 2022a) utilizes the vision encoder of Lynch & Sermanet (2020) trained on the CALVIN dataset (Mees et al., 2022b) and some wre-trained language encoder models such as sentence transformer (Reimers & Gurevych, 2019), and their HULC++ (Mees et al., 2023) also fine-tunes these encoders. Besides, Brohan et al. (2022) proposed RT-1, ie., robotics transformers, a 35M vision-language-actiinn action and aligns the vision, language, and action in the token space and is trained on large amount of real-world manipulation dataset, using the Universal Sentence Encoder (Cer et al., 2018) to obtain the language embedding and the pre-trained EfficientNet-B3 (Tan & Le, 2019) the vision tokenizer.
model (VLA)) which tokenizes the
LLM planning. Some approaches have exploited large language models (LLMs) as a powerful zero-shot planner, e.g., SayCan Ahn et al. (2022), to generate step-by-step pre-defined plans with human- intaaninteracctvve prompts on given tasks, subsequently instructing different pre-trained low-level skill policies to execute those plans and finish multiple tasks. Compared to other works, the controlling policies do not require any ability to understand instructions, but rely on the pre-trained frozen L.M to select necessary skills.
Co-Fine-Tunnin Tuning. Driess al. (2023) proposed 540B PaLM-E model, showing different way of utilizing the pre-trained vision and language model. Specifically, they choose different pre-trained models to encode the input scene, and the PaI.M (Chowdhery al., 2022) model the hase model, train the model to generate pre-defined multi-step plans described by language by fine-tuning the whole VLM end-to-end using both mobile manipulation question-answering data and auxiliary vision-languss training data such as image captioning and visual question answering data collected from the web. Similar to SayCan (Ahn et al., 2022), they require low-level control policies to execute the generated plans. Motivated by PaLM-E, Brohan et al. (2023) further introduced RT-2, which is based on RT- but is adapted to use large vision-language backbones like PaLI-X (Chen et al., 2023) and M-L (Driess et al., 2023), training the policy utilizing both robot manipulation data and web data. Their method reveals that VLMs have the potential to be adapted into robot manipulation, yet their key co- co-fine-tuning training strategy requires a large amount of both web-scale data vision-language data and low-level robot actions. Additionally, the VLMs and the data they use private, making it hard for every robotics practitioner to play such solution for their own.
Although these previous models somehow bridge the gap between vision and language on robot manipulation tasks, they either reply on low-level skill policies, like SayCan and PaLM-E; or train whole large model, such as RT-1; or require huge amount of vision-language data and computational resources ensure the model learns the manipulation policy without forgetting the great alignment of vision and language. Compared with these works, our proposed RoboFlamingo is simple and intuitive solution to casily adapt existing VLMs (OpenFlamingo (Alayrac al., 2022; Awadalla et al., 2023) used in this paper). only requiring fine-tuning on a small number of manipulation demonstrations. We hope RoboFlamingo provides a different perspective on fully leveraging the ability of VLMs, while requiring less data collection costs and computing consumption to make it an open and easy-to-use solution for everyone.
3 BACKGROUND
Robot manipulation. In this paper, we mainly consider robot manipulation tasks, where the agent (robot) does not have access to the ground-truth state of the environment, but visual observations from different cameras and its own proprioception states. As for the action space, it often includes the relative target pose and open/closed state of the gripper. For instance, in the testbed of CALVIN (Mees et al., 2022b), the observations consist of simulated camera captures from two different views, and the action is a 7-DoF control of a Franka Emika Panda robot arm with a parallel gripper, and the instructions are reaching goals, i.e., the tfter-the-fact descriptions.
3

---

Published as a conference paper at ICLR 2024
Pooling
Vision Enooder
Self Altention (ULM)
Resampler
Cross Aftention
82
Policy Head Stecked Adiera
Acton
ViI
Tokenizer
and pat 11 1800 Lhe dr drawer.
Figure The illustration of the proposed RoboFlaminge framework. The Flamingo backbone models single-step observations, and the temporal features are modeled by the policy head.
Imitation learning. Imitation learning (Pomerleau, 1988; Zhang et al., 2018; Liu et al., 2020; Jang al., 2022) allows the agent to mimic the manipulation plans from instruction-labeled expert play data D = {(T, } where D is the number of trajectories. / is the language instruction, and T {or, at)] contains preceding states and actions to reach the goal described by the given instruction. The learning objective can be simply concluded as a maximum likelihood goal-conditioned imitation objective to learn the policy Te.
|r|
Σ
log No (0.))
(I)
4 ROBOFLAMINGO
RoboFlamingo, a generalized robotics agent, excels in resolving language-conditioned manipulation tasks. The key idea to draw help from pre-trained vision-language models (VLMs) and adapt them to manipulation policies, acquiring the ability of object grounding, language comprehension, vision-language alignment, and long-horizon planning. Particularly, Flamingo looks into one of the popular VLMs, Flamingo (Alayrac et al., 2022), and takes its open- open-source model Open- Flamingo (Awadalla et al., 2023) as the backbone. The overview of RoboFlamingo is shown in Fig. 2. To adapt large-scale vision-language models to robotic manipulation, RoboFlaminge simply adds policy head for and-to-end finetuning. It addresses three main challenges: 1) it adapts vision-language models with static image inputs to video observations; 2) it generates robot control signals instead of text-only outputs; 3) it requires a limited amount of downstream robotic manipulation data to achieve high performance and generality with billions of trainable parameters. We will elaborate the design of Flaminno in this section.
4.1 LANGUAGE-CONDDITONED ROBOT CONTROL
The problem of Language-conditionel robot control can be modeled a goal-conditioned partially ob- servable Markov decision process (GC-POMDP) (Liu et al., 2022a): M = (S, O, T, TT Po, L, o. where S and O are the set of states and observations separately, is the action space, S x is the environment dynamics function, Po S + 0, is the initial state distribution, d(s) indi- cate if the task is successful, and f(lss S + Ó the observation function. Specifically, for each controlling episode, the robot is given goal, represented by a length-M M free-form language instruction 1 C at every time step t, and the observations o are typically two images It G2 from phire-per camera and a gripper camera. The controlling policy can be modeled as a goal-conditioned policy (aao,, S x >> A and the action is typically the desired relative position and pose of the gripper, along with its open/close status.
In our RoboFlamingo, the policy ne (alo, () is parameterized by 0. It consists of a backbone based on Flamingo fe and policy head pa. The backbone takes visual observations and language-represente goals as the input and provides a latent fused representation at each time step for the policy head: X, for l). Then the policy head further predicts the action to fulfill the specified goal for the robot: a, = Po (X,, 1), where h,-1 is the hidden state from the last step that encodes the history information for decision-making We will introduce each module in detail in the following sections.
4

---

Published as a conference paper at ICLR 2024
4.2 THE FLAMINGO BACKBONE
We adopt the Flamingo backbone So for understanding the vision and language inputs at every decision step. Overall, Flamingo encodes the vision observations to the latent tokens by vision encoder; and then fuses them with language goals through the feature fusion decoder. We explain these parts in detail below.
4.2. VISION ENCODER
The vision encoder consists of a vision transformer (ViT) (Yuan et al., 2021) and perceiver resampler (Alayrac al., 2022). At every time step f, the two-view camera images 4t G, are encoded to X,, consisting of a visual token sequence, through the ViT module:
X = VIT(A, G.),
where X;= of the encoded output. After encoding, RoboFlamingo utilizes perceiver resampler to compress the number of visual tokens from to N,. In detail, the resampler maintains a set of learnable parameters and utilizes the attention mechanism to reduce the number of token sequences to K. Formally, the resampler is formulated as:
.ii) (f ( represents the visual token sequence at t, N represents the token number
Qn R VVn
Kn= = X Vn= X;W X;" softmax
Vd
where QR Rd corresponds the learnable parameters of the resampler and serves the query vector, d is the hidden dimension size, W W E Rd represents the linear transformation matrix of key and value, de is the feature dimension of the visual token, KR and VR are the transformed key and value vector of vision input V.
4.2.2 FEATURE FUSION DECODER
The compressed visual tokens output from the resampler X; E N..d are further passed to the feature fusion decoder, which is designed to generate the vision-language joint embedding by fusing the language instruction with the encoded vision feature X²- In RoboFlamingo we utilize the pre-trained decoder from OpenFlamingo (Awadalla al., 2023) and fine-tune the decoder module following the way as in Awadalla et al. (2023). Specifically, the decoder consists of L layers, each of which involves transformer decoder layer and cross-aftention layer. The transformer layers are directly copied from a pre- trained language model (such as LlaMA (Touvron et al., 2023), GPT- Neox (Black al., 2022) and MPT (Team al., 2023)) and are frozen during the whole training process; the cross-attention layer takes the language token as query, and the encoded visual token as key and value, which fine-tuned by imitation learning objectives on manipulation data (see following sub-sections). Formally, if we denote E Rdd the i - th embedded token of the instruction, M the instruction length, and X E xd is the embedded matrix of the instruction, then the embedded natural language instruction should be = (211 x2, ". ( and output X of the I-th decoder layer given the input X; is computed by:
X = Tanh(a) MLP(A(X (.W.x + X x!+1 = MLP(A({)WS X/WS X(WS) X!
where X = X, X corresponds to the output of the gated cross- -otssnattention layer at time instant t, We W. W E ddd represents the learnable parameters of the cross-attention layer. a E R is learnable gate parameter to control the mixing weights for stability. W W W e Rdxd represents the parameters of the self-attention layer and MLP represents a multi-layer perceptron network. With the deep interaction of the vision and language token, expect the output X, X = (xf -/2,,.. ... xfMM at time step t to be an informative languae joint embedding for robot manipulation.
4.3 POLICY HEAD
The output X from the feature fusion decoder is trained as the representation of the vision obser- vation and language instruction, which will be further translated into low-level control signals. To
5

---

Published as a conference paper at ICLR 2024
achieve this, we simply adopt additional policy head pe to predict the action, e.g., the 7 DoF end-effector pose and gripper status. We test various strategies to model the historical observation sequences and behave the policy head, e.g., long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) network with an MLP for the final prediction; a decoder-only transformer (Brown al., 2020) similarly with an MIP; Section 5 for more details). Taking the LSTM version as an example, with the vision-language joint embedding sequence Xt' we obtain an aggregated embedding through max-pooling operation over the token dimension and predict the action as:
single ML.P that only models single-step information (see
X MaxPooling h4 LSTM(XX, h,, af aoose gripper MLP(ha),
(5)
where h4 represents the hidden state at i, and dt ag agripper are the predicted end-effector pose and gripper status.
4.4 TRAINING OBJECTIVE
We utilize maximum likelihood imitation learning objectives to fine-tune the proposed pre-trained backbone and the policy head. Concretely, the desired relative pose is optimized via regression loss (we use mean squared error (MSE) loss) and the gripper status uses classification loss (we use binary cross-entropy (BCE) loss):
,44 Σ MSE(a (S _(to0t) Agripper BCE(a (agripaeeeer ,04
gripper agripper),
agripper is the demonstration for end effector pose and gripper status at timestep 4 where af ,0, Agripper corresponds to the weight of gripper loss.
In the training procedure, we follow the fine-tuning paradigm of OpenFlamingo by only training the parameters of the resampler, the gated cross-attention module of each decoder layer, and the policy head while freezing all other parameters.
EXPERIMENTS
We conduct extensive experiments to examine the proposed RoboFlamingo solution, and answer how pre-trained VL models (VLMs) benefit aanuuage-oonitiions robotic manipulation. In short, investigate RoboFlamingo from the following perspectives:
Effectiveness. We wonder the imitation learning performance of RoboFlamingo by training on the given demonstration data.
2. Zero-shot Generalization. We focus on generalization unseen tasks. In other words, we study how the model will behave given unseen vision contexts like different objects, even with unseen instructions.
3. Ablation Studies. We further explore the essential factors that matter in adapting VLMs to robot control policy in the framework of RoboFlamingo.
5.1 BENCHMARK AND BASELINES
We choose CALVIN (Mees et al., 2022b), an open-source simulated benchmark to learn long-horizon lunguage-condttioned tasks, as our testbed, and the corresponding datasets as our imitation learning demonstration data. CALVIN encompasses total of 34 distinct tasks and evaluates 1000 unique instruction chains for sequential tasks. each experiment, the robot is required to successfully complete sequences of up to five language instructions consecutively. The policy for each consecutive task is dependent on goal instruction, and the agent advances to the subsequent goal only successfully accomplishes the current task. The dataset contains four splits for environments A, B, C, and D. Each consists of hours of human-ter teleoperaeeee recording data (more than 2 million steps) that might contain sub-optimal behavior, and only 1% of that data is annotated with language instructions (~22 thousand steps). See Fig. 4 in Appendix A. 1 for a more detailed description and visualized examples of the benchmark.

---

Published as a conference paper at ICLR 2024
Table 1: The imitation performance on various settings, all results are reported using the best-behaved model checkpoints. Full and Lang denote if the model is trained using unpaired vision data (i.e., vision data without language pairs); Freeze-emb refers to freezing the embedding layer of the fusion decoder; Enriched denote using GPT-4 enriched instructions. The gray rows denote numerical results evaluated by our re-trained model. We re-implement RT-1 and take the original code of HULC provided by Mees et al. (2022a). All other results are reported by Mees et al. (2022a).
Training
Task Completed in Sequence (Success Rate)
Method
Data
Split
4
Avg Len
MCTL
ABCD (Full)
0.373
0.027
0.002
0.000
0.000
HUILC
ARCD (Full)
0.889
0.753
0.587
0.475
0.383
3.06
HULC
ABCD
0.892
0.701
01:548
0.420
0.335
2.90
ABCD dlang)
0.323
0.227
Romiiinn (Chrr)
ARCD (1 angs
0.896
0.524
0.740
0.66
MCIL
(Pall)
0.013
0,002
0.000
0.300
0.31
HULC
ABC (Pull)
0.418
0.165
0,087
0.019
0.011
0.67
RT
ABC (Lang)
0.533
0.222
0.094
0.038
0.013
0.90
Robol Planinggo (Ouns)
ABC (Lamg)
0.824
0,619
0.466
0.331
0.235
2.48
HULC
ABCD (Full)
D (Fariched))
0.715
0:470
0.308
0.199
0.130
1.82
ARCI (.ang)
0:017
ABCD (1.ang)
(Eariched)
0.480
0.299
0.211
0.144
Ours (frezze-emb)
ABCD (1.ang)
D (Feriched)
0.737
0.530
0.385
0,275
0.192
2.12
We compare a set of well rell-performed baselines in CALVIN: (1) MCIL (Lynch & Sermanet, 2020): a scalable framework combining multitask imitation with free-form text conditioning, which learns language-conditionn
visuomotor policies, and is capable of following multiple human instructions over long horizon in dynamically accurate 3D tabletop setting. (2) HULC (Mees et al., 2022a): a hierarchical method that combines different observation and action spaces, auxiliary losses, and latent representations, which achieved the SoTA performance on CALVIN. (3) RT-1 (Brohan et al., 2022): robotics transformer, which directly predicts the controlling actions by action tokens, as well as vision and language inputs. RT-2 (Brohan et al., 2023) not experimentally compared since we have no access to their code, data, and model weights.
5.2 IMITATION PERFORMANCE
We train Roboinamingo (with the M-3B-IFT backbone) using demonstrations only with language annotation from all 4 splits (A, B, C, and D), and evaluate the imitation performance on episodes sampled split D) (ABCD + D).The performance comparison is shown in Tab. 1. RoboFlamingo outperforms all baseline methods over all metrics by a large margin, even for those methods that are trained on the full set of data. This demonstrates the effectiveness of RoboFlamingo as the solution for robotics manipulation, enabling VLMs to become effective robot imitators.
In addition, the success rate of the subsequent tasks can be regarded as a notion of the generalizability of the manipulation policies, since the initial state of subsequent task highly relies on the ending state of its former task. The later a task is arranged in the task sequence, the more diverse initial state is, which will need more powerful visual-language alignment abilities to successfully complete the task. Among all methods, Robo Flamingo achieves the highest success rate over the latter tasks. This demonstrates that RoboFlamingo is able to utilize the visual-languase grounding ability of pre-trained VLMs. In the appendix, we further include the results of RoboFlamingo co-trained with COCO and VQA data (Appendix B. and compare with recent roboties representation works (Appendix B.2). Appendix B. also reveals how the original VL abilities change after tee-tunngg.
5.3 ZERO-SHOR GENERALIZATION
To assess the zero-shot generalization ability, we evaluate RoboFlamingo in two aspects: vision and language. For vision generalization, we train models on splits A, B, and C and test on split D, which presents a different vision context. Our method significantly outperforms baselines in this vision generalization scenario (ABC + D), as shown in Tab. 1. Regarding language generalization, enrich the language setting by generating 50 synonymous instructions for each task using GPT- 4 (Achiam et al., 2023). We then randomly sample instructions during evaluation. Our method exhibits superior performance compared all baselines in this language generalization setting.
7

---

Published as a conference paper at ICLR 2024
Table 2: Variants of VLMs tested. Pre-train denotes the original performance of VLM on the pre-training VL dataset, Best Avg. Len. denotes the best performance of the average success length of VLMs within epochs, and Mean Avg. Len. denotes the mean performance of the average success length of VLMs of the last 3 epochs on CALVIN.
Buckbone
LLM
Total
Triinble
lasir
Pre-inined (Public, 4-4hnt)
Name
VQAx2
77.3
MI-B-1177
82.7
45.7
4.02
G 4B IFT
G-48
GPT
4B 30
1B
X
85.8
81.8
49.0
49.0
2.79
1.67
3.72
3.53
L. 9B
LLMMA
4B
8900
76.33
2.71
Sunciaa Patto NN:
7A0 5.42 Taa
4mg
Teke Te4
5.6 0:67 D-a.1
D-44
(a) Various policy formulation.
(b) Different training paradigms.
(c) Open loop control.
Figure 3: Ablation studies on the ABCD - D setting.
Note that the success rate of RoboFlamingo on subsequent tasks dropped more than HULC does. This may be due to our approach directly using word tokens as input during training, which can result in larger variations for synonymous sentences compared to HULC using a frozen sentence model for embedding instructions. To address this, we freeze the embedding layer the feature fusion decoder in our method, leading to improved generalization and reduced performance drop.
5.4 ABLATION STUDIES
In this section, we conduct ablation studies for RoboFlamingo to answer the following questions:
1) How does RoboFlamingo perform with different policy heads/formulation??
2) Does vision-language (VL) pre-training improve downstream robotic tasks?
3) How do critical factors in VL pre-training affect robotic tasks?
5.4.1 How DOES RoboFlamingo PERFORM WITH DIFFERENT POLICY FORMULATIONS?
We test RoboFlamingo with different policy heads/formulations. particular, we compare 4 different implementations: (a) MLP w/o hist takes only the current observation as input to predict actions, which ignores the observation history. (b) M LP w hist takes the history frames into the vision encoder with position embedding, and encodes the history information through the cross-attention layers in the feature fusion decoder. (c) GPT and (d) LSTM both utilize the VLM backbone to
process single- frame observations and integrate the history with the policy head. GPT explicitly takes the visual history as input to predict the next action. LST M implicitly maintains a hidden state encode memory and predict the action. See Appendix C.1 for detailed illustration. We compare their best performance on the ABCD + D setting in Fig. 3 (a). M ML w/o hist performs the worst, indicating the importance of the history information in the manipulation task. M LP w hist performs better than L.P w/o hist, but is still much worse than GPT and L.TM. We hypothesize that has only seen image-text pairs during
this may stem from the fact that the VLM pre-training and cannot process consequent frames effectively. Further, the performance of GPT and LSTM are similar, we choose LSTM as the default choice due to its simplicity.
5.4.2 DOES VL PRE-TRAINING IMPROVE DOWNSTREAM ROBOTIC TASKS?
To verify the necessity of VL pre-training we train the same model without loading the pre-trained parameters of the cross-attention layers and the resampler trained by OpenFlamingo models (denoted
8

---

Published as a conference paper at ICLR 2024
Table 3: The performance on 10% language annotated data on ABCD + D setting, All variants are trained and evaluated for the same training epochs.
Task Completed in Sequence (Success Rate)
Method
2
3
5
Avg Len
M-3B
0.047
0.003
0.000
0.000
0.000
0.05
M-3B-IFT
0.120
0.007
0.000
0.000
0.000
G-4B
0,420
0.054
0.003
0,000
0.000
0.48
G-4B-IFT
0,448
0.084
0.014
0.003
0.001
0.55
M-9B
0.547
0.190
0.067
0.020
0.003
0.83
as No VL Pre-train). Besides, we also conduct an ablation study to freeze the pre-trained VLM and only train the policy head (denoted as No VL Finetune). As shown in Fig. (b), we can see that vision-language pre-training crucially improves the downstream robotic manipulation by a large margin. Besides, tuning on the VI model itself on robotic tasks is indispensable due to the limited capacity of the policy head.
5.4.3 How DO CRITICAL FACTORS IN VL PRE-TRAINING AFFECT ROBOTIC TASKS?
Model size. A larger model usually results in better VL performance. Yet, with full training data in CALVIN, find that the smaller model is competitive with the larger model (see the comparison in Tab. 2 and Appendix B.4). To further validate the impact of model size on downstream robotic tasks, we train different variants with 10% of language annotated data in CALVIN, which is only 0. 1% of the full data. From Tab. 3 we can observe that with limited training data, the performance of VLMs is highly related to the model size. The larger model achieves much higher performance, indicating that a larger VLM can be more data-efficient.
Instruction fine-tuning. Dnstruction-Finttuingg is a specialized technique that utilizes a further pre- training enhancement on the LLM with the IFT dataset (Conover et al., 2023; Peng et al., 2023), which provides a rich repertoire of instruction-folloiin behaviors that inform its capabilities in language- conditioned tasks. We find that LLMs with such training stage can improve the performance of the policy both seen and unseen scenarios, revealed by the performance improvements of M-3B-IFT against M-3B, and G-4B against G-4B shown in Tab. 2.
5.5 FLEXIBILITY OF DEPLOYMENT
Since our RoboFlamingo adopts a structure that separates the perception and policy module and leaves the main computation the perception module, we could perform open loop control to accelcrate the inference of RoboFlamingo. Instead of taking only the next action to execute and performing VLM inference every time for new observations predict future actions, open-loop control can be achieved by predicting an action sequence (stacked actions) with only one inference given the current observation, therefore alleviating the delay and the test-time computing requirement. However, indicated in Fig. 3 (c), directly implementing open loop control without re-training may lead to deteriorated performance, retraining the model with jump step demonstration could alleviate the performance drop.
6 CONCLUSION AND FUTURE WORK
This paper explores the potential of pre-trained vision-language models in advancing language- conditioned robotic manipulation. Our proposed RoboFlamingo, based on the pre-trained Open- Flamingo model, showcases state-of-the-at performance on benchmark dataset. Moreover, our experimental findings highlight the benefits of pre-trained models terms of data efficiency and zero-shot generalization ability. This research contributes to the ongoing efforts to develop intelligent robotic systems that can seamlessly understand and respond to human language instructions, paving the way for more intuitive and efficient human-robot collaboration. Due to the lack of real-robot data, this paper does not deploy on real-world robotics. To our delight, recent progress on sarge-scale real roboties data (Padalkar et al., 2023) has shown the potential of fine-tuning large VLMs for real robots, and the most exciting future work is to see how Robo CoboFlamingo will behave in real- worldworld tasks combined with such amount of data.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported by the National Natural Science Foundation of China under Grant 62025304. The Shanghai Jiao Tong University team is partially supported by National Key R&D Program of China (2022ZD01 4800)), Shanghai Municipal Science and Technology Major Project (0211H2DZX0112) and National Natural Science Foundation of China (62322603, 62076161). The author Minghuan Liu is also supported by the ByteDance Scholarship and Wu Wen Jun Honorary Doctoral Scholarship.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv, pp. 2303-0333800 00774, 2023.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do can, not say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0169) 2022.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308. 01390, 2023.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao. Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An open-source autoregressive language model. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models, 2022. URL https / aaaiiiiiiivv///ii/ org/ / 06745.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint 22w::::::2222666:: 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski. Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision- Vision-language-attoon models transfer web knowledge robotic control. arXiv preprint arXiv:2307.158/1 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario -eeppeess Steve Yuan, Chris Tar, et al. Universal sentence encoder. arXiv preprint arXiv:1803.11175. 2018.
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Car- los Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arX 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXii:2204.023111 2022.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open com 12/dol 1 y- ffrfiree Frrirrt-oop rrst-open- commercia a o
instruction-tune tunded 1lm, 2023. URL https: : htmp:://wwwee /tww:/www......o dat
10

---

Published as a conference paper at ICLR 2024
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzzan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXtv:2303.03378, 2023.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()):
1735-1780, 1997.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pp. 991-1002. PMLR, 2022.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. In Fortieth International Conference Machine Learning. 2023.
Siddharth Karamcheti, Suraj Nair, Annie Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. arXiv preprini 1rXiv:2302.12766, 2023.
Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang. Jingjing Xu, Xu Sun, et al. m it: A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv preprint arXiv:2306. 04387,, 2023.
Xinghang Li, Di Guo, Huaping Liu, and Fuchun Sun. Robotic indoor scene captioning from streaming video. 2021 IEEE International Conference on Robotics and Automation (ICRA). pp. 6109-6115, IEEE, 2021.
Minghuan Liu, Tairan He, Minkai Xu, and Weinan Zhang. Energy-based imitation learning, arXiv preprint vrXiv:2004.09399 2020.
Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-condiiooned reinforcement learning: Problems and solutions. arXiv preprint 2xXi::2001.0899, 2022a.
Xinzhu Liu, Xinghang Li, Di Guo, Sinan Tan, Huaping Liu, and Fuchun Sun. Embodied multi-agent task planning from ambiguous instruction. Proceedings of robotics: science and systems, New York City, NY, USA, pp. 1-14, 2022b.
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.07644 2020.
Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters (RA-L), 7(4): 11205-11212, 2022a.
Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L), 73)7337-7334, 2022b.
Oier Mees, Jessica Borja-Diaz, and Wolfram Burgard. Grounding language with visual affordances over unstructured data. 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11576-11582. IEEE. 2023.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint 12X1v 2022.
Catharine Oertel, Ginevra Castellano, Mohamed Chetouani, Jauwairia Nasir, Mohammad Obaid, Catherine Pelachaud, and Christopher Peters. Engagement in human-agen interaction: An overview. Frontiers in Robotics and AI, 7:92, 2020.
Abhishek Padalkar, Acom Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:23 10.08644, 2023.

---

Published as a conference paper at ICLR 2024
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, arXiv preprint arXiv:2304.03277, 2023.
Dean A Pomerieau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140): 2020. URL
http:// wrg.pppers seteer//2111 http///-/2.///////e//// 074.htma
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084, 2019.
Katie Seaborn, Norihisa P Miyake, Peter Pennefather, and Mihoko Stake-Matsuura Voice in human-ape aament interaction: survey, ACM Computing Surveys (CSUR), 54(4): 2021.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks.
In International conference machine learning, pp. 6105-61 14. PMLR, 2019.
MosaicML NLP Team al. Introducing mpt-7b: A standard for open-source, commercially usable Ilms, 2023. URL www. mosaicml. com/blog/mppt-77 Accessed, pp. 05-05, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1 139711 2023.
Tiannan Wang, Wangchunshu Zhou, Yan Zeng, and Xinsong Zhang. Efficientvlm: Fast and accurate vision-language models via knowledge distillation and modal-adaptive pruning. arXiv preprint arXiv:2210.07795, 2022.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokenn-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference computer vision, pp. 558-567, 2021.
Yan Zeng, Xinsong Zhang. Hang Li, Jiawei Wang, Jipeng Zhang, and Wangchunshu Zhou. 2--mmm: All-in-one pre-trained model for vision-language tasks. arXiv preprint 1r 2022.
Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality telcoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 5628-5633 IEEE, 2018.
Wangchunshu Zhou, Yan Zeng, Shizhe Diao, and Xinsong Zhang. Vlue: A multi-task multi-dimension benchmark for evaluating vision-language pre-training. In International Conference Machine Learning, pp. 37395-27411 PMLR, 2022.
12

---

Published as a conference paper at ICLR 2024
A ENVIRONMENTAL SETUPS
A.1 THE CALVIN BENCHMARK
CALVIN (Mees et al., 2022b) is an open-source simulated benchmark for evaluating long-horizon language-condttioned tasks.
As shown in Fig. 4, CALVIN includes four different environments A, B, C, and D, each of which consists of hours of tuman-tclcoperated recording data (more than 2 million trajectories) that might contain sub-optimal behavior, and only 1% of that data is annotated with language instructions (around 24 thousand trajectories). Each split is settled with different settings of objects and environments, aiming to validate the performance, robustness, and generality of policies trained with different data combinations.
This benchmark requires 7-DOF Franka Emika Panda robot arm with parallel gripper, utilizing onboard sensors and images from two camera views to successfully complete sequences of up to five language instructions consecutively. This setup further challenges the robot's ability to transition between various goals. CALVIN encompasses total of 34 distinct tasks and evaluates 1000 unique instruction chains for sequences. The robot is reset to a neutral position after each sequence to prevent any policy bias resulting from its initial pose. This neutral initialization eliminates any correlation between the initial state and the task, compelling the agent to rely solely on language cues to comprehend and solve the given task. The policy for each consecutive task is dependent on the instruction of the current goal, and the agent advances to the subsequent goal only it successfully accomplishes the current task.
The task sequenoe starts
take the pink block and ratwoe the iight
yr and lift the pink block
Split A
Split B
Split C
Split D
store grasoed bick siding cabinet
pull the handle
lift the blum black from sliding cabinet
Figure 4: The visualization of the four splits (left) and a full task sequence demonstration in CALVIN (right), The ID in the blue circle represents the end of which task among the five tasks. When the task is finished, the instructions for the next task will be given. For instance, the second image in the first row of the right side denotes the end of task 1, and the next step the instruction shown in the blue circle "2" will be given. Blue circle 0 is not an instruction.
A.2 EXAMPLES OF ENRICHED INSTRUCTIONS
Table 4: Examples of original and enriched instructions in the CALVIN benchmark.
Type
rotate zed blook right
pash pink block
move ciider left
open draver
1ift block 511der
and
pust
the sliding
Pull handle
Instruction
the right
pink
open drawer
sliding
the pink bock
glicing
Grep handle firsly and
Carefully bist the
clockoiie direction
doorway
dideddee the driwer
of drawer
Enricheed Instruction
Cive rightwurd spin to
the pink cube
your
Grip the humlic eusi force
upward
door
undakd the drawer
the slddin closet
Change the position the rod right
Dialcate the pink cube
Guide diding
Tug handle make
Grasp lim Bluz box
your left
prosageesyy the
the drwwer slile out
from volling drawer
13

---

Published as a conference paper at ICLR 2024
To validate the performance of the policies over diversified language expressions, utilize GPT4 to augment the language instruction in CALVIN. We showcase the enriched language instructions in Tab. 4. We they are organized with different words. As shown in Table 1, RohoFlamingo can still achieve better performance compared HUL.C
that the enriched instructions do have the same meaning the original one, yet
A.3 COMPUTING RESOURCE
All experiments involved in this paper are conducted on a single GPU server with 8 NVIDIA Tesla A100 GPUs, and the default batch size is 6 on each GPU. The MPT-3B model takes 13 hours of training per epoch and achieves the best performance at the 3rd epoch, while the MPT-9B model also takes hours of training per epoch and achieves the best performance the 4rd epoch.
B EXTENDED EXPERIMENTAL RESULTS
B.1 CO-TRAINING
From Tab. 1, the Enriched setting, we have noticed some evidence that the model may lose some foundation capabilities as the performance loss, which indicates that there is over-fitting during the fine-tuning, To further understand the phenomenon, we conduct further experiments by testing the fine-tuned RoboFlamingo model (the M-3B-IFT variant) on the COCO image caption and VQAv2, which verify our conjecture (see Tab. 6). To prevent such problems, we choose to co-train RoboFlamingo (the M-3B-IFT variant) with VQA and COCO datasets during fine-tuning on the robotics dataset. We test the co-train model on CALVIN, and the COCO image caption, VQAv2 tasks as well, as shown in Tab. 6 and Tab. This provides solution for fine-tuning VL.Ms to robotics models while preserving the ability deteriorate the performance on robotic tasks. In our implementation, we ensure that the model equally incorporates batches of VL and robot data in each epoch. From Fig. and Fig. 5 could observe similar performance curve of the Co-trained and Fine-tune version of our model, while Co-trained model achieves higher performance in the early epochs and Fine-tune model ends up higher in the later epochs. One interesting observation is that under the Enriched setting, the performance of the co-trained model also drops, this may indicate the difference between understanding different sentences and aligning vision- laniua-a representations (as the pre-trained tasks do).
vision-language tasks, even though may slightly
Costained
Figure 5: The performance of Co- Co-Trained and Fine-tune model of MPT-3B-IFT each epoch on ABC' D split.
Co-tralted
Fin-tunt
Cooker
Figure 6: The performance of Co-TCoiTiia Trained and Fine-tune model of MPT-3B-IFT each epoch on
ABCD D split.
B.2 COMPARISON WITH PRE-TRAINED ROBOTICS REPRESENTATION MODELS
We consider comparing our RoboFlamingo with recent pre-trained robotics representation models, such as R3M (Nair et al., 2022) and Voltron (Karamcheti al., 2023). We loaded the pre-train
14

---

Published as a conference paper at ICLR 2024
Table Comparison of co-trained models and fine-tuned models on the CALVIN benchmark. All results are selected from the best of the last 5 epochs.
Training Data
Test Split
Task Completed Sequence
Method
2
3
4
5
Avg Lcn
Co-trained
ABC
D
0,829
0.636
0.453
0.321
0.234
2.47
Fine-tune
ABC
D
0.824
0.619
0.466
0.331
0.235
2.48
Co-trained
ABCD
D 0,957
0.858
0.737
0.645
0.561
3.76
Fine-tune
ABCD
D
0.964
0.896
0.824
0.740
0.66
4.09
Co-trained
ABCD
(Enriched)
0.678
0.452
0.294
0.189
0.117
1.73
Fine-tune
ABCD
D (Enriched)
0.720
0.480
0.299
0.211
0.144
1.85
Table Comparison of co-trained models and fine-tuned models on the COCO image caption and VQAv2 evaluation dataset. All results are selected from the epoch in Tab. 5.
COCO
VQA
BLEU--
BLEU-2
BLEL--
BLEFU-
METEOR
ROUGE_
CIDEr
SPICE
Acc
Finc-tunc (3B, zcro-shot)
0.157
0.052
0.018
0.008
0.038
0,147
0.005
0.006
4.09
Finc-tune (3B, 4-shot)
0,168
0.057
0.020
0.008
0.043
0.161
0.005
0.007
3.87
OpenFlamingo (3B, zero-shot)
OpenFlamingo (3B, 4-shol)
0.612
0.461
0.332
0.234
0.220
0.491
0,822
0.162
43.86
Co-Train (3B, zero-shot)
0.157
0.124
0,346
0.084
36.37
Co-Train (3B, 4-shot)
0,284
0,204
0.142
0.098
0.142
0.364
0,426
0,100
38.73
Original Flamingo (80B, fine luned)
1.381
82.0
weights of R3M and Voltron and fine-tuned them on CALVIN data, only training the policy head while freezing their representation parameters. As for Voltron, we also include a version that fine- tunes the representation layers. The results shown in Tab. 7, which reveals the clear advantage of fine-tuning pre-trained VLMs compared with these specific robotics representation models.
Table 7: Comparative performance of various VI. representation models on various settings, all results are selected from the best of the last 5 epochs.
Training Data
Test Split
Task Completed in Sequence
Method
2
3
4
5
Avg Len
Voltron (Frozen)
ABC
0.026
0.001
0.000
0.000
0.000
0.03
Voltron (Fine-tuned)
ABC
D
0.569
0.272
0.105
0.038
0.014
1.00
RoboFlamingo (Ours)
ABC
0.824
0.619
0.466
0.331
0.235
2.48
R3M (Frozen)
ABCD
0.085
0.005
0.001
0.000
0.000
0.10
Voltron (Frozen)
ABCD
0.101
0.003
0.001
0.000
0.000
0.11
Voltron (Fine-tuned)
ABCD
0,837
0.566
0.352
0,208
0.115
2.08
RoboFlamingo (Ours)
ABCD
D
0.964
0.896
0.824
0.740
0.662
4.09
B.3 FINE-TUNE THE FULL MODEL
In the fine-tuning of RoboFlamingo, we follow the training of Flamingo (Alayrac al., 2022; Awadalla et al., 2023) that only trains the parameters of the resampler, the gated cross-attention module of each decoder layer, and the policy head while freezing all other parameters. This leads RoboFlamingo to have IB trainable parameters (as shown in Tab. 2). In this part, we show the results of training the full model (the MPT-3B-IFT variant), which has 3B trainable parameters in Tab. revealing obvious performance deterioration.
B.4 PERFORMANCE CURVES IN TRAINING OF DIFFERENT BACKBONES
Fig. 8 and Fig. 9 show the performance of RoboFlamingo with different VLMs on both ABC + D and ABCD D settings in 5 training epochs. It noticed that most variants converge in 5-
15

---

Published as a conference paper at ICLR 2024
Table Comparison between full-model fine-tuning (3B trainable parameters) and Robo Flamingo- style fine-tuning (IB trainable parameters)
Training Data
Test Split
Task Completed a Sequence (Success
Ratc)
Method
2
3
5
Avg Len
Full model fine-tuned
0.415
0.070
0.009
0.002
0.001
0.50
ABCD (Lang)
D
RoboFlamingo
0.964
0.896
0.824
0.740
0.66
4.09
epoch training and achieve the best performance, benefiting from the pre-training on extensive vision-language tasks.
go push the pink block right
grasp and the pink bicos
store the grosped block In slicing cablnet
sidd the blcok that falls into the crawer
push the hondle to o.oce the drawer
Bure 36
Transo Se
frmm 154
fromr
126
furre 185
Tame 152
hann
Frams Sap
frane 580
furne 206
Initial State
go pine right
graap and lift the pink block
store the grasped block in the dlicing cabinet
ANLC
Figure 7: The visualization of RoboFlamingo and HULC excuting the same task sequence in the ABC - D split.
M-36
M-36-IF7
Figure 8: The performance of VLMs each epoch on ABC split.
B.5 QUALITATIVE EXAMPLES
We visualize the task frames and analyze how RoboFlamingo achieve such a great performance. As the example shown in Fig. 7, where RoboFlamingo successfully finishes the entire task sequence, while HULC stucks at the third one. RoboF Famingo only takes a dozen steps to locate and move to the top of the drawer, and simultaneously releases the gripper to complete the task: while HULC keeps moving above the desktop for hundreds of steps and fails to locate the drawer. Furthermore, although both methods are successful for the first two tasks, RoboFlamingo uses significantly fewer steps. This representative episode vividly illustrates that our method is much more effective and efficient and could better generalize to unseen vision context.
B.6 DETAILED IMITATION PERFORMANCES ON EACH TASK
We present the detailed imitation performances by tasks in Tab. All model are reported by their best checkpoint.
B.7 ROLLOUT EXAMPLES
We present some rollout examples of RoboFlamingo the ABCD D split.
16

---

Published as a conference paper at ICLR 2024
W-36
M:3D-FF
G-4b
C-40-FF
L-90
R-90
Lod
wo0ker
000
Figure 9: The performance of VLMS at each epoch ABCD D split.
Table 9: Success rates by task of variants of RoboFlamingo. Each task is evaluated 100 times.
Task Name
M-3B
M-3B-IFT
G-4B
G-4B-IFT
L-9B
M-9B
rotate blue block right
0.947
0.893
0.729
0.770
0.493
0.882
slider right
0.996
0.993
0.996
0.999
0.987
0.996
lift red block slider
0.890
0.970
0.967
0.858
0.856
0.927
place slider off lightbulb off led push into drawer 0.777
0.904
0.828
0.582
0.911
0.874
0.910
0,972
1.000
0.992
0.956
0.927
0.964
0.988
1.000
1.000
0.994
0.970
0.981
0.821
0.731
0.770
0.705
0.703
1fft blue block drawer
1,000
0.950
1.000
1.000
0.917
0.737
close drawer lift pink block slider
1.000
1.000
1.000
1.000
0,986
0.995
0.940
0.971
0.944
0.862
0.861
0.918
lift pink block table
0.859
0.851
0.905
0.892
0.543
0.899
move slider left
0.996
0.996
1.000
1.000
0.970
0.996
open drawer turn on lightbulb rotate blue block left
0.976
0.997
0.997
0.982
0.980
0.997
0.988
0.994
1.000
0.988
0.949
0.988
0.923
0,939
0.820
0.925
0.636
0.848
push blue block left
0.746
0.955
0.841
0.836
0.677
0.909
rotate red block right
0.926
0.972
0.853
0.905
0.591
0.959
turn on led
0.988
0.988
0.994
0.976
0.985
0.994
push pink block right
0.652
0.754
0.833
0.651
0.627
0.750
push red blook left
0.949
0.920
0.849
0.949
0.562
0.908
lift blue block table
0.891
0.956
0.925
0.927
0.611
0.931
place in drawer
0.988
0.989
0,988
0.975
0,971
0.976
rotate red block left
0.970
0.908
0.950
0.953
0.677
0.952
push pink block left
0,947
0.920
0.915
0.973
0.747
0.933
stack block
0.612
0.641
0.608
0.595
0.569
0.604
lift blue block slider
0.847
0.963
0. 0.908
0.826
0.769
0.869
push red block right
0.657
0.732
0.797
0.451
0.457
0.653
lift zed block table
0,948
0.939
0.942
0.975
0.606
0.989
lift pink block drawer
0.857
0.800
0.929
0.714
0.778
0.923
rotate pink block right
0.917
0.896
0.714
0.794
0.478
0.789
unstack block
1.000
0.982
0.957
0.980
0.946
0.979
rotate pink block left
0.929
0.839
0.906
0.818
0.698
0.927
push blue block right
0.479
0.597
0.471
0.478
0.400
0.594
lift block drawer
0.947
1.000
1.000
1.000
0.769
0.933
17

---

Published as a conference paper at ICLR 2024
ABCD-D
push
open
greep and 1iitt the red block
store the grasped block the drawer sliding
take the red bleck and zotate it to the right
puah the hardle close the drawer
Bliding
graap
store the grasped block in the slicing zabinet
Figure 10: Rollouts on the ABCD - D split of the CALVIN benchmark.
18

---

Published as a conference paper at ICLR 2024
C ADDITIONAL DETAILS
C.I ILLUSTRATION OF POLICY HEADS/FORMULATION
We illustrate the details of the four policy heads/formulation mentioned in Section 5.4: (a) MLP w/o hist takes only the current observation as input to predict actions, which ignores the observation history. (b) MLP hist takes the history frames into the vision encoder with position embedding, and encodes the history information through the cross-attention layers in the feature fusion decoder. (c) GPT and (d) LSTM both utilize the VLM backbone to process single-frame observations and integrate the history with the policy head. GPT explicitly takes the visual history as imput to predict the next action. LSTM implicitly maintains a hidden state to encode memory and predict the action.
Features
+
Pooling
Stacked Actions
LSTM
Feature Fusion Decoder
Features
Vision Encoder
@
Self Attention (LLM)
Transformer
Cross Attention
Resamplor
MLP
MLP
ViT
+
Tokenizer
GPT
Stacked Actions
Instructiion: Pick up the red block put
Current obseration
Feature
Stacked Actions
MLP w/o hist
Pooling
Feature
Vision Encoder
MLP
Feature Fusion Decoder
Resampler
#
Solf Attention [LLM)
Stock Actions
Cross Attention
MLP
VIT
32
Tokenizer
History observation sequence
and
into the
Figure 11: Implementation details of all policy heads/formulation involved.
19


Reviews:
[{"review_id": "hmE35ats6k", "reviewer": "Reviewer_8RDp", "summary": "- Proposes RoboFlamingo, a framework for adapting large vision-language models (VLMs) like OpenFlamingo to robot manipulation policies.\n- Achieves state-of-the-art performance on the CALVIN benchmark by fine-tuning VLMs with only a small amount of robotic demonstration data.\n- Shows VLMs can enable effective vision-language comprehension and long-horizon planning for robot control when combined with a simple policy head, while demonstrating strong generalization ability to unseen tasks and environments. Comprehensive analysis and ablation studies on using VLMs for robotic manipulation are conducted.", "questions": "- What steps would be needed to transfer RoboFlamingo to real robotic systems? How realistic are the CALVIN simulations?\n- Is there scope to incorporate offline robotic data to improve sample efficiency?\n- Experiments are with visual and language modalities. Robotic manipulation often relies on additional sensing (e.g. force, tactile). How can RoboFlamingo incorporate other modalities?\n- How flexible is the decoupled design? Will it be possible to incorporating RoboFlamingo into hierarchical frameworks like in PaLM-E?\n- How does the computational overhead of RoboFlamingo compare to other VLM-based methods?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- RoboFlamingo outperforms considerably prior methods on CALVIN\n- Requires much less data and compute than methods like RT-2 that co-train on extensive internet-scal data.\n- Decouples perception and policy to enable flexibility like open-loop control, while maintaining relatively strong zero-shot generalization ability.", "weaknesses": "- Relies on simulated robot environment, may be challenging to transfer to real world.\n- The evaluation is limited to a single simulated benchmark environment (CALVIN). Testing on more diverse robotic platforms and tasks in simulation could help validate the generalizability of the method.\n- Less sample efficient than methods leveraging offline robot data like MCIL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PbUyzpOfgn", "reviewer": "Reviewer_qVoh", "summary": "This work introduces RoboFlamingo, a language-conditioned manipulation method that finetunes an open-source VLM OpenFlamingo to output low-level robot control actions. The method builds upon the pre-trained and frozen OpenFlamingo Transformer backbone: 1) adds an LSTM policy head after the pooled visual-language embedding output from the OpenFlamingo backbone, 2) adds first-person and third-person camera image tokens to the ViT for the Resampler. Following the OpenFlamingo finetuning procedure, the ViT, Tokenizer, and Self-Attention Layers of the backbone are frozen during training; only the resampler, cross-attention, and policy head parameters are updated during finetuning on robot imitation learning datasets. In evaluations, RoboFlamingo is evaluated on: 1) in-distribution training performance and out-of-distribution generalization on the CALVIN benchmark where it achieves SOTA over HULC and RT-1, 2) ablations that show history based policy heads (GPT and LSTM) outperform MLPs, vision-language pretraining is critical for good performance, and 3) larger models and instruction finetuned base models performing better. The authors commit to releasing code upon acceptance.", "questions": "- Clarifications to my concerns above would be appreciated.\n- Will checkpoints be released as well?\n- How does performance on pre-training tasks change during finetuning? That is, is there catastrophic forgetting occurring, where the base foundation capabilities are lost?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The motivation is clear for a low-cost alternative solution to large closed Vision Language Action models (VLAs) like RT-2, which motivate this work. The study which incorporates open-source design components like the different LLMs of various architectures and sizes in OpenFlamingo is a great contribution to the open-sourced community as well.\n- The results on CALVIN, a well established and difficult robot control and generalization benchmark, are very compelling\n- Open loop results are intriguing for pragmatic on-robot deployment\n- The presentation is largely very easy to follow and a pleasure to read", "weaknesses": "- Other ways of incorporating VL pre-training are not considered, such as utilizing VL representations like R3M or VOLTRON or MVP. These baselines are relevant given the frozen-backbone + robot finetuning setup in RoboFlamingo. Essentially, a baseline should study different ways of incorporating \"web data\", which the current baselines do not study.\n- A core claim of RT-2 was the benefit of co-fine-tuning on robotics data in addition to the original VL data. This core claim is not studied in RoboFlamingo.\n- Another claim of RT-2 was measuring the transfer of internet knowledge to robotics, in addition to in-domain performance. This seems like a major benefit of utilizing VLMs for robotics generalization. However, this is not studied in this work; the setting in the ABC => D CALVIN environment seems insufficient to measure how much transfer is occuring from internet-scale VL pre-training to robotics. \n- Another claim of RT-2 was the benefit of mixing robot action tokens explicitly with VL tokens. In contrast, RoboFlamingo introduces a new policy head that directly only predicts action tokens. It would be interesting to compare an explicit action-only policy head with the multi-modal output token prediction setting in RT-2.\n- The presentation can be improved a bit in Section 4.2.1 and 4.2.2, where the notation is unwieldy. For example, the notation of $K$ is overloaded.\n- Writing nits:\n    - Section 2: \"models to encoder\" => \"models to encode\", \"train the policy\" => \"training the policy\", \"utilizing robot manipulation data both the web data\" => \"utilizing both robot manipulation data and web data\", \"We hope RoboFlamingo provide\" => \"We hope RoboFlamingo provides\"\n    - Section 4: \"Particularlly\" => \"Particularly\", \"look into one\" => \"looks into one\", \"and take its\" => \"and takes its\"\n    - Section 5: \"We wonder\" => \"We study\", \"24 thousand trajectories\" => weird ~ added\n    - Section 5.4.1: \"single-frame observation\" => \"single-frame observations\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WDHsBMvlaP", "reviewer": "Reviewer_fijq", "summary": "This paper proposes the RoboFlamingo architecture for effective language-conditioned robot manipulation task learning through behavior cloning. Specifically, the paper shows that by initiating model weights from pretrained VLMs and finetuning them in the OpenFlamingo-style using a minimal amount of downstream robot manipulation data, the policy can achieve good performance on the CALVIN benchmark for both seen task and unseen task variations. Such performance also outperforms previous baselines like RT-1. The authors further provide ablations on the effect of different backbone scales, different architectures, and training paradigms on agent performance.", "questions": "Page 3: `Compared to other works, the controlling policies do not require any ability to understand instructions, but rely on the pre-trained frozen LLM to select necessary skills.`. This sentence is inaccurate. The author's proposed approach still need to understand language instructions as the LLM backbone still needs to fuse language representations with visual input representations. I believe the authors' actual meaning is that the policy head does not explicitly take language instructions as input.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The authors agree to open-source their code and implementations, which I really appreciate. This will greatly facilitate efforts to scale up foundation model training for robotic manipulation tasks.\n- The ablation section is very helpful for readers to understand the critical components of the proposed RoboFlamingo architecture.\n- RoboFlamingo significantly outperforms prior baselines like RT-1 on the CALVIN benchmark.", "weaknesses": "Firstly, the presentation of the paper can be improved, and some parts of the method is unclear, which hinders reader's understanding.\n- According to Section 4.2.2, the finetuned language model backbone will output a fused vision-language representation $X_t^L=\\{x_{t,1}^L,\\dots, x_{t,M}^L\\}$ for *each* time step $t \\in [1...T]$, where $M$ is the length of the input language instruction $l$. To produce such outputs, it seems necessary that the same language instruction needs to be tiled $T$ times and fed into the language model for a sequence of $T$ images. However, such detail is not illustrated in Figure 2, and Figure 2 only illustrates the model behavior when $T=1$. It would be a lot more helpful if Figure 2 illustrates model behavior when $T>1$. \n- Sec. 5.4.2 of the ablation study shows that \"loading the pre-trained parameters of the cross-attention layers\" is crucial for model performance. From which model are these cross-attention layer weights being loaded? Also why not load the perceiver resampler weights from a pretrained model? In addition, the design to load pretrained cross-attention weights is never described in the methodology section.\n- In Section 5.4.3, what is the setup for the \"instruction fine-tuning\" experiment? I also didn't find the descriptions for the instruction finetuning designs in the methodology section.\n\nSecondly, authors choose to only train RoboFlamingo on the CALVIN benchmark throughout the paper. Even though authors claim that they want to showcase RoboFlamingo's ability to produce good language-conditioned manipulation policies given a small amount of finetune data, I'm afraid that by only finetuning on the CALVIN benchmark, the model overfits to the dataset and loses some crucial abilities like spatial reasoning and object relation understanding (which may be crucial for other robot manipulation tasks that are not present in the CALVIN benchmark). By training RoboFlamingo on a mixture of downstream robot datasets and large-scale datasets used to pretrain e.g., OpenFlamingo, InstructBLIP, IDEFICS, LLaVA, authors might alleviate such phenomenon, and even improve upon the CALVIN benchmark performance they achieved in this paper. Therefore, I do not quite agree with the author's claim that \"only a minimal amount of data is required to adapt the model to downstream manipulation tasks\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Y1aVgii6b", "reviewer": "Reviewer_iuWF", "summary": "The paper uses a VLM as a backbone for visual imitation learning of language-conditioned policies. In contrast to prior work, it builds on open-source VLMs and limits the number of finetuned parameters to make VLM-policy training feasible for conventional compute budgets. They demonstrate that the VLM backbone leads to superior imitation performance on the CALVIN simulated robotic manipulation benchmark and compare a few different design choices.", "questions": "- for the enriched language evaluations, the authors mention that they sample language instruction synonyms randomly from the GPT-4 generations -- did you ensure that all methods are evaluated on the same randomly sampled set of instructions to make the comparison fair?\n\n- can you explain in more detail the experiment on open-loop execution in Section 5.5? How can you open-loop execute the policy without re-training?\n\n\n## Review Summary\n\nOverall I think this paper is an interesting contribution to democratizing the access to large vision language models for policy learning. I believe that many in the community will be interested in this and thus recommend acceptance. However, the empirical analysis in the paper could be significantly improved by addressing the points raised above. Concretely, the authors can:\n- add details about the required compute\n- add comparison to co-finetuning and full model finetuning\n- add comparison to other pre-trained representations\n- add evaluations on at least one additional environment", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Using VLM backbones for policy learning is a promising direction for robot learning, but prior work was confined to proprietary models and required large compute for policy training. This work's focus on open-source models and parameter-efficient finetuning brings those models within the reach of academic compute budgets and thus is very valuable. The demonstrated results on the CALVIN benchmark are strong and support the claim that pre-trained VLM backbones are good features for imitation learning.\n\nI appreciate that the authors analyzed several of the design decisions experimentally and showed which choices have a larger influence on final performance. Particularly the results in Table 3 are interesting in that they show that larger VLM backbones are particularly beneficial in a low-data regime. I also appreciated the separate investigation of generalizability in the visual domain and to diverse language instructions.\n\nThe paper is easy to follow and most of the experiments are easily understandable.", "weaknesses": "A main selling point of the paper is that it claims the introduced method can forego expensive co-finetuning by restricting the number of finetuned weights and freezing most of the VLM weights. However, if I understand correctly, the paper finetunes all weights that were also finetuned in the Flamingo VLM grounding stage, i.e. like Flamingo they froze the vision and language model features, but finetuned all cross-attention features that perform the vision-to-language grounding (and Fig 3b shows that this is crucial). This however suggest that the model may still forget most of the knowledge obtained in the VLM pretraining stage, ie the OpenFlamingo training. The experimental section of the paper lacks comparison to (A) co-training with the current parameter-freezing scheme, (B) full model finetuning w/ and w/o co-training to support the claim that their partial finetuning scheme is actually key to enable good performance without co-finetuning.\n\nAnother comparison that would be good to add is to a simpler, pre-trained visual representation, like VC-1, Voltron etc. These models also use internet data to train good representations for imitation learning, but are arguably easier to use than the billion-parameter scale models introduced here, so it would be good to experimentally show the benefits.\n\nOne notable difference to prior work is that instead of predicting actions as tokens in the VLM's output vocabulary, the proposed method trains a separate action head. It would be good to analyze this choice and compare to directly outputting actions as \"text tokens\".\n\nSince the paper is mainly an empirical study, it would be good to evaluate the policy on more than one environment, e.g. the IKEA Furniture assembly environment could be a nice and challenging testbed with ~ photorealistic rendering.\n\nThe paper also lacks details on the computational requirements for training policies with the VLM backbones (required GPU + training time), which seems crucial given the focus on making VLM policies more accessible.\n\nFinally, Section 5.5 on open-loop execution lacks some detail on what exactly was tried, so I was a bit confused about these experimental results (see questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)