OpenVLA: An Open-Source Vision-Language-Action Model
Authors: Moo Jin Kim
Karl Pertsch
Siddharth Karamcheti
Ted Xiao
Ashwin Balakrishna
Suraj Nair
Rafael Rafailov
Ethan P Foster
Pannag R Sanketi
Quan Vuong
Thomas Kollar
Benjamin Burchfiel
Russ Tedrake
Dorsa Sadigh
Sergey Levine
Percy Liang
Chelsea Finn
Venue: CoRL 2024 (2024)
Keywords: Vision-Language-Action Models
Generalist Policies
Large-scale Robot Learning
Robotic Manipulation
Robotics
Vision-Language Models
TL;DR: We introduce OpenVLA, a state-of-the-art, open-source 7B-parameter VLA model that obtains strong performance for cross-embodiment robot control out-of-the-box and can be easily adapted to new robot setups via parameter-efficient fine-tuning.

Abstract: Large policies pretrained on a combination of Internet-scale vision-language data and diverse robot demonstrations have the potential to change how we teach robots new skills: rather than training new behaviors from scratch, we can fine-tune such vision-language-action (VLA) models to obtain robust, generalizable policies for visuomotor control. Yet, widespread adoption of VLAs for robotics has been challenging as 1) existing VLAs are largely closed and inaccessible to the public, and 2) prior work fails to explore methods for efficiently fine-tuning VLAs for new tasks, a key component for adoption. Addressing these challenges, we introduce OpenVLA, a 7B-parameter open-source VLA trained on a diverse collection of 970k real-world robot demonstrations. OpenVLA builds on a Llama 2 language model combined with a visual encoder that fuses pretrained features from DINOv2 and SigLIP. As a product of the added data diversity and new model components, OpenVLA demonstrates strong results for generalist manipulation, outperforming closed models such as RT-2-X (55B) by 16.5\% in absolute task success rate across 29 tasks and multiple robot embodiments, with 7x fewer parameters. We further show that we can effectively fine-tune OpenVLA for new settings, with especially strong generalization results in multi-task environments involving multiple objects and strong language grounding abilities, where we outperform expressive from-scratch imitation learning methods such as Diffusion Policy by 20.4\%. We also explore compute efficiency; as a separate contribution, we show that OpenVLA can be fine-tuned on consumer GPUs via modern low-rank adaptation methods and served efficiently via quantization without a hit to downstream success rate. Finally, we release model checkpoints, fine-tuning notebooks, and our PyTorch codebase with built-in support for training VLAs at scale on Open X-Embodiment datasets.

Open VLA:
An Open-Source Vision-Language-Action Model
Moo Jin Kim".l
Karl Pertsch" 1..2
Siddharth Karamcheti "
Ted Xiao Ashwin Balakrishna"
Suraj Nair
Rafael Rafailov
Ethan Foster
Pannag Sanketi
Quan Vuong Sergey Levine
Thomas Kollar
Benjamin Burchfiel Russ Tedrake Dorsa Sadigh'
Percy Liang'
Chelsea Finn*
gi github.io io
Large-Scale Robot
OpenVLA Vision- Vision-Language Lae--Langgggg-ttiion Model
Closed-Loop Robot Control Policy
Training Data
User: Wipe the table.
Fine-tune VLM w/ Robot Actions:
OpenVLA:
970k Robot Episodes
[AX, 58, &Grip)
Llama 2 7B
VIT
Base VLM
Multi-Robot Control & Efficient Fine-Tuning
Fully Open-Source
Data
Weights
Code
Figure I: We present OpenVLA, a 7B-parameter open-source lantuagggnaa model (VLA), trained on 970k robot episodes from the Open X-Embodiment dataset []]. VLA sets a new state of the art for generalist robot manipulation policies. It supports controlling multiple robots out of the box and can be quickly adapted new robot domains via parameter-efficient fine-tuning. The Open checkpoints and PyTorch training pipeline are fully open-source and models can downloaded and fine-tuned from HuggingFace.
Abstract: Large policies pretrained on combination of Internet-scale vision- language data and diverse robot demonstrations have the potential to change how we teach robots new skills: rather than training new behaviors from scratch, we can fine-tune such laroonnlnngagg-aatte policies for visuomotor control. Yet, widespread adoption of VLAs for robotics has been challenging as 1) existing VLAs are largely closed and inaccessible the public, and 2) prior work fails to explore methods for efficiently fine-tuning VLAs for new tasks, a key component for adoption. Addressing these challenges, we introduce VL. a 7B-parameter open-source VI.A trained on a diverse collection of 970k real-world robot demonstrations. Open VLA builds on a Llama 2 language model combined with a visual encoder that fuses pretrained features from DINOv2 and SigLIP. As product of the added data diversity and new model components, OpenVLA demonstrates strong results for generalist manipulation, outperforming closed models such as RT-2-X (55B) by 16.5% in absolute task success rate across 29 tasks and multiple robot embodiments, with 7x fewer param- eters. We further show that we can effectively fine-tune Open for new settings,
(VLA) models to obtain robust, generalizable
": denotes equal contribution
Correspondencee mooj wnwkstanford.....mmmooooo edu, pertach@berkeley. edu, skaramcheti@@taanfrro edu Stannoor University, 'UC Berkeley, TToyota Research Institute, 'Google Deepmind, "Physical Intelligence. MIT, Work done in part while at Google Decpmind
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

with especially strong generalization results in multi-task environments involving multiple objects and strong language grounding abilities, where we outperform expressive from-scratch imitation learning methods such as Diffusion Policy by 20.4%. We also explore compute efficiency; as a separate contribution, we show that Open can be fine-tuned on consumer GPUs via modern low-rank adapta- tion methods and served efficiently via quantization without hit to downstream success rate. Finally, we release model checkpoints, fine-tuning notebooks, and our PyToorch codebase with built-in support for training VIAs at scale on Open X- X-Embodiment datasets.
Keywords: Vision-Language-Actionn Models, Generalist Policies, Large-scale Robot Learning
Introduction
key weakness of learned policies for robotic manipulation is their inability to generalize beyond their training data: while existing policies trained for individual skills or language instructions have the capacity to extrapolate behaviors to new initial conditions such object positions or lighting [2, 3], they lack robustness to scene distractors or novel objects [4, 5] and struggle to execute unseen task instructions [6, 7]. Yet beyond robotics, existing foundation models for vision and language such as CLIP [8]. SigLIP [9], and Llama 2 [10] are capable of these types of generalization and more, stemming from the priors captured by their Internet-scale pretraining datasets. While reproducing this scale of pretraining for robotics is still an open challenge - even the largest robot manipulation datasets [1, 11] only have 100K 1M examples this imbalance suggests opportunity: using existing foundation models for vision and language
core building block for training robotic policies that can generalize to objects, scenes, and tasks beyond their training data.
Towards this goal, existing work has explored integrating pretrained language and vision- language models for robotic representation learning [12-14] and as a component in modular systems for task planning and execution [15, 16]. More recently, they have been used for directly learning vision- language-action models [VLAs; 1, 17. 18] for control. VLAS provide direct instantiation of using pretrained bosoo-and-languaad foundation models for robotics. directly fine-tuning visually- conditioned language models (VLMs) such PaLI [19, 20] generate robot control actions. By building off of strong foundation models trained on Internetsccale data, VLAs such RT-2 [7] demonstrate impressive robustness results, as well as an ability to generalize to novel objects and tasks, setting a new standard for generalist robot policies. Yet, there are two key reasons preventing the widespread use of existing VLAs: 1) current models [1, 7, 17, 18] are closed, with limited visibility into model architecture, training procedures, and data mixture, and 2) existing works do not provide best practices for deploying and adapting VLAs new robots, environments, and tasks especially on commodity hardware (e.g., consumer-grade GPUs). To develop rich foundation for future research, robotics needs open-source, generalist VLAs that support effective fine-tuning and adaptation, akin to the existing ecosystem around open-source language models [21-24].
To this end, we introduce Open VLA, 7B-parameter open-source VLA that establishes a new state of the art for generalist robot manipulation policies. OpenVLA consists of pretrained visually- conditioned language model backbone that captures visual features at multiple granularities, fine-tuned on large, diverse dataset of 970k robot manipulation trajectories from the Open-X Embodiment [1] dataset- dataset that spans wide range of robot embodiments, tasks, and scenes. As a product of increased data diversity and new model components, Open outperforms the 55B- parameter RT-2-X model [1, 7]. the prior -nth-offthhe-at VI.,, by 16.5% absolute success rate across 29 evaluation tasks on the WidowX and Google Robot embodiments, We additionally investigate efficient fine-tuning strategies for VLAs, a new contribution not explored in prior work, across 7 diverse manipulation tasks spanning behaviors from object pick-and-place to eleaning a table. We find that fine-tuned VLe policies clearly outperform fine-tuned pretrained policies such as Octo [5]. Compared to from-scratch imitation learning with diffusion policies [3], fine-tuned OpenVLA shows substantial improvement tasks involving grounding language behavior in multi-task settings with multiple objects. Following these results, we are the first to demonstrate the effectiveness

---

of compute efficient fine-tuning methods leveraging low-rank adaptation [LoRA; 25] and model quantization [26] to facilitate adapting Open VLA models on consumer-grade GPUs instead of large server nodes without compromising performance. As a final contribution, we open-source all models, deployment and fine-tuning notebooks, and the OpenVLA codebase for training VLAs at scale, with the hope that these resources enable future work exploring and adapting VLAs for robotics.
Related Work
IisuallyCConditione Language Models Visually-connitionee language models (VLMs), which are trained on Internet-scale data to generate natural language from input image(s) and language prompts, have been adopted for myriad applications from visual question answering [27-30] to object localization [31, 32]. VLMs typically models that bridge features from pretrained vision encoders [8, 9, 33] with pretrained language models [10, 23, 34-36]. While early work explored various architectures for cross-attending between vision and language features [37-41], new open- source VLMs [20, 42-44] have converged on a simpler "patch-as-token" approach, in which patch features from pretrained visual transformers are treated as tokens, and are then projected into the input space of a language model. These models build the backbone of our OpenVL.A policy.
Generalist Robot Policies A recent trend in robotics works towards training multi-task "generalist" robot policies [2, 6, 45-49] large diverse robot datasets [1, 2, 6, 11, 45, 49-56], spanning many different robot embodiments [1, 5, 53, 57-66]. Notably, Octo [5] trains generalist policy that can control multiple robots out-of-the-bol and allows for flexible fine-tuning to new robot setups. A key difference between these approaches and Open is the model architecture. Prior works like Octo typically compose pretrained components such as language embeddings or visual encoders with additional model components initialized from scratch [2, 5, 6]. learning to "stitch" them together during the course of policy training. OpenVLA adopts a more end-to-end approach, directly fine- tuning tisually-conditione language models to generate robot actions by treating them tokens in the language model vocabulary. Our experimental evaluation shows that this simple yet scalable pipeline substantially boosts performance and generalization ability over prior generalist policies.
Vision-Language-Action Models A number of works have explored the use of VLMs for robotics, e.g., for visual state representations [12, 13). object detection [67], high-level planning [16], and for providing a feedback signal [68-71]. Others integrate VLMs directly into end-to-end visuomotor manipulation policies [14, 15], but incorporate significant structure into the policy architecture or require calibrated cameras. Multiple works directly fine-tune large pretrained VLMs for predicting robot actions [1, 7, 17, 18, 72-74]. Such models are often referred to as vision-language-actionn models (VLAs), since they fuse robot control actions directly into VLM backbones. Existing works on VLAs either train and evaluate in single robot or simulated setups [72-75] and thus lack generality, or are closed and do not support efficient fine-tuning to new robot setups [1, 7, 17, 18]. Most closely related, RT-2-X [1] trains 55B-parameter VLA policy on the Open X-Embodiment dataset [1] and demonstrates state-of-the-art generalist manipulation policy performance. However, our work differs from RT-2-X in multiple important aspects: (1) by combining a strong open VLM backbone with richer robot pretraining dataset, Open VLA matches outperforms RT-2-X in our experiments while being 7x smaller; (2) we thoroughly investigate fine-tuning of Open VLA models to new target setups, while RT-2-X does not investigate the fine-tuning setting; (3) we are the first to demonstrate the effectiveness of modern trrameter-efffcien fine-tuning and quantization approaches for VLAs; and (4) VLA is the first generalist VLA that is open-source and thus supports future research on VLA training, data mixtures, objectives, and inference.
3 The Open VLA Model
We introduce the VLE model, a 7B- 18-parameter lisioo-language--ee model (VLA) trained on 970k robot demonstrations from the Open X-Embodiment dataset [1]. In this section, we first provide a brief overview of modern VLMs, which form the backbone of OLe (Section 3.1), describe our training recipe (Section 3.2) and dataset (Section 3.3), and discuss key design decisions (Section 3.4). Details of the infrastructure used for training and inference are in Appendix C, and description of our codebase is in Appendix D.
3

---

OpenVLA
Action De- Tokenizer
Ax AO Arrip
Llama 7B
Rober Aation
Input Image
MLP Projector
Llama Tokenizer
"Put eggplant bowl"
DinoV2
SigLIP
Language Instruction
should do to (task)7
Figure 2: Open model architecture. Given an image observation and language instruction, the model predicts 7-dimensional robot control actions. The architecture consists three key components: (1) a vision encoder that concatenates Dino V2 [33] and SigLIP [76] features, (2) a projector that maps visual features to the language embedding space, and (3) the LLM backbone, a Llama 2 7B p-raameter large language model [10].
3.1 Preliminaries: Vision-Languaae Models
The architecture of most recent VLMs [20, 42-44] consists of three main parts (see Fig. 2): (1) visual encoder that maps image inputs to "image patch embeddings", (2) a projector that visual embeddings and maps them into the input space of large language model, and (3) a language model backbone. During VLM training, the model is trained end-to-end with a next text token prediction objective on paired or interleaved vision and language data curated from various Internet sources.
In this work, we build on the Prismatic-7B VIMM [44]. Prismatic follows the same standard architec- ture described above, with 6000-0000000 parameetr visual encoder, a small 2-layer MLP projector, and 7B-parameter Llama 2 language model backbone [10]. Notably, Prismatic uses two-parl visual en- coder, consisting of pretrained SigLIP [76] and D2 [33] models. Input image patches are passed separately through both encoders and the resulting feature vectors are concatenated channel-wise. In contrast to the more commonly used vision encoders such as CLIP [77] or SigLIP-only encoders, the addition of D2 features has been shown to be helpful for improved spatial reasoning [44].
3.2 Open VLA Training Procedure
To train Open VLA, we fine-tune a pretrained Prismatic-7B VLM backbone for robot action prediction (see Fig. 2). We formulate the action prediction problem as a "vision-languue
task, where an input observation image and a natural language task instruction are mapped to string of predicted robot actions [7]. To enable the VLM's language model backbone to predict robot actions, we represent the actions in the output space of the L.LM by mapping continuous robot actions to discrete tokens used by the language model's ss tokenizer. Following Brohan et al. [7], we discretize each dimension of the robot actions separately into one of 256 bins. For each action dimension, we set the bin width to uniformly divide the interval between the 1st and 99th quantile of the actions in the training data, Using quantiles instead of the min-max bounds Brohan et al. [7] used allows us to ignore outlier actions in the data that could otherwise drastically expand the discretization interval and reduce the effective granularity of our action discretization.
Using this discretization, we obtain discrete integers E [O... 255] for an N -ddimensional robot action. We opt for simplicity and follow Brohan et al. [7]'s approach by simply overwriting the 256 least used tokens in the Llama tokenizer's vocabulary (which corresponds to the last 256 tokens) with our action tokens. Open VLA is trained with standard next-token prediction objective, evaluating the cross-entropy loss on the predicted action tokens only.
3.3 Training Data
The goal in constructing the OpenVLA training dataset is to capture large diversity of robot embodiments, scenes, and tasks. This enables the final model to control various robots out of the box and admits efficient fine-tuning to new robot setups. We leverage the Open X-Embodiment dataset [1] (OpenX) as base to curate our training dataset. The full OpenX dataset, at the time of writing, consists of more than 70 individual robot datasets, with more than 2M robot trajectories, that were pooled into coherent and easy-to to--eet data format in large community effort. To make training on this data practical, apply multiple steps of data curation to the raw datasct.

---

The goals of this curation are to ensure (1) coherent input and output space across all training datasets, and (2) a balanced mix of embodiments, tasks, and scenes in the final training mixture. To address (I), we follow [1, 5] and restrict our training dataset to contain only manipulation datasets with at least one 3rd person camera and use single-arm end-effector control. For (2), we leverage the data mixture weights of Octo [5] for all datasets that pass the first round of filtering. Octo heuristically down-weights or removes less diverse datasets and up-weights datasets with larger task and scene diversity; see Octo Model Team et al. [5] for details.
We also incorporate multiple datasets into training mixture that were added to the OpenX dataset since the release of Octo, including the DROID dataset [11], although at a conservative mixture weight of 10%. In practice, we found that the action token accuracy on DROID remained low throughout training, suggesting a larger mixture weight or model may be required to fit it. To not jeopardize the quality of the final model, we removed DROID from the data mix for the final third of training. We provide a complete overview of the used datasets and mixture weights in Appendix A.
3.4 OpenVLA Design Decisions
When developing the OpenVLA model, we explored various design decisions in smaller-scale experiments before starting the final model training run. Concretely, we trained and evaluated OpenVLA models on BrtageData V2 [6] for our initial experiments, instead of training the full OpenX mixture, increase iteration speed and reduce computational cost. We summarize key learnings from these explorations below.
VLM Backbone. Initially, we experimented with multiple VLM backbones. Apart from Pris- matic [44], we tested fine-tuning IDEFICS-1 [78] and LLaVA [79] for robot action prediction. We found all three models to be suitable backbones for VLA training, with similar downstream perfor- mance in our initial Bridge evaluations. We ultimately chose Prismatic for its improved spatial reasoning capabilities via fused SigLIP-DinoV2 backbones (see Section 3.1).
Image Resolution. The resolution of input images has significant impact on the computational requirements of VLA training, since righer-resolution images result in more image patch tokens and thus longer context lengths that quadratically increase training compute. We compared VLAs with 224 x 224px and 384 x 384px inputs, but found no performance difference in our evaluations, while the latter takes 3x longer to train. We thus opt for a resolution of 224 x 224px for the final VLE model. Note that on many VLM benchmarks, increased resolution does improve performance [44, 80, 81], but we did not see this trend (yet) for VLAs.
Fine-Tuning Vision Encoder. Prior work on VLMs found that freezing vision encoders during VLM training typically leads higher performance [44]. Intuitively, a frozen vision encoder may better preserve the robust features learned from its Internet-scale pretraining. However, we found fine-tuning the vision encoder during VLA training to be crucial for good VLA performance. A possible explanation is that the pretrained vision backbone may not capture sufficient fine-grained spatial details about important parts of the scene to enable precise robotic control.
Training Epochs. Typical LLM or VLM training runs complete at most one or two epochs. In contrast, we found it important for VLA training to iterate through the training dataset significantly more times, with real robot performance continually increasing until training action token accuracy surpasses 95%. Our final training run completes 27 epochs through its training dataset.
Learning Rate. We swept the learning rate across multiple orders of magnitude for VLA training and achieved the best results using a fixed learning rate of 2e-5 (the same learning rate used during VLM pretraining [44]). We did not find learning rate warmup to provide benefits.
Experiments
The goal of our experimental evaluations is to test Open VLA's 's ability to serve as a powerful multi- robot control policy out of the box, as well as be a good initialization for fine-tuning to new robot tasks. Concretely, we aim to answer the following questions: (1) How does Open VLA compare to prior generalist robot policies, when evaluating on multiple robots and various types of generalization? (2) Can Open VLA be effectively fine-tuned new robot setup and task, and how does it compare to oftt-oofthh-art data-efficient imitation learning approaches? (3) Can we use paramettr-efffiient
5

---

37.0
50.0
80
Succid Rote 15)
60.0
50.6
$2.0
38.8
20.0
20.0
26-7
26.3
30.0
8.0
0.0
Average
Viuu Gereral ietion
Motion Gene sizettonooon
Physicel Generalizetion
Sementic Genereltzetoo
Languegge Grounding
[Jnsoon backgrounds, stractors, object appesratioa)
(1 (irosenn sejjctt proftions onfortatonsl
[Urowen objert sinwe A shapes:
(1 (Iissenn objects, Instructione. concopra from
(Abiliy menipuee compulare object speaifeed language porpli
RI-1-X
Octo
RT-2-X
OpenVLA (ours)
Pot Yallote Com Pirk Piate
Black Blue Cup on
Put (Peed Bottle Epgpiant ino Pot
Lif Eogplant
Fip Unright
Figure 3: Bridge V2 WidowX evaluation task categories and results. We evaluate Open VLA and prior state-of- the-art generalist robot policies comprehensive suite of tasks covering several axes of generalization. well tasks that specifically assess language conditioning ability. OpenVLA achieves highest overall performance even outperforms closed-source model RT-2-X all categories except for semantic generalization. Average success rates + StdErr are computed across 170 total rollouts per approach.
fine-tuning and quantization to reduce the computational requirements for training and inference of OpenVLA models and make them more accessible? What are the performance-come compute trade-offs?
4.1 Direct Evaluations on Multiple Robot Platforms
Robot Setups and Tasks. We evaluate Open VLA's performance "out-of-the- the-box" on the WidowX robot from the Bridge Data V2 evaluations [6] (see Fig. 1, left) and the mobile manipulator from the RT-1 and RT-2 evaluations [2, 7] ("Google robot"; see Fig. 1, middle). We define a comprehensive set of evaluation tasks each environment that covers various axes generalization, such as visual (unseen backgrounds, distractor objects, colors/appearancee of objects); motion (unseen object positions/orrinntations)) physical (unseen object sizes/shapes); and semantic (unseen target objects, instructions, and concepts from the Internet) generalization. We also assess language conditioning ability in scenes with multiple distractor objects. See Fig. and Fig. 4 for example task images and Appendix B for detailed description of all tasks. Overall, we evaluated each method with 170 Bridge V2 rollouts and 60 Google robot rollouts, all conducted as A/B evaluations.
Comparisons. We compare Open per- formance to three prior generalist manipulation policies: RT-1-X [1], RT-2-X [1], and Octo [5]. RT-1-X (35M parameters) and Octo (93M pa-
05.0
A0
72.0
60
Success
rameters) transformer policies trained from scratch on subsets of the OpenX dataset; Octo is the state of the art among open-source ma- nipulation policies. RT-2-X (55B parameters) a state-of-the-art, closed VIAA that leverages Internet- pretrainde vision-language backbones.
25,7
32.0
20
14.3
Average
Iny Dost but
00O Generatization buckgrounds corceppii
(Teais
baining doaa)
Rm2
OpenVLA (ours)
The results are summarized Fig. for Bridge V2 evaluations and Fig. 4 for Google robot eval- uations task breakdown in Appendix, Ta- ble 4 and Table 6). We find that both RT-1-X and Octo struggle on the tested tasks, often failing to manipulate the correct object, especially when distractors are present. RT-2-X clearly outper- forms both RT-1-X and Octo, demonstrating the benefits of large, pretrained VLMSs for robotics.
Move Cole Call
Figure Google robot evaluation results. evaluate generalist rohot policies on in-distribution and out-of- distribution (OOD) tasks the mobile manipulator used RT-1 and RT-2 evaluations 7J. We find that Open- VLA and RT-2-X attain comparable performance and significantly outperform RT-1-X and Octo overall.
Notably, Open VLA performs comparably to RT-2-X on Google robot evaluations and significantly outperforms RT-2-X on Bridge V2 evaluations despite being 7x smaller (7B vs. 55B parameters).

---

Franka Tabletop
DROID
Rate 191
60
172
Awerage
Carrer
Nour com
Fil Set
Tove <oNects>
Khots Cockett> Overr
Covtr <ohlect>
Wipe Toole
Oifican Polior
Philcy imazthed!
OsenilL tours)
Divanon Mull
Figure Fine-tuning to new robot setups. We fine-tune OpenVLA on 10-100 demonstrations across 7 Franka Emika Panda tasks, ranging from single-instruction tasks to diverse multi-instruction tasks. Diffusion Policy exhibits strong performance on single- ingllcinstrucctin tasks, while OpenVI performs better on diverse fine-tuning tasks with multiple instructions and distractor objects, achieving highest aggregate performance. Mcan success = StdErr computed across 99 and rollouts per approach for Franka- Franka-Tabletop and Franka-DROID, respectively,
Qualitatively, both RT-2-X and OpenVLA exhibit markedly more robust behaviors than the other tested models, such as approaching the correct object when distractor objects present, properly orienting the robot's end-effector to align with the orientation of the target object, and even recovering from mistakes such as insecurely grasping objects." The higher performance of OpenVLA over RT-2-X can be attributed to a combination factors: we curated a much larger training dataset for OpenVLA with 970k trajectories vs. 350k for RT-2-X; we performed more careful cleaning of the training dataset and, e.g., filter out all-zero actions in the Bridge dataset (see Appendix for detailed discussion); and OpenVLA uses fused vision encoder that combines pretrained semantic and spatial features. See Appendix F for ablation analyses of these components.
4.2 Data-Efficient Adaptation New Robot Setups
While prior works mainly focused evaluating VLAs sitt-of-thhbboot [1, 16]. effective fine- tuning of VLAS to new tasks and robot setups is largely unexplored, yet is key for their widespread adoption. In this section, we investigate VLe''LA's 's ability to be adapted to new robot setups.
Robot setups and tasks. We perform full fine-tuning of all OpenVLA parameters, using small datasets with 10-100 demonstrations of a target task (see Fig. 5; we explore parameter-efficient fine-tuning approaches in Section 4.3). We test Open in two setups: Franka-Tie Franka-Tabletop, stationary, table-mounted Franka Emika Panda 7-DoF robot arm; and Franka-DROID, the Franka setup from the DROID dataset 1J, mounted widely used in the robot learning community and thus likely "target" of Open VLA fine-tuning.
movable standing desk. Franka robot arms
Comparisons. We compare to Diffusion Policy [3]. offte-of-the-ar data-efficient imitation learning approach, trained from scratch. We also compare to Diffusion Policy (matched), a version of Diffusion Policy that matches the input and output specifications of Open VLA (i.e., no history, no action chunking). Additionally, we evaluate Octo [5] fine-tuned on the target dataset (RT-2-X does not support fine-tuning). We also fine-tune Open on the same target dataset, and the resulting policy is denoted by VLe Finally, omits OpenX pretraining and directly fine-tunes base Prismatic VLM on the target robot setup.
ablation, compare to Open VLA (scratch), which
We present the results in Fig. 5 (per-task breakdown in Appendix, Table 7). We find that both versions of Diffusion Policy are competitive with or outperform the generalist policies Octo and Open VLA on narrower single-instruction tasks like "Put Carrot in Bowl" and "Pour Corn into Pot", but the pretrained generalist policies perform better more diverse fine-tuning tasks that involve multiple objects the scene and require language conditioning. OpenX pretraining for Octo and Open VLA enables the models better adapt to these more diverse tasks where language grounding important; see evidence for this in the lower performance of OpenVLA (scratch).
Overall, we find that Open VLA achieves the highest average performance. It clearly outperforms Octo while being trained on the same robot data, attesting to the benefits of Internetsccale pretraining. Notably, prior works achieve strong performance only in either precise or diverse tasks, resulting in widely varying success rates. Open VLA is the only approach that achieves at least 50% success
See https::///poeee www..g.ttttu................. io for videos of qualitative results.
7

---

rate across all tested tasks, suggesting that it can be a strong default option for imitation learning tasks, particularly if they involve a diverse set of language instructions. For narrower but highly dexterous tasks, Diffusion Policy still shows smoother and more precise trajectories; incorporating action chunking and temporal smoothing, as implemented Diffusion Policy, may help Open VLA attain the same level of dexterity and may be promising direction for future work.
4.3 Efficient Open VLA Fine-Tuning and Inference
Strategy
Success Rate
Train Params (x 10")
VRAM (batch 16)
Precision
Bridge Success
VRAM
Full FT
69.7 + 7.2 %
7,188.1
GB
163.3 GB
bfloat16
71.3 4.8%
16.8 GB
Last layer only
30.3 - 6.1 %
465.1
51.4 GB
int8
58.1 5.1%
10.2
Frozen vision
47.0 : 6.9 %
6,760.4
156.2 GB
int4
71.9 1 4.7%
7.0 GB
Sandwich
62.1 1: 7,9
914.2
64.0 GB
LaRA, rank=32
68.2
7.5%
97.6
59.7 GB
6a
68.2 + 7.8%
195.2
60.5 GB
Table 2: Performance with quantized in- ference. 4-bit quantization matches the per- formance biloat16 inference (our default approach) while reducing the GPU memory footprint by more than half. Mean success rate + StdErr computed across 8 representa- tive Bridge V2 tasks [6] and 80 rollouts per approach (see Table 5 for details).
Table I: Paraieter-Efffiieent Fine-Tuning Evaluation. LoRA fine-tuning achieves the best coctommncc-coompun trade-off. matching full fine-tuning performance while training only 1.4% of the parameters. Mean success + StdErr across 33 rollouts per approach on select Franka-Tabletop tasks (sec Table for details). :: Sharded across GPUs with FSDP [82].
We test various parameter-efficien fine-tuning approaches for Open across multiple Franka- Tabletop tasks in Table 1: last layer only fine-tunes only the last layer of OpenVLA 'ss transformer backbone and the token embedding matrix; frozen vision freezes the vision encoder but fine-tunes all other weights; sandwich fine-tuning unfreezes the vision encoder, token embedding matrix, and last layer; and LoRA uses the popular low-rank adaptation technique of Hu et al. [25] with multiple rank values r, applied to all linear layers of the model. We find that fine-tuning of the vision encoder is key (sandwich and LoRA). Importantly, LoRA matches full fine- ane-ttnnng performance while fine-tuning only 1.4% of the parameters (r 32), enabling us to fine-tune OpenVLA on a new task within 10-15 hours on a single A100 GPU an 8x compute reduction vs. full fine-tumin fne-tuning.
We also test efficient serving of OpenVLA via quantized inference [26, 83). 8-bit quantization slows down inference across most GPUs due the overhead of the quantization operations (see Appendix, Fig. 11 for inference speeds various GPUs). This results in reduced performance when evaluated across 8 representative Bridge V2 tasks (Table 2), as a slower model suffers distribution shift in system dynamics relative to the training data (see Appendix F.4 for supporting details). 4-bit inference achieves higher throughput due to reduced GPU memory transfer and thus recovers performance of the original bfloat16 model, while requiring less than half the amount of GPU memory.
Conclusion and Limitations
In this work, we presented VLAA, a state-of that obtains strong performance for cross-embodiimg robot control out-of-the-boo We also demon- strated that VLe can be easily adapted new robot setups via parameter-efffcient fine-tuning techniques. The current Open VLA model has several limitations. First, it currently only supports aigge--mage observations; expanding Vpe support multi-image and proprioceptive inputs is an important avenue for future work. Secondly, Open VLA runs at relatively low control frequencies due to its large size, so improving the inference throughput of Open VLA is critical to enable VLA control for high-frequency control setups such as ALOHA [84], which runs at 50Hz. Finally. due to compute limitations, many VLA design decisions remain underexplored, such as varying base VLM size, co-training on robot action data and Internet-scale VLM data, and other types of visual features. We hope that the release of the VLA model and codebase will enable the community to jointly investigate these decisions.
tf-the-r open- open-source vision- lssionlaaguaageactioo model
In Section 4.3, we experiment with a version of the OpenVLA model that pretrained with a smaller robot data mixture (the same OpenX dataset mixture as Octo) and has slightly smaller architecture which only uses a SigLIP [76] vision backbone instead of the fused DinoSigLIP encoder. We find that this simpler architecture still achieves strong performance both fine-tuning tasks and the--o--he-box tasks.
8

---

Acknowledgments
We are grateful to the Toyota Research Institute for providing significant funding and compute resources required to carry out this research. We also thank the Stanford Center for Research on Foundation Models for providing additional compute resources and Google DeepMind for alpha access the RT-2-X API for our evaluations. We acknowledge additional support from Volkswagen, Physical Intelligence, ONR grants N00014-22-1-22 and N00014-22-1-2293, the National Science Foundation through IIS-22463 1, and DARPA ANSR.
References
[1] Open X-Embodiment Collaboration, A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, A. Raffin, A. Wahid, B. Burgess-Limerick, B. Kim, B. Schölkopf, B. Ichter, C. Lu, C. Xu, C. Finn, C. Xu, C. Chi, C. Huang, C. Chan, C. Pan, C. Fu, C. Devin, D. Driess, D. Pathak, D. Shah, D. Büchler, D. Kalashnikov, D. Sadigh, E. Johns, F. Ceola, F. Xia, F. Stulp, G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Schiavi, H. Su, H.-S. Fang, H. Shi, H. B. Amor, H. I. Christensen, H. Furuta, H. Walke, H. Fang, I. Mordatch, I. Radosavovic, I. Leal, J. Liang. J. Kim, Schneider, J. Hsu, Bohg J. Bingham, J. Wu, J. Wu, J. Luo, Gu, J. Tan, J. Oh, Malik, Tompson, J. Yang. J. J. Lim, J. Silvério, J. Han, K. Rao, K. Pertsch, K. Hausman, K. Go, K. Gopalakrishnan, K. Goldberg, K. Byrne, K. Oslund, K. Kawaharazuka, K. Zhang, K. Majd, K. Rana, K. Srinivasan, L. Y. Chen, L. Pinto, L. Tan, L. Ott, L. Lee, M. Tomizuka, M. Du M. Ahn, M. Zhang, M. Ding, M. K. Srirama, M. Sharma, M. J. Kim, N. Kanazawa, N. Hansen, N. Heess, N. J. Joshi, N. Suenderhauf, N. D. Palo, N. M. M. Shafiullah, O. Mees, O. Kroemer, P. R. Sanketi, P. Wohlhart, P. Xu, P. Sermanet, P. Sundaresan, Q. Vuong, R. Rafailov, R. Tian, R. Doshi, R. Martín-Martin, R. Mendonca, R. Shah, R. Hoque, R. Julian, S. Bustamante, S. Kirmani, S. Levine, S. Moore, S. Bahl, S. Dass, S. Song, S. Xu, Haldar, S. Adebola, S. Guist, S. Nasiriany, S. Schaal, S. Welker, S. Tian, S. Dasari, S. Belkhale, Osa, Harada, T. Matsushima, T. Xiao, T. Yu, T. Ding, T. Davchev, T. Z. Zhao, T. Armstrong, T. Darrell, V. Jain, V. Vanhoucke, W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Wang. X. Zhu, X. Li, Y. Lu, Y. Chebotar, Y. Zhou, Y. Zhu, Y. Xu, Y. Wang, Y. Bisk, Y. Cho, Y. Lee, Y. Cui, Y. hua Wu, Y. Tang, Y. Zhu, Y. Li, Y. Iwasawa, Y. Matsuo, Z. Xu, and Z. J. Cui. Open X-Embodiment: Robotic learning datasets and RT-X models. https: / arxiv org/ abs 2310. 08864, 2023.
[2] A. Brohan, N. Brown, Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, Leal, K.-H. Lee, S. Levine, Y. La, U. Malla, D. Manju- nath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-1: Robotics transformer for real-world control at scale. In arXiv preprint 2rXiv:2212.0681/1 2022.
[3] Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[4] A. Xie, L. Lee, T. Xiao, and C. Finn. Decomposing the generalization gap in imitation learning for visual robotic manipulation. arXiv preprint 1.Xiv:2.07703659) 2023.
[5] Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y. Tan, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy. https:// htttp:://otoo-mmdelss github. .ioub.io,,.. 2023.
[6] H. Walke, K. Black, A. Lee, M. J. Kim, M. Du, C. Zheng, T. Zhao, P. Hansen-Estrude Q. Vuong, A. He, V. Myers, K. Fang, C. Finn, and S. Levine. Bridgedata v2: A dataset for robot learning at scale, 2023.

---

[7] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess,
A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman,
A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T.-W. E. Lee, S. Levine, Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, Salazar, Sanketi, P. Sermanet, Singh, Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In arXiv preprint 20t.Xiv223077188888 2023.
[8] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), volume 139, pages 8748-8763 2021.
[9] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre- training. In International Conference on Computer Vision (ICCV), 2023.
[10] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Babaei, N. Bashlykov. S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint (XXx:2307.092880 2023.
[1]] A. Khazatsky, K. Pertsch, Nair, A. Balakrishna, Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y. Ma. P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y. Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. Lu. J. Mercat, A. Rehman, P. R. Sanketi, A. Sharma, C. Simpson, Q. Vuong, H. R. Walke, B. Wulfe, T. Xiao, J. H. Yang, A. Yavary, T. Z. Zhao, C. Agia, R. Baijal, M. G. Castro, D. Chen, Q. Chen, T. Chung, J. Drake, E. P. Foster, J. Gao, D. A. Herrera, M. Heo, K. Hsu, J. Hu, D. Jackson, C. Le, Y. Li, Lin, R. Lin, 7. Ma, A. Maddukuri, Mirchandani, D. Morton, Nguyen, A. O'Neill, Scalise, Seale, Son, S. Tian, E. Tran, A. E. Wang, Y. Wu, A. Xie, Yang. P. Yin, Y. Zhang, O. Bastani, G. Berseth, Bohg, K. Goldberg, A. Gupta, A. Gupta, D. Jayaraman, J. Lim, J. Malik, R. Martin-Martin S. Ramamoorthy, D. Sadigh, S. Song, J. Wu, M. C. Yip, Y. Zhu, T. Kollar, S. Levine, and C. Finn. Droid: A large-scale in-the-wild robot manipulation dataset. 2024.
[12] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual representa- tion for robot manipulation. In CoRL, 2022.
[13] S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang. Language- driven representation learning for robotics. ArXiv, 1222232..1766,, 2023. URL https: // //api. seppltiemmtiii Bawll.ogg/ooooo http//ooppsID::52205777
[14] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipulation. In Conference on robot learning, pages 894-906. PMLR, 2022.
[15] A. Stone, T. Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong. P. Wohlhart, B. Zitkovich, F. Xia, C. Finn, et al. Open-world object manipulation using pre-trained vasion-language models. arXiv preprint arXiv:2303.00905 2023.
[16] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378. 2023.
[17] A. S. al. Introducing rfm-l: ing capabilities, 2024.
Giving robots human-like reason-
URL https://covariant ai/insights/
introducing- sewecinnnn----iee 1-wiiing -i-giiigg-ooootssssss wooots-humannn.....ooo like- www.oniiieeeeeeinnneeeeene.nnnne cwwppiiiiii..g.iiiiiiii...ee
10

---

[18] Wayve, Lingo-2: Driving with natural language. 2024. URL lingo- 2irgo-2-driing-- weaeengwwithhaaaeen thh-anguage//
ai/thinking/
[19] X. Chen, X. Wang. S. Changpinyo, A. Piergiovanni, P. Padlewski, D. M. Salz, S. Goodman,
A. Gryener, B. Mustafa, L. Beyer, A. Kolesnikov, J. Puigeerver, N. Ding, K. Rong, H. Akbari, G. Mishra, L. Xue, A. V. Thapliyal, J. Bradbury, W. Kuo, M. Seyedhosseini, C. Jia, B. K. Ayan, C. Riquelme, A. Steiner, A. Angelova, X. Zhai, N. Houlsby, and R. Soricut. Pali: A jointly-scaled multilingual language-image model. ArXiv, abs/2209.06794, 2022. URL seeppiiseemaiiicccccieaa .rg. Corpus :2222223320.
[20] X. Chen, X. Wang, L. Beyer, A. Kolesnikov, J. Wu, P. Voigtlaender, B. Mustafa, S. Goodman, I. M. Alabdulmohsin, P. Padlewski, D. M. Salz, X. Xiong, D. Vlasic, F. Pavetic, K. Rong, T. Yu, D. Keysers, X.-Q. Zhai, and R. Soricut, PaLI-3 vision language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.0119, 2023.
[21] T. Wolf, I.. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, and Transformers: State-of-the-ae natural language processing. In Proceedings of the 6th International Conference on Learning Representations, 2020. URL https: pttps://aaee /awxivvvvvvaa///////a////aaaaa org/ abs wtw.////........ 09910003777
[22] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[23] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. I. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 2023.
[24] G. Team, T. Mesnard, Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403..m 08295, 2024.
[25] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
[26] T. Dettmers, A. Pagnoni, A. Holtzman, and L.. Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
[27] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Computer Vision and Pattern Recognition (CVPR), 2017.
[28] A. Hudson and C. D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Computer Vision and Pattern Recognition (CVPR), 2019.
[29] A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach. Towards VQA models that can read. Computer Vision and Pattern Recognition (CVPR), 2019.
[30] J. P. Bigham, C. Jayant, H. Ji, G. Little, A. Miller, R. C. Miller, R. Miller, A. Tatarowicz, B. White, S. White, and T. Yeh. VizWiz: nearly real-time answers to visual questions. In User Interface Software and Technology (UIST), pages 333-342, 2010.
[31] S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg. ReferltGame: Referring to objects
in photographs of natural scenes. In Empirical Methods in Natural Language Processing (EMNLP), pages 787-798, 2014.
[32] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg. Modeling context in referring expressions. In European Conference on Computer Vision (ECCV),
11

---

[33] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
[34] T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, P. Tafti, L. Hussenot, P. G. Sessa, A. Chowdhery, A. Roberts, A. Barua, A. Botev, A. Castro-Roer Ros, A. Slone, A. Héliou, A. Tacchetti, A. Bulanova, A. Paterson, B. Tsai, Shahriari, C. L.. Lan, C. A. Choquette-Chon C. Crepy, D. Cer, D. Ippolito, D. Reid, E. Buchatskaya, E. Ni, E. Noland, G. Yan, G. Tucker, G.-C. Muraru, G. Rozhdestvenskly, H. Michalewski, I. Tenney, I. Grishchenko, J. Austin, J. Keeling, J. Labanowski, J.-B. Lespiau, J. Stanway, J. Brennan, J. Chen, J. Ferret, J. Chiu, J. Mao-Jones, K. Lee, K. Yu, K. Millican, L. L. Sjoesund, L. Lee, L. Dixon, M. Reid, M. Mikula, M. Wirth, M. Sharman, N. Chinaev, N. Thain, O. Bachem, O. Chang, O. Wahltinez, P. Bailey, P. Michel, P. Yotov, R. Chaabouni, R. Comanescu, R. Jana, R. Anil, R. Mcllroy, R. Liu, R. Mullins, S. L. Smith, S. Borgeaud,
S. Girgin, S. Douglas, S. Pandya, Shakeri, S. De, T. Klimenko, Hennigan, V. Feinberg, W. Stokowiec, Y. hui Chen, Z. Ahmed, Z. Gong, T. Warkentin, L. Peran, M. Giang, C. Farabet,
O. Vinyals, J. Dean, K. Kavukcuoglu, D. Hassabis, Z. Ghahramani, D. Bck, J. Barral, F. Pereira, E. Collins, A. Joulin, N. Fiedel, E. Senter, A. Andreev, and K. Kenealy, Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0822
2024.
[35] Y. Li, S. Bubeck, R. Eldan, A. D. Giorno, S. Gunasekar, and Y. 'T. Lee. Texthooks are all you need ii: phi-1.5 technical report. arXiv preprint (rXiv:2309.05466 2023,
[36] J. Bai, S. Bai, Y. Chu, Cui, K. Dang. X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[37] J. Li, D. Li, C. Xiong, and S. C. H. Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning (ICML), 2022.
[38] J. Li, D. S. Savarese, and S. C. H. Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning (ICML), 2023.
[39] W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. A. Li, P. Fung, and S. C. H. Hoi. general-purpose
InstructBLIPP Towards general- purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.063: 06500, 2023.
[40] H. H. Tan and M. Bansal. LXMERT: Learning cross-modality encoder representations from transformers. In Empirical Methods in Natural Language Processing (EMNLP), 2019.
[41] H. Laurençon, L. Saulnier, L. Tronchon, S. Bekman, A. Singh, A. Lozhkov, T. Wang, S. Karam- cheti, A. M. Rush, D. Kiela, M. Cord, and V. Sanh. OBELICS: An open web-scale filtered dataset of interleaved image-text documents. In Neural Information Processing Systems Track Datasets and Benchmarks (NeurIPS Datasets and Benchmarks), 2023.
[42] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[43] H. Liu, C. Li, Y. Li, and Y. Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.0377. 2023.
[44] S. Karamcheti, S. Nair, A. Balakrishna, P. Liang, Kollar, and D. Sadigh. Prismatic vlms: Investigating the design space of cisually-conditioned language models. arXiv preprint wwXiv:2402.07865.. 2024.
[45] D. Kalashnikov, A. Irpan, Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, al. QT-Opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: 10293, 2018.
12

---

[46] D. Kalashnkov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and
K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv, 2021.
[47] F. Ebert, Y. Yang. K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109.
2021.
[48] K. Ehsani, T. Gupta, R. Hendrix, J. Salvador, L. Weihs, K.-H. Zeng, K. P. Singh, Y. Kim, W. Han, A. Herrasti, et al. Imitating shortest paths in simulation enables effective navigation and manipulation the real world. arXiv preprint arXiv:23/2.02977 2023.
[49] H. Bharadhwaj. J. Vakil, M. Sharma, A. Gupta, S. Tulsiani, and V. Kumar. Roboagent: Generalization and efficiency robot manipulation via semantic augmentations and action chunking. arXiv preprint arXiv:2309.01918. 2023.
[50] I.. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA), pages 3406-3413. IEEE, 2016.
[51] A. Mandlekar, Y. Zhu, A. Garg, Booher, M. Spero, A. Tung, Gao, J. Emmons, Gupta, E. Orbay, al. Roboturk: crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, pages 879-895 PMLR, 2018.
[52] A. Gupta, A. Murali, D. Gandhi, and L. Pinto. Robot learning in homes: Improving generalization and reducing dataset bias. Advances in neural information processing systems, 31, 2018.
[53] S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and C. Finn. Robonet: Large-scale multi-robot learning. CoRL, 2019.
[54] S. Cabi, S. G. Colmenarejo, A. Novikov, K. Konyushkova, S. Reed, R. Jeong, K. Zolna, Y. Aytar, D. Budden, M. Vecerik, O. Sushkov, D. Barker, J. Scholz, M. Denil, N. de Freitas, and Z. Wang. Scaling data-driven robotics with reward sketching and batch reinforcement learning. RSS, 2019.
[55] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991-1002. PMLR, 2022.
[56] H.-S. Fang, H. Fang. Z. Tang, J. Liu, C. Wang. J. Wang, H. Zhu, and C. Lu. Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot. Towards Generalist Learning Paradigms for Scalable Skill Acquisition @ CoRL2023, 2023.
Robots:
[57] C. Devin, A. Gupta, T. Darrell, P. Abbeel, and S. Levine. Learning modular neural net- work policies for task and multi-robot transfer. In Proceedings of IEEE International Conference on Robotics and Automation, 2017.
[58] E. S. Hu, K. Huang, O. Rybkin, and D. Jayaraman. Know thyself: Transferable visual control policies through robot-awareness. In International Conference on Learning Representations, 2022.
[59] J. H. Yang, Sadigh, and C. Finn. Polybot:
Training one policy across robots while embracing variability. In 7th Annual Conference on Robot Learning, 2023. URL https:
// /oppneevieww...//// www.ee..eeeeeeeeeee wwwp/ftrmemidieeeeeeeeeeeeteeeee id-HEIRj511cs..
[60] S. Reed, K. Zolna, Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-maron, M. Giménez, Y. Sulsky, J. Kay, T. Springenberg. T. Eccles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y. Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas. A generalist agent. Transactions
Machine Learning Research, 2022. ISSN 2835-8856.
13

---

[61] G. Salhotra, I.-C. A. Liu, and G. Sukhatme. Bridging action space mismatch in learning from demonstrations. arXiv preprint arXiv:2304.03833, 2023.
[62] I. Radosavovic, B. Shi, L. Fu, K. Goldberg, T. Darrell, and Malik. Robot learning with sensorimotor pre-training. In Conference Robot Learning, 2023.
[63] D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. Gnm: A general navigation model to drive any robot. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7226-7233 IEEE, 2023.
[64] K. Bousmalis, G. Vezzani, D. Rao, C. Devin, A. X. Lee, M. Bauza, T. Davchev, Y. Zhou, A. Gupta, A. Raju, et al. Robocat: A self- foundation agent for robotic manipulation. arXiv preprint arXiv:2306. 1706, 2023.
[65] D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine. ViNT: A foundation model for visual navigation. In 7th Annual Conference on Robot Learning, 2023.
URL https::/ttaattte /aaxivv org/ abs/ 2aobbs/2330 2200. 14846.
[66] J. Yang, C. Glossop, A. Bhorkar, D. Shah, Q. Vuong, C. Finn, D. Sadigh, and Levine. Pushing the limits of coos-emmbodiment learning for manipulation and navigation. arXiv preprint arXiv:2402. 19432, 2024.
[67] S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song. Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation. Proceedings of the IEFF/CVE Conference on Computer Vision and Puttern Recognition, pages 23171-23181, 2023.
[68] Y. Du, K. Konyushkova, M. Denil, A. Raju, J. Landon, F. Hill, N. de Freitas, and S. Cabi. Vision-language models as success detectors. arXiv preprint arXiv:2303.07280, 2023.
[69] Y. J. Ma, V. Kumar, A. Zhang, O. Bastani, and D. Jayaraman. Liv: Language-image represen- tations and rewards for robotic control. In International Conference on Machine Learning, pages 23301-23320 PMLR, 2023.
[70] Zhang, Ding, S. Amiri, H. Yang. Kaminski, C. Esselink, and Zhang. Grounding classical task planners via vision-language models. arXiv preprint arXiv:2304. 00004.0887.... 2023.
[71] S. Sontakke, J. Zhang. S. Amold, K. Pertsch, E. BIyIk, D. Sadigh, C. Finn, and L. Itti. Roboclip: One demonstration is enough to learn robot policies. Advances in Neural Information Processing Systems, 36, 2024.
[72] J. Huang, S. Yong, X. Ma, X. Linghu, P. Li, Y. Wang, Q. Li, S.-C. Zhu, B. Jia, and S. Huang. An embodied generalist agent in 3d world. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
[73] X. Li, M. Liu, H. Zhang, Yu, J. Xu, H. Wu, C. Cheang, Jing, W. Zhang, Liu, al. Vision-language foundation models effective robot imitators. arXiv preprint arXiv:2311.01378, 2023.
[74] H. Zhen, X. Qiu, P. Chen, J. Yang, X. Yan, Y. Du, Y. Hong, and C. Gan. 3d-vla: 3d vision- language-action generative world model. arXiv preprint arXiv:2403 24t.02403.0931 2024.
[75] N. Dorka, C. Huang, T. Welschehold, and W. Burgard. What matters in employing vision language models for tokenizing actions in robot control? In First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024.
[76] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975- 1986, 2023.
14

---

[77] A. Radford. J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[78] H. Face. Introducing idefics: An open reproduction of state-of-the-an visual langage model. Hugging Face Blog, 2024.
[79] Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances neural information processing systems, 36, 2024.
[80] McKinzie, Z. Gan, J.-P. Fauconnier, S. Dodge, B. Zhang, P. Dufter, D. Shah, X. Du, Peng, F. Weers, et al. Mml: Methods, analysis insights from multimodal Ilm pre-training, arXiv preprint arXiv:2403. 000000961,, 2024.
[81] J. Lin, H. Yin, W. Ping, Y. Lu, P. Molchanov, A. Tao, H. Mao, Kautz, M. Shoeybi, and S. Han. Vila: On pre-training for visual language models. arXiv preprint arXiv:2312.07533, 2023.
[82] Y. Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304. 1277, 2023.
[83] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. Gpt3. int8 O: 8-bit matrix multi- plication for transformers at scale. Advances in Neural Information Processing Systems, 35: 10338-3033, 2022.
[84] T. Z. Zhao, V. Kumar, Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 13705, 2023.
[85] A. Brohan, N. Brown, Carbajal, Y. Chebotar, J. Dabis, Finn, K. Gopalakrishnan, K. Haus- man, Herzog, Hsu, et Rt-1: Robotics transformer for real-world control scale. arXiv preprint arXiv:22 12.0:22212.01117 2022.
[86] E. Rosete-Beas, O. Mees, G. Kalweit, J. Boedecker, and W. Burgard. Latent plans for task agnostic offline reinforcement learning. In Proceedings of the 6th Conference Robot Learning (CoRI.), 2022.
[87] O. Mees, J. Beaja,Diaz,aaa and W. Burgard. Grounding language with visual affordances over unstructured data. In Proceedings of the IEEE International Conference on Roboties and Automation (ICRA), London, UK, 2023.
[88] S. Dass, J. Yapeter, J. Zhang, J. Zhang, K. Pertsch, S. Nikolaidis, and J. Lim. CLVR jaco play dataset, 2023. URL https: //tt///tttttt/t com/ cttcc//caaacccccccccccccccacccaa wwt.aaa/cvv..jacoopppp.. htw.jaaco_pll_dattssett.ttt dataset.
[89] J. Luo, C. Xu, X. Geng, G. Feng, K. Fang, L. Tan, S. Schaal, and Levine. Multi-stage cable routing through hicrarchical imitation learning. arXiv preprint arXhv:2307.08927 2023.
[90] A. Mandlekar, Y. Zhu, A. Garg, Booher, M. Spero, A. Tung. J. Gao, J. Emmons, A. Gupta, E. Orbay, S. Savarese, and L. Fei-Fei. RoboTurk: A crowdsourcing platform for robotic skill learning through imitation. CoRR, abs/181 1.00999 2018. URL pestp://arxtt org/ /bs// 1811.02796
[91] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors, 2023.
[92] L. Y. Chen, S. Adebola, and K. Goldberg. Berkeley UR5 demonstration dataset. //sites. www.gl..ooo..oooooo.ooo.o....... www/wvwwwwerreeeleeeeeeeeeeeeeee home.
https:
15

---

[93] G. Zhou, V. Dean, M. K. Srirama, A. Rajeswaran, J. Pari, K. Hatch, A. Jain, T. Yu, P. Abbeel, I. Pinto, C. Finn, and A, Gupta. Train offiin, test online: real robot learning benchmark, 2023.
[94] C. Lynch, A. Wahid, Tompson, T. Ding, J. Betker, R. Baruch, Armstrong, and P. Florence. Interactive language: Talking robots real time, IEEE Roboties and Automation Letters, 2023.
[95] S. Belkhale, Y. Cui, and D. Sadigh. Hydra: Hybrid robot actions for imitation learning. arxiv, 2023.
[96] Y. Zhu, P. Stone, and Y. Zhu. Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation. IEEE Robotics and Automation Letters, 7(2):4126-4133 2022.
[97] Z. J. Cui, Y. Wang. N. M. M. Shafiullah, and L. Pinto. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint wrXhv:2210.10047 2022.
[98] M. Heo, Y. Lee, D. Lee, and J. Lim. Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation. In Robotics: Science and Systems, 2023.
[99] K. W. Ge Yan and X. Wang. uesd kitchens Dataset. August 2023.
[100] S. Nasiriany, T. Gao, A. Mandlekar, and Y. Zhu. Learning and retrieval from prior data for skill-based imitation learning. In Conference on Robot Learning (CoRL), 2022.
[101] H. Liu, S. Nasiriany, L. Zhang, Z. Bao, and Y. Zhu. Robot learning on the job: Human-in- the-loop autonomy and learning during deployment. In Robotics: Science and Systems (RSS), 2023.
[102] Quere, Hagengruber. M. Iskandar, Bustamante, Leidner, F. Stulp, and Vogel. Shared Control Templates for Assistive Robotics. In 2020 IEEE International Conference on Robotics and Automation (ICRA), page 7, Paris, France, 2020.
[103] S. Saxena, M. Sharma, and O. Kroemer. Multi-resolution sensing for real-time control with vision-language models. In 7th Anmual Conference on Robot Learning, 2023, URL https://openrewe
net orrmmorrmm w Mid-
[104] R. Shah, R. Martin-Martin, and Zhu, MUTEX: Learning unified policies from multimodal
task specifications. In 7th Annual Conference on Robot Learning, 2023. URL https:
[105] X. Zhu, R. Tian, C. Xu, M. Ding, W. Zhan, and M. Tomizuka. Fanuc manipulation: dataset for dearning-based manipulation with fanuc mate 200id robot. 2023.
[106] R. Mendonca, S. Bahl, and D. Pathak. Structured world models from human videos. CoRL, 2023.
[107] N. M. M. Shafiullah, A. Rai, H. Etukur,, Y. Liu, I. Misra, S. Chintala, and L. Pinto. On bringing robots home, 2023.
[108] NVIDIA, Tensorrt-llm. URL htt.:/rpee///ggha.aa.a///////e/r
[109] PyTorch.
Automatic mixed precision. URL httpss:/pytotrchh wwt.:oossts/tooo//////o//oo/o/ee htt.:/pps//yrrrer..rr/////////// html.
[110] T. Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv
preprint arXiv:2307.0869) 08691, 2023.
16

---

[111] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36, 2024.
[112] V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:/910.01100. 108, 2019.
17

---

A Data Mixture Details
We list our used data mixture in Table 3. The mixture mostly follows [5], with few additional datasets.
OpenVLA Training Dataset Mixture
Fractal [85]
12.7%
Kuka [45]
12.7%
Bridge[6, 47]
13.3%
Taco Play [86, 87]
3.0%
Jaco Play [88]
0.4%
Berkeley Cable Routing [89]
0.2%
Roboturk [90]
2.3%
Viola [91]
0.9%
Berkeley Autolab UR5 [92]
1.2%
Toto [93]
2.0%
Language Table [94]
4.4%
Stanford Hydra Dataset [95]
4.4%
Austin Buds Dataset [96]
0.2%
NYU Franka Play Dataset [97]
0.8%
Furniture Bench Dataset [98]
2.4%
UCSD Kitchen Dataset [99]
<0.1%
Austin Sailor Dataset [100]
2.2%
Austin Sirius Dataset [101]
1.7%
DLR EDAN Shared Control [102]
<0.1%
IAMLab CMU Pickup Insert [103]
0.9%
UTAuutiin Mutex [104]
2.2%
Berkeley Fanuc Manipulation [105]
0.7%
CMU Stretch [106]
0.2%
BC-Z [55]
7.5%
FMB Dataset [?
7.1%
DobbE [107]
1.4%
DROID [11]
10.0%
Table VLe training data mixture using datasets from the Open X E-Eboodiment dataset [1], following [5] with a few additions.
B Evaluation Tasks and Detailed Results
In this section, we provide more details on the BridgeData V2 WidowX and Google robot evaluations discussed in Section 4.1, well as the Franka-Tableton and Franka-DROID fine-tuning evaluations discussed in Section 4.2.
B.1 Bridge V2 WidowX Evaluation Details
Here we focus specifically on BridgeDatt V2 evaluations discussed in Section 4.1.
B.1.1 Bridge V2 Evaluation Tasks
As described in Section 4.1, we evaluate each generalist robot manipulation policy on 17 tasks with 10 trials each. In this section, we provide details the task categories and individual tasks.
In total, we evaluate 5 visual generalization tasks, 2 motion generalization tasks, physical generalization tasks, 4 semantic generalization tasks, and 3 language grounding tasks. Note that all tasks we evaluate on introduce some form of distribution shift since we are unable to procure the exact objects used in the original dataset (other distribution shifts naturally arise as we reproduce real-world test environment originally constructed at a different location; see Appendix 1.1. for detailed discussion on such distribution shifts). All 17 tasks are depicted Fig. 6. Each rollout is
We remove DROID for the last third of training due to slow learning progress (see Section 3.3) and re- distribute its mixture weights across all other datasets.
18

---

Eggplant Pot (Easy
Eggolant
Visual Gen.
Put from Counter into Sink
Eggplant into (w/í Clutteri
Yellow
Motion Gen.
Lift Eggplent
Put Carrot on Plate Height Change)
Put Carrot on Plate
Flip Pot Upright
Physical Gen.
Lift AAA Battery
Move Skull into Drying Rack
Lift White Tape
Semantic Gen.
Purple Pot
Blue Cup
Put Eggplant into Pot
Put Red Bottle Pot
Language Grounding
Lift Cheese
Lift Red Ghili Pepper
on
Figure 6: BridgeData V2 WidowX robot evaluation tasks. We evaluate every generalist robot policy on 4 types out-of-distribution (OOD) generalization tasks: visual, motion, physical, and semantic (as defined in Section 4.1). Every pair of images shows the start state and example end state after the robot completes the task. also rigorously assess language grounding the tasks shown the bottom rows, by changing prompt while tixing the initial state and testing whether the policy can approach the correct target object.
19

---

marked as a failure (0) or success (1). In some more difficult tasks, we record partial successes (0.5); we describe the conditions for partial credit in the task descriptions below.
Below we describe each of the 17 tasks, in the order shown in Fig. 6:
1. Put Eggplant into Pot (Easy Version): The robot's goal is to pick up the eggplant and drop into the pot. This is visual generalization task because we use a handcrafted paper pot that has different appearance than the pot used in the original BridgeData V2 training dataset (since we are unable to procure the original pot). Unlike all 16 other tasks, for this particular task we initialize the robot's end-effector directly above the eggplant before rolling out the policy; hence, we call this the "Easy Version" of the "Put Eggplant into Pot" task.
2. Put Eggplant into Pot: This is the same task as described above, except that the robot's s end-effector is not initialized directly above the eggplant. Instead, we initialize it a position that is fixed across all rollouts, which means that the robot must horizontally reach for the eggplant first before manipulating it. (Note: The same applies all other tasks described below.) This is visual generalization task for the same reason as above.
3. Put Cup from Counter into Sink: The robot's goal is to pick up the pink cup from either the kitchen countertop or drying rack and place it into the sink on the right. This is a visual generalization task because we use a pink cup rather than a blue cup (a blue cup is used in the original BridgeData V2 dataset, but we find that none of the methods we evaluate is able to manipulate reliably most likely because the color of the cup blends with the color of the sink).
4. Put Eggplant into Pot (w/ Clutter): This is the same task as the "Put Eggplant into Pot" task, except that it is more difficult due to the presence of several distractor objects. It is visual generalization task for the same reason discussed in the normal "Put Eggplant into Pot" task, and even more so given unseen distractors in the scene. Partial credit (0.5 out of 1) is rewarded when the robot moves towards the correct target object.
5. Put Yellow Corn on Pink Plate: The robot's goal is to pick up the yellow corn and place it on the pink plate. This is a visual generalization task due to the presence of unseen distractor objects in the scene, such green dinosaur on the countertop in the back section of the sink. Partial credit (0.5 out of 1) is rewarded when the robot moves towards the correct target object.
6. Lift Eggplant: generalization task because the eggplant is initialized in unseen positions and/or orienta- tions, and the robot forced to move beyond its training distribution of positions and/or orientations and often perform long-range reaching in order to complete the task. (Note: Long-range reaching is not demonstrated in this environment in the original BridgeData V2 demonstrations; see Appendix 1.2 for details.) We find that this task, though seemingly simple, is deceptively challenging for many policies. Partial credit (0.5 out of 1) is rewarded when the robot makes contact with the eggplant.
The robot's goal is to grasp and lift the eggplant into the air. This is motion
7. Put Carrot on Plate (w/ Height Change): The robot's goal is to pick up the carrot and place it on the yellow plate. This is a motion generalization task because the plate is clevated from its usual position at the bottom of the sink, and the robot must adjust its trajectory correctly place the carrot on the elevated platform (without knocking down the plate in the process). Partial credit (0.5 out of I) is rewarded when the robot grasps the carrot and touches the plate with it.
8. Put Carrot on Plate: This is the same task as above, except that the plate is at its normal position (at the bottom of the sink or drying rack). We consider this a physical generaliza- tion task because the carrot has a different size and shape than the one used in the original BrtageData V2 dataset, which shorter and narrower. (Note that the previous version of this task listed above would also technically be a physical generalization task since it involves the same carrot, but we list it under the "motion generalization" category since that is the focus there.)
20

---

9. Flip Pot Upright: The robot's goal is to manipulate the pot such that it is oriented upright in the sink at the end of the episode. This is a physical generalization task because this pot has a different size and shape than the one used in the original Bata V2 training demonstrations (the pot we use is wider and shorter).
10. Lift AAA Battery:
The robot's s goal is simply to grasp the AAA battery and lift it up into the air. This is considered a physical generalization task because the battery is much smaller and thinner than target objects seen the BridgeData V2 training demonstrations this environment; see Appendix B.1.2 for details. (Note that this target object does not exist in the original BridgeData V2 demonstrations this environment, so this is also an instance of "semantic generalization", but classify it solely as "physical generalization" since that is the main focus here).
11. Move Skull into Drying Rack:
The robot's goal is to grasp the skull windup toy and drop it into the yellow drying rack in the left part of the sink. This is semantic generalization task since the skull is an unseen target object (does not appear in the BridgeData V2 training demonstrations).
12. Lift White Tape: The robot's goal is to grasp and lift the white roll of tape into the air. This is semantic generalization task since the white tape roll is an unseen target object (does not appear in the BridgeData V2 training demonstrations), (Note that this task may also be considered as "physical generalization" because of its shape being different than the objects seen in the training demonstrations in this environment; most policies struggle to grasp objects with this ring structure, and they often move the robot's end-effector directly into the center region.)
13. Take Purple Grapes out of Pot: The robot's goal is to grasp the purple grapes lying inside the steel pot and remove it from the pot (by lifting out and/or dropping it anywhere outside the pot). This is a semantic generalization task because is an unseen language instruction; the robot has never seen this task in the original BridgeData V2 training dataset.
14. Stack Blue Cup on Pink Cup: The robot's goal is to grasp the blue cup and place it securely on top of the pink cup. This is a semantic generalization task because it is an unseen language instruction; the robot has never seen this task in this environment in the original BridgeData V2 training dataset. Partial credit (0.5 out of 1) is rewarded when the robot grasps the blue cup and touches the pink cup with the blue cup.
15. Put (Eggplant, Red Bottle) into Pot:
This is a language grounding task. The robot's 'ss goal is to put the specified target object into the pot. Both the eggplant and red bottle are present in the scene. We conduct paired evaluations: for the same initial state, we prompt the policy target the eggplant in one episode, and then the red bottle in the next episode. We test each method 5 times with the eggplant and 5 times with the red bottle, using the same set of 5 initial states for both target objects. Partial credit (0.5 out of 1) is rewarded when the robot moves towards the correct target object.
16. Lift (Cheese, Red Chili Pepper): This is language grounding task. The robot's goal is to grasp and lift the specified target object. We conduct paired evaluations as described in the task above. Partial credit (0.5 out of 1) is rewarded when the robot moves towards the correct target object.
17. Put (Blue Cup, Pink Cup) on Plate: This is language grounding task. The robot's goal is to grasp the specified target object and place it onto the plate. We conduct paired evaluations as described in other language grounding tasks. Partial credit (0.5 out of 1) is rewarded when the robot moves fowards the correct target object.
B.1.2 Comparing Evaluation Tasks to Original Bridge V2 Training Data
We conduct our evaluations in a sink environment used in the original BridgeData V2 dataset [6]. We reproduce the environment to match the original environment in the BrtdgeData V2 dataset with rough approximations for the robot's 's location relative to the sink, well as the camera's 's placement
21

---

relative to the scene. Given the lack of precise measurements of these positions in the original dataset, we are unable to reproduce the exact environment setup, and natural distribution shifts arise due to slightly different robot, sink, and camera placements. In addition, since we evaluate robot policies in a different location than where the training demonstrations were collected from, other natural distribution shifts arise. For example, the lighting conditions and background (e.g., visible areas behind the sink) are inevitably different than what was seen in the training dataset. Furthermore, are unable to procure the exact set of objects used in the original BridgeData V2 dataset, so there are distribution shifts between the objects used at train time and those used at test time.
Despite all these challenges, we find that certain generalist policies, such as OpenVLA and RT-2-X, can still generalize and perform various tasks fairly reliably cet-oofthhe-box- Other generalist policies, such as RT-1-X and Octo, can also complete some tasks, though they struggle when tested with more difficult generalization tasks in our BridgeData V2 evaluation suite.
The original BridgeData V2 dataset includes demonstrations of the following seven tasks in this specific sink environment: "Flip Pot Upright", "Put Carrot on Plate", "Put Cup from Counter (or Drying Rack) into Sink", "Put Eggplant into Pot", "Put Knife on Cutting Board", "Put Spoon in Pot", and "Turn Lever Vertical to Front". See Fig. 7 for samples images of all these tasks from the original dataset. Note that all training demonstrations collected in this environment are initialized such that the robot's end-effector is positioned directly above the target object in the beginning of the episode. (However, this is not the case across all environments in the BridgeData V2 dataset; in some other environments, the robot is initialized farther away from the target object, so it must horizontally reach for the object first before manipulating it.)
Flip Pot Upright
Put Carrot on Plate
Put Cup from Counter into Sink
Turn Lever Vertical Front
Put Eggplant Into Pot
Put Knife Cutting Board
Put Spoon in
Figure 7: Original BridgeData V2 sink environment tasks. Images from sample demonstrations in the sink environment from the original BridgeData V2 dataset reveal that all demonstrations this environment were initialized such that the robot's 's end-effector was positioned immediately above the target object. Note that these initial states are different from the initial states we use in our BridgeData V2 evaluation tasks shown in Fig. 6. In our evaluations, always initialize the robot's end-effector to a fixed location above the sink, rather than positioning it directly above the target object (except for task: "Put Eggplant into Pot (Easy Version)").
In our BridgeData V2 evaluation suite, only one task "Put Eggplant into Pot (Easy Version") is initialized with the robot's end-effector hovering directly over the target object; in all 16 other tasks, the end-effector is initialized at a fixed location above the sink such that the robot must horizontally reach towards the object. This initial condition, in combination with the distribution shifts we introduce in the various types of OOD generalization in our evaluation suite, challenges the generalist policies and requires high degree of robustness in order to complete the tasks successfully. Hence, the success rates for policies like RT-1-X and Octo are lower than what is reported in prior works. However, find that other policies such as RT-2-X and Open VLA still achieve relatively strong performance despite all these distribution shifts and challenges.
22

---

B.1.3 Detailed Bridge V2 Evaluation Results
See Table 4 for the full Brta V2 WidowX evaluation results. The number of successes for each method, out of 10 trials, is listed for each of tasks. OpenVLA achieves strongest performance in the majority of the tasks and has the highest aggregate success rate among the generalist policies. RT-2-X also shows good performance, outperfoming RT-1-X and Octo, though it does not perform as well Open RT-1-X and Octo generally experience difficulty in these generalization tasks.
Table Detailed BridgeData V2 WidowX evaluation results. We report performance on the full evaluation suite of 17 tasks (discussed in Section 4.1), including visual/motion/ htt////ooooooiiiiieoooooo physica//emmatti generalization tasks and language grounding tasks. Note that partial success (score of 0.5) is possible for some tasks; see Appendix B. 1 for details. We find that OpenVLA performs best in most tasks and achieves highest performance overall, followed by RT-2-X. On the other hand, RT-1-X and Octo struggle in the evaluations, only getting 0-2 successes several tasks. See Fig. for illustrations of all tasks.
Category
Task
Trials
RT-1-X
Octo Successes
RI-2-X Successes
OpenVLA (ours)
Successes
Visual gen
Put Eggplant into Poi (Pacy Version)
10
10
Visual gen
Put Eggplant into Pot
10
1
10
gen
Put from
gen
Fpgplant
Clutte)
3.5
7.5
Visual gen
Put Yollow Corn on Pink Plate
10
4 0.5
9 T.5
Motion gen
Lift Eggplant
10
6.5
gen
Height Change)
Physical gen
Put Carrot on Plate
10
0 6
Physical gen
Flip Put Upright
10
8
Physical gen
Lift AAA Battery
10
0
Semantic gen
Move Skall Drying Rack
10
0
5
Semantic gen
Lan White Thie
10
0
Semantic
Take Purple Grapes
Pot
10
0
4
Semantic gen
Blue Cup
Cup
Language grounding
Put (Eggplant, Red Bottle] into Pot
10
4
7.5
Language grounding
Laft [Cheese, Red Chili Pepper)
10
1.5
2.5
8,5
10
Language grounding
Put (Bine Cup. Pink Cup) Plate
10
5.5
8.5
9.5
Mean Success Rate
18.5
20.022.65
50.6++33%
70.6%33.%%
Additionally, in Table 5, we provide the full evaluation results for the quantized inference experi- ments that were summarized in Table 2. For these evaluations, we test policies on representative BridgeData V2 tasks spanning all task categories the full evaluation suite.
Table 5: Full quantized inference results. Here we present the detailed version of the results shown in Table 2.
blooat16 Successes
int8 Successes
int4 Successes
Category
Task
# Trials
Visual gen
Put Eggplant into Pot (Easy Version)
10
9
9
Visual gen
Put Eggplant into Pot
10
7
Visual gen
Put Cup from Counter into Sink
10
Motion gen
Lift Eigppaant
10
6
7,5
Physical gen
Put Carrot on Plate
10
Physical gen
Lift AAA Battery
10
Semantic gen
Take Purple Grapes out of Pot
10
8
Language grounding
Put (Eggplant, Red Bottle) into Pot
10
Mean Success Rate
71.3 : 4.8%
58.1 # 5.1%
71.9 =: 4.7%
B.2
Google Robot Evaluation Details
In this section, we provide more details on the Google robot evaluations introduced in Section 4.1.
B.2.1 Google Robot Evaluation Tasks
On the Google robot, we evaluate each generalist robot policy on 12 tasks with 5 rollouts each, for total of 60 rollouts. The first five tasks test on in-distribution conditions, and the last seven tasks test on more difficult is marked as a failure (0) success (1).
(OOD) conditions. All tasks are depicted in Fig. 8. Each rollout
We describe the 12 tasks below:
1. Pick Coke Can (in-distribution)): The robot is positioned in front of platform with can of Coke top of it. The robot's goal is to grasp and lift the Coke can.
23

---

Apple
In In-Distribution Tasks
Open Middlo Drawwer
Apple
Upright
Move Orenge neer Brown Chip (w/ Unseen Background)
Pick Pepsi Can (w Unseen Background)
Plok Benane (Unseen Targel Object)
Put-of-Distributte Tasks
Move Coke Can to Taylor Swift (Uneeen Internet Photo]
Plak Green Cup (Unseen Target Object)
Place Apple P (Unseen Instruction)
Place Banana in Pan (Unseen Instructian)
Figure 8: Google robot evaluation tasks. evaluate every generalist rohot policy on in-distribution tasks and out-of-distribution (OOD) generalization tasks. OOD tasks involve unseen backgrounds. target objects, instractions/object relations, and semantic concepts (e.g., photos from the Internet that do not appear in robot action data).
2. Move Apple near Green Can (in (in-distiibution): The robot is positioned front a platform with an apple and green soda can on top of it. The robot's goal is to grasp the apple and move it next to the green can.
3. Move Blue Chip Bag near Apple (in-distrbbution): The robot is positioned in front of platform with blue bag of chips and apple top of it. The robot's goal is grasp the blue bag of chips and move it close to the apple.
4. Place Coke Can Upright (in-distribution): The robot is positioned in front of platform with can of Coke top of it, and the can is oriented horizontally on its side. The robot's goal is to grasp the Coke can and orient it to be in vertical position.
5. Open Middle Drawer (in-distribution): The robot is positioned in front of a set of three drawers. The robot's goal is to grasp the middle drawer handle and pull the drawer open.
6. Move Orange near Brown Chip Bag (OOD): The robot is positioned in front of a platform with a brown bag chips and an orange on top of it. A tablecloth with blue sky and white cloud patterns covers the platform underneath the objects. The robot's goal is grasp the orange and bring next to the bag of chips. This task is OOD because the orange is an unseen object relative to the training dataset, and the tablecloth is unseen background."
'eee Appendix of Brohan et al. [7] for detailed list of OOD conditions in Google robot evaluations.

---

7. Pick Pepsi Can (OOD): The robot is positioned in front of a platform with a can of Pepsi on top of it. A tablecloth with bright yellow/brown patterms covers the platform underneath the can. The robot's goal is to grasp and lift the can. This task is OOD because the Pepsi can an unseen object, and the tablecloth is an unseen background.
8. Pick Banana (OOD): The robot is positioned front of platform with an apple, a can of Coke, and banana. The robot's goal is grasp and lift the banana. This task is OOD because the banana is unseen target object.
9. Pick Green Cup (OOD): The robot is positioned in front of platform with banana, can of Pepsi, and green cup. The robot's goal is grasp and lift the green cup. This task is OOD because all objects in the scene are unseen in the training data.
10. Place Apple on Plate (OOD): The robot is positioned in front of a platform with plate and an apple. The robot's goal is to grasp the apple and move it onto the plate. This task is OOD because is a novel instruction describing an unseen object relation: training demonstrations only cover moving the apple near the plate, rather than placing it on top of the plate.
11. Place Banana in Pan (OOD): The robot is positioned in front of platform with a pan and banana. The robot's goal grasp the banana and move it into the pan. This task is OOD because the banana is an unseen target object, and it is a novel instruction describing an unseen object relation, as explained in the previous task.
12. Move Coke Can to Taylor Swift (OOD): The robot is positioned in front of a platform with a can of Coke and photos of three different celebrities, including Taylor Swift. The robot's goal is to grasp the can and move it to the photo Taylor Swift. This task is OOD because the photos of the celebrities are unseen in the robot interaction data.
B.2.2 Detailed Google Robot Evaluation Results
Table 6: Detailed Google robot evaluation results. We report full evaluation results for Google robot evaluations discussed in Section 4.1. Each generalist policy is evaluated with 60 rollouts across 12 tasks, covering both in-distribution and out-of-distribution (OOD) testing conditions. In the bottom row, report mean success rate StdErr each policy. Open VLA and RT-2-X both significantly outperform RT-1-X and overall (we bold the mean success rate for both due to overlapping error bars). See Fig. 8 for illustrations of all tasks.
KT-1-X
Octo
RT-2-X
OpenVLA (ours)
Category
Task
# Trials
Successes
Successes
Successes
Successes
In-distribation
Piek Coke Can
In-distribarion
Move Apple near Green Can
In di-iistibttion
Move Blue Chip Bag Apple
In-distribution
Place Coke Can Upright
In-distribution
Opea Middle Drawer
2
0OD
Move Orange near Brown Chip Bag
0OD
Pick Pepsi Can
OOD
Banana Pick Green Cup
0OD
0OD
Place Apple Plate
00D
Place Banana in Pan
00D
Move Coke Can near Taylor Swift
3
Mean Success Rate
33. 26.7 2..8% 7..3%...4%
85.0.14.6%
Full results for the Google robot evaluations are shown in Table 6. Overall, find that RT-1-X and Octo experience difficulty the evaluation tasks; they often unable to achieve single success out of five trials in several tasks. On the other hand, RT-2-X and Ope demonstrate strong performance, completing every task at least two times out of five trials; these two VLA policies perform comparably with each other on this particular evaluation suite.
B.3
Adaptation Experiment Details
In this section, we provide more details on the data-efficient adaptation experiments discussed in Section 4.2, where investigate the effectiveness of fine-tuned Open VLA policies new robot setups such as Franka- Franka-Tabletop and Tranka-DROID
25

---

B.3.1 Franka-Tabletop and Franka-DROID Tasks
collect 10-150 demonstrations of each of seven tasks. The first tasks correspond to a robot setup which denote
(Franka Emika Panda robot mounted on top table), and the final task corresponds to robot setup which call "Franka-DROIID"
In the Franka- Tranka-Tabletop setup, the first three six tasks correspond to single-instruction innssructtttinn tasks and are narrow, while are present in the scene and the robot must manipulate the correct one depending on the language instruction.
three tasks correspond to multi-instruction tasks in which multiple objects
In-Distribution
Out-of-Distribution
Carrot
Narrow Single insstrucctionn Tasks
Pour Corn into Pot
Flip Pot Upright
Move <object> onto Plate
Diverse Multi-instruction Tasks
Knock cobject> Over
Cover <objects with Towel
Figure 9: Franka- Franka-Tabletop fine-tuning tasks. Franka-Tabletop tasks used in the data-efficient adaptation experiments in Section 4.2 and described in detail in Fig. are depicted above. The first three of six tasks, shown in the top three rows, only involve single instruction, while the last three tasks in the bottom three rows involve multiple objects and instructions (the instructions specify the target object or target location). first column shows sample initial states matching the training data distribution, while the second column shows out-of-distribution (OOD) initial states (e.g.. unseen backgrounds, target objects, distractors, and object positiioss/rrienaationn)). Every policy in Section 4.2 is evaluated with 10-12 rollouts in-distribution tasks and 5 6 rollouts OOD tasks.
26

---

Below we describe each of the six Tranka-Tableiie tasks shown in Fig. 9:
1. Put Carrot in Bowl :inngle-nstrucctonn:: The robot's goal is to grasp the carrot and place it into the bowl. We collect 50 demonstrations of this task for the training dataset, randomly placing the carrot and the bowl at different locations on the table in every episode. The carrot is always initialized on the left side of the bowl. During evaluation, each trial is recorded as a success (1) or failure (0); there is no partial credit.
2. Pour Corn into Pot (single-instructinn instruction): The robot's goal is to grasp the red bowl, move towards the steel pot, and pour the contents (a yellow corn) into the pot. We collect 50 demonstrations this task for the training dataset, randomly placing the bowl and the pot at different locations on the table in every episode. The bowl always initialized on the right side of the pot. During evaluation, each trial is recorded as a success (1) or failure (0); there is partial credit.
3. Flip Pot Upright (single-insrruction): The robot's goal is to grasp the steel pot (which is initially oriented vertically). rotate to be in the upright position, and place it back onto the table. We collect only 10 demonstrations of this task for the training dataset, randomly placing the steel pot various locations within a small section of the table. During evaluation, each trial is recorded as a success (1), failure (0), or partial success (0.5). Partial successes include grasping the pot but not orienting it upright, or knocking it over to the upright position but not carefully guiding it. The robot must release the pot at the end of the episode for full credit.
4. Move <object> onto Plate conlti-instruction)) The robot's goal is to grasp one out of three objects (depending on the target specified in the language instruction) and place on the plate on the right side of the table. We collect 150 demonstrations of this task for the training dataset, randomly placing different combinations of three objects on the table and selecting one the target. The plate is always initialized on the right side of the table. During evaluation, each trial is recorded as a success (I), failure (0), or partial success (0.5). Partial success is recorded when the first object that the robot makes contact with is the correct target object (i.c., the object specified in the language instruction), but the robot does not complete the task.
5. Knock <object> Over
The robot's goal is to approach one out of three objects (depending on the target specified in the language instruction) and push it until it falls over. We collect 70 demonstrations of this task for the training dataset, randomly placing different combinations of three objects on the table and selecting one as the target. During evaluation, each trial is recorded as a success (I), failure (0), or partial success (0.5). Partial success recorded when the first object that the robot makes contact with is the correct target object (i.e., the object specified in the language instruction). but the robot does not complete the task.
6. Cover <object> with Towel imulti-instructtonn The robot's goal to grasp the blue towel and place it instruction). We collect 45 demonstrations of this task for the training dataset, randomly placing different combinations of three objects on the table. During evaluation, each trial is recorded as success (I), failure (0), or partial success (0.5). Partial success is recorded when the first object that the robot touches with the towel is the correct target object (i.e., the object specified the language instruction), but the robot does not complete the task (e.g., drops the towel onto the table instead of on top of the target object). Full credit is given when any part of the towel is resting over the top surface of the target object, i.e., the object does not need to be fully covered.
out of three objects (depending the target specified in the language
For every Franka- Tabketoo task, we evaluate each method with 10 12 in-distribution trials and 5-6 OOD generalization trials. The in-distribution and OOD test conditions are depicted in Fig. 9 (second column).
We describe the OOD test conditions for each of the six tasks below:
27

---

1. Put Carrot in Bowl (OOD): An eggplant (unseen object) replaces the carrot.
2. Pour Corn into Pot (OOD); An unseen brown tablecloth covers the tabletop.
3. Flip Pot Upright (OOD): An unseen white tablecloth covers the tabletop
4. Move <object> onto Plate (OOD): A set of three unseen objects are placed on the table.
5. Knock <object> Over (OOD): Two unseen distractor objects (red plastic cup and brown box) are positioned behind the set of three seen objects.
6. Cover <object> with Towel (OOD): The three objects on the table are placed upside-down and at unseen positions.
Finally, in the BRonD environment, we experiment with one task and variants of it: Wipe Table (see Fig. 10). In this task, the robot's goal is to grab the brush and sweep all three small brown objects into the dustpan. We collect 70 demonstrations for this task for the training dataset, varying the positions of all the objects.
Wipe Table (In-Distributicn)
Wipe Table w/ Distractors
Figure 10: Franka-DROID fine-tuning task. The "Wipe Table" task shown here the final task used in the data-efficient adaptation experiments in Section 4.2. The left image shows the initial conditions for an in-distribution trial. The right image shows out-of-distribttion trial in which unseen distractor objects are present on the table. To fully complete the task, the robot must grab the brush and sweep all three objects into the dustpan.
At test time, we evaluate on in-distribution conditions matching the training data (Fig. 10, left), as well as out-of-distribution (OOD) conditions in which distractor objects are also present in the scene on the table (Fig. 10, right). Since there are various possible outcomes for each trial, we define scoring rubric as follows: The maximum score for each trial 2 points. The policy receives the full 2 points if the robot sweeps all three objects into the dustpan. It receives point for successfully sweeping one or two objects into the dustpan. Otherwise, it receives points. We evaluate each policy with 18 in-distribution trials and OOD trials, so each policy receives an aggregate score out of 60 points.
B.3.2 Detailed Tabletp and Franka-DROID Evaluation Results
Full evaluation results for both Tralee-Tabletop and Franka-DROI) evaluations are shown in Table 7. We evaluate the methods discussed in Section 4.2. We find that Diffusion Policy demonstrates strong performance on the single-instruction Franka- Tabletop tasks ("Put Carrot in Bowl", "Pour Corn in Pot", and "Flip Pot Upright"), outperforming other methods. However, OpenVLA (Octo Mix) and Octo achieve higher performance the more diverse multi-instruction tasks ("Move <object> onto Plate", "Knock <object> Over". and "Cover <object> with Towel"). In the Franka-DROID environment, Open VLA (Octo Mix) obtains strong results. Overall, we find that Open (Octo Mix) achieves the highest average performance across both tasks.
fine-tuning experiment results summarized in Table 1. In these experiments, we use a representative subset of two Franka- Tabletop tasks, with both in-distribution and OOD variants: one narrow single-instruction task ("Put Carrot in Bowl') and one diverse multi-instruction task ("Move <object> onto Plate"). We use the same number of training demonstrations used in Section 4.2 (50 and 150, respectively). which is delineated in Appendix B.3.I.
Additionally, in Table 8, we show the detailed version of the parameter-
28

---

Table 7: Detailed data-efficient adaptation experiment results. We report the performance of Diffusion Policy trained from scratch on new robot tasks, as well as generalist policies fine-tuned on the same data. Each policy is tested against both in-distribution and out-of-distribution (OOD) generalization conditions. We find that no single policy performs best on all tasks: Diffusion Policy achieves high success rates on ingle-instruction tasks, while OLe (Octo Mix) and Octo performs well diverse multi-instruction tasks. In terms of aggregate performance, however, OL (Octo Mix) obtains the highest average success rate across both environments.
tiralss Difiusion Policy
Difiniinn Pulicy
(mencheal)
Uc00
OpeNVIAA
(ecrsich)
OpenVI.A (Orto Mis)
[cam)
Tibletp (SHa)
"Put Cianot Bool" disuPattio)
31.0
100.0
0.0
10,0
"Phe
W0.0
0.0
"Flip Upeigh" (ineiittiuiiio)
100.0
$5.0
90.0
"Hipp Upright (OOD)
500
200
0.0
60.0
40.0
"Mowe cobyecto onto Plate" tindieenbutioo))
12
25.0
250
41.7
50.0
79.2
"Mose oshyects onto Plate" 100D)
833
1533
8.3
58.3
41.7
"Kaock <abject- Over" (1 distabuttio))
12
53.5
25.0
53.3
62.5
$7.5
"Kookk <otject> Over" (OOD)
16.7
16.7
33.3
25.0
$3.3
"Covrr sobyect> with Towel (n distrbutionn
12
16.7
208
91.7
20,8
02.5
"Cover colyec with Tooal (OOI)
16.7
35.3
91.7
0.0
41.7
Average
18.514.95
43.424.74
4344:4.8%
38.914.5%
68.234.25
Frank-DROID (15H))
"Wipe Table" (indistriatioo)
50.0%
27.8%
30.69%
58.6%
"Wipe Tabk" Distractos (0DD)
12.9%
25.0%
16.7%
20,8%
54.2%
Avergg
26797.58
76747.58
Table 8: Detailed parameter-efficient fine-tuning experiment results. Here we present the detailed task performance results summarized in Table 1.
Puisls
FT
laycr only
Frozen vision
Sandwich
LaRA, mm32
LoRA, c-64
Pranka Tablecop (5832)
"Put Canoc diszriunion)
90,0
Per Canoe (OOD)
conjects (i-diiitnbutio))
corgort Plat" (OOI)
41.7
Avenge
69.727.2%
19
47,0125
62.197
78.2775%
68217.8%
C Infrastructure for Training and Inference
The final VLE model is trained on a cluster of 64 A100 GPUs for 14 days, total of 21,500 A100-hours, using a batch size of 2048. During inference, OpenVLA requires 15GB of GPU memory when loaded in ffla1116 precision, i.e., without quantization, and runs at approximately 6Hz on one NVIDIA RTX 4090 GPU (without compilation, speculative decoding, or other inference speed-up tricks). We can further reduce the memory footprint of OpenVLA during inference via quantization, and we demonstrate minimal performance regression with quantized VLA execution in Section 4.3. We report inference speed various consumer- and server-grade GPUs in Fig. 11, For convenience, we implement remote VLA inference server to allow realtime remote streaming of action predictions to the robot - removing the need for the robot to have a powerful GPU. We release this remote inference solution as part of our open-source code release (Appendix D).
D
The Open VLA Codebase
Along with our model, we will release the Open VLA codebase, a modular PyTorch codebase for training VLA models (see https:://penr//
github..... b io). It scales from fine-tuning VLAs on individual GPUs to training billion-parameter VLAs on multi-node GPU clusters, and supports modern techniques for large transformer model training such as automatic mixed precision (AMP, PyTorch [109]), FlashAttention [110] and fully sharded data parallelism (FSDP, Zhao al. [82]). Out of the box, the Open codebase has full support for training on the Open X-Embodiment dataset, integrates with HuggingFace's 's [21) AutoModel class, supports LoRA fine-tuning [25] and quantized model inference [26, 83J. Everything will be released under a permissive MIT license.
E RT-2-X vs. Open VLA in Bridge V2 Evaluations
In this section, we provide additional details on RT-2-X vs. Open comparisons in BridgeData V2 evaluations discussed in Section 4.1. As discussed previously, Open is pretrained on a larger subset of OpenX data than RT-2-X and uses a fused SigL.P-DDnoV2 vision backbone rather than a single visual encoder. However, in addition these factors, we believe that Open 's significant improvement upon RT-2-X specifically in BridgeData V2 evaluations (as shown in Fig. 3) also stems from more careful preprocessing of the Bridge dataset.
29

---

bfloat16
8 bit
Actions sec
4 bit
N/A
1080Ti
2080TI
A50D0
A100
RTX 1090
H100
Figure 11: OLe inference speed for various GPUs. Both 6 and int4 quantization achieve high throughput, especially on GPUs with Ada Lovelace architecture (RTX 4090, H100). Further speed-ups are possible with modern LLM inference frameworks like TensorRT-LLM [108].
Model sharded across two
GPUs to
During the development of the Open model, discovered that the original version of the BridgeData V2 dataset contained many transitions with all-zero (no-op) actions. For instance, in every demonstration, an all-zero action was recorded as the ground-truth action in the first timestep. Consequently, training a highly expressive VLA model on the original dataset without any data preprocessing led to a policy that frequently predicted all-zero actions and froze during evaluations. Therefore, simply filtered
the first transition every demonstration when training the OpenVLA model, and this was sufficient for mitigating the freezing behavior in most cases.
However, the RT-2-X model was trained without such data preprocessing, so it often suffers the aforementioned freezing behavior if deployed out of the box without modifying the model querying procedure which severely deteriorates rollout performance. Since this is proprictary model that is infeasible for to re- trairain (e.g., with preprocessed version of the BridgeData V2 dataset), mitigated this issue by simply querying the second second-most-Nee most-likely action from the model, since the first-mostliike likell action was often all zeros while the second-move likely action was not. (Note that this is the same workaround that was applied by the developers of the RT-2-X model for BridgeData V2 evaluations reported in the Open X-Embodiment experiments [1].) This workaround led to much stronger RT-2-X performance on BridgeData V2 evaluations though we believe that it is still suboptimal compared to re-training the model on the preprocessed version of the dataset.
We also tried to dynamically query RT-2-X, i.e., by first sampling the first-most-likely action and then sampling the second-most-likes action the first one was all zeros. However, we empirically found that dynamic querying led to worse performance than simply querying the second-most likely action all times. We hypothesize that this is due change in the robot's dynamics that arises from dynamic querying: pausing in the middle of trajectory re-query the model leads to slight interruptions in the robot's movement due to non-neglible latency in the querying pipeline, and this leads to subtle performance degradation. Therefore, we report the performance of RT-2-X when always querying the second-most-likely action, as done in the Open X-Embodiment project [1].
F Additional Experiments and Ablations
In this section, we conduct several additional experiments to analyze the effects of individual components of the OpenVLA model architecture and training scheme, as well as provide quantitative evidence for claims made in earlier sections of this work. We aim to answer the following questions:
1. How important is OpenX training and how does it impact Open VLA's performance (Ap- pendix F.1)?
30

---

2. What effect does using a fused SigLIP-Dino'' vision encoder have OpenVLA's perfor- mance, compared to using SigL.PPonlly vision encoder (Appendix F.2)?
3. Is it better to fine-tune freeze the vision encoder in OpenVL. (Appendix F.3)?
4. How do the quantized inference results discussed in Section 4.3 change when policy perfor- mance disentangled from model inference speed (Appendix F.4)?
We discuss the experimental setup and results addressing each of the above questions sequentially in the following sections.
F.1 OpenX Training Data Ablation Experiments
As discussed in Section 3.3, OLe is trained on a large dataset of robot embodiments, scenes, and tasks from the Open X-Embodiment dataset [1] (OpenX). In this section, we ablate the OpenX mixture and train a VLA policy solely on one robot dataset, to assess the impact of OpenX training on policy performance. Note that we have already observed the negative effect of ablating OpenX training in the fine-tuning regime, as discussed in Section 4.2 (see OpenVLA (Scratch)), but we discuss additional experiments on another robot embodiment in this section to provide more supporting evidence.
Experimental setup and tasks. compare the original OpenVLA model with Open VLA- -Bridge, which produced by taking the same pretrained VLM as OpenVLA (Prismatic VLM [44]) and fine-tuning solely BridgeData V2 [6] rather than the entire OpenX training mixture diseussed in Appendix A. We evaluate Open VLA and Open DgeeVVLA-Brdde
a subset of representative tasks from the BridgeData V2 WidowX robot evaluation suite discussed in Appendix B.1.1. The tasks are
listed in Table 9.
Results. Results for the OpenX training mixture ablation are shown in Table 9. By comparing OpenVLA with Open VLA-Brre,,,,,,,, we see that performance drops drastically (reduction of 30 percent in absolute success rate), which demonstrates the importance of OpenX pretraining on final policy performance. Although the language grounding performance not impacted, we observe performance reduction across all generalization categories. This result suggests that the large diversity of scenes, objects, and tasks in the OpenX training mixture is essential for unlocking improved generalization capabilities in the VLA model.
Table BridgeData V2 WidowX ablation experiment results. We evaluate various methods subset of representative tasks to assess the importance different components and training scheme. VLA-Bride is a version of Open VLA without OpenX training (it is trained only on BridgeData V2), and Open VppenVLA-Be LAA-rrigg-SSigL.. additionally ablates the fused vision backbone by removing the Dino V2 encoder (its vision backbone only consists of the SigLIP encoder). We observe that both OpenX training and the fused vision encoder improve policy performance, though former much greater effect than the latter.
Open model architecture
OpenVLA Successes
OpenVLA--Biige OpenVL.A-Brigge SigI.IP Suuceessss
Category
Trials
Successess
Visual gen
Put Eggplant into Pot (Easy Version)
10
10
8
8
Visual gen
Put Eggplant
2
gen
Put Cup from Counter
Motion gen
Lift Eggplont
10
7.5
5.5
6.5
Physical gen
Put Carrot on Plate
4
Physical gen
Battery
Semantic gen
Take Purple Grapes out of Pot
3
Language grounding
Put (Egyplant, Red Bottle) into Pot
10
7.5
8
5.6%
5.5%
F22 Dual vs. Single Vision Encoder Experiments
The OpenVLA model architecture consists of a fused vision backbone that combines the SigLIP [9] and DinoV2 [33] encoders. In this section, we ablate the D2noV2 component to assess the importance of using dual vision encoder.
Experimental setup and tasks. We instantiate a model, Open VLA- Bridegiig--- which is a version of VLA that is trained only on BridgeData V2 and consists of only the SigLIP encoder as the vision backbone. We compare this model with the Dee--LA-Bride model discussed in the previous section (Appendix F.I), which shares the same model architecture the original OLe
31

---

model and is only trained on Bridge robot data. Therefore, the only difference between Open Bridge-SigL.IP and Open BeenVLA--Brdde that the former omits the DinoV2 encoder in the vision backbone. We evaluate these models on the same subset of 8 Bridge tasks described in the previous section.
Results. Results for the dual vision encoder ablation shown in Table 9. The drop in performance from Open VLA-Bridge to encoder in the vision backbone improves policy performance. However, the 5 percent reduction in performance here is not as significant as the 30 percent drop in performance observed from ablating OpenX training. The low-level spatial features represented in D2 appear to aid generalization in only some cases.
implies that additionally including the Di
F.3 Fine-Tuned vs. Frozen Vision Encoder Experiments
As discussed in Section 3.4, prior work VLMs observed higher performance from freezing the vision encoder than fine-tuning its parameters [44]. However, when training OpenVLA, we fine-tuned all 7B parameters in the model, including the SigLIP-DinoV2 vision backbone, as we discovered early on during development that fine-tuning the vision encoder led to higher-performing VLAs - a finding which held across various pretrained VL.Ms and model architectures. We discuss details of such findings below.
Experimental setup and tasks. In this section, we report the performance of two VLA policies produced by fine-tuning two different pretrained models from the Prismatic VLMs [44] repository on BridgeData V2. The two pretrained models are named SigLIP ViT-SO 224px and LLaVa v1.5 7B (Reproduction); see Karamcheti et al. [44] for details on their architectures and training mixtures. We evaluate both policies on various Bridge tasks shown Table 10. Note that the evaluation configurations here differ from previously discussed Bridge evaluations, the results are not directly comparable results in other similar experiments.
Results. Results for the fine-tuned vs. frozen vision encoder experiments are shown in Table 10. We find that for both VLAs tested, fine-tuning the vision encoder leads to significantly higher success rates across various tasks. Qualitatively, in some cases, deploying the frozen vision encoder policies leads to unstable robot behaviors that are clearly suboptimal. Consequently, we decided early on during development not conduct further experimentation with frozen vision encoders.
Table 10: Fine-tuned vs. frozen vision encoder experiment results. We evaluate the performance of line-tuning vs. freezing the vision encoder ("Frozen Vision") in VIAA policies built top of two different pretrained VL.Ms from the Prismatic VL.Ms [44] repository. BridgeData V2 WidowX tasks shown here are performed in the same sink environment used for other Bridge experiments in this work (however, the initial environment configurations here differ, as these evaluations were conducted at earlier stage in the project). We find that fine-tuning the vision encoder is crucial to obtain good policy performance. Certain frozen vision encoder evaluations were discontinued due to very poor (near-zero) performance and unstable robot behaviors. Among the evaluations where both frozen vision and fine-tuned approaches are tested, fine-tuning the vision encoder lcads to 80.0% average success versus 46.7% average success from leaving it frozen.
SigLIP VIT-SO 224px
LLaVa v1.5 7B (Reproduction)
Frozen Vision
Fine- Tuned Successes
Frozen Vision Successes
Fine- Tuned Successes
Task
Trials
# Successes
Put Eggplant into Pol
10
7
10
5
9
Put Corn on Plate
10 Mean Success Rate
10
9
0
9
85
95
25
90
Put Eggplant, Red Bottle } into Pot
2
4
3
Put Blue Cup. Pink Cup ] on Plate
0
0
0
Lift Cheese, Red Chili Pepper }
3
Put Strawberry, Lime into Pot
1
0
3
Move { Sushi, Grapes
3
4
Mean Success Rate
30
55
55
F.4 Additional Quantized Inference Experiments: Disentangling Policy Performance and Model Inference Speed
In Section 4.3, we evaluated VLA with different levels of precision at inference time: half precision (bfloat16), 8-bit quantization, and 4-bit quantization. 8-bit quantization led to lower
32

---

BridgeData V2 performance relative to the other two approaches, and we hypothesized that the reduction in performance was caused by lower model inference speed from the operations used in 8-bit quantization. In this section, we conduct experiments to assess the veracity of this claim.
Specifically, we evaluate OpenVLA again with the three different levels of precision listed above, but now with blocking control. In other words, each action is fully executed on the robot before the next one is predicted by the policy and executed by the controller. This scheme controls system dynamics across methods with varying amounts of latency and thus allows us test the quality of a policy's s action predictions, independent of its prediction speed. Effectively, the precision levels that have higher throughput - bffoat 6 and 4-bit quantization - are forced to run slower to match the dynamics observed when deploying Open VLA with 8-bit precision. Therefore, we expect Open VLA's performance with 8-bit precision to match the performance of b and 4-bit precision under blocking control.
Experimental setup and tasks. We report the performance of Open with blocking control and quantized inference the same subset of 8 BridgeData V2 tasks used in Appendix F.1 and Appendix F.2.
Results. Quantized inference experiment results with blocking control are shown in Table 11. Unlike in Table 2, where 8-bit quantization led to the worst rollout performance due to low inference speed, here we observe that 8-bit quantization performs comparably to b precision and 4-bit quantization given that we evaluate with blocking control remove the influence varying inference speeds on task performance. This confirms hypothesis about the effect of inference speed on 8-bit quantization performance in previous experiments (when using non-blocking control). We also see no substantial performance degradation when using the lowest precision, 4-bit, as also observed in Section 4.3.
Table 11: Quantized inference experiment results with blocking control. We report the rate and standard error OpenVL.A on various BridgeData V2 WidowX tasks with bfloat16 precision (the default approach), 8-bit quantization (int8), and 4-bit quantization (int4) at inference time. All average success rates have overlapping error bars, which suggests that all methods perform comparably.
bflout16 Successes
int8 Successes
int4 Successes
Category
Task
Trials
Visual gen
Put Eggplant Pot (Easy Version)
10
10
10
Visual gen
Put Eggplant into Pot
10
10
10
Visual gen
Put Cup from Counter into Sink
Motion gen
Lin Figplaant
7.5
Physical gen
Put Carrot on Plate
10
10
10
10
Physical gen
Rattery
Semantic gen
Take Purple Grapes out of Pot
10
2
2
Language grounding
Put (Eggplant, Red Bottl] into Pot
10
9.5
8.5
Mean Success Rate
70.0 +:
5.1%
74.4
4.9%
68.8
5.2%
G LIBERO Simulation Experiments
Our previous discussions in Section 4.2 and Section 4.3 focused on adapting Ope to novel real-world robot setups and tasks. This section explores adapting Open VLA to simulated robot setups and tasks, specifically utilizing the LIBERO benchmark [111]. J. Our experimentation simulation offers two key advantages:
Demonstration of versatility: We show that Open despite having been pretrained exclusively on real-world robot data, can effectively adapt simulated domains, overcoming potential disparities between real-world and simulated environments and dynamics.
2. Enhanced accessibility and reprodueibility: Integration of OpenVLA into a publicly available simulation platform makes our model more accessible to other researchers, especially those who may not have access to robotic hardware. Additionally, simulated experiments are more casily reproduced than their real-world counterparts.
We discuss the experimental setup in Appendix G.1 and the results in Appendix G.2. We release the materials required to reproduce the experiments along with the Open codebase.
33

---

G.1 LIBERO Simulation Experimental Setup
Simulation setup and tasks. The LIBERO benchmark [111] consists of four task suites designed for studying lifelong learning in robotic manipulation, and the original paper therefore investigates both forward and backward transfer to a variety of tasks. In our experiments, we focus solely on supervised fine-tuning on the target task suite, measuring the performance of various policies trained via behavioral cloning successful demonstrations of the tasks.
We perform experiments with the following four task suites, which each contain 10 tasks with 50 human-teleoperatee demonstrations each:
LIBERO-Spatial consists of the same set of objects but different layouts, and tests the model's understanding spatial relationships.
LIBERO-Object consists of the same scene layouts but different objects, and tests the model's understanding of object types.
LIBERO-Goal consists of the same objects and layouts but different task goals, and tests the model's s knowledge of different task-oriented behaviors.
LIBERO-Long (also called LIBERO-10) consists of long- onnghhoizoon tasks with diverse objects, layouts, and tasks.
We make the following modifications to each of the training datasets above:
1. To accommodate methods requiring higher-resolution images (such as 256 x 256px or 224 x 224px), we regenerate all demonstrations at an increased resolution of 256 x 256px. Originally, the dataset provided by the benchmark consists of 128 x 128px images. We find that simply upscaling these images to 256 x 256px results in poor image quality, Therefore, we choose to begin with higher-resolution images, which can downscaled as necessary, ensuring higher image quality across various resolution requirements. These higher- nigher-resolution images with the actions stored in the provided human-collected demonstrations and saving the images rendered by the simulator.
obtained by stepping through the simulation environments
2.
We filter out all "no-op" actions from the dataset, i.e., actions that have near-zero magnitude in the translation and rotation components and do not change the state of the robot's s gripper. We find that this simple data cleaning step is crucial for highly expressive single-step policies such as Open VLA, which otherwise learn to imitate these no-op actions and consequently freeze indefinitely at certain states during evaluation.
3. We rotate all third-person images at both train and test time by 180 degrees because we observe that the LIBERO environments return images that are upside down on our hardware.
4. Since train policies via imitation learning, which expects demonstrations be successful, we replay all demonstrations in the corresponding simulation environments and filter out the demonstrations that fail to complete the task (as determined by the environments' success criteria). As a result, we remove 68 of 500 LIBERO-Spatin demonstrations, 46 of 500 LIBERO-Object demonstrations, 72 500 LIBERO-Goal demonstrations, and 121 500 LIBERO-Long demonstratinos.
5. For all methods in our comparisons, we only utilize the static third-person camera images; we do not use the wrist camera images that are additionally provided in the original datasets. This is for sake of having fair comparisons, as OpenVLA' visual inputs only consist of third-person camera images.
Comparisons. The methods that we compare include Diffusion Policy? [3] trained from scratch, Octo [5] fine-tuned on the target dataset, and VLe fine-tuned on the target dataset via LoRA = 32) described in Section 4.3, Each policy trained independently on each of the task suites
'We use the implementation of Diffusion Policy that is described in the DROID dataset paper [11], which conditions action generation DistilBERT [112] language embeddings of the task label.
34

---

above (rather than training a single policy on all four suites combined). All policies are trained with the same set of demonstrations, so all methods benefit from the data cleaning steps described above. Evaluation details. To ensure lower variance in the experimental results, all methods are evaluated across 500 trials for each task suite, and the reported performance the average success rate over three random seeds (resulting in 1500 total trials per statistic). Although we modify the training datasets, described earlier, do not change the test environments but rather use the same initial environment configurations provided by the original LIBERO benchmark.
G.2 LIBERO Simulation Experimental Results
We present the LIBERO experimental results in Table 12. Importantly, we observe that Open VLA can be effectively adapted to tasks in the LIBERO simulation environments, as it obtains highest average success rate and rank among the tested methods. However, we find that the overall margin between OpenVL.A and the other methods are tighter here than in the real-world fine-tuning experiments discussed in Section 4.2. We attribute this to the fact that OpenVLA was pretrained with purely real-world robot data and no simulation data, which suggests that fine-tuning the model simulated robot tasks may not be as effective as fine-tuning it on real-world tasks due to the domain gap between simulated and real-world environments and dynamics. We see evidence for this notion in the results obtained by Octo another policy pretrained on large amounts of real-world robot data which also only achieves small boost in overall performance relative to a simple, strong baseline such as Diffusion Policy trained from scratch, We expect increased gains in performance for the pretrained and fine-tuned methods if simulation data is added the pretraining data mixture.
Table LIBERO simulation benchmark results. report success rate (SR) standard error each method for the four task suites in the LIBERO benchmark, averaged over three random seeds with 500 trials each. In addition, strongest method in the suite and rank of indicates the weakest method. (The average ranking is important note since it informs which method may most suitable use as default for a variety of tasks; it more informative than the average success rate, which not normalized by individual task suite difficulty.) Overall, we find that fine-tuned VLA VLA achieves highest average success rate and rank, followed by fine-tuned Octo and then Diffusion Policy trained from scratch.
show the ranking of each method within each task suite, where a rank of I indicates the
LIRFRN-Spuiial Kink
LIRERO-Gbjest SR
LIBERD-Gol SRC0
LIBERO- Leng
SR
(1)
Rink
SR
Rink (
Diffision cuned (ours)
1.1%
0.7%
1.28
50.5 1.3%
22.4 0.7%
2.5
OpenVLA
1.0%
35


Reviews:
[{"review_id": "tF0bolqpBU", "reviewer": "Reviewer_GEDn", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes an open-sourced Vision Language Action model which generates discrete action tokens auto regressively. The model utilizes only robotic data and surpass the SOTA end2end models like RT-2-X. All the code and data are released.", "summary_of_recommendation": "This paper proposes an open-sourced Vision Language Action model which generates discrete action tokens auto regressively. The model utilizes only robotic data and surpass the SOTA end2end models like RT-2-X. All the code and data are released.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "See weakness.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fHy3DJzUGm", "reviewer": "Reviewer_t3E9", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors proposed a novel approach called OpenVLA an open-source vision language action (VLA) model that uses large scale robotics dataset to finetune a pre-trained VML model for robotics tasks.", "summary_of_recommendation": "While it would be great to adress current weak points of the paper, overall OpenVLA would be useful in empovering many robotics researcher to start from robotics foundational model and build on top of it.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "# Minor comments/questions: \n\n- In table 2 (Bridge Success) it would be great to split the sources of the decreased performance, for example by testing on some of the task in the simulation and see to what extent the problem in int8 is not appearing there. Also, it would be great to do even more extreme quantization to show the limits of such approach. Besides, it would be useful to include original performance for not quantized model in the same table \n\n> \"Neither SigLIP / DinoV2, nor Llama 2 release details about their training data, which likely consists of trillions of tokens of Internet-sourced image-text and text-only data respectively.\"\n- DINOv2 is not using image-text pair but only images so the claim could be more careful about this.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GBhXWGu4Gh", "reviewer": "Reviewer_LnaQ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces OpenVLA, a 7B-parameter open-source Vision-Language-Action (VLA) model, trained on 970k robot episodes. The model aims to obtain generalizable policies of multiple robots and tasks for visuomotor control.", "summary_of_recommendation": "I highly evaluate this paper.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "See above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths:
+ OpenVLA achieves compelling performance over RT-2-X despite being smaller.
+ Code, data, and checkpoints are open-sourced.
+ The paper is written clearly.
+ The experimental setup is thorough.
+ A parameter-efficient fine-tuning process is provided.

Weaknesses:
- No simulation results for fair comparison to existing works.
- Experiments could include: more varied robot setups (different manipulators?), comparison between frozen and fine-tuned vision encoder, ablation with only SigLIP features.

Post rebuttal:   
The authors have addressed the concerns. In general, the paper received very positive feedback.
Decision: Accept