Vision-Language-Action Instruction Tuning: From Understanding to Manipulation

## Abstract

Abstract To operate effectively in the real world, robots must integrate multimodal reasoning with precise action generation. However, existing vision-language-action (VLA) models often sacrifice one for the other, narrow their abilities to task-specific manipulation data, and suffer catastrophic forgetting of pre-trained vision-language capabilities. To bridge this gap, we introduce InstructVLA , an end-to-end VLA model that preserves the flexible reasoning of large vision-language models (VLMs) while delivering leading manipulation performance. InstructVLA introduces a novel training paradigm, Vision-Language-Action Instruction Tuning (VLA-IT) , which employs multimodal training with mixture-of-experts adaptation to jointly optimize textual reasoning and action generation on both standard VLM corpora and a curated 650K-sample VLA-IT dataset. On in-domain SimplerEnv tasks, InstructVLA achieves 30.5% improvement over SpatialVLA. To evaluate generalization, we introduce SimplerEnv-Instruct, an 80-task benchmark requiring closed-loop control and high-level instruction understanding, where it outperforms a fine-tuned OpenVLA by 92% and an action expert aided by GPT-4o by 29%. Additionally, InstructVLA surpasses baseline VLMs on multimodal tasks and exhibits inference-time scaling by leveraging textual reasoning to boost manipulation performance in both simulated and real-world settings. These results demonstrate InstructVLA’s potential for bridging intuitive and steerable human-robot interaction with efficient policy learning. Project website.

## 1 Introduction

Large-scale pre-training has produced versatile foundation models in computer vision (CV) [oquab2023dinov2, siglip, clip, sam, sam2, groundedsam] and natural language processing (NLP) [bert, T5, gpt2, qwen, touvron2023llama]. Inspired by this success, recent Vision-Language-Action (VLA) models [openvla, RT-2, ecot, qu2025spatialvla, robovlms, cogact, li2025cronusvla] initialize from large vision-language models (VLMs) [llava, llava_next, karamcheti2024prismatic, beyer2024paligemma, alayrac2022flamingo, peng2023kosmos] and train on large-scale embodied data  [open_x_embodiment, khazatsky2024droid] to enhance generalization in robotic manipulation. While these VLAs demonstrate strong performance in robotic manipulation tasks, they are susceptible to *catastrophic forgetting*[mcclelland1995there, french1999catastrophic], which gradually diminishes the rich multimodal reasoning capabilities inherited from their web-scale pre-trained vision-language backbones. Two challenges contribute to this issue: (1) existing large-scale real-world robotic datasets mostly lack diverse human instructions across varied task scenarios, restricting training to simple, templated commands (e.g., “open the drawer”); and (2) training solely on domain-specific robotic data accelerates the erosion of general multimodal understanding, limiting the model’s ability to handle diverse inputs, user feedback, and free-form instructions[hirobot].

To mitigate catastrophic forgetting when finetuning VLMs into VLAs, prior work primarily adopts two strategies. The first aims to jointly preserve general multimodal capabilities while learning diverse manipulation skills. Models such as ChatVLA [zhou2025chatvla] and Magma [magma] follow this approach by jointly training on vision-language and manipulation data. However, this approach often neglects complex embodied reasoning. The second strategy focuses on tightly integrating embodied reasoning into manipulation datasets to transfer VLM capabilities. Methods such as ECoT [ecot] and Emma-X [sun2024emma] embed chain-of-thought (CoT) reasoning into manipulation datasets. However, these methods are built on action-pretrained architectures [openvla] and structured reasoning patterns (plan, subtask, etc.), which inherently constrain general multimodal capabilities. The extent to which the VLM capabilities translate into action generation in embodied contexts remains largely unexplored.

Figure: Figure 1: Method overview. InstructVLA integrates robust multimodal understanding with precise instruction-driven robotic control, leveraging the world knowledge of VLMs. The core training strategy, vision-language-action instruction tuning, enhances manipulation by enabling the model to perform vision language reasoning before generating actions.
Refer to caption: x1.png

To tackle this issue, we propose InstructVLA, a generalist VLA model that extends pretrained VLM for accurate action generation while retaining strong multimodal understanding, as illustrated in [Figure 1](https://arxiv.org/html/2507.17520v1#S1.F1). InstructVLA adopts a training paradigm specifically designed to bridge vision-language understanding with action generation by treating language-steered action generation as an integral component of instruction following.
To this end, we curate the Vision-Language-Action Instruction Tuning (VLA-IT) dataset, consisting of 650K human-robot interactions annotated with diverse instructions, scene captions, and question-answer pairs grounded in high-quality manipulation tasks [Bridge_data, RT-1]. The training process follows a two-stage paradigm:
(1) Action Pretraining, which trains a VLM-driven action expert using latent action representations distilled from language-based motion descriptions; and
(2) Vision-Language-Action Instruction Tuning, which unifies language and latent action generation through a trainable mixture-of-experts(MoE) adaptation framework. This framework is jointly trained on multimodal datasets [llava, allava, bunny], manipulation datasets, and the curated VLA-IT corpus, enabling the automatic switch between textual reasoning and action generation, thereby effectively leveraging vision-language understanding for action execution.

To validate the generalist performance of InstructVLA, we introduce the SimplerEnv-Instruct benchmark, a manually designed evaluation suite featuring 80 zero-shot manipulation tasks. It encompasses both closed-loop manipulation tasks and high-level instruction reasoning, involving either situated understanding or decomposition into actionable subtasks. With its thinking ability during manipulation, InstructVLA outperforms the fine-tuned OpenVLA baseline by 92% and achieves a 29% improvement over an action expert model assisted by GPT-4o on SimplerEnv-Instruct, demonstrating its effectiveness in instruction following and task decomposition. Furthermore, InstructVLA surpasses similarly sized VLMs in multimodal performance and shows a 27% improvement over Magma in closed-loop manipulation [simpleenv]. Our contributions can be summarized as follows:

- •
We propose InstructVLA, a VLA architecture and training pipeline that emphasizes the importance of language capability in VLAs by efficiently preserving pretrained vision-language knowledge from VLMs while integrating manipulation as a component of instruction following.
- •
We design a practical data and evaluation pipeline for vision-language-action instruction following, supported by 650K tailored VLA-IT annotations and a manually curated benchmark suite, enabling evaluation of VLAs’ instruction generalization capabilities.
- •
InstructVLA achieves leading performance across robotic manipulation tasks, multimodal benchmarks, and real-world deployments, enabling intuitive and controllable manipulation.

## 2 Related Works

Policy learning at scale. Following the success of CV [oquab2023dinov2, siglip]and NLP [touvron2023llama, gpt2], recent research [hpt, RT-1, RT-2, zheng2025universalactionsenhancedembodied, wang2024poco] shows that robot policies improve when trained in large heterogeneous datasets. RT1 [RT-1] and RT-2 [RT-2], trained in large-scale real-world demonstrations, achieve strong in-domain accuracy and zero-shot transfer. Works such as Octo [octo] and RT-X [open_x_embodiment] extend this approach by aggregating the largest open-source manipulation datasets [open_x_embodiment]. Some methods, such as LAPA [lapa], Seer [tian2024predictive], and Moto [chen2024moto], use video generation and inverse dynamics to learn scalable motor representations. In the VLA domain, models are typically initialized from pretrained vision-language models [openvla, qu2025spatialvla, RT-2] leveraging prior visual-linguistic alignment instead of learning from scratch. Further, methods such as RT-Trajectory [rt-trajectory] and GraspVLA [deng2025graspvla] jointly train intermediate manipulation representations such as trajectories or bounding boxes using a combination of real, simulated, and web data to guide action generation and enhance generalization.

Vision-language-action models. Recent foundation models [RT-2, openvla, qu2025spatialvla, pi_0, chen2024moto, bjorck2025gr00t, pertsch2025fast] integrate perception, language, and robot manipulation into a single network, using two main architectures. Autoregressive models treat actions as discrete tokens: RT-2 [RT-2] co-trains a web-scale VLM on robot trajectories, transferring semantic knowledge to manipulation, while OpenVLA [openvla] and SpatialVLA [qu2025spatialvla] follow a similar token-based control approach. FAST tokenization [pertsch2025fast] compresses motion sequences to manage length. In contrast, flow-based VLAs avoid discretization; for example, $\pi_{0}$ [pi_0] and GR00T [bjorck2025gr00t] generate actions through continuous flow matching [flowmatching], while CogACT [cogact] and CronusVLA [li2025cronusvla] use diffusion [DiT]. Hybrid approaches, like RoboDual [robodual], combine generalist action models with specialist action experts. Although flow-based methods [pi_0, bjorck2025gr00t, li2025cronusvla, cogact] often achieve superior performance, they typically neglect the integration of autoregressive text reasoning [RT-2], which is crucial for leveraging the VLM’s semantic capabilities. In contrast, our model unifies autoregressive VLM language generation with the flow-based action generation, demonstrating efficient co-training of language and action at scale.

Robot policies with hierarchical decision making. Training multitasking policies for decision making and planning in complex environments remains a significant challenge. Leveraging the capabilities of pretrained VLMs and LLMs provides a simple yet effective solution. For example, SayCan [saycan] uses a frozen LLM to identify subtasks. RT-H [rth], Steer [smith2024steer], and Hi-robot [hirobot] ground actions through language, training a language-steerable VLA model. RT-Trajectory [rt-trajectory] and RoboGround [huang2025roboground] bridge planning and action generation with intermediate representations. These methods use language, trajectories, or bounding boxes as interfaces between high-level understanding and low-level action generation. Designing a general interface without ambiguity remains difficult; with LCB [lcb] and Helix [helix] inserting learnable latent tokens between high-level reasoning backbones and low-level policy heads, enabling end-to-end finetuning without manual skill libraries. However, these methods do not integrate textual reasoning and action planning in a single model, and designs of dual models [hirobot] often incur additional computational costs. Our approach preserves the VLM’s reasoning capabilities while enabling dynamic switching between reasoning and action execution, resulting in efficient and controllable manipulation.

## 3 InstructVLA

We propose InstructVLA ([Figure 2](https://arxiv.org/html/2507.17520v1#S3.F2)), a unified model for joint language and action generation. [Section 3.1](https://arxiv.org/html/2507.17520v1#S3.SS1) details the architecture with dynamic reasoning and execution switching, along with inference strategies, while [Section 3.2](https://arxiv.org/html/2507.17520v1#S3.SS2) outlines the training paradigm for VLA instruction following.

### 3.1 Architecture

Embodied VLM for textual and latent action generation. We propose a unified framework that enables simultaneous multimodal reasoning and language-steered latent action planning using a single VLM ([Figure 2](https://arxiv.org/html/2507.17520v1#S3.F2) (1) and (2)). The model produces textual outputs to preserve the strong language understanding and multimodal inference capabilities of the pretrained VLM, while subsequently generating latent action representations for downstream manipulation. To support action planning, we introduce $N$ learnable action queries $Q\in\mathbb{R}^{N\times D}$, which attend to the VLM’s hidden states and extract task-relevant latent action $C\in\mathbb{R}^{N\times D}$, where $D$ is the VLM hidden dimension. Our implementation builds on the compact and efficient Eagle2-2B backbone [li2025eagle], with a tailored training strategy described in [Section 3.2](https://arxiv.org/html/2507.17520v1#S3.SS2). The model is supervised with cross-entropy on language output with loss $\mathcal{L}_{LM}$.

Figure: Figure 2: Overview of the InstructVLA. InstructVLA integrates the multimodal reasoning capabilities of a vision-language model with robotic manipulation. Generation consists of three steps: (1) asynchronous auto-regressive reasoning by the VLM, (2) latent action generation, and (3) action decoding. A MoE adaptation enables the VLM to alternate between reasoning and latent action prediction. The flow matching action expert decodes the final actions, conditioned on latent actions.
Refer to caption: x2.png

Mixture of adaptation experts for language-steered latent action. A key challenge is enabling the model to seamlessly alternate between reasoning and manipulation at inference time. To this end, we adopt a Mixture-of-Experts (MoE) design [moe], which allows adaptive reweighting of expert modules based on input context and reasoning mode, thereby integrating multimodal reasoning with language-steered latent action. Specifically, LoRA [lora] modules are employed as experts within the LLM backbone, preserving pretrained capabilities while ensuring efficient inference. A scale head [xlora] predicts gating coefficients $\lambda_{i}$ for each expert by classifying the hidden state, enabling the model to adaptively blend their outputs. The resulting hidden states for $K$ experts are computed as $h=W_{0}x+\sum_{i=0}^{K}B_{i}A_{i}x\cdot\alpha_{i}\cdot\lambda_{i}$, where $W_{0}$ is the original weight, $x$ denotes input, $A_{i}\in\mathbb{R}^{r\times d}$ and $B_{i}\in\mathbb{R}^{d\times r}$ are the LoRA parameters, $\alpha_{i}$ is the LoRA scaling factor.

Flow model as an efficient action expert. To further decouple low-level control from high-level understanding, the action expert is designed to generate actions from image observations conditioned on VLM-derived intentions. It takes image features from DINOv2 [oquab2023dinov2], latent actions from the VLM, noisy action embeddings and optional information such as proprioception, and fuses these with a simple transformer architecture [touvron2023llama] with block-wise causal attention. Specifically, non-causal attention is applied within each input, and causal attention between input types. The DINOv2 vision encoder, further enhanced with feature-wise linear modulation (FiLM) [perez2018film], plays a crucial role in directing actions to spatial and contextual input. The flow matching objective [pi_0] is used to supervise action learning, as detailed in [Section E.2](https://arxiv.org/html/2507.17520v1#A5.SS2).

Inference. InstructVLA integrates language and action generation in a single model. The following techniques are designed for InstructVLA to enhance speed.
(1) Decoding strategies. To mitigate the latency of autoregressive decoding, textual responses are generated via greedy search until the first action query token appears. The remaining action queries are then decoded in parallel within a single forward pass of the VLM.
(2) Language response and latent action caching. We decouple language response from action generation by caching textual outputs across multiple action steps, leveraging their temporal stability. InstructVLA also supports cache latent actions, which reduces the number of VLM forward with minimal performance impact compared with ECoT [ecot] (see [Section A.3](https://arxiv.org/html/2507.17520v1#A1.SS3)).

### 3.2 Two-Stage Training Recipe

The first stage involves efficient pretraining of an action expert aligned with latent action embeddings from the VLM via Action Pretraining, followed by Vision-Language-Action Instruction Tuning to bootstrap the action generation process by reactivating the VLM’s multimodal reasoning abilities.

Stage 1: Action pre-training. InstructVLA is pre-trained using heterogeneous manipulation data [RT-1, Bridge_data]. To distall the knowledge from the VLM for manipulation, the model is trained to predict both actions and rule-based annotated language motion ([Section 4.1](https://arxiv.org/html/2507.17520v1#S4.SS1)), with the latter supervised via cross-entropy loss. Due to the stability of flow matching and the next token prediction, the final loss is the direct sum of both losses as $\mathcal{L}=\mathcal{L}_{LM}+\mathcal{L}_{FM}$. During this stage, only the input and output embedding of the action queries and action LoRA adapter on the LLM backbone are tuned, consisting of 650M parameters. The model trained in this stage is named the “Expert”.

Stage 2: Vision-language-action instruction tuning. We extend the concept of visual instruction tuning [llava] with a simple approach to train InstructVLA. Our observation is that once the action expert is pretrained to follow the latent actions from the VLM, further adapting the LLM backbone enables the model to handle manipulation tasks with more complex instructions and generate appropriate responses. In this stage, the action expert remains frozen, a new language LoRA adapter and scale head of the MoE-adaptation are added. The MoE module is the only trainable parts, comprising 220M parameters. We detail the data pipeline for vision-language-action instruction tuning in [Figure 3](https://arxiv.org/html/2507.17520v1#S4.F3); this data bridges pretrained vision-language capabilities with embodied task scenarios. We further co-train the model using multimodal datasets [bunny, llava, allava] to bootstrap multimodal understanding. The resulting model is referred to as the “Generalist”, reflecting its combined vision-language and manipulation capabilities.

## 4 VLA Dataset and Benchmark

### 4.1 InstructVLA Tuning Dataset

Figure: Figure 3: Vision-language-action instruction tuning data examples. Annotations focus on: (1) improving scene understanding and (2) learning instruction following and planning.
Refer to caption: x3.png

We curate diverse hierarchical language annotations from large-scale manipulation datasets [RT-1, Bridge_data], including language motion [rth] and rule-based labels for pretraining, along with the VLA-IT (Vision-Language-Action Instruction Tuning) dataset for instruction tuning and reasoning transferring.

Language motion pre-training data. Language motion [rth] provides intuitive linguistic descriptions of basic end-effector movements, which can be distilled into latent actions. We compute the relative displacement of the end-effector between the $t$-th and $(t+W)$-th steps, using a window size $W$. The final labels, such as “move right and open gripper,” provide supervision for VLM.

Vision-language-action instruction tuning data. To enable language-steerable VLA models, it is essential to curate diverse instructions, model responses, and reasoning patterns. We categorize our data into four types as illustrated in [Figure 3](https://arxiv.org/html/2507.17520v1#S4.F3). For embodied scene understanding: (1) Scenario captioning provides descriptions of the robot’s environment (2) Question answering targets scene understanding through consistent QA pairs across an episode. Together, they bridge vision-language annotations with embodied scenes. For instruction understanding and latent action planning: (3) Command rewriting introduces instructional diversity through paraphrasing, attribute-based references and varied vocabulary.
(4) Context creation generates implicit user goals or progress cues in multi-step tasks, requiring the robot to infer intent. These annotations support joint VLA reasoning.

We use GPT-4o [GPT-4] to annotate data with three frames from each episode, along with the corresponding instruction. Ground-truth instruction is crucial for annotation accuracy, emphasizing that even state-of-the-art VLMs can make errors in embodied tasks, leading to a performance gap when using GPT-4o as an instruction interpreter for such tasks. Additional details of the dataset analysis and prompt templates are provided in [Appendix C](https://arxiv.org/html/2507.17520v1#A3).

### 4.2 SimplerEnv-Instruct

Figure: Figure 4: Simpler-Instruct. We visualize six representative test cases, with instructions and responses from InstructVLA during evaluation. Top four failure modes of other VLAs are listed.
Refer to caption: x4.png

Building upon the real-to-sim SimplerEnv platform [simpleenv], we propose SimplerEnv-Instruct, a manually designed benchmark for evaluating the instruction-following and reasoning capabilities of VLA models in a zero-shot setting. The benchmark comprises two hierarchical levels: instruction aggregation (50 tasks) and situated reasoning (30 tasks), totaling 1.1K trials, as shown in [Figure 4](https://arxiv.org/html/2507.17520v1#S4.F4).

Instruction aggregation. The instruction aggregation tasks focus on command diversity, including new verbs, multilingual expressions, object references, sentence rephrasing, and novel objects. Situated reasoning. Situated reasoning tasks evaluate the model’s ability to infer intent when instructions are implicit. For example, “I want to clean the table. Pick a suitable tool for me.” requires the model to identify and retrieve the correct object (e.g. a sponge) through reasoning. We further incorporate subtask identification, where each subgoal is conditioned on both the instruction and the environment, capturing the complexity of long-horizon tasks.

The benchmark task design adheres to two principles: (1) tasks should evaluate the transfer of in-domain manipulation skills to novel scenarios, and (2) instructions must be human-interpretable. To assess the first, we filter basic tasks and objects to ensure models exhibit correct intent, and then annotate novel instructions to increase task difficulty. To validate the second, we conduct cross-checks among human annotators to ensure instruction clarity and naturalness.

## 5 Experiment

Benchmarks. (a) Multimodal. We use the automatic evaluation from VLMEvalKit [duan2024vlmevalkit] including MMMU(Val) [mmmu], MMStar [MMstar], MME [mme], OCRBench [ocrbench], HallB(Avg) [guan2024hallusionbench], MMB(Dev En V1.1) [liu2024mmbench], TextVQA [singh2019towards], DoCVQA [mathew2021docvqa], InfoVQA [mathew2022infographicvqa], AI2D [ai2d], ChartQA [masry2022chartqa] and RWQA [rwqa]. These benchmarks collectively evaluate diverse multimodal capabilities, including general visual question answering, document, infographic and chart understanding, OCR reasoning, and hallucination robustness. (b) SimplerEnv [simpleenv] features real-to-sim evaluation on large-scale manipulation datasets [RT-1, Bridge_data] with visual matching and variance aggregation settings to evaluate generalization ability. (c) SimplerEnv-Instruct, detailed in [Section 4.2](https://arxiv.org/html/2507.17520v1#S4.SS2), extends the SimplerEnv with more novel objects, tasks, and instructions, providing a broader testbed to evaluate the instruction generalization of VLAs. We additionally evaluate language-capable baselines as detailed in [Section A.1](https://arxiv.org/html/2507.17520v1#A1.SS1).

Training details.
The VLM is trained with a resolution of $448\times 448$ following [li2025eagle], while the action expert operates at $224\times 224$ as in [openvla], using a fixed learning rate of 5e-5 without warm-up. The action expert employs a 12-layer transformer backbone with a hidden size of 768. Following [pi_0], a $\beta$ distribution is used to enhance accuracy on the noisier time steps. During Stage 2 finetuning, manipulation and multimodal understanding are trained in an interleaved manner. Owing to InstructVLA’s design, multimodal capabilities are retained without requiring tuning of the multimodal-to-manipulation ratio; we use a 1:7 ratio, twice the imbalance used in ECoT (1:3)[ecot]. [Appendix E](https://arxiv.org/html/2507.17520v1#A5) gives more details.

Baselines. We categorize the baselines into three groups: (1) Multimodal VLMs, including LLaVA-OV[llavaov], Bunny[bunny], Eagle2 [li2025eagle], and Qwen2-VL [qwen2vl]; (2) VLA models, including RT-1-X and RT-2-X from OXE [open_x_embodiment], Octo [octo], RoboVLMs [robovlms], SpatialVLA [qu2025spatialvla], TraceVLA [tracevla], and OpenVLA [openvla];
(3) Generalist VLA models, including Magma [yang2025magma], OpenVLA fine-tuned (FT) from its official generalist pretrained model on both robotic and multimodal data, and ECoT(Bridge) [ecot]. For both language and action generation, InstructVLA and other baselines use a temperature of 0 and greedy search without sampling to expedite generation. We re-evaluate Magma using its official checkpoint(^1^11We observe a notable performance gain for Magma when using sampling. Accordingly, we report its official score on SimplerEnv and re-evaluate its performance on SimplerEnv-Instruct under the sampling setting.). For ECoT, we report only its multimodal results due to its real-to-sim domain gap [ecot].

### 5.1 Main Results

**Table 1: Multimodal understanding. #Params is the size of LLM backbone. S. denotes robot state.**
| Methods | #Params | Multi-modal Understanding Benchmarks | VQA Benchmarks |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MMMU${}^{\text{Val}}$ | MM-Vet | MMStar | MME^P | OCRBench | HallB | MMB | TextVQA | DocVQA | InfoVQA | AI2D | ChartQA | RWQA |  |  |
| LLaVA-OV [llavaov] | 8B | 47.9 | 50.6 | 61.9 | 1993.6 | 622 | 31.6 | 80.9 | - | - | - | 82.4 | 80.9 | 69.9 |
| Bunny [bunny] | 8B | 43.4 | 39.1 | 45.4 | 1987.7 | 444 | 37.7 | 72.9 | - | - | - | 69.4 | 30.1 | 60.4 |
| Eagle2 [li2025eagle] | 2B | 43.1 | 53.8 | 56.4 | 1572.1 | 818 | 45.8 | 74.9 | 79.1 | 88.0 | 65.8 | 79.3 | 82.3 | 63.1 |
| Qwen2-VL [qwen2vl] | 2B | 41.1 | 51.5 | 48.0 | 1872.0 | 809 | 41.7 | 74.9 | 74.9 | 88.6 | 61.4 | 74.7 | 73.5 | 62.9 |
| OpenVLA [openvla] | 7B | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| OpenVLA (FT) | 7B | 26.0 | 9.1 | 28.2 | 87.6 | 2.5 | 8.4 | 18.9 | 2.5 | 29.2 | 43.4 | 35.8 | 1.4 | 47.2 |
| ECoT [ecot] | 7B | 16.2 | 0.0 | 19.1 | 0.0 | 0.0 | 3.1 | 0.9 | 0.0 | 2.2 | 0.0 | 0.0 | 0.0 | 29.8 |
| Magma [yang2025magma] | 8B | 38.8 | 34.1 | 41.3 | 1496.5 | 518 | 38.0 | 69.7 | 66.5 | 65.4 | 45.2 | 66.1 | 61.8 | 56.5 |
| Generalist | 2B | 44.8 | 47.5 | 54.9 | 1611.2 | 795 | 47.0 | 76.6 | 75.6 | 84.4 | 63.8 | 78.1 | 79.7 | 64.4 |
| Generalist(S.) | 2B | 44.2 | 51.4 | 55.6 | 1612.6 | 816 | 43.4 | 77.7 | 76.6 | 85.5 | 64.7 | 78.9 | 81.5 | 63.7 |

We present our main results in [Tables 2](https://arxiv.org/html/2507.17520v1#S5.T2) and [2](https://arxiv.org/html/2507.17520v1#S5.T2). In [Table 2](https://arxiv.org/html/2507.17520v1#S5.T2), using the same generalist model InstructVLA (generalist), it not only outperforms the co-trained baseline Magma-8B [magma], but also exceeds its base model Eagle2 [li2025eagle] and Bunny (VLM data corpus) [bunny] across multiple multimodal benchmarks including MMMU, MMB, and RWQA. In [Table 2](https://arxiv.org/html/2507.17520v1#S5.T2), InstructVLA (expert) outperforms expert baselines SpatialVLA by 30.5%, while InstructVLA (generalist) maintains strong performance on SimplerEnv’s atomic instructions and further achieves a 29.5% improvement on the SimplerEnv-Instruct over SOTA baseline (OpenVLA with GPT-4o).

However, we observe that finetuning OpenVLA on multimodal and manipulation datasets does not fully restore its original multimodal capabilities, although it does improve task performance. Its performance can be further enhanced by integrating GPT-4o as an API-based system-2 module to rephrase instructions (OpenVLA (FT&GPT)). However, GPT-4o faces the same challenges in accurate instruction rewriting as noted in [Figure 3](https://arxiv.org/html/2507.17520v1#S4.F3), and fails to outperform InstructVLA (Generalist). Methods such as Magma, which co-train both abilities of the VLM, better preserve multimodal ability, but still fail to match the performance of our approach. ECoT relies solely on textual chain-of-thought reasoning over manipulation datasets and lacks the capability for multimodal question answering. We observe that it consistently generates manipulation-style CoT responses, without demonstrating effective instruction-following ability.

### 5.2 Real-world Experiments

To evaluate InstructVLA in real-world scenarios, we conduct zero-shot experiments on the WidowX-250 Arm and few-shot experiments on the Franka Research 3 robot, as shown in [Figure 5](https://arxiv.org/html/2507.17520v1#S5.F5). The few-shot tasks involve spatially demanding pick-and-place from a rack and cluttered tabletop organization. We use GPT-4o to annotate VLA-IT dataset on the collected few-shot data. InstructVLA is fine-tuned using the proposed training recipe, while OpenVLA is jointly trained on both atomic skill and VLA-IT datasets until the action token accuracy reaches 95% [openvla]. The zero-shot tasks are set in the kitchen environment following Bridge Dataset [Bridge_data].

Each scenario includes both atomic and reasoning instructions. Atomic settings focus on in-domain objects and instructions with an emphasis on spatial generalization to assess baseline VLA capabilities. Both models perform comparably on direct instruction with in-domain objects; InstructVLA achieves a 23.3% success rate improvement over OpenVLA. For reasoning settings such as celebrity recognition, OCR, and tool-use inference, OpenVLA exhibits a substantial performance drop. In contrast, InstructVLA outperforms it by 41.7% in few-shot and 46.7% in zero-shot settings. Detailed experimental setups are provided in [Appendix G](https://arxiv.org/html/2507.17520v1#A7).

Figure: Figure 5: Real-world experiments. “Atomic” refers to atomic instructions, while “Reasoning” denotes situated reasoning. For the Bridge settings, InstructVLA’s responses are presented.
Refer to caption: x5.png

### 5.3 Ablation Studies

We conduct ablation studies guided by two central questions: (1) How can manipulation and multimodal understanding be effectively integrated into a single model through architectural design and training strategies? (2) To what extent does vision-language comprehension enhance manipulation performance in complex scenarios? Through targeted ablations, we examine the impact of key architectural and training decisions on these capabilities.

Figure: Figure 6: Ablation Studies. Ablation studies are grouped into two perspectives: (a-d) Action ability integration: analysis of how design choices in data (language motion), representation (latent action tokens), vision encoders, and finetuning strategies influence manipulation performance. (e-g) Multimodal ability transfer: analysis of how vision-language understanding contributes to manipulation bt VL-to-action learning, instruction data scaling, and inference time thinking.
Refer to caption: x6.png

#### 5.3.1 Action Ability Integration

Effects of language motion data for pre-training. As shown in [Figure 6](https://arxiv.org/html/2507.17520v1#S5.F6) (a), introducing “language motion” (auxiliary textual descriptions of low-level actions) enhances the VLM’s ability to associate visual cues with manipulation primitives, leading to a 10.5% improvement in overall success rate.

Effects of latent action queries. Latent action tokens are a key design component for decoupling high-level VLM planning from low-level action generation. As shown in [Figure 6](https://arxiv.org/html/2507.17520v1#S5.F6) (b), we vary the number of tokens from 16 to 128. Too few tokens limit behavioral diversity, while too many reduce training efficiency. A setting of 64 offers a good trade-off under our configuration.

Ablation on action expert designs. As shown in [Figure 6](https://arxiv.org/html/2507.17520v1#S5.F6)(c), while the base VLM offers general visual understanding, fine-grained perception for manipulation tasks demands richer representations. Removing the DINOv2-based ViT encoder from the action expert results in a 50.0% performance drop, highlighting its critical role in capturing task-relevant visual cues. Incorporating FiLM enhancement to the ViT encoder yields a further 15.3% improvement by modulating visual features with latent actions, enhancing task alignment. As shown in [Table 2](https://arxiv.org/html/2507.17520v1#S5.T2) The expert model with robot state generally performs better.

Comparison with fully finetuning VLA.
As shown in [Figure 6](https://arxiv.org/html/2507.17520v1#S5.F6) (d), FFT denotes full finetuning of the VLM backbone with latent actions using the same data recipe, but without MoE adaptation or multi-stage training. In contrast, InstructVLA employs our proposed architecture and two-stage training strategy, yielding 12.5% performance gain over Magma on SimplerEnv. This highlights the effectiveness of our design for integrating manipulation capabilities into VLMs.

#### 5.3.2 Multimodal Ability Transfer

Ablation on VL-to-action learning. As shown in [Figure 6](https://arxiv.org/html/2507.17520v1#S5.F6) (e),
to isolate the effect of instruction adaptation, we compare two settings: (i) finetuning only the VLM with the action expert frozen, and (ii) jointly finetuning both components. Freezing the action expert yields performance comparable to joint tuning, while reducing the number of trainable parameters and accelerating training. This suggests that InstructVLA can effectively adapt to complex textual inputs by fine-tuning only the VLM, without altering the pretrained action expert.

Effects of instruction data scaling. As shown in [Figure 6](https://arxiv.org/html/2507.17520v1#S5.F6) (f),
we evaluate performance on the SimplerEnv-Instruct benchmark as we scale the amount of VLA-IT annotations from 25% to 100% examples. Instruction-following accuracy exhibits a logarithmic improvement. Notably, situated reasoning tasks, where the model must ground objects and goals in context, benefit more from larger annotation sets, underscoring the bootstrapped reasoning ability from VLMs. In contrast, pertrained OpenVLA fine-tuned on VLA-IT benefits primarily from increased instruction diversity, but exhibits limited improvement on situated reasoning tasks, due to catastrophic forgetting of its original vision-language capabilities. These results indicate that InstructVLA’s advantage stems not only from additional training data but also from its model design. Further ablations and discussions for OpenVLA are provided in [Section A.2](https://arxiv.org/html/2507.17520v1#A1.SS2).

Training and inference strategies in different models. As shown in [Figure 6](https://arxiv.org/html/2507.17520v1#S5.F6) (g), OpenVLA suffers from catastrophic forgetting, leading to suboptimal performance when directly fine-tuned with VL or VLA-IT dataset. Magma, despite being co-trained on multimodal datasets, demonstrates limited benefits from its vision-language capabilities on reasoning tasks. In contrast, our generalist model, trained on the VLA-IT corpus, outperforms the expert model, which is capable for atomic instructions, on the SimplerEnv-Instruct benchmark. We denote language generation during manipulation as Think. Enabling thinking in the generalist model results in a further 36.1% performance gain over direct instruction execution and surpasses InstructVLA-expert paired with GPT-4o as an external interpreter. Further analysis of the role of thinking is discussed in [Section A.3](https://arxiv.org/html/2507.17520v1#A1.SS3).

## 6 Conclusion

We present InstructVLA, a unified VLA model that integrates multimodal reasoning and action generation. By preserving the generalization capability of pretrained VLMs and aligning language, perception, and control in a cohesive process, InstructVLA offers a solution to catastrophic forgetting and disjoint reasoning for VLAs. While effective, the current implementation leverages only minimal inputs: a single image and instruction. Incorporating additional sensory modalities (e.g., depth) could further enhance performance. Despite this, our end-to-end data and training pipeline enables state-of-the-art performance across manipulation tasks, multimodal benchmarks, and real-world deployments, paving the way for more generalizable, interpretable, and interactive robots.

Appendix

The supplementary material is organized as follows:

- •
[Appendix A](https://arxiv.org/html/2507.17520v1#A1) presents: (1) additional benchmarks on language responses, (2) finetuning of OpenVLA under the same settings as InstructVLA, and (3) extended analysis of InstructVLA.
- •
[Appendix B](https://arxiv.org/html/2507.17520v1#A2) provides additional case analysis for InstructVLA, OpenVLA, and GPT-4o System2.
- •
[Appendix C](https://arxiv.org/html/2507.17520v1#A3) lists data annotation details, including GPT-4o prompt and dataset statistics. We further analyse the distribution of the instructions from two dimensions: task diversity and language diversity.
- •
[Appendix D](https://arxiv.org/html/2507.17520v1#A4) visualizes the SimplerEnv-Instruct benchmark and the acknowledgements of 3D assets.
- •
[Appendix E](https://arxiv.org/html/2507.17520v1#A5) details the model architecture, training configurations, inference speeds under different settings, and compute resources used.
- •
[Appendix F](https://arxiv.org/html/2507.17520v1#A6) shows several multimodal question answering examples.
- •
[Appendix G](https://arxiv.org/html/2507.17520v1#A7) describes the real-world experimental setup and provides example executions.
- •
[Appendix H](https://arxiv.org/html/2507.17520v1#A8) discusses the broader impacts and outlines future directions for InstructVLA.

## Appendix A More Experiments and Analysis

### A.1 Vision-Language Evaluation

**Table 3: VLA-IT captioning evaluation. “Sentence-BERT” and “SimCSE” represent learning-based evaluation methods, while the remaining metrics are traditional n-gram-based evaluations focused on word distribution.**
| Methods | # Params | Sentence-BERT | SimCSE | BLEU-1 | BLEU-4 | METEOR | CIDER |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Qwen2-VL [qwen2vl] | 2B | 61.3 | 67.5 | 16.8 | 1.5 | 12.4 | 0.30 |
| GPT4o [GPT-4] | - | 60.7 | 67.1 | 16.3 | 1.8 | 16.2 | 0.09 |
| OpenVLA(VLA-IT) [openvla] | 7B | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.00 |
| Magma [magma] | 8B | 59.8 | 66.7 | 12.4 | 1.2 | 12.3 | 0.12 |
| InstructVLA(Generalist) | 2B | 72.0 | 77.0 | 44.3 | 8.2 | 18.7 | 0.84 |

In addition to the multimodal and closed-loop evaluations presented in the main results, we conduct supplementary language evaluations on the proposed VLA-IT dataset. This evaluation uses manually verified VLA-IT annotations on the Bridge dataset [Bridge_data], chosen for its diversity and distinct validation split. We generate 1,000 annotations following the method described in the VLA-IT dataset generation section. Two evaluation metrics are employed: (1) learning-based methods [Sentence-BERT, simcse], and (2) traditional metrics [bleu, young2023cider, meteor].

The captioning, question-answering and instruction-following results are presented in [Tables 5](https://arxiv.org/html/2507.17520v1#A1.T5), [5](https://arxiv.org/html/2507.17520v1#A1.T5) and [5](https://arxiv.org/html/2507.17520v1#A1.T5). We select Qwen2-VL [qwen2vl] and GPT-4o [GPT-4] as zero-shot VLM baselines, and include Magma [magma] (zero-shot) and OpenVLA [openvla] fine-tuned on the VLA-IT dataset as baselines for VLA models.

Although OpenVLA is fine-tuned on the VLA-IT dataset, it fails to generate complete sentences under the same evaluation setting as InstructVLA, despite the performance on multiple-choice benchmarks reported in our main results. This suggests a significant loss of its free-form dialogue capability. Magma performs well on question answering and captioning tasks. However, it struggles with instruction response ([Figure 7](https://arxiv.org/html/2507.17520v1#A1.F7)), often generating outputs misaligned with the given image. We hypothesize that this failure stems from the similarity between these instructions and the atomic commands used in finetuning manipulation datasets, which disrupts the coherence of the language latent space near the action latent space. This suggests a limited capacity to interpret and generalize free-form instructions, hindering effective transfer of vision-language capabilities.

InstructVLA achieves state-of-the-art performance, while GPT4o demonstrates competitive results. We visualize three episodes in [Figure 8](https://arxiv.org/html/2507.17520v1#A1.F8). GPT-4o generates more detailed captions but occasionally exhibits minor hallucinations. In the instruction response task, InstructVLA produces clearer and more grounded responses compared to GPT-4o, benefiting from the integration of ground truth atomic instructions during the data annotation process, as discussed in [Section C.3](https://arxiv.org/html/2507.17520v1#A3.SS3).

Figure: Figure 7: Magma results. Magma’s responses collapse when given instructions resembling those in its manipulation tasks, possibly due to learned actions interfering with its language latent space.
Refer to caption: x7.png

Figure: Figure 8: Comparison with GPT-4o. We visualize three examples from the VLA-IT language validation set. Each example includes a scenario caption (top), instruction response (middle), and question answering (bottom). The GPT-4o column displays responses only, as the instructions are identical across models.
Refer to caption: x8.png

### A.2 Data Ablation on OpenVLA

**Table 6: Data ablation on OpenVLA. “+VL” indicates finetuning OpenVLA with the same multimodal dataset used by InstructVLA. “+VLA-IT” refers to finetuning OpenVLA with the same VLA-IT dataset as InstructVLA. “+GPT4o” denotes using GPT4o as system 2 to translate free-form instructions into atomic ones.**
|  | OpenVLA (OXE) | OpenVLA + VL | OpenVLA + VL + VLA-IT | OpenVLA + VL + GPT4o | InstructVLA |
| --- | --- | --- | --- | --- | --- |
| Instruction Aggregation | 14.8 | 28.3 | 30.5 | 38.8 | 43.3 |
| Situated Reasoning | 13.6 | 19.5 | 17.4 | 32.4 | 48.8 |
| Average | 14.2 | 23.9 | 24.0 | 35.6 | 46.0 |

To investigate whether the performance gain of VLA-IT arises solely from the dataset itself, we reimplement the training procedure of the InstructVLA on OpenVLA [openvla], which represents a class of models trained under the action-only paradigm. As shown in [Table 6](https://arxiv.org/html/2507.17520v1#A1.T6), OpenVLA benefits from both vision-language and VLA instruction tuning data, with the latter showing greater improvement in the instruction aggregation setting. This is attributed to exposure to more diverse instructions. However, performance on the situated reasoning setting remains unchanged, likely due to catastrophic forgetting caused by the action-only training paradigm, which limits OpenVLA’s ability to leverage the VLM’s reasoning ability through simple finetuning.

The greatest performance gain is observed when GPT-4o is introduced as an auxiliary System 2 in both evaluation settings. However, overall performance remains inferior to InstructVLA, as GPT-4o cannot fully ground free-form instructions to the atomic skills on which OpenVLA is pretrained.

### A.3 Further discussions

Figure: Figure 9: Test-time tinking and dual-frequency evaluation. “Expert” refers to the model after action pretraining, while “Generalist” denotes the model after VLA-IT tuning. For dual-frequency evaluation, the horizontal axis represents the ratio of VLM executions to expert model executions.
Refer to caption: x9.png

Figure: Figure 10: Performance visualization of 30 situated reasoning tasks with and without reasoning enabled. Activating reasoning in our generalist model generally improves performance. For clarity, tasks are grouped into three categories: Subtask, involving subtask identification; Commonsense Reasoning, requiring broad world knowledge; and Commonsense for Tool Use, focusing on tool-related reasoning.
Refer to caption: x10.png

Role of VLA-IT training.
As shown in [Table 2](https://arxiv.org/html/2507.17520v1#S5.T2), although the InstructVLA-expert model does not outperform the OpenVLA(OXE) on Situated Reasoning of SimplerEnv-Instruct, which benefits from direct full fine-tuning of the VLM backbone, InstructVLA-expert shows promising scaling ability in understanding complex instructions and performing test-time thinking after stage-2 VLA-IT training. This result reflects a deliberate design choice in InstructVLA, where latent action learning during pretraining focuses on querying from visual and simple instruction features rather than relying on the full semantic space of the VLM too early. This design offers two significant advantages. First, it preserves the original semantic space of the pretrained VLM, maintaining its vision-language capabilities. Second, it enables the model to integrate diverse reasoning contexts during VLA-IT training. These properties contribute to the strong performance gains achieved by our generalist model and demonstrate the effectiveness of this training paradigm.

Test-time thinking. Allowing the model to perform test-time thinking by generating textual analysis of the given instruction can improve performance, particularly on situated reasoning tasks, as shown in [Figure 9](https://arxiv.org/html/2507.17520v1#A1.F9) (left). Notably, while the model with access to robot state outperforms the one without state when no instruction response is required, it provides limited performance gains when instruction following is involved. We hypothesize that state information helps the model retain manipulation skills but compromises its generalization to OOD environments and instructions.

Dual frequency inference. To further analyze the relationship between latent actions generated by the VLM and the final decoded actions, we decouple the inference frequencies of the VLM and the action expert, as illustrated in [Figure 9](https://arxiv.org/html/2507.17520v1#A1.F9) right. The results show that performance remains stable at a 1:2 ratio (VLM:expert), but begins to degrade at higher ratios. This suggests that latent actions offer relatively stable guidance to the action expert, reducing the need for frequent VLM queries.

A closer look at reasoning in manipulation tasks. We compare the performance of the generalist model on SimplerEnv-Instruct with and without vision language reasoning, as shown in [Figure 10](https://arxiv.org/html/2507.17520v1#A1.F10). A clear performance gap emerges in tasks involving commonsense tool use and interaction with articulated objects. This may result from instructions that do not explicitly state the intended actions and objects. For example, retrieving a cleaning tool from a drawer requires the robot to infer whether the prerequisite of an open drawer is satisfied, and to identify the sponge as the appropriate tool among several options. In addition to these cases, the reasoning process also improves performance on other situated reasoning tasks by grounding unfamiliar instructions using the pretrained in-domain knowledge of the vision language model.

VLA instruction tuning for cross-embodiment understanding. To assess whether InstructVLA retains this capability, we evaluate three variants on SimplerEnv-Instruct (see [Table 7](https://arxiv.org/html/2507.17520v1#A1.T7)): InstructVLA-Expert, trained solely on atomic instructions without test-time thinking; InstructVLA Generalist (Bridge), trained with the VLA-IT dataset on Bridge and the original Fractal dataset; and InstructVLA Generalist, trained with the full VLA-IT datasets across both environments. Adding the Bridge dataset results in a 139.4% improvement in Situated Reasoning performance for Generalist (Bridge) over the expert baseline, while Instruction Aggregation performance remains comparable. This discrepancy reflects differing generalization requirements: Instruction Aggregation emphasizes linguistic robustness, whereas Situated Reasoning demands vision-language grounding prior to action. The latter particularly benefits from the preserved reasoning capabilities of the pretrained VLM. As illustrated in [Figure 11](https://arxiv.org/html/2507.17520v1#A1.F11), the zero-shot model generates more diverse and accurate outputs than its fine-tuned counterpart.

**Table 7: Instruction tuning data ablation. We evaluate three settings: without VLA-IT data, with data only on Bridge, and with VLA-IT data on both Fractal and Bridge. This ablation examines the contribution of the VLA-IT dataset and the cross-embodiment generalization of InstructVLA on SimplerEnv-Instruct.**
| Instruction Tuning Data | Name | Insturction Aggregation | Situated Reasoning | Overall |  |
| --- | --- | --- | --- | --- | --- |
| Bridge | Fractal |  |  |  |  |
| ✗ | ✗ | Expert | 20.8 | 10.4 | 15.6 |
| ✓ | ✗ | Generalist (Bridge) | 18.4 | 24.9 | 21.7 |
| ✓ | ✓ | Generalist | 43.3 | 48.8 | 46.0 |

Figure: Figure 11: Case study on cross-embodiment. Top left: rollouts on SimplerEnv-Instruct. Top right: similar scenarios from the Bridge dataset with corresponding instructions. Bottom left: zero-shot results trained only on Bridge instructions. Bottom right: rollouts from the fine-tuned model.
Refer to caption: x11.png

Case study on multimodal capability transfer. As shown in [Figure 12](https://arxiv.org/html/2507.17520v1#A1.F12), we compare InstructVLA with OpenVLA [openvla], Magma [magma], and CogACT [cogact], all using the same input (language instruction and a single image). InstructVLA-Expert, though trained without multimodal datasets, retains the OCR capability of the underlying VLM and achieves the best performance among baselines trained solely on manipulation data. Finetuning InstructVLA-Expert into InstructVLA-Generalist with multimodal and VLA-IT datasets further enhances performance. For autoregressive models such as OpenVLA and Magma, multimodal finetuning improves OCR ability. In contrast, CogACT, when fine-tuned from OpenVLA(OXE) only on manipulation data with an action head, shows improved in-domain performance (on SimplerEnv) but suffers in generalization.

Figure: Figure 12: Case study on multimodal capabilities. OCR represents a unique multimodal skill of VLMs that is absent from typical manipulation datasets. We evaluate two tasks from the Instruction Aggregation set in SimplerEnv-Instruct, involving moving one letter to another (see [Figure 13](https://arxiv.org/html/2507.17520v1#A2.F13)(1)). By comparing different finetuning paradigms, we assess how effectively multimodal capabilities are integrated into VLA models.
Refer to caption: x12.png

Training at scale. A generalist VLA model with vision-language capabilities should be scalable across both manipulation and multimodal datasets. In this context, we compare datasets used by models claiming generalist abilities, as shown in [Table 8](https://arxiv.org/html/2507.17520v1#A1.T8). RoboMamba [liu2024robomamba] utilizes a limited manipulation dataset compared to other methods, while the dataset for ChatVLA [zhou2025chatvla] is not reported. $\pi_{0.5}$ [pi05] employs a significantly larger multimodal dataset than other approaches, though its multimodal performance is not disclosed. Magma uses more robot and multimodal data but achieves slightly worse performance on both multimodal and manipulation benchmarks compared to InstructVLA.

**Table 8: Data comparison of different methods. “Trans.” denotes transitions.**
|  | Magma[magma] | ChatVLA[zhou2025chatvla] | RoboMamba[liu2024robomamba] | $\pi_{0.5}$[pi05] | InstructVLA |
| --- | --- | --- | --- | --- | --- |
| Manipulation Data | 9.4M Trans. | - | 10K Trans. | 400 Hours | 469 Hours/ 5.9M Trans. |
| Multimodal Data | 1.2M Images + 4M Videos | 54K | 1.5M | >7M | 2M |

## Appendix B Case Study

### B.1 Reasoning Cases in SimplerEnv-Instruct

Figure: Figure 13: Reasoning cases in SimplerEnv-Instruct. Three cases of the VL fine-tuned OpenVLA and InstructVLA-Generalist. “SR” denotes success rate.
Refer to caption: x13.png

We present three representative reasoning cases in [Figure 13](https://arxiv.org/html/2507.17520v1#A2.F13). In the first example, OpenVLA fails to associate the letters “V” and “L” with their corresponding shapes in the image, resulting in consistent failure to grasp in all similar scenarios. In the second case, OpenVLA does not correctly associate the concept of "sour" with the corresponding fruit. As a result, its action is influenced by both the pear and lemon, leading to a grasp attempt between them that strikes the table. In the final example, OpenVLA fails to interpret the negation in the instruction and incorrectly grasps Coke instead of orange.

### B.2 Failure Cases

Figure: Figure 14: Failure case 1 of InstructVLA. The model receives only a third-person view image as visual input, making it difficult to estimate depth or the gripper’s relative position to the object. Consequently, it fails to grasp the object accurately, despite the gripper appearing aligned with the target in the image.
Refer to caption: x14.png

Figure: Figure 15: Failure case 2 of InstructVLA. The model fails to accurately estimate depth due to the real-to-sim gap, specifically the absence of arm reflection on the table, which causes the robot to become stuck in an out-of-distribution position.
Refer to caption: x15.png

We illustrate two representative failure cases of InstructVLA in [Figures 14](https://arxiv.org/html/2507.17520v1#A2.F14) and [15](https://arxiv.org/html/2507.17520v1#A2.F15). While some failures may result from the real-to-sim gap, incorporating additional sensory inputs such as depth information and robot state may enhance performance. We leave this exploration for future work. Additionally, we observe that the model achieves higher success rates in language responses than in action execution, suggesting that multimodal understanding is more readily transferable than manipulation skills. This highlights a fundamental challenge in the development of embodied models.

### B.3 GPT4o as the Auxiliary System 2

Figure: Figure 16: GPT-4o as the auxiliary system 2. We prompt GPT-4o with the first image from the environment along with the instruction, asking it to rewrite the prompt in a simple and clear format.
Refer to caption: x16.png

A strong baseline for InstructVLA integrates an expert model capable of executing atomic instructions with GPT-4o as an instruction parser to decompose complex, free-form commands for decision-making [hirobot, gao2025genmanip]. The prompt used is listed in Prompt [1](https://arxiv.org/html/2507.17520v1#A2.SS3), and it was evaluated and refined on 20 test cases from the Instruction Aggregation to ensure reliable performance. Results on additional test cases are presented in [Figure 16](https://arxiv.org/html/2507.17520v1#A2.F16). GPT-4o successfully identified the atomic instruction in the second case but failed in the first.

During evaluation, GPT-4o is invoked only in the initial step to ensure an unobstructed view of the scene and to generate a free-form instruction. We do not provide a closed set of task-relevant instructions for selection, as the training set ([Figure 19](https://arxiv.org/html/2507.17520v1#A3.F19)) lacks sufficient diversity in instructions and objects, and therefore does not adequately cover the evaluation settings. Across 80 evaluation cases, GPT-4o frequently fails in physical grounding, maintaining coherence, and accurately interpreting the scene.

## Appendix C Data Annotation Details and Analysis

The data analysis and GPT4o prompt are listed as follows ([Figure 19](https://arxiv.org/html/2507.17520v1#A3.F19) and Prompt [2](https://arxiv.org/html/2507.17520v1#A3.SS2)).

### C.1 Task Diversity Analysis

We categorize tasks into two broad classes: Command Rewriting / Context Creation and Question Answering. Each class includes several common task types:

#### Command Rewriting / Context Creation

- •
Complex Object Referencing: Uses attributes, pronouns, or relational terms to reference an object.
*Example:* “Place the red item next to the box.”
- •
Novel Action Referencing: Rephrases a previously known action using a different verb or motion.
*Example:* “Shut the drawer” (instead of “Close the drawer”).
- •
Negative Task Specification: Specifies the correct action by negating incorrect alternatives.
*Example:* “I’m thirsty, but I don’t want sparkling water—bring me something else.”
- •
Subtask Identification: Isolates a step from a multi-step instruction with a clear sequential order.
*Example:* From “Take the spoon out of the top drawer,” execute only the first step.
- •
Situated Task Identification: Infers the required action based on contextual cues or situational conditions.
*Example:* “I want to clean the table. What should I use?”
- •
Direct Instruction: Provides an explicit and unambiguous command.
*Example:* “Organize the drinks by putting the green can next to the Coke can.”
- •
Tool-Use Understanding: Refers to an object by its utility or function rather than its name.
*Example:* “Hand me something to cut with” (instead of “Use the knife”).

#### Question Answering

- •
Quantitative Identification: Requires determining the number or quantity of items.
*Example:* “How many apples are on the table?”
- •
Spatial Identification: Involves spatial relationships between objects or with the user.
*Example:* “Is the cup on the left or the right of the plate?”
- •
Visual Identification: Focuses on appearance-based attributes such as color or shape.
*Example:* “Which one is the metallic silver object?”
- •
Commonsense Answering: Requires everyday reasoning or world knowledge.
*Example:* “Which of these would you use to cut paper?”
- •
State Identification: Determines the current condition or status of an object.
*Example:* “Is the drawer currently open or closed?”

The data examples for VIA-IT are provided in [Figures 19](https://arxiv.org/html/2507.17520v1#A3.F19) and [19](https://arxiv.org/html/2507.17520v1#A3.F19).

Figure: Figure 17: Data analysis. Left: We manually identify common task categories and calculate the distribution. The proportion of direct prompts is reduced in favor of more diverse, free-form expressions. Right: Word cloud and verb-noun analyses compare the original Fractal instructions with the VLA-IT corpus.
Refer to caption: x17.png

### C.2 Prompting

The Prompt [2](https://arxiv.org/html/2507.17520v1#A3.SS2), along with three images captured at the beginning, middle, and end of each episode, is packaged and sent to GPT-4o. Episodes from the Bridge dataset [Bridge_data] that lack valid instructions are excluded from annotation.

### C.3 Ground Truth Instruction for Data annotation

During data generation, we observe that GPT-4o often struggles to accurately interpret robot behavior using only the three provided images, performing noticeably worse than humans. To quantify this, we randomly sample 100 examples and prompt GPT-4o to generate our four types of annotations using a similar prompt (excluding the ground truth instruction from a human expert). We then manually evaluate the correctness of the results: a sample is scored as 1 if no obvious errors are found, 0.5 if minor errors are present, and 0 if completely incorrect.

The results are summarized in [Tables 10](https://arxiv.org/html/2507.17520v1#A3.T10) and [10](https://arxiv.org/html/2507.17520v1#A3.T10), with two representative cases illustrated in [Figures 21](https://arxiv.org/html/2507.17520v1#A3.F21) and [22](https://arxiv.org/html/2507.17520v1#A3.F22). In the first case, GPT-4o hallucinates the robotic arm as a bread roll, leading to an incorrect caption and instruction. In the second, it reverses the temporal order of actions, resulting in an inaccurate annotation.

We attribute this performance gap to GPT-4o’s lack of temporal grounding and the low visual quality of images in manipulation datasets. In contrast, human-provided instructions inherently encode temporal links across the image sequence by grounding the task in context, identifying target objects, and specifying corresponding robot actions. This finding underscores that, despite their impressive capabilities, even state-of-the-art VLMs lack embodied experience and temporal grounding, limiting their ability to infer fine-grained actions in robot manipulation tasks.

**Table 9: Data annotation success rate. GPT-4o shows a significant performance drop without ground truth instructions during data annotation.**
| Method | Success Rate |
| --- | --- |
| With GT Instruction | 95.4% |
| Without GT Instruction | 45.0% |

### C.4 Language Motion Examples

Language motion [rth] describes end-effector movements using natural language, enhancing the VLM’s understanding of robotic manipulation. To generate such annotations, we leverage proprioceptive data that captures the end-effector’s position and orientation relative to the robot base. While the Bridge dataset [Bridge_data] adopts annotations from ECoT [ecot], we additionally annotate the Fractal dataset [RT-1] using a similar approach. The examples on the Fractal dataset are presented in [Figure 20](https://arxiv.org/html/2507.17520v1#A3.F20).

Figure: Figure 20: Language motion examples
Refer to caption: x20.png

Figure: Figure 21: Comparison of GPT annotations with and without ground truth instruction. Errors are highlighted in red.
Refer to caption: x21.png

Figure: Figure 22: Comparison of GPT annotations with and without ground truth instruction. Errors are highlighted in red. In this case, GPT-4o incorrectly infers the temporal sequence of actions without access to the instruction.
Refer to caption: x22.png

## Appendix D Benchmark Visualization

As shown in [Table 11](https://arxiv.org/html/2507.17520v1#A4.T11), although SimplerEnv-Instruct is primarily designed for instruction generalization, we incorporate diverse out-of-distribution objects, environments, and distractors to prevent VLA models from exploiting the benchmark by disregarding the instructions.

**Table 11: Task distribution**
| Attr. | with OOD Obj. | with OOD Env. | with Distract Obj. | Only Language OOD |
| --- | --- | --- | --- | --- |
| Percentage(%) | 50.0 | 62.5 | 35.0 | 5.0 |

We select 10 task scenes with InstructVLA rollout actions and responses, as shown in [Figures 23](https://arxiv.org/html/2507.17520v1#A4.F23) and [24](https://arxiv.org/html/2507.17520v1#A4.F24), to illustrate its performance on both Instruction Aggregation and Situated Reasoning tasks.

Figure: Figure 23: Examples of Instruction Aggregation in SimplerEnv-Instruct. We list ten examples with corresponding instructions and responses. Notably, InstructVLA shows the strong zero-shot ability to interpret multilingual instructions, recognize novel objects, and leverage OCR capabilities.
Refer to caption: x23.png

Figure: Figure 24: Examples of Situated Reasoning in SimplerEnv-Instruct. The second example’s responses is recorded before and after the drawer is open.
Refer to caption: x24.png

Acknowledgements of 3D assets.
We gratefully acknowledge the creators of the following 3D assets used in SimplerEnv-Instruct. All assets are licensed under the Creative Commons Attribution license:

- •
Fruit Bowl Collection
[https://sketchfab.com/3d-models/fruit-bowl-collection-d89f6311cb0c4c37b0bf9cdd1e5abcb3](https://sketchfab.com/3d-models/fruit-bowl-collection-d89f6311cb0c4c37b0bf9cdd1e5abcb3)
- •
Coffee Mug
[https://sketchfab.com/3d-models/coffee-mug-school-project-5f5ccee1514c440887c072fae8e0d699](https://sketchfab.com/3d-models/coffee-mug-school-project-5f5ccee1514c440887c072fae8e0d699)
- •
Copper Tea Pot
[https://sketchfab.com/3d-models/copper-tea-pot-27f2ac58f7614f2796630bdc6f18ee2f](https://sketchfab.com/3d-models/copper-tea-pot-27f2ac58f7614f2796630bdc6f18ee2f)
- •
Game Boy Classic
[https://sketchfab.com/3d-models/game-boy-classic-0ae80019e6f046168923286d7e628f6f](https://sketchfab.com/3d-models/game-boy-classic-0ae80019e6f046168923286d7e628f6f)

All other assets are created using Blender or modified from SimplerEnv [simpleenv].

## Appendix E Model Design and Training Details

### E.1 Instruction Format

To train captioning, question answering, and instruction-following capabilities, we integrate all tasks into a unified dialogue format. For captioning and question answering, we adopt the template shown in Prompt [3](https://arxiv.org/html/2507.17520v1#A5.SS1), where the captioning instruction is sampled from Prompt [4](https://arxiv.org/html/2507.17520v1#A5.SS1). For free-form instructions, we append the postfix “First answer my question.” to elicit a direct response from the model, as illustrated in Prompt [5](https://arxiv.org/html/2507.17520v1#A5.SS1).

### E.2 Learning Objective and Inference Procedure

We adopt flow matching [pi_0, flowmatching] to learn the action chunk $\mathbf{A}\in\mathbb{R}^{H\times 7}$ [act] over a horizon $H$. The training objective is defined as the flow matching loss:

$$ $\mathcal{L}_{FM}=\mathbb{E}\left[\left\|V{\theta}(\mathbf{A}^{\tau},q_{t})-( \epsilon-\mathbf{A})\right\|^{2}\right],$ (1) $$

where $\tau\in[0,1)$ denotes the flow step, and $V_{\theta}(\mathbf{A}^{\tau},q_{t})$ is the network output conditioned on $q_{t}$, which encodes information from DINOv2 [oquab2023dinov2] and a latent action $C$. The interpolated noisy action is given by $\mathbf{A}^{\tau}=\tau\mathbf{A}+(1-\tau)\epsilon$, with $\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$.

During inference, we generate the action chunk using forward Euler integration:

$$ $\mathbf{A}^{\tau+1/N}=\mathbf{A}^{\tau}+\frac{1}{N}V_{\theta}(\mathbf{A}^{\tau },q_{t}),$ (2) $$

starting from $\mathbf{A}^{0}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, with $N=10$ denoising steps.

### E.3 Model Parameters

**Table 12: Model parameters. “Adaptor” and “Scale Head” are used for MoE adaptation. Specifically, two LoRA adaptors are used to learn latent action generation and assistant response during VLA-IT.**
| Component | Parameter | Value |
| --- | --- | --- |
| Adaptor | Rank | 128 |
|  | Alpha | 256 |
|  | Dropout | 0.05 |
|  | Target | Attn. Q/K/V/O |
|  |  | MLP Up/Down |
| Scale Head | Depth | 4 |
|  | Size | 128 |
| Action Backbone | Depth | 12 |
|  | Head | 12 |
|  | Hidden Size | 768 |
|  | RoPE Theta | 1000 |
| Proprioception Encoder(Optional) | Hidden Size | 8 $\to$ 768 $\to$ 768 |
|  | Activation | SiLU |
| Action Encoder with Time Embedding | Hidden Size | 7+768 $\to$ 1536 $\to$ 768 |
|  | Activation | SiLU |

**Table 13: Flow matching parameters. The time steps is sampled from $p(\tau)=\beta(\frac{s-\tau}{s};1.5,1)$ [pi_0]**
| Component | Parameter | Value |
| --- | --- | --- |
| Flow Sampling | s | 0.999 |
|  | Inference Steps | 10 |
| Sinusoidal Time Embed | Max Period | 100 |

Additional model parameters are provided in [Table 12](https://arxiv.org/html/2507.17520v1#A5.T12), with flow-matching sampling settings detailed in [Table 13](https://arxiv.org/html/2507.17520v1#A5.T13). All projectors—including those aligning latent actions and DINO-ViT visual features to the action expert’s dimension—use a simple two-layer MLP with SiLU activation. The action head, also a shallow MLP with SiLU, maps the action expert’s hidden states to $\mathbb{R}^{N\times 7}$, where $N=16$ is the prediction horizon and 7 denotes the action dimension, including the gripper.

### E.4 Inference Speed

We evaluate the inference speed of InstructVLA on a single A100 GPU with BF16 precision, as shown in [Table 14](https://arxiv.org/html/2507.17520v1#A5.T14). To support language feedback during evaluation (i.e., CoT inference), in the “Thinking” setting, we enable VLM auto-regressive generation every 20 action expert steps. The “Action Only” setting bypasses language generation and directly decodes latent actions via a single VLM forward pass. In the “Latent Action Caching”, latent actions are generated every two expert steps; this introduces minimal performance impact. All settings are tested without action chunking. Note that although the model predicts 16-step action sequences, only one step is executed.

**Table 14: Inference speed. Inference speed is evaluated under three settings without using action chunking. Each evaluation includes a 50-step warm-up followed by 200 steps for stable measurement.**
|  | With Language | Action Only | Latent Action Caching |
| --- | --- | --- | --- |
| Inference Frequency(Hz) | 2.07 | 3.50 | 4.96 |

### E.5 Experiments Compute Resources

The action pretraining phase takes approximately 27 hours on 64 A100 GPUs, each node equipped with 1 TB of CPU memory. The VLA-IT phase requires around 12 hours under the same GPU configuration. Simulator-based evaluations are conducted using 8 A100 GPUs. For real-world experiments, training is performed over 4 hours on 32 A100 GPUs, and deployment is carried out on a single A100 GPU.

## Appendix F Multimodal Examples

[Figure 25](https://arxiv.org/html/2507.17520v1#A6.F25) illustrates InstructVLA’s multimodal and embodied commonsense reasoning across diverse scenarios. The model demonstrates accurate visual inference (e.g., recognizing a dog via reflection, identifying synthetic images), basic scene text recognition, and reliable grounding of objects and colors. In manipulation tasks, it interprets high-level goals, predicts appropriate next actions, and verifies task completion. These capabilities showcase its integration of perception, language, and manipulation, enabling effective performance in complex daily-life scenarios.

Figure: Figure 25: Zero-shot multimodal question answering. Four commonsense and four embodied examples are selected.
Refer to caption: x25.png

## Appendix G Real-world Experiments Setup and Analysis

We collect data exclusively for few-shot settings as shown in [Figure 26](https://arxiv.org/html/2507.17520v1#A7.F26). In the first setting, which focuses on grasping objects in a clustered arrangement, the robot is instructed to classify objects within a $20\times 40$ cm region on the table—placing all cubic objects into a plate and all others into a box. This setting includes 70 complete episodes, totaling 677 pick-and-place actions. In the second setting, which emphasizes spatial actions, the robot is instructed to randomly grasp three objects from the top of a rack and place them into a plate. We collect 60 complete episodes for this setting, comprising 180 pick-and-place actions. The experimental setups are depicted in [Figure 30](https://arxiv.org/html/2507.17520v1#A7.F30).

Figure: Figure 26: Real-world dataset examples. Four examples from the few-shot training set, illustrating cluster classification tasks (left) and rack pick-and-place tasks (right).
Refer to caption: x26.png

Figure: Figure 27: Zero-shot grounding. In a clustered pick-and-place setting, InstructVLA accurately places the blue cube by semantically grounding the reference to the celebrity.
Refer to caption: x27.png

Figure: Figure 28: Light distraction. Stable visual features from DINO and SigLIP enable the model to operate robustly under extreme out-of-distribution lighting conditions.
Refer to caption: x28.png

Figure: Figure 29: Zero-shot evaluation. We perform zero-shot evaluation in the Bridge kitchen environment with augmented background and novel objects. The instruction and model response are visualized in the first image.
Refer to caption: x29.png

To assess semantic grounding in novel contexts, we replace the plate and box in the cluster classification setting with images of celebrities. As illustrated in [Figure 27](https://arxiv.org/html/2507.17520v1#A7.F27), the model accurately interprets instructions and places the blue cube correctly by leveraging object and celebrity recognition.

[Figure 28](https://arxiv.org/html/2507.17520v1#A7.F28) shows that InstructVLA remains robust under extreme lighting conditions, supported by stable visual features from DINO and SigLIP. Finally, we evaluate zero-shot generalization in the Bridge kitchen environment with augmented backgrounds and unfamiliar objects. As shown in [Figure 29](https://arxiv.org/html/2507.17520v1#A7.F29), the model successfully follows novel instructions and completes the tasks.

Figure: Figure 30: Real-world settings. A third-person view is captured using an Intel D435i camera for the Franka (few-shot) and WidowX (zero-shot) settings.
Refer to caption: x30.png

## Appendix H Broader Impacts and Future Work

### H.1 Broader Impacts

InstructVLA contributes to the advancement of general-purpose embodied agents by integrating vision-language understanding with action generation. Its ability to follow free-form instructions and generalize to novel tasks supports applications in assistive robotics and human-robot collaboration. Nonetheless, as with other large pretrained models, careful attention must be given to potential limitations such as dataset bias and safety in real-world deployment. Ensuring responsible use and reliable performance across diverse environments is essential.

### H.2 Future Work

We plan to incorporate additional sensory modalities, such as depth and tactile feedback, to enhance safety and reliability in physical interactions. Leveraging recent advances in digital twins and simulation technologies, we aim to reduce reliance on real-world data by utilizing large-scale synthetic datasets. Finally, we will extend the evaluation and deployment of InstructVLA to a broader range of environments to further assess its generalization capabilities.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: Please clarify the 31.7% increase in the performance part is a relative percentage or an absolute percentage increase. In my calculation its a relative percentage, but you sould clarify it for easier understanding.
soundness: 4
strengths: Effective Resolution of Core VLA Challenges and Strong Performance: The work successfully addresses major VLA obstacles, namely catastrophic forgetting of multimodal skills and the difficulty of tightly integrating high-level reasoning with precise low-level control. Also itsresults are compelling, validating the model’s efficacy.

Novelty in Training Paradigm and Architecture Design: The core technical strength lies in the Vision-Language-Action Instruction Tuning (VLA-IT) paradigm and its architectural implementation. This approach skillfully unifies autoregressive language generation with flow-based action generation using a Mixture-of-Experts (MoE) adaptation framework within a two-stage training strategy. This design allows for the dynamic alternation between textual reasoning and action execution , providing an efficient solution to task interference while preserving general VLM knowledge. and the diagram of this part is very clear.

The authors also curate a large-scale, customized 650K-sample VLA-IT dataset featuring detailed annotations for embodied scene understanding and planning, essential for bridging VLM knowledge with embodied scenes.
summary: The article proposes a novel VLA model architecture that uses MoE architecture to effectively address the critical challenge of catastrophic forgetting of pre-trained VLA capabilities. The work is exciting and solid, providing a clear narrative that successfully addresses the challenge of integrating multimodal reasoning ability into VLA models. The introduction of a dedicated evaluation task, SimplerEnv-Instruct, is also a substantial contribution to the field.
weaknesses: Lack of explaination of Architecture part. The proposed architecture, particularly the reliance on the Mixture-of-Experts (MoE) adaptation and the design of the latent action space, is central to the paper's claims of solving catastrophic forgetting and enabling unified reasoning-guided manipulation. However, key mechanistic details and the representational quality require further clarification to substantiate the claims.

Please provide a detailed description of the Scale Head (gating network) component of the MoE. Specifically, what is its architecture, what precise input does it take (e.g., is it the hidden state of the VLM?), and how is it trained during the Vision-Language-Action Instruction Tuning (VLA-IT) stage?

Also, the core challenge in co-training VLM capabilities and action skills is catastrophic interference. Did the training process, besides the standard $\mathcal{L}_{LM} + \mathcal{L}_{FM}$ loss3333, include any regularization loss (e.g., sparsity or orthogonality constraints) specifically designed to encourage or enforce the dynamic switching between the Language Adapter and the Action Adapter? Such detail is crucial to validate the claim that MoE effectively enables adaptation while preserving pre-trained knowledge.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: **1.** The description of the architecture and training paradigm is quite confusing.

   **1.1** For example, they mention an action lora, a language lora, MoE adaptation, and a scalar head. However, it is hard to find details about them in Figure 2. Are the MoE and language lora only activated in stage 2? 

   **1.2** I'm also confused about the training details of stage 1. The authors mention that they use the data of RT-1, and the model needs to predict both action and language, so what is the input of the model in stage 1?  If the task prompt and images are the input, is the output language generated by the authors? 

   **1.3** In stage 1, what is the meaning of the action lora? Where exactly is this action lora applied? Do both language tokens and action tokens pass through the action lora? If only action tokens go through it, then what’s the point of the language loss? Is it just used as a simple regularizer?

**2**. The metric of the SimplerEnv-Instruct benchmark is success rate? 
**3.** Table 2 shows that the training stage 2 is harmful to the normal simpler tasks, and there is little explanation about it. Since the necessity of reasoning VLA data is critical, can the authors provide more discussion?
soundness: 3
strengths: This paper shows a good example of how to effectively keep the capability of LLM/VLM in a VLA system, which is a very critical problem for current VLA models. It also demonstrates how to further enhance the reasoning ability of VLA. The proposed dataset and benchmark can also be a good reference for later works.
summary: This paper introduces InstructVLA, which is a VLA model that preserves large vision-language reasoning capabilities while achieving state-of-the-art manipulation performance. To achieve this, it introduces VLA-IT training paradigm with a 650K dataset. To evaluate generalization, this paper also proposes SimplerEnv-Instruct benchmark. InstructVLA achieves good performance on multimodal, Simpler, and SimplerEnv-Instruct benchmarks.
weaknesses: Please see the questions.

---

confidence: 5
contribution: 3
initial_rating: 4
presentation: 3
questions: See weakness.
soundness: 2
strengths: 1. VLA-IT Paradigm: It introduces a novel training paradigm called "Vision-Language-Action Instruction Tuning (VLA-IT)". This paradigm uses a Mixture-of-Experts (MoE) adaptation framework to jointly optimize textual reasoning and action generation, trained on standard VLM corpora and a new 650K-sample VLA-IT dataset.
2. Two-Stage Training: The training is divided into two stages: (1) Action Pretraining, which trains a VLM-driven "action expert" ; and (2) VLA Instruction Tuning, which freezes the action expert and fine-tunes only the VLM backbone via the MoE module to handle complex instructions and multimodal reasoning.
3. New Dataset and Benchmark: The paper contributes two significant resources: (a) a 650K VLA-IT dataset with diverse instructions, scene captions, and Q&A pairs ; and (b) a new benchmark, SimplerEnv-Instruct, featuring 80 tasks designed to evaluate generalization, closed-loop control, and high-level instruction understanding in VLAs.
summary: This paper introduces InstructVLA , a model designed to solve "catastrophic forgetting" in Vision-Language-Action (VLA) models, a common problem where models lose their pre-trained multimodal reasoning abilities after being fine-tuned on manipulation tasks. The authors propose a novel training paradigm called "Vision-Language-Action Instruction Tuning (VLA-IT)" , which uses a Mixture-of-Experts (MoE) framework and a new 650K-sample dataset to jointly optimize both textual reasoning and action generation. As a result, InstructVLA successfully preserves its powerful language and reasoning capabilities while delivering state-of-the-art manipulation performance , significantly outperforming baselines like OpenVLA on the new SimplerEnv-Instruct benchmark.
weaknesses: 1. This work is the combination of MoE and the fine-grained instruction finetuning. It seems so incremental. Especially, I can't see any advantages from the design. The biggest contribution is that the author gives the a 650K VLA-IT dataset with diverse instructions, scene captions, and Q&A pairs.
2. More related works containing the step by step reasoning ability should be added and discussed like CoT-VLA.
3. Authors should give more attention to the motivation. Particularly when the paper said their problem about ''how to train robotic manipulation skills without suffering "catastrophic forgetting" of the powerful multimodal reasoning capabilities inherent in pre-trained Vision-Language Models (VLMs)'', I don't know how instruction tuning can solve this kind of problems.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: When creating SimplerEnv-Instruct, what variations were considered? I see that there are around 80 tasks + 1.1k trials which necessarily limits coverage for practicality reasons, but I am curious what things were decided to not be covered.
soundness: 3
strengths: Paper does a good job of arguing that existing VLAs perform subpar at more complex queries, and the idea that the LLM backbone needs to be finetuned to exploit the action expert for embodied understanding seems reasonable. The proposed SimplerEnvInstruct benchmark is generally reasonable and gains on this benchmark are pretty significant. Ablations on different parts of the architecture, dataset, etc. are pretty extensive.
summary: The authors argue that prior iterations of VLA models have not placed as much focus on embodied reasoning as they should. By embodied reasoning, we mean the ability of the model to move from being told exactly what manipulation to do, to a robot that can use context clues from the environment to decide what manipulation to do.

To address this, the authors introduce a new evaluation benchmark, based on augmenting the SimplerEnv setup with multilingual instructions, more indirect descriptions of objects (i.e. "I need something to clean with"), etc. They also train the VLA in a 2 stage process. First, an action expert is pretrained on a large manipulation dataset, to predict both learnable latent actions and language descriptions of the actions. This  is tuned against a fixed VLM backbone, using a flow matching loss. This approach was popularized by works like pi_0, but then as a 2nd stage, the authors train LoRAs on the LLM backbone against the fixed action expert. (Some experiments were done with finetuning the action expert too, and results were generally similar / slightly worse). The authors argue this 2nd stage can be viewed as a form of instruction tuning, as mediated by the action expert, to answer more complex questions.

This model is shown to perform reasonably on two prior benchmarks, a Google Robot and WidowX Robot. On the new SimplerEnv-Instruct benchmark, InstructVLA-Expert (the model trained only with the 1st stage) is only around on par with finetuned OpenVLA, but performs signficantly better after the stage 2 LLM backbone finetuning.
weaknesses: I think the paper is somewhat overreaching on how much it contributes to embodied reasoning. Even in the pre RT-1 days, there were some papers trying to do embodied reasoning. In my opinion the main difference these days is that the smaller, low latency VLMs are now more capable of answering these questions on their own.

Similarly, I find it difficult to attribute how much gain comes from the general evolved understanding that MoE backbones are better than dense backbones (many baseline models date to before this time).

Last, there is always some benchmark bias, where in general you expect a paper to do better on its self proposed benchmark compared to other models, the largest gains coming from the self-proposed benchmark gives me some pause.

nit: please label the y-axes in Figure 5, Figure 6. Please make it clearer how many trials were done for each eval number and/or include confidence intervals (why do these only exist in the SimplerEnv-Instruct columns?). In Table 1/2, I think including robot state is a significantly more important variable than the rest and would prefer if that were separated from the other results more in some way. (It is also unclear which baselines methods use robot state and which do not.)


Decision: Accept (Poster)