Vision-Language-Action Instruction Tuning: From Understanding to Manipulation
Authors: Shuai Yang
Hao Li
Bin Wang
Yilun Chen
Yang Tian
Tai Wang
Hanqing Wang
Feng Zhao
Yiyi Liao
Jiangmiao Pang
Venue: ICLR 2026 Poster (2026)
Keywords: vision-language-action model
vision-language model
large language model
Primary area: applications to robotics, autonomy, planning
TL;DR: InstructVLA unifies VLM reasoning and precise action generation, achieving leading results in manipulation and multimodal tasks.

Abstract: To operate effectively in the real world, robots should integrate multimodal reasoning with precise action generation. However, existing vision-language-action (VLA) models often sacrifice one for the other, narrow their abilities to task-specific manipulation data, and suffer catastrophic forgetting of pre-trained vision-language capabilities. To bridge this gap, we introduce **InstructVLA**, an end-to-end VLA model that preserves the flexible reasoning of large vision-language models (VLMs) while delivering leading manipulation performance with the help of embodied reasoning. InstructVLA introduces a novel training paradigm, *Vision-Language-Action Instruction Tuning (VLA-IT)*, which employs multimodal training with mixture-of-experts adaptation to jointly optimize embodied reasoning and action generation on both standard VLM corpora and a curated 650K-sample VLA-IT dataset. On in-domain SimplerEnv tasks, InstructVLA achieves 33.3% improvement over SpatialVLA. To evaluate generalization, we introduce SimplerEnv-Instruct, an 80-task benchmark requiring closed-loop control and high-level instruction understanding, where it outperforms a fine-tuned OpenVLA by 96% and an action expert aided by GPT-4o by 29%. Additionally, InstructVLA surpasses baseline VLMs on multimodal tasks and exhibits inference-time scaling by leveraging textual reasoning to boost manipulation performance in both simulated and real-world settings. These results demonstrate InstructVLA's potential for bridging intuitive and steerable human-robot interaction with efficient policy learning.

Published as a conference paper at ICLR 2026
VISIONLLANGUAGE-ACTION INSTRUCTION TUNING: FROM UNDERSTANDING TO MANIPULATION
Shuai Yang?.>- Hao Li Bin Wang
Yilun Chen 31 Yang Tian
Tai Wang
Hanqing Wang² Feng Zhao
Yiyi Li
Jiangmiao Pang"
*University of Science and Technology of China, "Zhejiang University.
SShanghai Artificial Intelligence Laboratory
ABSTRACT
To operate effectively in the real world, robots should integrate multimodal rea- soning with precise action generation. However, existing ission-laguaage-cttonn (VLA) models often sacrifice one for the other, narrow their abilities to task-specific manipulation data, and suffer catastrophic forgetting of preitrained vision- -aaguul capabilities. To bridge this gap. we introduce InstructVLA. an end-to-ent VLA model that preserves the flexible reasoning of large vision-langume language models (VLMs) while delivering leading manipulation performance with the help of embodied rea- soning. InstructVL.A introduces novel training paradigm, Aasion-LanguggeAAce Instruction Tuning (VLA-IT), which employs multimodal training with mixture-of- experts adaptation to jointly optimize embodied reasoning and action generation on both standard VLM corpora and a curated 650K-sample sample VLA-IT dataset. On in-domain SimplerEnv tasks, InstructVLA achieves 33% improvement over Spa- Li.. To evaluate generalization, we introduce SimplerEnv-Instreee benchmark requiring closed-loop control and high-level instruction understanding. where it outperforms a fine-tuned Open VLA by 96% and an action expert aided by GPT-4o by 29%. Additionally, InstructVLA surpasses baseline VLMs on multi- modal tasks and exhibits inference-ting scaling by leveraging textual reasoning boost manipulation performance in both simulated and real-world settings. These results demonstrate InstructVLA' potential for bridging intuitive and steerable human-robot interaction with efficient policy learning. Project website.
an 80-task
INTRODUCTION
Large-scale pretraining has produced versatile foundation models in computer vision (CV) (Qquab
ett al.
2023
Radford et al., (2021) and natural language processing (NLP) (Bai et al. 2023 Touvron
it al.
2023
(VLA) models (Black et al. Kim et al. (024) adapt large vinion-language models (VLMs) (Karamcheti et
Building on this progress, recent Dision-Language
2024
2024 Beyer
ct al.
2024) and finetune them on embodied datasets to achieve generalizable manipulation. While
the integration of multimodal reasoning has led to significant advances in VLMs (Wei al, 2022,
Liu et al.] 2024a), such reasoning remains largely unexplored in VLA settings. Fully leveraging manipulation beyond VLA initialization remains an open challenge. Current attempts to incorporate the reasoning capabilities of VLMs into action learning face three main obstacles: (1) task interference, catastrophic forgetting (French 1999) of multimodal ability during action training; (2) data scarcity, particularly the limited availability of manipulation datasets with rich multimodal supervision; and (3) methodological gaps, specifically the lack of effective mechanisms and training paradigm to translate multimodal reasoning into action generation. These limitations lead to a fundamental question for VLA-basee manipulation:
VLMs for reasoning-guide
How acquire manipulation skills without eroding the VLM's multimodal reasoning, and how can such reasoning, in turn, enhunce manipulation?
To address these challenges and utilize VLMS more effectively, prior work has primarily adopted two strategies. The first aims to retain general multimodal capabilities while learning manipulation
Equual contributions. 'Proecct leader. CCoreespondig authors: Feng Zhao, Yiyi Liao.

---

Published as a conference paper at ICLR 2026
InstructVLA
Embodied Understanding
What ope of ofstendd to bakert
There
fire nasking dises red coss from backit
Ilgaa the spoc and ala tuy
The enaene aqgic th aagiss soe
Muttimofall Web
ACD
Vision
Lang
Largesscale Interaction Are otation
Vision- Language Knowledge
Insrie-nntrruetinn Manipulation
Instruction Tuning
Action
Sound Faaien
NOVAISE
Clownd-lopp Beassening Evalaation
guiddd guided Manipelation
Heleragmmng Em Embaaimmnii
Figure I: Method overview. InstructVLA integrates vision- understanding with precise robotic control to achieve reasoning guided manipulation. Its core training strategy, Vision-Language- Action Instruction Tuning, enhances manipulation by unifying general multimodal knowledge, embodied reasoning, and atomic instruction-based manipulation into coherent chain of thought.
skills through unified auto-regressive modeling. Models such as RT-2 (Brohan et al. 2023 and Magma (Yang et al., 2025) follow this approach by co-training on vision-language and manipulation data. Yet, this paradigm often overlooks complex embodied reasoning. and our ablations reveal that the general VLM corpus exhibits a domain gap in embodied scenarios. The second strategy tightly integrates embodied reasoning into manipulation datasets to transfer VLM capabilities. Methods such ECoT (Zawalski et al. 2024) and Emma-X (Sun et al. 2024) embed chain-of-thout (CoT) reasoning into manipulation datasets. While promising, these methods rely on action- prettrined architectures (Kim al. 2024) and structured reasoning formats (e.g.. subtasks, grounding), which limit expressiveness, suffer from catastrophic forgetting, and fail to demonstrate general multimodal capabilities-even with additional finetuning. Consequently, the extent which VLM capabilities can be effectively translated into action generation in embodied contexts remains largely unexplored.
Building these observations, we propose InstructVLA, generalist VLA model that extends pretrained VLMS for accurate action generation while preserving strong multimodal understand- ing. Building on this unified modeling. we conduct extensive experiments to investigate how multimodal capabilities contribute to manipulation. Motivated by these insights, we design the Intruction Tuning (VLA--IT) paradigm specifically tailored to bridge vision-language knowledge with action generation, treating an integral component of instruction following, as illustrated in Figure To support this paradigm, we Instruction Tuning dataset, consisting of 650K human-robot
Vision-laaggageeAAcie
action generation as
curate the Assion-Language interactions annotated with diverse instructions, scene captions, and question-answer pairs grounded in high high-quality manipulation tasks (Ebert et al.) 2021 Brohan et al., (022). The training follows two stages: (1) Action Pretraining, which trains a VLM-driven action expert using latent action queries distilled from language-bane based motion descriptions, providing learnable interface to the VLM while decoupling low-level control learning from the VL.M backbone to preserve its multimodal reasoning capabilities; (2) Aision-LanguageActtinn Instruction Tuning, which unities language and latent action generation through a trainable -exture-of-expertssMteee
adaptation framework. This framework jointly trained on multimodal datasets (He al. 2024), manipulation datasets, and the curated VLA-IT corpus, enabling the automatic switch between textual reasoning and action generation, thereby effectively leveraging lisoun-aagguage understanding and reasoning for action generation.
To validate the performance of InstructVLA, we introduce the SimplerEnv-Intruct benchmark. a manually designed evaluation suite featuring 80 zero-shot manipulation tasks. It encompasses both closed-loop manipulation tasks and high-level instruction reasoning, involving either situated understanding or decomposition into actionable subtasks. With its thinking ability during manip- ulation, Ln outperforms the fine-tuned VLe baseline by 96% and achieves a 29% improvement over an action expert model assisted by GPT-4o on ComplerEnvIInstruee demonstrating its effectiveness in instruction following and task decomposition. Furthermore, InstructVLA LA surpasses similarly sized VLMs in multimodal performance and shows a 33% improvement over SpatialVLA in closed-loop manipulation (Li et al. 2024d). Our contributions can be summarized follows:
2

---

Published as a conference paper at ICLR 2026
Model. We propose InstructVLA, VLA architecture and training pipeline that supports studying language capability in VLAS by efficiently preserving pretrained vision -aiiuge knowledge from VLMs while integrating manipulation
component of instruction following.
Dataset & Benchmark. We design a practical data and evaluation pipeline for vision- -isoonlaagguage- action instruction following, supported by 650K tailored VLA-IT annotations and manually curated benchmark suite, enabling evaluation of VLAs' instruction generalization capabilities.
Validation. InstructVLA achieves leading performance across robotic manipulation tasks, multi- modal benchmarks, and real-world deployments, enabling intuitive and controllable manipulation.
2 RELATED WORKS
Policy learning at scale. Following the success of CV (Oquab et al. 2023 Zhai et al. 2023 aadd
NLP Touvron al. 2023). recent research (Wang et al. 2024a |Bohhan et al. 2022 2023 Zheng
ct al. 2025; Wang et al. 2024bb] Niu et al. 2025) shows that robot policies improve when trained
in large heterogeneous datasets. RT-1 (Brohan et al. 2022) and RT-2 (Brohan al.
2023), trained
in large-scale real-world demonstrations, achieve strong in-domain accuracy and zero-shot transfer.
Works such as Octo (Octo Model Team et al. 2024) and RT-X (Collaboration et al. (2023) extend this
approach by aggregating the largest open-source manipulation datasets (Collaboration et al. (002)).
Some methods, such as LAPA eYe al. 2024). Seer (Tian et al.) (024), and Moto (Chen et al. 2024b)
use video generation and inverse dynamics to learn scalable motor representations. In the VLA domain, models are typically initialized from pretrained lision-language models (Kim al. 2024]
Qu et al. 2025 Broohan et al., (2023) leveraging prior visual-linguisti alignment instead of learning from scratch. Further, methods such as RT-Trajectory (Gu et al., 2023) and GraspVLA (Deng et al.. 2025b) jointly train intermediate manipulation representations such as trajectories or bounding boxes using a combination of real and simulated data to guide action generation and enhance generalization.
models. Recent foundation models (Brohan et al. 2023 Kim al. 2024 Qu et al. 2025 Black et al., 2024) (hen et al.) 2024b Bjorck et al. [0225] Pertsch et al. 2025 Wu
Vision-language-acll
al. 2026) integrate perception, language, and robot manipulation into single network, using two
main architectures. Autoregressive models such as RT-2 (Brohan et al.] 2023), Open (Kim et al.
2024) and SpatialVLA (Qu al. 2025) treat actions as discrete tokens. LLARVA (Niu al. 2024) introduces 2D trace for pretraining. FAST tokenization (Pertsch et al. (025) further compresses
motion sequences. In contrast, blsed VLAS avoid discretization; for example, To (Black et al.
2024) and GROOT (Bjorck et al. (2025) generate actions through continuous flow matching (Lipman al. 2022). while CogACT et al., 2004a)) and Cronus (Li al. [025a] use diffusion (Peebles & Xie, (023). ). Hybrid approaches, like RoboDual (Bu et al.) 2024). combine generalist action models with specialist action experts. Although flow-based methods (Black et al. [024] Bjorck et al. 2025) Li et al. 2025a 2024a) often achieve superior performance, they typically neglect the integration of autoregressive text reasoning (Brohan et al. 2023), which is crucial for leveraging the VLM's semantic capabilities. In contrast, our model unifies autoregressive VLM language generation with the flow-based action generation, demonstrating efficient co- co-training of language and action.
Bringing step-by-stey reasoning ability to manipulation. Bridging pre-trained world knowledge to enhance the generalization of robot policies a promising direction. One line of work standardizes intermediate representations (Li et al. (026), such primitive (Chen et al., 2024c). trajectories (Li et al. 2025b), keypoints (Li et al. (2024b) and masks (Huang et al., 2025). However, these approaches often rely on rule-based decomposition and hand-crafted planning heuristics, whose rigid separation from low-level control limits scalability and hinders end-to-end policy learning. CoT-VLA (Zhao) al. 2025) instead treats future video generation as an implicit chain-of- to-thhhggh, but predicting image tokens stcp-by-step introduces computational overhead, limiting practicality for fast closed-loop control. More recently, unified modeling of perception, reasoning, and manipulation (Intelligence et al. 2023 AI 2024 Shentu et al., 2024), along with other generative formulations (Pan et al.) 2025 Zhou et al. 2024). ). has demonstrated the potential leveraging pre-trained VLMs and LLMS for reasoning-guider generation, revealing emerging capabilities (Deng et al., 2025a). Yet, many prior studies depend on closed-soure source data (Intelligence et al. 2025) or conduct limited evaluation in real-world settings (Belkhale et al.) 2024 Zhou et al. 2025), constraining reproducibility and large-scale assessment. Our work provides a pioneering exploration. supported by open data and benchmarks, study reasoning- aasooinngguudee manipulation through the integration of reasoning and action.
3

---

Published as a conference paper at ICLR 2026
VL Understanding
Manipulation
(1) Language Response
Latent Action
(3) Action
will place the pot the burner.
Condittions
an AGrpp
X.
Pre-trained VLM
MoE Adaptation
Action Expert
Put the left heating.
SigLIP
High-level Prompt
RLM
DinoV2
Concatenate
Noisy Actions
Optioonal
Propriccapiio
Figure 2: Overview of the InstructVLA. Instruct VLA integrates the multimodal reasoning capa- bilities of vision-language model with robotic manipulation. Generation consists of three steps: (1) asynchronous auto-regressive reasoning by the VLM, (2) latent action generation, and (3) action decoding. A MoE. adaptation enables the VLM to alternate between reasoning and latent action prediction. The flow matching action expert decodes the final actions, conditioned on latent actions.
3 INSTRUCTVLA
We propose InstructVLA (Figure 2) unified model for joint language-action generation that also mitigates task interference and catastrophic forgetting. Section Se the architecture, including dynamic switching between reasoning and execution modes, well inference strategies. Section 3 presents the training paradigm for the instruction following of VLAs.
3.1 ARCHITECTURE
Embodied VLM for text and latent action generation. We propose unified framework that enables simultaneous multimodal reasoning and language-steeree latent action planning using single VLM (Figure (1) and (2)). The model produces textual outputs preserve the strong language understanding and multimodal inference capabilities of the pretrained VLM,, while subsequently generating latent action representations for downstream manipulation. To support action planning, we introduce N learnable action queries Q E ND which attend to the VLM's hidden states and extract task-relevant latent action C RDD where D is the VLM hidden dimension. Our implementation builds on the compact and efficient Eagle2-2B backbone (Li et al. (025c), with tailored training strategy described in Section 3.2 The VLM is supervised with cross-entropy on language output with loss L..M-
MoE adaptation harmonize reasoning and action. seamlessly alternate between reasoning and manipulation. To this end, we adopt a MoE design (Zhou (ct al. 2022). which allows adaptive reweighting of expert modules based on input context and reasoning mode, thereby integrating multimodal reasoning with language-steered latent action. Specifically, LoRA (Hu et al., 2022) modules are employed as experts within the LLM backbone. preserving pretrained capabilities while ensuring efficient inference. A scalar head (Buehler & Buehler 2024) predicts gating coefficients X; for each expert by classifying the hidden state, enabling the model adaptively blend their outputs. The resulting hidden states for K experts are computed as h Wa + ΣK B,A, a; >,, where Wo is the original weight, x denotes input, A, I rxx and B, dxx are the LoRA parameters, a is the LoRA scalar factor, as detailed in Section F.2
A key challenge is enabling the model to
Flow model as an efficient action expert. To further decouple low-level control from high-level understanding, the action expert is designed to generate actions from image observations conditioned on VLM-derived intentions. It takes image features from DINOv2 (Oquab et al. (2023) vision encoder, latent actions, noisy action embeddings and optional information such as proprioception, and fuses these with a simple transformer architecture (Touvron al. 2023) with block-wise causal attention. Specifically, non-causal attention is applied within each input, and causal attention between input types. The vision encoder, further enhanced with feature-wit wise linear modulation (FiLM) (Perez et al.) 2018), plays a crucial role in directing actions to spatial and contextual input. The flow matching objective (Black al. 2024) used supervise action learning, detailed in Section
4

---

Published as a conference paper at ICLR 2026
Original Dataset
Embodied Scene Understanding
Instraction Understanding and Planzing
bat' co Sces
Secnario Caption
Question Answering
Command Rewriting
Context Creation
Aiitchen nounter wi varisu oigacts, inclddnng ca priu placs food herr, coa, une a
wher ourber is outt bourd placed on? Thesloes.
Ploce the cutting toail the cotting bound.
Fin abs perpare candwicch
Crudd you tood enady
the ieste cutting
(1l placs lorfle (Stusted board.
Unility
rable with dent, nedt vor Lirtte teets induding microwaee and cans
Wiee stendl mode of thoe the tatte? medic motal and plasti.
plastic put
spoion
(Mottiial
(Novsl Action
Von rabch colaar and onur Sar
drwaecic toep,,, meaasing moikk white
Wi boon tn mikile diower?
Radi une sedeoedion from the droseer and sot the counter.
Pieus open micde srewer arts
Colee and tnitt
Coke can.
lwillget the soda can from the dowwer:
The vidile drover suppas
table.
Caka chithull drawerr
Appeorince
wxth valo
(Long hortson
Figure 3: Vision Vision-aagguage-acteee instruction tuning data examples. Annotations focus on: (I) improving scene understanding and (2) learning instruction following and planning.
Inference. InstructVLA integrates language and action generation in single model with the follow- ing techniques to improve speed. (1) Decoding strategies. To mitigate the latency of autoregressive decoding, textual responses are generated via greedy search until the first action query token appears. The remaining action queries are then decoded in parallel within a single forward pass of the VLM. (2) Language response and latent action caching. We decouple language response from action generation by caching textual outputs across multiple action steps, leveraging their temporal stability. InstructVLA LL also supports cache latent actions, which reduces the number of VLM forward with minimal performance impact compared with HCoT (Zawalski et al. 2024) (see Section A.1)
3.2 TRAINING RECIPE
Direct co-training of vision, language, and action often leads unstable optimization and slow convergence. We therefore adopt principled two-stage training paradigm: first, action pretraining to align with the VLM's latent action embeddings; second, assion-language-ctiinn instruction tuning to integrate multimodal reasoning with manipulation.
Stage 1: Action pre-training. InstructVLA is pre-trained using heterogeneous manipulation data (Brohan et al. 20222] Ebert et al., 202T). To distill the knowledge from the VLM for ma- nipulation, the model is trained to predict both actions and language motion (Section [41)). with the latter supervised via cross-entropy loss. Due to the stability of flow matching and the next token prediction, the final loss is the direct sum of both losses as L = LLM + LrM- During this stage, only the embedding of the latent action and action LoRA adapter the LLM backbone are tuned, consisting of 650M parameters. The model trained is named the "Expert".
Stage 2: Vision laston-anguagg--cctin instruction tuning. We extend visual instruction tuning (Liu et al. 2023) with a simple and efficient approach to train InstructVLA. LA. Our key observation is that once the action expert has been pretrained to follow latent actions generated by the VLM, further adapting the LLM backbone enables the model to handle manipulation tasks with more complex instructions. In this stage, a language LoRA and scalar head are added, which together with the stage action LoRA constitute the MoE adaptation (Buehler & Buehler 2024). This MoE module is the only trainable component Stage 2, totaling 220M parameters. We detail the data pipeline for lision-language-an
instruction tuning in Section 4.1] this data bridges pretrained vision-language capabilities with embodied task scenarios. To further bootstrap multimodal understanding, we co-train the model with additional multimodal datasets (He et al. (024).. The resulting model, referred to as the "Generalist", integrates both vision-langueen language reasoning and manipulation capabilities.
4 VLA DATASET AND BENCHMARK
4.1 INSTRUCTVLA TUNING DATASET
We curate diverse hierarchical language annotations from large-scale manipulation datasets (Brohan et al.) 2022 Ebert et al., 202T). including language motion (Belkhale et al.) 2024) as detailed in Section D. along with the VLA-IT dataset for instruction tuning and reasoning transferring.
5

---

Published as a conference paper at ICLR 2026
Task Aggregation
Situated Reasoning
After tins my drink, can you pi the truit oeer For stark?
laat dean sable suitable tool
will seker the eponge for dleaningg E
move the apple
Muttiliggsiie Riter
Subriaak End
Vesilize recupieri form
Fut
susurig foce
1stl pec the green pear.
B
Pesfemmase
Sabttakk Start
+31.7
Erab the Cake into the drmwnr, then The Edtar the saaa yuull paoseir wnliss tue drswer
Negathe ai thistr bus do not west iiks. Poase gloseggaasso grassonctiing for mo. wwll piok smange 25
Figure Simpler-Instruct. Six representative test cases with instructions and InstructVLA responses. Prior VLAs exhibit limited generalization compared to InstructVLA.
Vision- Iisionllagguage-attion instruction tuning data. To enable language-steerable VLA models, it essential to curate diverse instructions, model responses, and reasoning patterns. We categorize data into four types as illustrated in Figure For embodied scene understanding: (1) Scenario captioning provides deseriptions the robot's s environment (2) Question answering targets scene understanding through consistent QA pairs across episode. Together, they bridge vision-language annotations with embodied scenes. For instruction understanding and latent action planning: (3) Command rewriting introduces instructional diversity through paraphrasing, attribute-based references and varied vocabulary. (4) Context creation generates implicit user goals or progress cues in multi-step tasks, requiring the robot infer intent. These annotations support joint VLA reasoning.
We use GPT-4o (OpenAl, 2023) to annotate data with three frames from each episode, along with the corresponding instruction. Ground-truth instruction is crucial for annotation accuracy. emphasizing that even state-of-the-arr VLMS can make errors in embodied tasks, leading to performance gap when using GPT-4o as an instruction interpreter for such tasks. Additional details of the dataset analysis and prompt templates are provided in Section D
4.2 SIMPLERENY-INSTRUCE
Building upon the SimplerEnv platform, we introduce SimplerEnv-Instruce benchmark specifi- cally designed to evaluate the instruction-folloiing and reasoning capabilities of rision-language action (VL.A) models zero-shot setting. Unlike prior manipulation benchmarks that primarily focus atomic actions low-level control, SimplerEnv-Instruv captures two essential yet un- derexplored abilities: (1) policy generalization to linguistic and visual diversity, and (2) contextual reasoning in situated environments, evaluated in the situated reasoning suite.
Task creation. We remove trivial cases and design novel tasks requiring genuine generalization rather than memorization. Novel objects and instructions are strictly out-of-distributio from the originals, and all tasks are cross-validated by three annotators for clarity and consistency. In total, we curated 80 tasks with 1.1K trials, about one third the size of SimplerEnv, keeping evaluation practical.
Task aggregation. (50 tasks; examples shown in Figure left). This suite assesses a model's s ability to consistently interpret and execute core tasks based on both instructions and environmental context, despite variations in visual or linguistic forms. Tasks cover phenomena such as novel verbs, multilingual expressions, diverse object references, sentence rephrasings, and OOD objects.
Situated reasoning. (30 tasks; examples shown in Figure 4 right). Beyond task aggregation, this suite evaluates model's ability to reason contextual cues or indirect instructions and to decompose commands into sub-goals. For example, "I want to clean the table. Pick a suitable tool for me." requires selecting the correct object (e.g., sponge) from context.
Together, by leveraging the large-scale real-world training dataset, SimplerEnv-Instee provides a reproducible benchmark that evaluates VLA generalization unseen tasks. It achieves an affordable evaluation cost while systematically probing both task generalization and reasoning, filling a critical gap in VLA evaluation with diagnostic, hummnninterppettblle, and standardized benchmark.

---

Published as a conference paper at ICLR 2026
Table Multimodal understanding. #Params is the size of LLM backbone. S. denotes robot state.
Methods
Muttimrdal Marteradal Undentaniing Benchnaress
VQA Berehmerks
M3HMU
MMd-V
OC3ticncc
[talll
MMS
ToctVAA
InseVQ
Chariq4
IWQQ
2026
43.4
45.4
1587.7
37.7
72.9
30.1
Pal
25
34.9
33.1
48.3
1686.1
6014
52.2
65.5
68.1
74.0
14.0
03.3
33.1
55.2
Bagle
122
1-40
03.1
6338
56.4
1572.1
515
45.8
74.9
79.
80.0
66.8
75.3
83.1
65.1
Qward
HD4
1:30
41.1
31.3
43.0
1872J)
41.7
34.0
24.9
1808
26.2
23.3
Opa
2024
0.0
0.0
Opca SAAAAA
79
26.0
9.1
28.2
67.6
255
5.4
18.9
2.5
20.2
43.4
35,8
1.4
47.2
BCuT
78
16.2
0.0
19.1
08
0.0
3,8
0.9
80
2.2
0.8
0,0
0.0
29.5
14352
71.2
81-5
67A
99-8
Mayma
1.58
44.2
51.3
56.3
1529
814
45.6
76.1
77.0
65.8
63.7
29.1
$1.3
63.1
1.5
43.8
5400
56.0
1548.0
829
42.8
76.3
78.2
86.0
63.7
76.9
82.9
63.5
Table 2: Robotic manipulation. Google and WidowX Robot denote two embodiments in SimplerEnv. For SapllerEn--Instrucc we focus on two reasoning levels instead of embodiments. Magma' denotes evaluation with sampling. The results of InstructVLA are averaged over three random seeds.
Google Rebot
WidooX Robet
Simpler EmeplerEEmiie Instruct
Mathoos
OgeniCkec Drawer
Drawer
Put
Coke Cun
Pick
Move Near
Put Spons
Put Canoe
Stzck Blocks
Avg
Tisk Aggrepaiion
Sinuttal Reasuning
Arg
VM
VM
VM
RE-1X
Cullabocation e
1023
19.4
0.0
26.8
MT-2-X
35.5
3.7
78.7
79.3
RoboVE Ai merieeesppp Amp
10.6
77.3
60.0
45.8
38.8
OpesVLA-7B
Kim
29.8
18.0
67.7
4.2
27.2
Mengere RDPm
72.7
16.7
29.2
45.9
13.2
11.7
45.8
(S)
37.5
41.7
12.1
1.4
(ntrertVVAA-ExpetttS)
82.4
Magnu-88
Yang
1005
46.0
45.8
30.5
15.5
Magma-BB
Tang
53.4
53.7
68.8
65.7
35.5
31.0
12.7
43.6
26.2
23.8
OpenVLA
42.6
3.7
62.3
$9.7
67.7
12.5
33.3
39,0
28.3
23.9
OpenVLA (FT&GPT)
38.8
InstrieVV A-Geneallet
IestrerVI GrGereali)))S)
18.244
45.5-+00.5
5 EXPERIMENT
Benchmarks. (a) Multimodal: We adopt automatic evaluation from VLMEvalKit (Duan et al., 2024). as detailed in Section E. (b) SimplerEnv: This benchmark (Li et al.) 2024d) provides cal--to-sim evaluation on large-scale manipulation datasets, incorporating visual matching and variance aggrega- tion to assess generalization. (c) SimplerEnn-nsstruct As described in Section 4.2 this extension of
SimplerEnv introduces novel objects, tasks, and instructions, offering broader testbed for evaluating instruction generalization in VLAs. In addition, assess embodied understanding in Section A.2
and manipulation performance on the LIBERO (Liu et al. 2024b) benchmark in Section A.3
Training details. The VLM is trained with resolution of 448 448 following Li et al. (2025c), while the action expert operates at 224 x 224 as in (Kim et al. 2024), using a fixed learning rate of 5e-5 without warm-up. The action expert employs 12-layer transformer backbone with a hidden size of 768. Following Black et al. 2024). a 3 distribution is used to enhance accuracy on the noisier time steps. During Stage 2 finetuning, manipulation and multimodal understanding are trained in an interleaved manner. Owing to our training paradigm, multimodal capabilities are preserved easily. We adopt a 1:7 multimodall-o-action training ratio, twice the ratio in ECoT and ChatVLA (1:3), reducing the additional computation needed to maintain multimodal ability. More details are in Section
Baselines. We categorize the baselines into three groups: (1) Multimodal VLMs, including Bunny (e
et al. 2024). PaliGemma (Beyer et al.) 2024), Eagle2 (Li et al. 2025c). and Qwen2- VL (Wang et al.
2024ch; (2) VLA models, including RT- 1-x and RT-2-X Coolabooratoon al. 2023). RoboVLMs (Li
et al. 2024c), SpatialVI.A (Qu et al. 2025) Tp (Black et al.) 2024). GROOT-NI.S (Bjorck et al.
2025), and OpenVL.A (Kim et al., (024); (3) Generalist VLA models, including Magma (Yang et al.
2025), OLe fine-tuned (FT) from generalist pretrained model on both robotic and multimodal
data, and ECoT(Bridge) (Zawalski et al. 2024). During evaluation, InstructVLA and other baselines temperature of ( without sampling to expedite generation. We re-evaluate Magma with official checkpoin For ECoT, we report only its multimodal results due to its real-to-sim domain gap.
WWe observe notable performance gain for Magma when using sampling. Accordingly, report its official score on SimplerEny and re-evaluate its performance on SimplerFan-Instrucc under the sampling setting.
7

---

Published as a conference paper at ICLR 2026
62
Pith fan
Irein
Tark
price
Pubal
Por
Figure 5: Real-world experiments. "Atomic" refers to atomic instructions. For the Kitchen and math settings, InstructVLA's responses are presented.
5.1
MAIN RESULTS
We present our main results and In Table using the same generalist model InstructVLA (generalist), it not only outperforms the co-trained baseline Magma, but is also comparable to its base model Eagle2 and Bunny (VLM data corpus). InstructVL.A further demonstrates stronger embodied understanding detailed in Section A.2 In Table InstructVLA (expert) outperforms the expert baseline SpatialVLA by 33.3% on SimplerEnv. Meanwhile, Instruct (generalist) not only maintains strong performance SimplerEnv's 's atomic instructions but also achieves 31.7% relative improvement on SimplerEnv-Instrr Instrucct over the state-of-ther -fhte--rrr baseline (OpenVLA with GPT-4o)..
2
However. we observe that finetuning OpenVLA on multimodal and manipulation datasets does not fully restore its original multimodal capabilities, although it does improve task performance, Its performance can be further enhanced by integrating GPT-4o as an API-based system-2 module to rephrase instructions (OpenVLL. VLA (FT&GPT)). However, GPT-4o faces the same challenges in accurate instruction rewriting as noted in Section and fails to outperform InstructVLA (Generalist). Methods such as Magma, which co-train both abilities of the VLM, better preserve multimodal ability, but still fail to match the performance of our approach. Although it also adapts two-stage training, ECoT relies solely on textual capability for multimodal question answering. We observe that it consistently generates manipulation- style CoT responses, without demonstrating effective instruction-following ability.
reasoning
manipulation datasets and lacks the
5.2 REAL-WORLD EXPERIMENTS
To evaluate InstructVLA in real-world scenarios, we conduct zero-shot experiments on the WidowX- 250 Arm and few-shot experiments on the Franka Research 3 robot, as shown in Figure [5] The few-shot tasks involve spatial pick-and-place from rack and cluttered tabletop setting and math- centric tasks detailed Section AA.3 to demonstrate the role of multimodal data. The zero-shot tasks set in kitchen environment following the Bridge dataset. InstructVLA is fine-tuned using the proposed training recipe, while VLe is jointly trained on atomic skill and VLA-IT datasets with extra language supervision. The To is finetuned using the official repository.
Each scenario includes both atomic and reasoning instructions. Atomic tasks emphasize in-domain objects and instructions with focus spatial generalization to assess baseline VLA capabilities. Both models perform comparably on direct in-domain instructions, but Instruct achieves a 23.3 3% improvement over Open VLA. For reasoning tasks such celebrity recognition, OCR, and tool-use inference, OpenVLA shows a substantial performance drop, whereas InstructVLAA outperforms it by 41.7% in few-shot and 46.7% zero-shot settings. On reasoning and math tasks, InstructVLA. achieves 2.5x improvement over To, which behaves close to random guessing. Additional ablations and experimental setups are provided Sections A.5 and
5.3 ABLATION STUDIES
We conduct ablation studies guided by two central questions: (1) Section How can manipulation and multimodal understanding be effectively integrated into a single model through architectural design and training strategies? (2) Section 5.3.2 To what extent does vision-language comprehension
8

---

Published as a conference paper at ICLR 2026
Table 3: Ablation of action expert vision design and language motion. "w/o Lang." denotes without using lan- guage motion. "w/o FiLM" denotes us- ing only DINO. "w/o DINO" denotes action expert without the vision input.
#Map Milver
Wireere Expe
Widdowd tcbet Google Fobot
Overal
IntructtA-eE
(A)
Exports
WidowX Bot
Google Bot
Ave.
Figure Finetuning strategies. (a) Freezing or finetun-
DINO
32.4
23.0
ing the action head during VLA-IT training. (b) Training
w/o FILM
25.0
56.3
45.9
Lang.
15.3
65.0
48.4
strategies when multimodal and manipulation tasks co-cxist.
InstructVLA
29.1
64.8
52,9
"FFT" denotes full finetuning. "AR" denotes auto-regressive.
Data Scaling SimplerEnv-Insrruct
Training and Inference Ablation CumplerEnv-Instruee
Average Task Completion
InstructVLA.Arg
45
OpeniiAAAvg Aug.
Task Agg
Situated
Average
40
Aps.
SitgkteB
30
25
20
15
0.00%
25.00%
50.00%
75.00%
100.00%
OpenVLA+
OpenVLA
Magma
Expert
Generalist
Generalist w
(a)
VL
VLA
(b)
w/oo Think
Think
Figure 7: Data scaling and multimodal training. Impact of scaling and training strategies manipulation with multimodal reasoning.
enhance manipulation performance in complex scenarios? Through targeted ablations, we examine the impact of key architectural and training decisions on these capabilities.
5.3.1 MULTIMODAL AND MANIPULATION CO-TRAINING
Strategies for multimodal and manipulation co-training. As shown in Figure (b)), four paradigms are compared. (1) Following Open VLA-OFF, FFT denotes full finetuning of the model with latent actions but without MoE adaptation and multi-stage training. With comparable computational resources, this setting yields suboptimal performance both manipulation and understanding tasks. (2) The AR paradigm (Magma, RT-2) supports co-training but has limited performance. (3) Removing the MoE design while keeping the training paradigm preserves multimodal performance but reduces manipulation capability. (4) In contrast, InstructVLA leverages our proposed architecture and two-stage training strategy, achieving 12.5% improvement over Magma SimplerEnv.
Language motion helps action understanding. As shown in Table introducing "language motion" (textual descriptions of low-level actions) supervision enhances the VLM's ability associate visual cues with manipulation primitives, leading to a 9.3% improvement in overall success rate.
Enhanced expert perception helps policy learning. Incorporating richer perception into the action expert is efficient due to its compact design compared to the VLM backbone. As shown in Table 3 while the base VLM offers general visual understanding, fine-grained perception for manipulation tasks demands richer representations. Removing the DINOv2-basee ViT encoder from the action expert results in 50.0% performance drop, highlighting its critical role capturing task-relevant visual cues. Incorporating FiL.M to the VIT encoder yields a further 15.3% improvement by modulating visual features with latent actions. As shown in Table the expert model with robot state generally performs better.
5.3.2 MULTIMODAL ABILITY TRANSFERS TO MANIPULATION
VLM-only instruction tuning handles situdated understanding. As shown in Figure 6 6(a),
examine the effect of VLA instruction tuning by comparing two configurations: (1) finetuning only the VLM, and (2) jointly finetuning both the VLM and the action expert. Freezing the action expert

---

Published as a conference paper at ICLR 2026
achieves performance comparable to joint finetuning while substantially reducing the number of trainable parameters. This suggests that InstructVLA can effectively adapt complex and situated manipulation tasks by fine-tuning only the VLM, without altering the pretrained action expert.
How VLA-IT scale and multimodal diversity affect reasoning-guided manipulation. As shown in Figure aa),, we first evaluate the scaling behavior of VLA-IT annotations on the SimplerEnv-Instren Instruct benchmark. Situated reasoning tasks, which require grounding objects and goals in context, benefit most from larger annotation sets, highlighting the bootstrapped reasoning abilities inherited from VLMs. In contrast, pretrained OpenVI.A fine-tuned on VI-A-IT gains primarily from increased instruction diversity but shows limited improvement on situated reasoning tasks due to catastrophic forgetting of VI. capabilities. These findings suggest that two-stage methods such as ECoT may be insufficient for fully leveraging the multimodal capacity of VLMs. We also examine the effeet annotation diversity, as shown in Table 4 where adding QA and captioning improves generalization of InstructVLA by 10.8%. Additional ablations are provided Section
Table 4: Effect of data dievrsity. "T.A." denotes task aggregation, and "S.R." denotes situated reason- ing on SimplerEnv-Instrue
QA & Cap.
T.A.
S.R.
Ave.
X
40.7
42.7
41.7
43.6
48.8
46.2
Ablating strategies for incorporating reasoning into manipulation. As shown in Figure (b), (1) Simply combining manipulation and general multimodal ability through co-training does not yield significant benefits. Magma, despite co-training multimodal datasets, shows limited transfer of vision-languaal capability to reasoning tasks on SimplerEnv Insttrruct. Although Open VLA suffers from catastrophic forgetting when finetuned with VLA-IT corpus, still achieves better performance than Magma. (2) Multimodal ability can implicitly benefit manipulation when preserved through embodied reasoning annotation. Our generalist model, trained on the VLA-IT corpus, surpasses fine-tuned VLe and Magma on the SimplerEnv-Instrue benchmark, even without explicit textual reasoning (Think). (3) Explicit textual reasoning further enhances manipulation. Enabling thinking in the generalist model brings a 36.1% performance gain over direct instruction execution and even outperforms InstructVLA-cxtr
paired with GPT-4o as an external interpreter. Further analysis of the role of thinking is presented in Section
CONCLUSION
We present InstructVLA, a unified VLA model that integrates multimodal reasoning and action generation. We further demonstrate how the embodied understanding ability directly benefit the maipulation tasks. Our data and training pipeline enables leading performance across manipulation tasks, multimodal benchmarks, and real-world deployments, paving the way for more generalizable, interpretable, and interactive robots.
ACKNOWLEDGMENTS
This work is funded in part by the National Key R&D Program of China (2022ZD0160201), Shang- hai Artificial Intelligence Laboratory, the Special Program of the Graduate School, University of Science and Technology of China, and Anhui Provincial Natural Science Foundation under Grant 2108085UD12. We acknowledge the research support from Shanghai Artificial Intelligence Labora- tory, including computational and hardware resources. We also acknowledge the support of GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC. The Al-driven experiments, simulations and model training were partly performed on the robotic AL-Scientist platform of Chinese Academy of Sciences.
REFERENCES
Figure AI. Helix, 2024. URL
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vision- lasiunnlanguage model with versatile abilities. arXiv preprint a30iv:2308. /2966, 2023.
10

---

Published as a conference paper at ICLR 2026
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65-72, 2005.
Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong. Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. Rt-h: Action hierarchies using language. arXiv preprint arXiv:2403.00 0000433.....,,, 2024.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024.
Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. GrOOt nl: An open foundation model for generalist humanoid robots. arXiv preprini arXiv:2503. 14734, 2025.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. Apig: A vision-language-acioon flow model for general robot control. arXiv preprint arXiv:2410.24166 2024.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068/ 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision- laiio-language-action models transfer web knowledge robotic control. arXiv preprini XXXi:2307./58/1 2023.
Qingwen Bu, Hongyang Li, Li Chen, Jisong Cai, Jia Zeng, Heming Cui, Maoqing Yao, and Qiao. Towards synergistic, generalized, and efficient dual-system for robotic manipulation. arXiv preprint arXiv:2410.08001, 2024.
Eric L Buehler and Markus J Buehler. X-lora: Mixture of low-rank adapter experts, a flexible framework for large language models with applications in protein mechanies and molecular design. APL Machine Learning, 2(2), 2024.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision- laiiunla models? arXiv preprint arXiv:2403. 203300 2024a.
Yi Chen, Yuying Ge, Yizhuo Li, Yixiao Ge, Mingyu Ding, Ying Shan, and Xihui Liu. Moto: Latent motion token as the bridging language for robot manipulation. arXiv preprint S4ti:2412.04444 2024b.
Zeren Chen, Zhelun Shi, Xiaoya Lu, Lehan He, Sucheng Qian, Hao Shu Fang, Zhenfei Yin, Wanli Ouyang, Jing Shao, Yu Qiao, al. Rh20t-p: A primitive-leved robotic dataset towards composable generalization agents. arXiv preprint arXiv:2403.1 19622, 2024c.
Open X-Embodiment Collaboration, Abby O' O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acor Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh, Animesh Garg. Aniruddha Kembhavi, Annie Xie. Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerien Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn. Guangwen Yang Guanzhi Wang, Hao Su, Sho-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen,
11

---

Published as a conference paper at ICLR 2026
Hiroki Furuta, Homanga Bharadhwaj, Homer Walke, Hongjie Fang. Huy Ha, Igor Mordatch, Iliaa Radosavovic, Isabel Leal, Jacky Liang, Jad abou-Chakra, Jaehyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jay Vakil, Jeannette Bohg, Jeffrey Bingham, Jeffrey Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Bymne Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsami, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng. Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding. Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" " Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart in-Mart in. Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonea, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shubham Tulsiani, Shuran Song. Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park. Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vikash Kumar, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang. Yao Lu, Yecheng Jason Ma Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang. Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho. Youngwoon Lee, Yuchen Cui, Yue Cao, Yuch-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open X- E-Eoodiment: Robotic learning datasets and RT-X models. https://arxiv.. wwwp:/iowoogggggoee wwtp://..wrg/b//////e// 08864 2023.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint Xrvv:22005 /4683, 2025a.
Shengliang Deng, Mi Yan, Songlin Wei, Haixin Ma, Yuxin Yang, Jiayi Chen, Zhiqi Zhang, Taoyu Yang, Xuheng Zhang, Heming Cui, et al. Graspvla: grasping foundation model pre-trained on billion-scale synthetic action data. arXiv preprint arXiv:2505.032.. 2025b.
Danny Driess, Jost Tobias Springenberg, Brian Ichter, Lili Yu, Adrian Li-Bell, Karl Pertsch, Allen Z. Ren, Homer Walke, Quan Vuong, Lucy Xiaoyang Shi, et al. Knowledge insulating vision-language- action models: Train fast, run fast, generalize better. arXiv preprint wxXiv:2505.23705 2025.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang. Pan Zhang, Jiaqi Wang, et al. VImevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 11198-11200 11011, 2024.
Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniliddis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross- datasets. arXiv preprint arXiv:2109. 13996 2021.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 14)-128-135, 1999.
12

---

Published as a conference paper at ICLR 2026
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. URL 13394.
Ning Gao, Yilun Chen, Shuai Yang. Xinyi Chen, Yang Tian, Hao Li, Haifeng Huang, Hanqing Wang. simulation for generalizable instruction-
Tai Wang, and Jiangmiao Pang. Genmanip: following manipulation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 12187-12198, 2025.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.0882 2021.
Jiayuan Gu, Sean Kirmani, Paul Wohlhart, Yao Lu, Montserrat Gonzalez Arenas, Kanishka Rao, Wenhao Yu, Chuyuan Fu, Keerthana Gopalakrishnan, Zhuo Xu, al. Rt-trajectory Robotic task generalization via hindsight trajectory sketches. arXiv preprint arXiv:2311.01977, 2023.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entan- gled language hallucination and visual illusion large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14375-14385 2024.
Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. Efficient multimodal learning from data-centric perspective. arXiv preprint arXiv:2402. //530, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. International Conference on Learning Representations, 2022.
Haifeng Huang, Xinyi Chen, Yilun Chen, Hao Li, Xiaoshen Han, Zehan Wang, Tai Wang, Jiangmiao Pang, and Zhou Zhao. Roboground: Robotic manipulation with grounded vision-language priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22540-22551 2025.
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. pi-0.5: vision-language-attoo model with open-world generalization. arXiv preprint urXiv:2504 1605,, 2025.
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually conditicod language models. In Forty-first International Conference on Machine Learning. 2024.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In European Conference on Computer Vision, pp. 235-251. Springer, 2016.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.1294 12945, 2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti. Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source laaion-lnggage-acttion model. arXiv preprint arXiv:2406. 09246, 2024.
Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and success. arXiv preprint urXiv:2502. 19645, 2025.
Hao Li, Shuai Yang, Yilun Chen, Yang Tian, Xiaoda Yang, Xinyi Chen, Hanqing Wang, Tai Wang, Feng Zhao, Dahua Lin, et al. Cronusvla: Transferring latent motion across time for multi-frami prediction in manipulation. arXiv preprint arXiv:2506, 13550669866666 /98/6, 2025a.
13

---

Published as a conference paper at ICLR 2026
Hao Li, Ziqin Wang, Zi-han Ding, Shuai Yang, Yilun Chen, Yang Tian, Xiaolin Hu, Tai Wang, Dahua Lin, Feng Zhao, et al. Robointer: A holistic intermediate representation suite towards robotic manipulation. arXiv preprint arXiv:2602.09973, 2026.
Qixiu Li, Yaobo Liang, Zeyu Wang. Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, et al. Cogact: A foundational vision- synergizing cognition and action in robotic manipulation. arXiv preprint arXiiv:4 1.69600, 2024a.
model for
Xiang Li, Cristina Mata, Jongwoo Park, Kumara Kahatapitiya, Yoo Sung Jang, Jinghuan Shang, Kanchana Ranasinghe, Ryan Burgert, Mu Cai. Yong Jae Lee, et al. Llara: Supercharging robot learning data for vision-language policy. arXiv preprint arXiv: 20095, 2024b.
Xinghang Li, Peiyan Li, Minghuan Liu, Dong Wang, Jirong Liu, Bingyi Kang, Xiao Ma, Tao Kong, Hanbo Zhang, and Huaping Liu. Towards generalist robot policies: What matters in building vision-language-action models. arXiv preprint arXiv:2412. 14058, 2024c.
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, et al. Evaluating real-world robot manipulation policies in simulation. arXiv preprint arXiv:2405..m 05941, 2024d.
Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Raymond Yu, Caelan Reed Garrett, Fabio Ramos, Dieter Fox, Anqi Li, et al. Hamster: Hierarchical action models for open-world robot manipulation. arXiv preprint arXiv:2502.00588 2025b.
Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang. Vibashan VS, Yishen Ji, Shiyi Lan, Hao Zhang, Yilin Zhao, Subhashree Radhakrishnan, et al. Eagle 2: Building post-training data strategies from scratch for frontier vision-languag models. arXiv preprint 2rXiv:250I.1488 2025c.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamn Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng. Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36, 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 36:34892-3444 2023.
Jiaming Liu, Mengzhen Liu, Zhenyu Wang, Lily Lee, Kaichen Zhou, Pengju An, Senqiao Yang, Renrui Zhang, Yandong Guo, and Shanghang Zhang. Robomamba: Multimodal state space model for efficient robot reasoning and manipulation. arXiv e-prints, pp. MeXiv-24000 2024c
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference Computer Vision, pp. 216 233.. Springer, 2024d.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng- Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(1)), December 2024e. ISSN 1869-1919. doi: 10.1007/s114333 1410077/11442--044425555
URL http: doi org/10.111/ 1007/s1
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2200-2209, 2021.
14

---

Published as a conference paper at ICLR 2026
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Emnest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 1697-1706, 2022.
Siyuan Mu and Sen Lin. A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications. arXiv preprint arXiv:2503.07137, 2025.
Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell, and Roci Herzig. Llarva: Vision-action instruction tuning enhances robot learning. arXiv preprint arvi:22406. 1815, 2024.
Dantong Niu, Yuvan Sharma, Haoru Xue, Giscard Biamby, Junyi Zhang, Ziteng Ji, Trevor Darrell, and Roei Herzig. Pre-training auto-regressive robotic models with 4d representations. arXiv preprint arXiv:2502.13142 2025.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, 2024.
OpenAI. Gpt-4 technical report. arXiv:2303.085 087774 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint urXiv:2304.07199 2023.
Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, et al. Transfer between modalities with metaqueries. arXiv preprint arXiv:2504.066 0625666...6.. 2025.
Kishore Papineni, Salim Roukos, Todd Ward. and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 2023.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Karl Pertseh, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-language-action models. arXiv preprint rxXiv22001.09747 2025.
Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, Jia Yuan Gu, Bin Zhao, Dong Wang. et al. Spatialvla: Exploring spatial representations for disual-lagguag action model. arXiv preprint arXiv:2501.15830 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. PMLR, 2021.
N Reimers. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084, 2019,
Yide Shentu, Philipp Wu, Aravind Rajeswaran, and Pieter Abbeel. From llms to actions: latent codes as bridges in hierarchical robot control. In 2024 JEEF/RSI International Conference on Intelligent Robots and Systems (IROS), pp. 8539-8546. IEEE, 2024.
Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, et al. Hi robot: Open- Open-ended instruction following with hierarchical langu-aa-agggatiionn models. arXiv preprint 2rXiv:2502.1 19417, 2025.
15

---

Published as a conference paper at ICLR 2026
Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1717-8118866 2019.
Qi Sun, Pengfei Hong, Tej Deep Pala, Vernon Toh, U Tan, Deepanway Ghosal, Soujanya Poria, et al. Emma-x: An embodied multimodal action model with grounded chain of thought and look-ahead spatial reasoning. arXiv preprint arXiv:2412. /2. 1974, 2024.
Mcituan LongCat Team, Bei Li, Bingye Lei, Bo Wang. Bolin Rong, Chao Wang, Chao Zhang, Chen Gao. Chen andf Zhang, Cheng Sun, et al. Longcat- technical report. arXiv preprint arXiv:2509.01322, 2025.
RealWorld Team. Realworldqa, 2024. URL
Yang Tian, Sizhe Yang, Jia Zeng. Ping Wang, Dahua Lin, Hao Dong, and Jiangmiao Pang. Pre- dictive inverse dynamics models are scalable learners for robotic manipulation. arXiv preprint arXiv:2412.15109, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397/ 2023.
Lirui Wang, Xinlei Chen, Jialiang Zhao, and Kaiming He. Scaling propriaceptive--iuut learning with heterogeneous pre-trained transformers. Advances in Neural Information Processing Systems, 37:124420-12446 124450, 2024a.
Lirui Wang, Jialiang Zhao, Yilun Du, Edward H Adelson, and Russ Tedrake. Poco: Policy composi- tion from and for heterogeneous robot learning. arXiv preprint arXiv:2402.02511 2024b.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing I Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing rision-language model's perception of the world at any resolution. arXiv preprint 14Xx:2249912191,, 2024c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-though prompting clicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Wei Wu, Fan Lu, Yunnan Wang, Shuai Yang, Shi Liu, Fangjing Wang. Qian Zhu, He Sun, Yong Wang, Shuailei Ma, et al. A pragmatic vla foundation model. arXiv preprint arXiv:2601. 18692, 2026.
Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, et al. Magma: foundation model for multimodal ai agents. arXiv preprint arXiv:2502. 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference Learning Representations, 2023.
Sconghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Wei Chao, Bill Yuchen Lin, et al. Latent action pretraining from videos. arXiv preprint 2rXiv:2410.11758. 2024.
James C Young, Rudy Arthur, and Hywel TP Williams. Cider: Context sensitive sentiment analysis for short-form text. arXiv preprint arXiv:2307 078640 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang. Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang. Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the JEÉE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
16

---

Published as a conference paper at ICLR 2026
Michat Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of of-uuuught reasoning. arXiv preprint arXiv:2407.08693, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11975-11986. 2023.
Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, al. Cot-vla: Visual thain-of-thought reasoning for vision- language-action models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1702-1713, 2025.
Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-griieed bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023.
Jinliang Zheng, Jianxiong Li, Dongxiu Liu, Yinan Zheng, Zhihao Wang, Zhonghong Ou, Yu Liu, Jingjing Liu, Ya-Qin Zhang, and Xianyuan Zhan. Universal actions for enhanced embodied foundation models, 2025. URL
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuue images with one multi-modal model. arXiv preprint arXiv:2408.11033 2024.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoe V Le, James Laudon, et al. -f with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114, 2022.
Zhongyi Zhou, Yichen Zhu, Minjie Zhu, Junjie Wen, Ning Liu, Zhiyuan Xu, Weibin Meng, Ran Cheng, Yaxin Peng, Chaomin Shen, et al. Chatvla: Unified multimodal understanding and robot control with rision-language model. arXiv preprint a502 14420, 2025.
17

---

Published as a conference paper at ICLR 2026
Appendix
CONTENTS
More Experiments and Analysis
20
Further Discussions
20
A.1.1
Extra Model Design Analysis
20
1.2
Extra Reasoning-Manipulaiii Manipulation Analysis
21
A.3
Extra Interence and Training analysis
22
A.2
Embodied Understanding Evaluation
24
A.3
Extra Manipulation Benchmark
26
Data Ablation on Open
26
A.5
Real-world Ablation
27
Extra Related Works
28
B.
Embodied Instruction Tuning
28
B.2
Multi-stage Training
28
Case Study
29
Reasoning Cases in SimplerEnv-Instruee
29
Failure Cases
30
C.3 GPT4o as the Auxiliary System
31
Data Annotation Details and Analysis
32
D.I
Language motion pre-training data
32
D.2
Task Diversity Analysis
32
D.3 Prompting
33
D.4 Ground Truth Instruction for Data annotation
35
Benchmark Details
38
Multimodal
38
SimplerFnn-Insttuu
38
F
Model Design and Training Details
41
Instruction Format
41
F2
Moł Adaptation
42
Learning Objective and Inference Procedure
42
Model Parameters
43
F5
Inference Speed
44
F66
Experiments Compute Resources
44
18

---

Published as a conference paper at ICLR 2026
Multimodal Examples
45
Real-world Experiments Sctup and Analysis
46
Broader Impacts and Future Work
48
Limitation
48
I.2
LLM Usage Statement
48
I.3 Broader Impacts
48
Future Work
48
The supplementary material is organized as follows:
Section presents: (1) extended analysis of InstructVLA, (2) additional benchmarks on embodied understanding, (3) extra simulation benchmark and ablation study, (4) finetuning of Open under the same settings InstructVLA, and (5) extra real-world ablation study.
Section B discusses related concepts to InstructVLA and the proposed vision-language-aciion instruction tuning methods.
Section provides additional case analysis for InstructVLA, OpenVLA, and GPT-4o System2.
Section D lists data annotation details, including GPT-4o prompt and dataset statistics. We further analyse the distribution of the instructions from two dimensions: task diversity and language diversity.
Section visualizes the CraplerEnv-Instrunn benchmark and the acknowledgements of 3D assets.
Section details the model architecture, training configurations, inference speeds under different settings, and compute resources used.
Section G shows several multimodal question answering examples.
Section describes the real-world experimental setup and provides example executions.
Section discusses the broader impacts, limitations, and outlines future directions for InstructVLA.
19

---

Published as a conference paper at ICLR 2026
A MORE EXPERIMENTS AND ANALYSIS
A. FURTHER DISCUSSIONS
Our further analysis is threefold. First, we present visualizations and scaling curves to examine the MoE and latent action designs. Second, we provide a detailed analysis of reasoning gains in manipulation tasks and case studies. Finally, we demonstrate that InstructVLA supports zero-shot dual-frequency generation accelerate inference and compare the dataset scales used across different studies.
A.1.1 EXTRA MODEL DESIGN ANALYSIS
The MoE and latent action are our key design components. We present an example illustrating the role of MoF under different task settings, including simple and reasoning instructions, with and without model reasoning. For latent action, we analyze its scaling behavior to guide future tuning.
System Prompt
Image Tokens
Langunge Tokens
Lalent Action
Scalin Scatik
Execating simple instruction without ressoming
Reasoning Results
0.00
Executing simple instruction with ressoning
-022
Exccuting complex
Reasoning Results
1.00
Exccuting complex instruction with reasoning
Figure 8: Activation visualization. We evaluate a WidowX zero-shot example across four settings. Red indicates stronger activation in the language adapter, while blue indicates stronger activation in the action adapter. The horizontal axis lists each language token. The generated tokens are marked.
Analysis of MoE gating, From the example in Figure g we draw the following intuitive conclusions:
System prompts are primarily processed by the language adapter, reflecting its close connection to pretraining.
Visual information is processed by both the language and action adapters, indicating that both semantic understanding and manipulation decision-makee require visual inputs.
During language generation, the model engages not only in multimodal reasoning but also in manipulation planning, as evidenced by the activation of the action expert. Notably, the action expert attends more strongly to nouns and verbs in the generated tokens, highlighting its role in instruction following.
During latent action generation, the language expert plays less prominent role. Instead, with multimodal reasoning, the model concentrates more effectively on action generation, as shown by the stronger activation of the action expert (deeper blue).
To conclude, the MoE has demonstrated its effectiveness in improving efficiency and handling heterogeneous datasets (Mu & Lin 2025) Buchler & Buchler [2044] Zhou et al. 2022 Team et al.) 2025 Liu et al., 2024a). In Instruct VLA, we further investigate how the MoE facilitates interleaved multimodal reasoning and manipulation decision making.
Effects of latent action. Latent action tokens are a key design component for decoupling high-level VLM planning from low-level action generation. As shown in Figure we vary the number of tokens from 16 to 128. Too few tokens limit behavioral diversity, while too many reduce training efficiency. setting of 64 offers good trade-off under our current configuration.
20

---

Published as a conference paper at ICLR 2026
Average Task Completion
70
60
30
20
Widowx Robot
Google Overall
16
32
64
128
Figure 9: Impact of latent action token quantity on robot performance.
Sabtokk
Commonmenne
Commorsense Toal-sse
86
cous Mee then
owen thver
snei cant gali
crei aree gone jur con gin
Pst
Ove
Figure 10: Performance visualization of 30 situated reasoning tasks with and without reasoning enabled. Activating reasoning in our generalist model generally improves performance. For clarity, tasks are grouped into three categories: Subtask, involving subtask identification; Commonsense Reasoning, requiring broad world knowledge; and Commonsense for Tool Use, focusing on tool- related reasoning.
A.1.2 EXTRA GEASONING-MANIPULAANN ANALYSIS
In this section, we discuss the efficiency and design choices of VLA-IT training. We then analyze how multimodal reasoning benefits manipulation through fine-grained evaluation, examine its role in coos-emmboimeett generalization, and present case study illustrating how unique multimodal capability addresses challenging tasks.
Effect of VLA-IT on Scaling and Reasoning. As shown in Table although the Instruct, which model does not outperform the Open VLA(OE) on Situated Reasoning of SimplerEnv benefits from direct full fine-tuning of the VLM backbone, Instruet VeA-expert shows promising scaling ability in understanding complex instructions and performing test-time thinking after stage-2 VLA-IT training. This result reflects a deliberate design choice in InstructVLLA, where latent action learning during pretraining focuses on querying from visual and simple instruction features rather than relying on the full semantic space of the VLM too early. This design offers two significant advantages. First, it preserves the original semantic space of the pretrained VLM, maintaining its vision-language capabilities. Second, enables the model to integrate diverse reasoning contexts during VLA-IT training. These properties contribute to the strong performance gains achieved by our generalist model and demonstrate effectiveness of this training paradigm.
Embodied reasoning helps manipulation. Allowing the model to perform test-time thinking by generating textual analysis of the given instruction can improve performance, particularly on situated reasoning tasks, shown in Figure II (left). Notably, while the model with access to robot state outperforms the one without state when no instruction response required, provides limited performance gains when instruction following is involved. We hypothesize that state information helps the model retain manipulation skills but compromises its generalization to OOD environments and instructions.
Fine-grained analysis of reasoning gains in manipulation tasks. We compare the performance of the generalist model on SimplerEnv-Instrun with and without vision language reasoning, as
21

---

Published as a conference paper at ICLR 2026
shown in Figure 10 A clear performance gap emerges in tasks involving commonsense tool use and interaction with articulated objects. This may result from instructions that do not explicitly state the intended actions and objects. For example, retrieving cleaning tool from a drawer requires the robot to infer whether the prerequisite of an open drawer satisfied, and to identify the sponge as the appropriate tool among several options. In addition these cases, the reasoning process also improves performance on other situated reasoning tasks by grounding unfamiliar instructions using the pretrained in-domain knowledge of the vision language model.
72
a Sturde Prundcaanzky aTuutAgrapson
Conral
mean
hekin rean dineraai
33
Overal Eigen femerall
Ounanibit
Conneilt "snirg
Test-time tinking and dual-frequency evaluation. "Expert" refers the model after action pretraining, while "Generalist" denotes the model after VLA-IT tuning. For dual Sual-frequency evaluation, the horizontal axis represents the ratio of VLM executions expert model executions.
Figure 11:
VLA instruction tuning for cross-embodiment understanding. To assess whether Ins retains this capability, we evaluate three variants SimplerEnv-Istrruee (see Table 5): InstructVLA- Expert, trained solely on atomic instructions without test-time thinking; InstructVI.A Generalist (Bridge), trained with the VI-AIIT dataset Bridge and the original Fractal dataset; and InstructVI.A Generalist, trained with the full VLA-IT datasets across both environments. Adding the Bridge dataset results 139,4% improvement in Situated Reasoning performance for Generalist (Bridge) over the expert baseline, while task aggregation performance remains comparable. This discrepancy reflects differing generalization requirements: task aggregation emphasizes linguistic robustness, whereas Situated Reasoning demands vision- grounding prior to action. The latter particularly benefits from the preserved reasoning capabilities of the pretrained VLM. As illustrated Figure [2] the shoo model generates more diverse and accurate outputs than its fine-tuned counterpart.
Table 5: Instruction tuning data ablation. We evaluate three settings: without VLA-IT data, with data only Bridge, and with VLA-IT data on both Fractal and Bridge. This ablation examines the contribution of the VLA-IT dataset and the cross-embodiment generalization of InstructVL.A on SimplerEnv-Instrenn
Instruction Tuning Data Bridge Fractal
Name
Task Aggregation
Situated Reasoning
Overall
X
Expert Generalist (Bridge)
20.8
10.4
15.6
X
18.4
24.9
21.7
/
Generalist
43.3
48.8
46.0
Case study multimodal capability transfer. As shown in Figure 13 compare InstructVL.A with OpenVLA (Kim et al., 2024), Magma (Yang et al. (025). and CogACT (Li et al. 2024a), all using the same input (language instruction and a single image). InstrutVTLA-Expert though trained without multimodal and VLA-IT datasets, retains the OCR capability of the underlying VLM and achieves the best performance among baselines trained solely on manipulation data. Finetuning
with multimodal and VLA-IT datasets further enhances performance. For autoregressive models such as VLe and Magma, multimodal finetuning improves OCR ability. In contrast, CogACT, when fine-tuned from Open VLA(OXE) only on manipulation data with an action head, shows improved in-domain performance (on SimplerEnv) but suffers in generalization.
into SiaruuttVLA-Geneeee
A.1.3 EXTRA INFERENCE AND TRAINING ANALYSIS
Dual frequency inference. To further analyze the relationship between latent actions generated by the VLM and the final decoded actions, decouple the inference frequencies of the VLM and the
22

---

Published as a conference paper at ICLR 2026
something cold can you open fridge?
Anewer Opening fridge now comehhing refreshing.
Queetion: Plazee clock inside the fridge for any smacks.
Answer: I will the frägge what stacks available.
Question: need check the food ready the Please open for
open the
Queetion: waat take
from over Please open
SimplerEnv-Isttuet Rollout Google Robot
Anewer. open the over doer.
Bridge Instructions
Zero-shot
Fine- tuned
Inzd retrivve some stessis from the betlorn drwver
nead retrivve some stenells Eres the battom drawer
opea botomm
curendly Therefore, open bottom
start by opening the bottonn drower.
butleen is curentlly closed, open bottom
will open the drower find the ulenils.
The botton drewer is curienlly closed. I will open the botlom drower to aceess the vienaik.
will start by opening the botton drawer.
will by opening the bottom drowr the utensils.
open
Case study on cross-embodiment. Top left: rollouts on SimplerEny-nssrruct Top right: similar scenarios from the Bridge dataset with corresponding instructions. Bottom left: zero-shot results trained only on Bridge instructions. Bottom right: rollouts from the fine-tuned model.
Figure 12:
PlaceLNMe Vear
Place Near
SimplerBrn
OpenVtAVVL
Magna
ImbructAA-bppet
InetruetLAAGGnm Geneeddt--
Figure 13: Case study on multimodal capabilities. OCR represents a unique multimodal skill of VLMs that is absent from typical manipulation datasets. We evaluate two tasks from the task aggregation set SimplerEnv- ErnpterEn--Instruet, involving moving one letter to another (see Figure [8] 1)). By comparing different finetuning paradigms, assess how effectively multimodal capabilities integrated into VI.A models.
action expert, as illustrated in Figure right. The results show that performance remains stable at 1:2 ratio (VLM:expert), but begins to degrade at higher ratios. This suggests that latent actions offer relatively stable guidance to the action expert, reducing the need for frequent VLM queries.
Training at scale. A generalist VIAA model with vision-language capabilities should be scalable across both manipulation and multimodal datasets. this context, we compare datasets used by models claiming generalist abilities, as shown in Table RoboMamba (Liu et al., 2024c) utilizes a limited manipulation dataset compared to other methods, while the dataset for ChatVLA (Zhou al. (025)) is not reported. Wu.s (Intelligence al. 2025) employs significantly larger multimodal dataset than other approaches, though its multimodal performance is not disclosed. Magma uses more robot and multimodal data but achieves slightly worse performance both multimodal and manipulation benchmarks compared to InstructVLA.
Table 6: Data comparison of different methods. "Trans." denotes transitions.
Mapma Yung 2025
RobuMantt Liu
Rea Inselligncee
InstustLA
Manipolaton
Images
23

---

Published as a conference paper at ICLR 2026
A.2 EMBODIED UNDERSTANDING EVALUATION
Table 7: VLA-IT captioning evaluation. "Sentence- BERece-B-TTT and "SimCSE" represent learning- based evaluation methods, while the remaining metrics traditional n-gram-base based evaluations focused on word distribution.
Methods
Params
Sentence-BERT
SimCSE
BLEU-1
BLEU-4
METEOR
CIDER
Qwen2-VL Wang et al. 20246
1.5B
61.3
67.5
16.8
1.5
12.4
0.30
GPT4o OpenAl 2025
60.7
67.1
16.3
16.2
0.09
OpenVLA/VVLAII Kim
2024
7B
0.0
0.0
0.0
0.0
0.00
Magma Yang al. 2755
8B
59.8
66.7
12.4
1.2
12.3
0.12
Instma(VLAAGeeennlln
1.5B
72.0
77.0
44.3
8.2
18.7
0.84
Table 8: VLA-IT question-answering evaluation.
Params
Sentence-BERT
SimCSB
BLEU-1
BLEU-4
METEOR
CIDER
Qwen2 VL. Wang et
2024c
1.5B
53.4
15.3
0.82
GPT4o OpenAL 2023
63.6
63.6
29.6
19.9
9.8
1.16
LAAVIA Kim et al.
2024
7B
0.0
0.0
0.0
0.0
0.0
0.00
Magma Yang et al.
2025
8B
53.5
54.5
23.7
5.7
21.6
1.04
VI A(Generalist)
1.5B
65.9
Table 9: VLA-IT instruction response evaluation. We use "context creation" annotations, as they present a more challenging and diverse set of instructions.
Methods
Params
Sentence-BERR
SimCSE
BLEU-1
BLEU-4
METEOR
CIDER
Qwen2-VL Wang ct al.
2024c
1.5B
52.3
54.0
5.6
1.5
11.6
0.09
GPT4o OPenAA 2025
52.8
54.1
17.8
20.6
1.02
OpenVLA/VLA.ID Kim al.
2024
7B
0.0
0.0
0.0
0.0
0.00
Magma Yang al. 2TA
8B
10.9
13.6
3.7
0.8
1.6
0.00
MasmectVAA/enneralll
1.5B
71.6
73.1
50.2
24.1
25.8
2.26
In addition the multimodal and closed-loop evaluations presented in the main results, we conduct supplementary language evaluations on the proposed VLA-IT dataset. This evaluation uses manually verified VLA-IT annotations on the Bridge dataset (Ebert et al.] (021), chosen for its diversity and distinct validation split. We generate 1,000 annotations following the method described in
the VLA-IT dataset generation section. Two evaluation metrics are employed: (1) learning- based
methods (Reimers) 2019 Gao et al. [2021), and (2) traditional metrics (Papineni et al. 2002 Young ctt al.
2023 Banerjee & Lavie 2005).
The captioning, question-answeriigg and instruction results presented in Tables 7 to We select Qwen2-VL (Wang et al. 2024c) and GPT-4o (OpenAl) (023) as zero-shot VLM baselines, and include Magma (Yang et al.) 2025) (zero-shot) and Open VLA (Kim et al. 2024) fine-tuned on
the VLA-IT dataset as baselines for VLA models.
Although Open VLA is fine-tuned on the VLA-IT dataset, it fails to generate complete sentences under the same evaluation setting as InstructVI.A, despite the performance on multiple-choice benchmarks reported in our main results. This suggests significant loss of its free-form dialogue capability. Magma performs well on question answering and captioning tasks. However, it struggles with instruction response (Figure 14), often generating outputs misaligned with the given image. We hypothesize that this failure stems from the similarity between these instructions and the atomic commands used in finetuning manipulation datasets, which disrupts the coherence of the language latent spuce near the action latent space. This suggests limited capacity to interpret and generalize free-form instructions, hindering effective transfer of vasion-language capabilities.
InstructVI.A achieves late-offthe-art performance, while GPT4o demonstrates competitive results. We visualize three episodes in Figure 15 GPT-4o generates more detailed captions but occasionally exhibits minor hallucinations. In the instruction response task, InstructVLA produces clearer and more grounded responses compared to GPT-4o, benefiting from the integration of ground-truth atomic instructions during the data annotation process, discussed in Section D.4
24

---

Published as a conference paper at ICLR 2026
Which fruit curved and yallow? Bunana
QA
Make sure the banano occessible by pripper for the nost ludtu uow www.or/ueeaaaa uubuaa/uuttsa uuiuzee
Instruction
lue422 juek sar jur (uessa uddaa subabd (uadsoiuucc Flueus (ube29 (.6222
Check the contents of the brown box openinng flpp.
Instruction
When you open the flep of the brown hos, you oa find net tnihhen unwenllis indide The utrends may include tremcouuhh aeaa spatubs, a ladle, a gronee and other torn dee groed for cooting ng or food preparation on. These steneih are typically mace materta liee stannisos steel, silioon, or plantic, and they are often used together to perform specifie tasks the kirchen. The presamoe of these designed for easy accoss and organiation commoniy
utensil in the box supgesta that it used kischan took.
what color are the handles of the screwdrive?? The aand the screwditver are red.
QA
Orgacior the bonk by pirking the namnrrw Allen bey net finst yube yurrsaat touuu www.tttdyyooooooooooooo toduuuuo direilusttefl ait.rutteeeetttllllllll. unteeattch direelyuRDNn direnly www.ee/eitttteeeeeeeee
Instruction
Figure 14: Magma results. Magma's responses collapse when given instructions resembling those in its manipulation tasks, possibly due learned actions interfering with its language latent space.
Alaundry whise basket connaiiing dorvee wad westing
the sable, sosting rachins door open does drums
ed efforen colored favves ind blko, and cedi curchppeee dieped owc edoe of booker
Pleas prepare the leandry for weshing by putting in machins
contaiir ssed for holding the dith pleces on the ground? basbet.
The cantaine nur used hold eg the cash places ground typically beaket.
rod tocboo filed with verious tocks, inck dling srewerken and weenn has, on and and sorewdriver
The table cowere with vuriout tocks Induddng conwedtien with difirore tandins, hax ofll bis, and other astored ha took. The toot organine wetton wiitte drawne
We need
taut
proding the cotorial keys
will start by cking the rantic allen ves snt
Pick riedow Alles key set snorge a neats onvoee side the drower, then suriee
rokor handlies sonsedrkee?? The sorwerkerr hur red and welkow
The hondles ofeeee scendiieers are yellow with and accenss
Axoddee with sni caange, rerrod pet connsinar spices, and pooped
the rakle, meri ploced upa down, yallow sporge sggnne erribbing
engped
eevendi the the table The wnoden
Before we cook on recure the por sportees by uning sponge? wips with sponge.
dork purple and white died as
loing
What object used for dewning sooee?
the pollor sponge
yut on sponge os for dounrng in scone.
InstruciVAA
(5) GPTR
Figure 15: Comparison with GPT-4o. We visualize three examples from the VLA-IT language validation set. Each example includes scenario caption (top), instruction response (middle), and question answering (bottom). The GPT-4o column displays responses only, as the instructions are identical across models.
25

---

Published as a conference paper at ICLR 2026
A.3 EXTRA MANIPULATION BENCHMARK
Table 10: LIBERO benchmark results. We present the success rate and standard error for each method across four task suites, which are averaged over three random seeds with 500 trials. "KI" denotes knowledge insulating Driess et al. 2025).
Sputial
Object
Goal
10 (Long)
Average
OpenVLAA-2 7B Kim al. 2024
84.7 + 0.9
88.4 + 0.8
79.2
1.0
53.7
1.3
76.5
0.6
OpenVLA-OFF OFT-/B Ixa Luneera ara. 2025
97.6::0.9
98.4
0.8
97.9
1.0
94.5
1.3
97.1
0.6
SpatilVLA-2B et 2025
0.5
89.9 0.7
78.6
55.5
78.1
0.7
2B (Black
96.8 + 0.8
98.8 + 0.9
95.8 1.1
85.2 43.2+110
94.2
0.9
3g-FAST-2B (Perisch et 2025
06.4
96.8 = 0.7
88.6 1.0
60.2 1.4
85.5
1.0
CoT-VVLA /hio al
0.5
87.6
69.0
81.1
GROOT-NI-1.344 BBjorcked 2025
94.4
1.0
93.0
90.6
93.9
KI (from seraten) Intellgeece al 2025
96.6
97.2
94.6
84.8
93.3
To.6 Kl (from generalist modelj (nnellignnce et al. 2025
98.0
97.8
95.6
85.8
94.3
InstruetVLAA (w/n wrist view)
92.4
95.6
92.0
76.6
89.2
Inst1etVLA-1.5B
0.5
99.6
0.0
96.5
0.5
89.8
95.8
Benchmarks and baselines. We evaluate InstructVLA on the LIBERO simulation benchmark (Liu ct al. 2024b), which includes diverse robotic manipulation tasks in simulated environments. Follow- ing Open VLA (Kim et al., 2024), we conduct experiments four task suites, each containing 10 tasks with 50 human-teleoperated demonstrations. These suites assess spatial reasoning (LIBERO- Spatial), object type understanding (LIBERO-Object) task-oriented behaviors (LIBERO-Goal), and generalization to long-horizon tasks involving diverse objects, layouts, and goals (LIBERO-Long).
Our baselines fall into categories: (i) generalist manipulation policies, including OpenVLA (Kim) et al. 2024). Open VLA-OFF (Kim et al. 2025). SpatialVLA (Qu et al. (025),, To Black et al. (2024). and Too -T- FAST(Pertschh GROOT-NI (Bjorck et al. 2025), and To.5 [Iteelligeec et al. (2225 with knowledge insulation DDriess 2025)
2025): and (ii) manipulation policies with multimodal ability, including
al.
Training details. We augment InstructVLA with wrist-view images from the LIBERO training set (Liu al. (2024b). Specifically, both the main and wrist-view images are provided the VLM and the action expert. To reduce the tokenized input length, the two images are concatenated and resized into a single frame for VLM. Training follows the same hyperparameters experiments and is performed with evaluation every 1.5K steps.
the Simpler- single A800 node with 8 GPUs using global batch size of 256,
Results. As shown in Table 10 InstructVLA achieves competitive performance despite not being pretrained on large-scale manipulation datasets like Tu.s (Intelligence et al.) 2022 Driess et al. 2025) and using much smaller VLM backbone than OpenVLA OFT(Kim et al. 2025). Compared with recent VLAs such as wi InstructVLA attains higher performance with a substantially smaller action model (134M versus 300M).
A.4 DATA ABLATION ON OPENVLA
Table 11: Data ablation on Vpen "+VL" indicates finetuning Open with the same multimodal dataset used by InstructVLA. "+VLA-IT" refers to finetuning Open with the same VLA-IT dataset as InstructVLA. "4GPT40" denotes using GPT4o as system translate free-form instructions into atomic ones.
OpenVI.A (OXF) OpenVT.A VI.
OpenVI.A VI. VLA-IT
OpenVI.A VI. GPT4p
InstruerVI.A
Task Aggregation 14.8
28.3
30.5
38.8
43.3
Situated Reasoning
32.4
Average
14.2
23.9
24.0
35.6
46.0
To investigate whether the performance gain of VLA-IT arises solely from the dataset itself, we reimplement the training procedure of the InstructVLA OpenVLA (Kim al. 2024), which represents a class of models trained under the action-only paradigm. As shown in Table II Open VLA benefits from both vision-language and VLA instruction tuning data, with the latter showing greater
26

---

Published as a conference paper at ICLR 2026
improvement in the task aggregation setting. This is attributed to exposure to more diverse instructions. However, performance on the situated reasoning setting remains unchanged, likely due to catastrophic forgetting caused by the action-only training paradigm, which limits OpenVLA's ability to leverage the VLM's reasoning ability through simple finetuning.
The greatest performance gain is observed when GPT-4o introduced auxiliary System 2 in both evaluation settings. However, overall performance remains inferior to InstructVLA, as GPT-4o cannot fully ground free-form instructions the atomic skills which VLA
VLA is pretrained.
A.5 REAL-WORLD ABLATION
Answorthe the question stop by
is 3/0
18 al
18
$2 22
$2 H2
ESHAR 26, the are
lall
2. 9C
10-20
te Has
LA SE Re
8+0
Beyond Collection Range
Writtan-out Numbers
Maschatick N bumbers
Hard Case
Hard Clase 2
Figure 16: Real-world ablation study, The first row depicts the reasoning responses and the rolled- out actions, while the second row illustrates five categories of generalization.
Grara Falldd
Fallure
17 6
shote 4a..|
++0 +.9
The probiem 8 The cornect answer green
8, aprions are
will pick red cube. w/o multimodal data traiiininng
Success
The problem 6 m. cornect
The problem is correct
oot ons yellow
will
yelow multimodal data co training
Figure 17: Reasoning examples. Two evaluation cases are presented to illustrate the role of multimodal datasets. We further summarize the results of InstructVILA in Sankey diagram.
Setup. This case study evaluates the role of multimodal datasets in manipulation tasks, The robot setup follows our few-shot Frank evaluation. As shown in Figures and and 17 the model must first perform OCR to recognize the formula on the board and its answer options, then compute the result, and finally control the robot to grasp the correct object. This task mirrors shopping scenario where robots often need to read prices and perform simple caleulations to satisfy a requirement. The study jointly assesses OCR and calculation abilities, which are expected to benefit from multimodal data. To reduce bias, each case is evaluated three times with different target objects. In total, 250 training cases collected but excluded from evaluation.
The in-domain tasks are defined as calculations within the range of the training data and written in similar format. Generalization tasks are divided into five types: (1) Beyond Collection Range, (2) Written-out Numbers, (3) Matchstick Numbers, (4) Hard Case (digits partially occluded with superimposed lines), and (5) Hard Case 2 (involving more complex calculations).
27

---

Published as a conference paper at ICLR 2026
Analysis. By co-training with general multimodal dataset, observe that InstructVLA performs better on the tasks of Written-on out Numbers, Matchstick Numbers, and Hard Case 1. We attribute this improvement to the inclusion of general OCR data within the multimodal dataset. Although the multimodal dataset unfiltered (i.e., identical to the corpus used for training a VIM such as Bunny), it nonetheless enhances the instruction generalization for these specific tasks.
The SOTA VLA To (Black et al., 2024), although pretrained on DROID Khazatsky et al. (2024), however, produces near-random results: although each grasp is executed precisely, the model frequently selects the wrong target object. Interestingly, when the third-view camera, which capturing the board with expressions and options. masked, To still behaves similarly. This suggests that To largely ignores reasoning cues and overfits to the wrist view. While it performs precise grasping, the overall outcomes remain unsatisfactory.
B EXTRA RELATED WORKS
In this section, we delineate the distinctions between InstructVLA and several similarly named methods that differ substantially in their conceptual foundations and objectives.
B. EMBODIED INSTRUCTION TUNING
Vision-Action Instruction Tuning. The concept of Vision-Action Instruction Tuning is introduced in LLARVA (Niu et al. 2024), which unifies robotic tasks through structured prompts and 2D trace supervision for cross-embodiment pretraining. In contrast, InstructVLA extends this idea by focusing on preserving the multimodal knowledge of VLMs and bridging high-level human instructions with low-level manipulation skills, enabling generalization to diverse tasks that require common-sense reasoning.
Visuomotor Instruction Tuning. The concept of Visuomotor Instruction Tuning is purposed in LLaRA (Li et al. 2024b). This approach formulates robot policies visuo-textual conversations and produces 2D keypoints and rotations for manipulation. However, it functions primarily as a high-level planner, and its outputs require additional adaptation before being directly executed on robots.
B.2 MULTI-STAGE TRAINING
VLA-FFA- Open (Kim et al. 2025) extends Open (Kim et al., 2024) by incorporating FiL.M layers, Parallel decoding, MLP action head, and has been applied to fine-tuning on smaller simulation datasets such as LIBERO Liu et al. (2024b). This approach demonstrates the effectiveness of architectural enhancements for improving manipulation performance in specific domains. However, while these techniques improve in-domain performance, they fall short in reasoning-centrr settings such as SimplerEnv-Instruce as shown in Figure 6 (b). In contrast, our work moves beyond architectural modifications by emphasizing generalizable manipulation with textual reasoning through MoE adaptation, latent action methods, and a comprehensive data and evaluation pipeline. With the proposed VIA-ITT' training paradigm, our generalist model achieves nearly a 2 improvement over models that rely solely on architectural designs.
Embodied Otain-of-Thoughtt ECoT (Zawalski al. 2024) introduces chain-of-thoure (CoT) supervision to link reasoning with manipulation and follows standard "pretrain- sntrrrainnthe---nssuuttttt---t tuune"" paradigm. However, it relies full-model pretraining fine-tuning, as Open (Kim et al., 2024) which leads to catastrophic forgetting of vision- language capabilities. In contrast, InstructV LA adopts a two-stage design: the first stage injects action- iitionngeneration ability while deliberately preserving the multimodal knowledge of the pretrained VLM. This approach ensures that the model retains open-world understanding and general multimodal reasoning, both of which are largely lost in ECoT. The second stage then strengthens multimodal reasoning and manipulation alignment. Consequently, InstructVLA supports broader inference modes (reasoning + manipulation, direct manipulation, and multimodal VQA) and achieves stronger performance with substantially fewer trainable parameters.
Visual Chain-of- Thoughh.. CoT-VLA (Zhao et al. 2025) enhances manipulation by generating future image frames visual chain-of of sifin-ff-hhuuse thought goals before predicting actions. While effective for goal specification, this approach relies on heavy rideo-generatie supervision and does not exploit strong VLM pretraining for vangal-language reasoning.
28

---

Published as a conference paper at ICLR 2026
C
CASE STUDY
C.1 REASONING CASES IN SIMPLERENV-ISTTRCC
Openk 82.5%
InstructV SR 58,
the less
grah something
OpenUai 6.35
IntrecttVLA SR 56.1
Figure 18: Reasoning cases in SinpterEnn-Instrue. Three cases of the VL fine-tuned Open VLA and InstrattVLA-GGenerraistt "SR" denotes success rate.
We present three representative reasoning cases in Figure 18 In the first example, OpenVLA fails to associate the letters "V" and "I." with their corresponding shapes the image, resulting in consistent failure to grasp in all similar scenarios. In the second case, Open VLA does not correctly associate the concept of "sour" with the corresponding fruit. As a result, its action is influenced by both the pear and lemon, leading grasp attempt between them that strikes the table. In the final example. OpenVLA fails to interpret the negation in the instruction and incorrectly grasps Coke instead of orange.
29

---

Published as a conference paper at ICLR 2026
C.2 FAILURE CASES
Figure 19: Failure case of InstructVLA. The model receives only a third-person view image visual input, making it difficult to estimate depth the gripper's relative position the object. Consequently, it fails to grasp the object accurately, despite the gripper appearing aligned with the target in the image.
B3
L33
Figure 20: Failure case 2 of InstructVLA. The model fails to accurately estimate depth due to the real-to-sim gap, specifically the absence of reflection on the table, which causes the robot to become stuck in an -iittr--iitiitttonnn position.
We illustrate representative failure cases of InstructVLA Figures 19 and 20
While some failures may result from the real-to-sim gap. incorporating additional sensory inputs such as depth information and robot state may enhance performance. We leave this exploration for future work. Additionally, we observe that the model achieves higher success rates in language responses than in action execution, suggesting that multimodal understanding is more readily transferable than manipulation skills. This highlights fundamental challenge in the development of embodied models.
30

---

Published as a conference paper at ICLR 2026
C.3 GPT40 AS THE AUXILIARY SYSTEM 2
Instruction: Can you place the sourest fruit to the less sour one?
Instruction: Pick the apple from the middle drawer, then close the
image, concise
direct prompt describing, what da next.
Rased on the instruction and the imagn, write conciso and dirrect next.
directly pive the prompt.
prompt describingg
You should directly the prompt.
Place the lemon on the left, the pear the center, and the sponge right.
Figure 21: GPT-4o as the auxiliary system 2. We prompt GPT-4o with the first image from the environment along with the instruction, asking it to rewrite the prompt in simple and clear format.
strong baseline for InstructVI.A AA integrates an expert model capable executing atomic instructions with GPT-4o as an instruction parser to decompose complex, free-form commands for decision- making (Shi et al. 2025 Gao et al. (022).. The prompt used is listed in Prompt | and it was evaluated and refined on 20 test cases from the task aggregation ensure reliable performance. Results on additional test cases are presented in Figure 21 GPT-4o successfully identified the atomic instruction in the second case but failed in the first.
During evaluation, GPT-4o is invoked only in the initial step to ensure an unobstructed view of the scene and generate free-form instruction. We do not provide closed set task-relevant instructions for selection, as the training set (Figure[3) lacks sufficient diversity in instructions and objects, and therefore does not adequately cover the evaluation settings. Across 80 evaluation cases, GPT-4o frequently fails in physical grounding, maintaining coherence, and accurately interpreting the scene.
GPT-4o System-2 Prompt
Instraction: Can you place the sourest Fruit the least sour one? Based instraction image, write concise direct prompt to what do next. You should directly give the prompt.
31

---

Published as a conference paper at ICLR 2026
Table 12: Definition of the 6D motion-pins notion-primitive code e 1,1,0,13 and its natural- natural-language verbalization. If no motion is detected, the default label is step.
Index
Symbol
Physical meaning
Values
Verbalization
0
Translation along
-1,0, 3,0,1
backward /- forward
1
Uyy
Translation along
-1,0, 1
right / - leeett
2
Translation along
-1,0,
down / /u/up
3
Upitch
Pitch rotation
-1,0, 1
tilt down wwwn//// - tilt up
4
Byaw
Yaw rotation
-1,0,
rotate clockwise / rotate counterclockwise
5
Egrip
Gripper motion
-1,0,1 1
open gripper / close gripper
D DATA ANNOTATION DETAILS AND ANALYSIS
The data analysis and GPT4o prompt are listed as follows (Figure 23 and Prompt 2
D.1 LANGUAGE MOTION PRE-TRAN PRE-TRAINING DATA
Language motion (Belkhale 2024 provides intuitive linguistic descriptions of basic end-effector movements, which can be distilled into latent actions. For each episode, we extract sequence of low-level motion primitives from the robot state trajectory. Let 8; E R denote the state at time consisting of the end-effectorr position Pt R orientation quaternion q: E _ (in zyew order), and the scalar gripper state Jr E R
St
(pt, 9t
(1)
We process overlapping windows of length n, (st,..., St4N), and summarize each window by the displacement between the first and last state
Ap P++N N Pt. Aq O 04 Ag = It+N gr-
The rotational displacement Aq is converted to Euler angles (r, d. tb in the xyz convention (roll, pitch, yaw). We then form a 6D continuous motion descriptor
d= (dz, dy. dz. dginch dyaw: dgrip);
where dr.y. correspond to the clipped Ap, and dgrp Ag. Finally, we quantize each dimension with symmetric threshold 0:
+1,
d, > 0,
0
0,
|d; S A
11,
d, < -0,
obtaining discrete motion code u E (-1,0.1) This code is then mapped to natural-languue description (e.g., "move forward", "tilt up". "close gripper") using a fixed vocabulary, shown in Table 12 If all dimensions are zero, the primitive is labeled stop. We visualize example annotations from a representative episode Figure The language motions are concatenated into the model's response along with the corresponding prompt.
D.2 TASK DIVERSITY ANALYSIS
We categorize tasks into two broad classes: Command Rewriting Context Creation and Question Answering. Each class includes several common task types:
COMMAND REWRITING / CONTEXT CREATION
Complex Object Referencing: Uses attributes, pronouns, or relational terms to reference an object.
Example: "Place the red item next the box."
Novel Action Referencing: Rephrases previously known action using different verb motion.
Example: "Shut the drawer" (instead of "Close the drawer").
32

---

Published as a conference paper at ICLR 2026
dhen, robere cinduste
Move Owen
beckwars retats costorcbcteie
Figure 22: Language motion examples
Negative Task Specification:
Specifies the correct action by negating incorrect alternatives.
Example: "I'm thirsty, but don't want sparkling water water-brinn -brig me something else."
Subtask Identification: Isolates step from multi-step instruction with clear sequential order.
Example: From "Take the spoon out of the top drawer," execute only the first step.
Situated Task Identification:
Infers the required action based on contextual cues or situational
conditions.
Example: want to clean the table. What should use?"
Direct Instruction: Provides an explicit and unambiguous command.
Example: "Organize the drinks by putting the green can next to the Coke can."
Tool-Use Understanding: Refers to object by its utility or function rather than its name.
Example: "Hand something cut with" (instead of "Use the knife").
QUESTION ANSWERING
Requires determining the number or quantity of items. Example: "How many apples are on the table?"
Quantitative Identification:
Spatial Identification: Involves spatial relationships between objects or with the user. Example: "Is the cup on the left or the right of the plate?"
Visual Identification: Focuses on appearance-based attributes such color shape. Example: "Which one is the metallic silver object?"
Commonsense Answering: Requires everyday reasoning or world knowledge.
Example: "Which of these would you to cut paper?"
State Identification: Determines the current condition or status of an object. Example: "Is the drawer currently open or closed?"
The data examples for VIA-IT are provided Figures
and 25
D.3 PROMPTING
The Prompt 2
along with three images captured the beginning, middle, and end each episode, is packaged and sent to GPT-4o. Episodes from the Bridge dataset (Ebert et al. (2021) that lack valid instructions are excluded from annotation.
33

---

Published as a conference paper at ICLR 2026
From can TOP dele
bot ton di Lawer e
Quedito Anowring
Command Bewriling
Cantrct Cradioa
drew
Bag put po
PARESS Hp
pat andvin www.rnneetttll
Task Diversity Analysis
Vocabulary Diversity Comparison)
Figure 23: Data analysis. Left: We manually identify common task categories and calculate the distribution. The proportion of direct prompts is reduced in favor of more diverse, free-form expressions. Right: Word cloud and verb-noun analyses compare the original Fractal instructions with the VLA-IT corpus.
Malic sure the bottom drower opin.
will that the bottom drower.
Pleas froiiee setup by securing all open compartments woristation.
clous bettom drower
Organize the trivis by putting green can nont the Coke can,
postion grwen can near the
Please group togother starting tinntnng by moving groen one.
green
next
Situated Task kentification experiment.
bottle from I'm preporing smoothic and need bariona, please onc.
Organizz
Dinect Instractis
will pick banera from the white bowl.
Direct Instruciion
We need to set up baranas for display, could you help or handing them?
Sure, sick banana from the white bow!.
Novel Action Refer
Figure 24: More VLA instructions on Fractal dataset.
Sproed out folced green fabric from upper right the lower left
will unfold the dlch that direction.
Oson the tr orlangular shaped groen textile by lt clwwwords to the left.
Okay, well start unfolding it now.
Complex Ocject
After botling water in the part, please put the an simmer contents.
ready place
Subtask identifiation
weshinng dishes, oover pot aineady stovetop?
Subtcsk isentification
custing
Toni-une Undeeretand ng
Sperieue shorp and in compartment.
well find the cutter and store
Complex Ofject Reter
After you're done washiing dishes, can you place the blue spoon the wishelat??
wwll place the bive spoon on the orange diet, there no red clett.
Place the blue atwnal on the red febre.
will set the lue spoon the orange doth.
Figure 25: More VLA instructions on Bridge dataset.
34

---

Published as a conference paper at ICLR 2026
Data Annotation Prompt
Imagine mhot assistant operating in laboratory household environment. The rohot expeated follew diverse commands based on realistic tasks and human interactions. Your task is to:
1. Write caption to describe the visual scene shown in the first image. You should NOT include the robot itself here.
2. Based on the given robot task description and the images, generate new user instructions and corresponding robot responses with QA pairs.
The new user instructions should align with the actions performed by the robot in the images and environment shown in the images. You required produce three categories instructions:
Command Rewriting (CR): Rephrase the task description using diverse language styles and vocabulary. You may refer objocts by their utility. color, shape, or other attribates, but cnsure the attribure you use unique to each object
2. Context Creation (CC): Generate detailed scenarios where the robot needs to perform the given instruction. The situation necaasary. may also simulate long-horizon
should involve realistic surroundings
task bused on the conteat provided by the imuge. Your generated question should NOT include the answer itself.
3. Scene-related Commonsense QA (QA): Generate some other QA pairs that are related to the scene. The answer should be concise and consistent among the three images.
For instraction, provide concise
response that clearly (use simple words) communicates
robot
take. Do not chain multiple actions together using phrases like "and then." " nocessary. the response may include brief explanation the reasoning. Avoid repeating the instruction in response. Response Format: You MUST respond ISON format. You should include "Caption", "CR", "Oc", and "QA" in your response. You should create 1-3 cutries for each of CR, CC, and QA.
Example the instruction "Close midilee
(Coresponding three images omitted) Caption: "A table
Coke chips on top, with middle drawer open."
1 "Capiinn: "A table with a Cake and chips on top, with its middle drower opon.",
"CH": (1 "question": "Push midale
closed."
"answer::
will close #:"
"question": "Ensure the center drawer is closed.".
"answer": 11 will close the drawer." ')
'CC":(/
"question": 'I you
out the Coke from the middle drawer and closing it.",
"answer: "The Coke on the table, and the middle drawer empty. So, should close the middle drawer.") ),
"question: "Please push the middle drawer shuf so we can clear the workspace.",
"answer": "Okay. will close the midele drawer." ].
"GA":L
"What
middle drawer?",
"answer": middle drawer empty.").
"question:: "How many Coke cans are on the table?",
"answer: "One." [1]
Example For the instruction "move the apple near the Coke":
/Corresponding three images omitted)
Caption: "A table with Coke, apple, and soap it."
"Caption": table with Cake, apple, soap it.,
"CR":[1 "question": "Move the healthy food near the Coke."
"answer": "The healthy food refers to the apple. and will move the apple to the Coke."
"question: "Move the apple the cylindrical shaped abject.".
"answer" "Of course!"
*OC": [1 "question": "Gather objects near Coke, exoapt the soap.".
"answer": 'I wil move the apple to the Coke." ")
"QA":[1
question": 'Imm thirsty, what have?".
'answer:: Coke
table.").
"question:: "What the healthy food on the table?".
"answer": "The apple.") "
Your task description is "<placeholdrrs"
Now give your response ISON format.
D.4 GROUND TRUTH INSTRUCTION FOR DATA ANNOTATION
During data generation, we observe that GPT-4o often struggles to accurately interpret robot behavior using only the three provided images, performing noticeably worse than humans. To quantify this, we randomly sample 100 examples and prompt GPT-4o to generate our four types of annotations using similar prompt (excluding the ground truth instruction from human expert). We then manually evaluate the correctness the results: a sample is scored as no obvious errors are found, 0.5 minor emrrr present, and 0 if completely incorrect.
The results are summarized in Tables 1 and 14 with two representative cases illustrated in Figures 26 22 In the first case, GPT-4o hallucinates the robotic arm as a bread roll, leading to incorrect caption and instruction. In the second, reverses the temporal order actions, resulting in an inaccurate annotation.
35

---

Published as a conference paper at ICLR 2026
We attribute this performance gap GPT-4o's lack of temporal grounding and the low visual quality of images in manipulation datasets. In contrast, human Human--roovddd instructions inherently encode temporal links across the image sequence by grounding the task in context, identifying target objects, and specifying corresponding rohot actions. This finding underscores that, despite their impressive capabilities, even state-of-the-an VLMMs lack embodied experience and temporal grounding. limiting their ability to infer fine-grained actions in robot manipulation tasks.
Table 14: Distribution of common error types. Error analysis of GPT-4o annotations generated without access to ground truth instructions, with long-tail errors omitted.
Table 13: Data annotation success rate. GPT- 4o shows significant performance drop without ground truth instructions during data annotation.
Method
Success Rate
Error Type
Percentage
With GT Instruction 95.4%
Ignore Vision Context
32.5%
Without GT Instruction
45.0%
Reverse Temporal Order
10.2%
Minor Object Hallucination
5.7%
36

---

Published as a conference paper at ICLR 2026
pick apple white
Capáán:
Captant:
square sable with white bowl contaning an apple and two red fruns.
A tañle with bowl contaning bread roll and tomatoes.
CR:
CR:
Quation: Grah the sepple from the surving conoenee. bowl.
Quaiiion: Move the broad mill out of bowl.
Answer: pick the
Answer: will broad roll
fnui white Auswer: grab apple
you
from
Airigte, pick beige
Question: sound becad
Question: necll you pegpare apple cutting, please
Anowen Cleay, IIIl nomove the brown
Auswer: will take the apple our of the bowl.
CC:
Quention: Can you the apple on the counter to we can slice for the salad?
Question: need the breed roll for sondwicch Can netriese from the how??
Auswer: will take the apple of bowl.
Auswer: I will lift the buead roll out of the bow).
breed Answer: will take the roil the bowd.
Useetion: Please
the bowil dinner
Unestion: How fruits the bowi!!
Auswer:
Quaition: Could
table?
Answer:
QA
Questina: What is the shape of sable?
Question: What inside che bowl!
Answer: Square.
Answer: A bread and tomaates..
Quaation: Whene can find the hroad mill?
Answer: the bost oa the sable.
Ground Truth Instruction
Usention: oolor breed roll?
Ground Truth Instruction
X
Auswer.
Figure 26: Comparison of GPT annotations with and without ground truth instruction. Errors are highlighted in red.
pick orange can fridge
Caption:
Caption:
fiidge with several empty and fow stoned the sholvee.
vending mochine multiple and
including
Resieve colored
rack.
will orang
Question: cylindrical sliver contuiee place exppy Answer: pick up silver and place tryy beloww
Gödpe's lowest compartment Answer nior our orange can bottom
Quesinn Remove copper-uued
Quedion: Transfir the groen can to botton
Asswer: will move the green cas to botton shelff
OC:
Quosion: sood you to collest the ocange can frum the fridge for the party.
Qwestion: Put the Popsi can the lowest companment.. Answer put the Pepsi the buttom comportmene.
Anewer will pick the orange cur from the bettom shelf.
Question: Get the beverage flom the fridge can serve to the guests.
CC:
Atawerr I'll grah the umage can from the bottum shelf.
Question: rethgenator by boctons skeelf
from the thind shelf
shelf.
the
How
located the botona shelf? shelf.
Questio::
make space
Asswer: will lemon lime
is unique can Arswer Onang:
Quediun: rastadking the vending muchine and noed to place section Begin with Pepei can. Asswor: I'll place the Popel can in the bottens section.
QA:
Question: How mony Cola cass are is the retrigenator?
Asswor: One.
Questoo:: Which drinks placed the botton shelf!
Spriic.
Ground Truth Instruction
Ground Truth Instruction
Figure 27: Comparison of GPT annotations with and without ground truth instruction. Errors are highlighted in red. In this case, GPT-4o incorrectly infers the temporal sequence of actions without access to the instruction.
37

---

Published as a conference paper at ICLR 2026
E BENCHMARK DETAILS
E. MULTIMODAL.
We the automatic evaluation from VLMEvalKit (Duan al. 2024 including MMMU(Val) Yue
et al. 2024). MMStar (Chen et al. 2024a), MME (Fu et al. 2024). OCRBench (Lu et al. 2024e).
2024d). TextVQA (Singh al. 2019), DoCVQA (Mathew et al. 2021). VnA (Mathew et al. 2022). A12D (Kembhavi et al. 2016), ChartQA (Masry et al., 2022) and RWQA (Team] 2024). These benchmarks collectively
(Alg (Guan et al. |l.| (024)) MMB(Dev En V1.1) (Liu et al.
evaluate diverse multimodal capabilities, including general visual question answering, document, infographic and chart understanding, OCR reasoning, and hallucination robustness.
E.2 SIMPLERENVINNTRUCT
As shown in Table 13 although SimplerEnv-Instrues is primarily designed for instruction generaliza- tion, we incorporate diverse out-of-distribution objects, environments, and distractors to prevent VLA models from exploiting the benchmark by disregarding the instructions.
Table 15: Task distribution
Attr.
with OOD Obj.
with OOD Env.
with Distract Obj.
Only Language OOD
Percentage(%)
50.0
62.5
35.0
5.0
We select 10 task scenes with InstructVLAA rollout actions and responses, shown in Figures 28 and 29 to illustrate its performance on both task aggregation and Situated Reasoning tasks.
Acknowledgements of 3D assets. We gratefully acknowledge the creators of the following 3D assets used in SimplerEnn-Inttruct.. All assets are licensed under the Creative Commons Attribution license:
Fruit Bowl Collection
Coffee Mug
Copper Tea Pot
Game Boy Classic
All other assets are created using Blender or modified from SimplerEnv (Li et al. 2024d).
38

---

Published as a conference paper at ICLR 2026
Instruction
Response
Pleaze retrieve the green teardrop shoped fruit from the surtace.
will pick the pear.
Veuilica récupérer fruit on forms de goutte verte de la surfo
up yellow on
will piik the banann.
purple
will pick the grape.
Grab the green and white flzzy drink can from the workspace.
will pick the 7up can.
toma lato verde blonce
Please move the to
will move the 'L' noxt the V.
pick the plastic bottle and then place near the orange.
will pick the move orange.
you place
lenion
play
will pick the clover the
Pull the center crower open
will open the middle drawer new
will open the drawar
place apple draweryou you opened
will oparn drewer and place apple inside.
Figure 28: Examples of task aggregation in SimplerEnv-Instrue We list ten examples with corresponding instructions and responses. Notably, InstructVLA shows the strong zero-shot ability interpret multilingual instructions, recognize novel objects, and leverage OCR capabilities.
39

---

Published as a conference paper at ICLR 2026
Rollout Video
Instruction
Model Response
Ploose move the water bottle wet the sponge.
will pick up the water bottle holo
currently
Please check If items bettom
bottom radd open, iotents
health vegetable, Grab
epgalant.
Can you grab the fruit before preparing the salee??
up orange
suitable
spente reg
will pick up bottle and move to the orange.
Pick the mug from the top drawer, then diace the top drower
mug is already on bobll, top drowe:
feed
will
Aftor finish my drink, can you place the fruit ncor for snock?
Sune, will place the apple near the Peps can.
zm thirsty drinis, Please somothing
up orange
Examples of Situated Reasoning in SimplerEnv-Intrrue. The second example's responses is recorded before and after the drawer open.
Figure 29;
40

---

Published as a conference paper at ICLR 2026
F MODEL DESIGN AND TRAINING DETAILS
F.1 INSTRUCTION FORMAT
To train captioning. question answering, and instruction-following capabilities, we integrate tasks into unified dialogue format. For captioning and question answering. adopt the template shown in Prompt where the captioning instruction sampled from Prompt For free-form instructions, append the postlix "First answer my question." to elicit direct response from the model, as illustrated in P
Dialogue Format
(
"role": "systen", "content": SESAUL MESSAGE
"role": "user",
"content": "[Questian]",
"inage": inage
"role": "assistant", "content": "[Answer]"
"role": "user",
"content": "What action should the robot take to [Instruction]?"
"role": "assistant",
"content": "[Latent Action Queries]"
Caption Prompts
Describe what's 's the table. Don't mention the robot arm.
What objects are in the scene? Ignore the robot arm.
Tell me what you see on the table, not the robot.
Describe
their positions, skip robot. the table describe
include
Only talk about the objects, machine.
Give short description of the without (he robot.
Descrihe the setup the rable. Leave out the rehotic arm.
Focus on the objects and cavironment, Ignore the robot.
Describe the envirenment tabletop conients, excluding any robotic hardware.
Instrnction Format
"role": "systen", "contant": DEFAUL DEFAUL WWW.TTSSSSSSSSSSAAAAA MESSAGE
"role": "user",
"content": "What action should robot [Instruction]? question.".
"inage": inage
"role": "assistant", "content": "[Response] [Latent Action Queries]"
41

---

Published as a conference paper at ICLR 2026
Input Tokens
VLM Backbone
VIM Backhone
Transformer Black
Liner Layor
Last Hidden States
Transformer Black
Feature:
...
MrK.Anpestinlager
Weight
Action Adapler
Lang Alaptar
Scalar Head
Block
Scaling weights for token 2
Transformer Block
Language Latent Action Token
Language
Mud
Action Expert
Action
Figure 30: Detailed overview of the MoE adaptation architecture. The frozen VLM backbone's last hidden states are classified by scalar head produce gating weights Å₁ and X2. which control the weighted MoE adaptation. Similar to finetuning VLMs with multiple LoRA adapters, the MoE adaptation computes weighted differently based on their token type: language tokens are directly decoded as the model's 's response, while features corresponding action tokens decoded by the action expert (see Figure (right)) to produce continuous actions.
over the LoRA experts. The predicted tokens are then used
F.2 MoE ADAPTATION
We briefly outline the LoRA (Hu et al. 2022) mechanism, which serves as the basis for MoE adaptation design Buehler & Buehler (2024). LoRA models weight updates as lying in a low-
dimensional subspace by freezing the original weights Wo E pd and parameterizing updates through a low-rank decomposition:
Wo + AW, = W. + BA,
where RRd and A BRik with < min(d. (d, k). The forward pass becomes
Wor + AWoz Wor BAr.
(5)
In practice, LoRA introduces fixed scaling factor a, yielding
h Wor + aBAr.
(6)
As shown in Figure 30 the scalar head in MoE adaptation predicts a gating coefficient À to reweight two LoRA adapters, inspired by ofxtuure-of-epeere architectures (Zhou et al. 2022). This is achieved by rescaling each adapter's scaling factor
aj a; Ai
(7)
where indexes the LoRA adapters. The scalar head together with the LoRA adapters constitutes the MoE adaptation module.
Unlike X-LoRA (Buehler Buehler 2024). which trains the scalar head and LoRA adapters separately. We first pretrain the action adapter in Stage-1 following the standard LoRA pipeline. In Stage-2, we introduce the language adapter for embodied reasoning and the scalar head, and train the complete MoE adaptation module jointly. We instantiate the scalar head as 4-layer MLP with dimensions shown in Table 17 for simplicity, without any other auxiliary loss design.
F.3 LEARNING OBJECTIVE AND INFERENCE PROCEDURE
We adopt flow matching Black al. 2024 Lipman al. 2022 learn the action chunk A
R H x7 (Zhao et al. 2023 over a horizon H. The training objective is defined as the flow matching
loss:
LFM = [IVO (q) (4) (e- A)I
(8)
42

---

Published as a conference paper at ICLR 2026
Table 16: Overview data used in Stage-I action pretraining and Stage-2 VLA instruction tuning.
Supervision Type
Sclf-Annotated
Stage-1
Stage-2
Action
Language Motion
/
General Multimodal Datasets
X
Embodied Reasoning( VLA-IT dataset)
X
Table 17: Model parameters. "Adapter" and "Scalar Head" are used for MoE adaptation. Specifically, two LoRA adapters are used to learn latent action generation and assistant response during VLA-IT.
Component
Parameter
Value
Adapter
Rank
128
Alpha
256
Dropout
0.05
Target
Attn. Q/K/V/O
MLP Up/Down
Scalar Head
Hidden Size
2048
128
128
128 + 2
Activation
ReLU
Action Backbone
Depth
12
Head
12
Hidden Size
768
RoPE Theta
1000
Proprioception Encoder(Optionall
Hidden Size
8 - 768 - 768
Activation
SiLU
Action Encoder with Time Embedding
Hidden Size
7+768 - 1536 + 768
Activation
SiLU
where [0, 1) denotes the flow step, and (A qt is the network output conditioned gr, which encodes information from DINOv2 (Oquab et al. 2023) and a latent action C. The interpolated noisy
action is given by A TA + T)E, with e IO,
During inference, we generate the action chunk using forward Euler integration:
A T+/NN A + N Vo(A", .4),
starting from A ~ N/O, I), with N = 10 denoising steps. For language prediction, we use the standard cross-entropy loss. We simply sum the two losses with 1:1 weighting. The data we used are detailed in Ta6le
F.4 MODEL PARAMETERS
Additional model parameters are provided Table 17 with now-matching sampling settings detailed in Table 18 All projectors-includee those aligning latent actions and DINO-ViT visual features to
Table 18: Flow matching parameters. The time steps is sampled from p(T) = 3( (===;== 1.5, 1) Black
et al. 2024)
Component
Parameter
Value
Flow Sampling
0.999
Inference Steps
10
Sinusoidal Time Embed
Max Period
100
43

---

Published as a conference paper at ICLR 2026
the action expert's dimension- simple two-layer MLP with SiLU activation. The action head, also a shallow MLP with SiLU, maps the action expert's 'ss hidden states to RNx7, where N = 16 is the prediction horizon and 7 denotes the action dimension, including the gripper.
F.5 INFERENCE SPEED
We evaluate the inference speed of InstructVLA on single A100 GPU with BF16 precision, as shown in Table [9| To support language feedback during evaluation (i.e., CoT inference), in the "Thinking" setting, we enable VIM auto-regressive generation every 20 action expert steps. The "Action Only" setting bypasses language generation and directly decodes latent actions via a single VLM forward pass. In the "Latent Action Caching", latent actions are generated every two expert steps; this introduces minimal performance impact. All settings are tested without action chunking. Note that although the model predicts 16-step action sequences, only one step is executed.
Table 19: Inference speed. Inference speed is evaluated under three settings without using action chunking. Each evaluation includes a 50-step warm-up followed by 200 steps for stable measurement.
With Language
Action Only
Latent Action Caching
Inference Frequency(Hz)
2.51
3.50
4.96
F.6 EXPERIMENTS COMPUTE RESOURCES
The action pretraining phase requires approximately 27 hours on 64 A100 GPUs, with each node equipped with TB of CPU memory. The VLA-IT phase takes about 12 hours under the same GPU configuration. Simulator-based evaluations are conducted with 8 A100 GPUs, while real-world experiments involve 4 hours of training on A100 GPUs and deployment on single A100 GPU.
To assess minimal training resources, we further reproduce pretraining results using 8 A800 GPUs in 2.5 days as shown in Table 20
Table 20: Evaluation results under different training settings. We report mean success rates (%+ standard error) across tasks, with Overall denoting the average over all tasks. "Main" corresponds to the results reported in the main table.
Google Robot (VA)
Google Robot (V3)
WidooX Robo
Oszaal
Setiine
Coke
Drower
Applc
Cake
Dnawer
Apelc
PM Spoce
CarCorot
Pst Eprplasi
719613 769999
61.7008 628816
83286 83842
94.4224 9538441
562=19 63.513
GPUs spocha)
94.0
39.8443
88.737
07.4821
6L8525
31.7119
(2.e111
486624
44

---

Published as a conference paper at ICLR 2026
G
MULTIMODAL EXAMPLES
Figure 31 illustrates InstructVLA's multimodal and embodied commonsense reasoning across diverse scenarios. The model demonstrates accurate visual inference (e.g., recognizing dog via reflection, identifying synthetic images), basic scene text recognition, and reliable grounding of objects and colors. In manipulation tasks, interprets high-level goals, predicts appropriate next actions, and verifies task completion. These capabilities showcase its integration of perception, language, and manipulation, enabling effective performance in complex daily-life scenarios.
How dogs this image? Explain
What is on the top of the shelf is yellow block
dag image.
single
vioble mirrar
only one dag's The dog's clear and distinct, in ddicaiing that there is only one dog present in the scane.
Image? Explain
current goal open drawer out snacks, place close
drower what is the immedate next step based the image?
This appears dightal creation rather then real photograph, The smooth uniform texture of tree's bark, perfect symmetry the Additionally, the background solid, unverying blue, which not typical of natural landscapes.
proportions
The immediate next step is pick the brown chip baz from the top
drawer and place on the counter.
12
Can help me resogrize the words the advertisng board where bus on drop orange
The advertinng boards have words like 'midnight", *?", "12"
What is unusual about this image
I'm proparing lunch. What did just and what should do neat based
The unusual aspect this Image that man ironing cloches on a makeshift troning board ottached to the back of yellow taxi, This not commen sight, typically done indoors designated driving.
You've already surshed the vegetables and are the pracess preparing your lunch. The nost step would heat the cooked vegetables the microwaws, lunch additional ingredients
Figure 31: Zero-shot multimodal question answering. Four commonsense and four embodied examples are selected.
45

---

Published as a conference paper at ICLR 2026
H REAL-WORLD EXPERIMENTS SETUP AND ANALYSIS
We collect data exclusively for few-shot settings as shown in Figure 32 In the first setting, which focuses on grasping objects in a clustered arrangement, the robot is instructed to classify objects within a 20 x em region on the table-placing all cubic objects into a plate and all others into box. This setting includes 70 complete episodes, totaling 677 pick-and-play setting, which emphasizes spatial actions, the robot instructed to randomly grasp three objects from the top of rack and place them into plate. We collect 60 complete episodes for this setting, comprising 180 pick-and-ple and-lleee actions. The experimental setups are depicted in Figure 36
actions. In the second
Figure 32: Real-world dataset examples. Four examples from the few-shot training set, illustrating cluster classification tasks (left) and rack teck--nd--pace tasks (right).
Figure 33: Zero-shot grounding. In clustered pick-and-plas place setting, InstructVLA accurately places the blue cube by semantically grounding the reference to the celebrity.
Figure 34: Light distraction. Stable visual features from DINO and SigLIP enable the model to operate robustly under extreme out-of-distribution lighting conditions.
46

---

Published as a conference paper at ICLR 2026
Figure 35: Zero-shot evaluation. We perform zero-shot evaluation in the Bridge kitchen environment with augmented background and novel objects. The instruction and model response are visualized in the first image.
To assess semantic grounding in novel contexts, replace the plate and box in the cluster classifica- tion setting with images of celebrities. As illustrated Figure 33 the model accurately interprets instructions and places the blue cube correctly by leveraging object and celebrity recognition.
Figure 34 shows that InstructVLA remains robust under extreme lighting conditions, supported by stable visual features from DINO and SigLIP. Finally, evaluate zero-shot generalization in the Bridge kitchen environment with augmented backgrounds and unfamiliar objects. As shown Figure 35| the model successfully follows novel instructions and completes the tasks.
200
60B
Figure 36: Real-world settings. A third-person view is captured using an Intel D435i camera for the Franka (few-shot) and WidowX (zero-shot) settings.
47

---

Published as a conference paper at ICLR 2026
1 BROADER IMPACTS AND FUTURE WORK
I.1 LIMITATION
InstructVLA integrates world knowledge into manipulation tasks by performing multimodal reasoning prior to action generation. Recent VLMs also excel at long-context processing and multi-turn dialogue. This motivates curating interleaved manipulation and reasoning with multi-turn interaction to support long-horizon tasks involving user intervention reasoning-action alternation (Yao et al., 2023). Furthermore, the existing tasks are limited to basic primitives such as open/close and pick/place due to the constraints of the datasets use (Brohan et al., 20222 Ebert et al., (021]) and the capabilities of the simulator. In contrast, standard VLM benchmarks typically contain thousands of tasks. Extending InstructVLA and SimplerEnv Instrrcct bench to more dexterous skills is essential for real-world
deployment.
1.2 LLM USAGE STATEMENT
We employed large language models (LLMs) solely for grammar refinement and minor linguistic polishing. All L.M-assisted edits were carefully reviewed and verified by the authors ensure that no fabricated content or unintended alterations to the original meaning were introduced. The research ideas, experimental design, data analysis, and conclusions presented in this work were entirely conceived and executed by the authors without LLM assistance.
1.3 BROADER IMPACTS
InstructVLA contributes to the advancement of vision-language understanding with action generation. Its ability to follow free-form instructions and generalize novel tasks supports applications in assistive robotics and human-robot collaboration. Nonetheless, as with other large pretrained models, careful attention must be given to potential limitations such as dataset bias and safety in real-world deployment. Ensuring responsible use and reliable performance across diverse environments is essential.
embodied agents by integrating
1.4 FUTURE WORK
We plan to incorporate additional sensory modalities, such as depth and tactile feedback, to enhance safety and reliability in physical interactions. Leveraging recent advances in digital twins and simulation technologies, we aim to reduce reliance on real-world data by utilizing large-scale synthetic datasets. Finally, will extend the evaluation and deployment Instruct broader range of environments to further assess its generalization capabilities.
48


Reviews:
[{"review_id": "FHCs9SVXU0", "reviewer": "Reviewer_vAo8", "summary": "The article proposes a novel VLA model architecture that uses MoE architecture to effectively address the critical challenge of catastrophic forgetting of pre-trained VLA capabilities. The work is exciting and solid, providing a clear narrative that successfully addresses the challenge of integrating multimodal reasoning ability into VLA models. The introduction of a dedicated evaluation task, SimplerEnv-Instruct, is also a substantial contribution to the field.", "questions": "Please clarify the 31.7% increase in the performance part is a relative percentage or an absolute percentage increase. In my calculation its a relative percentage, but you sould clarify it for easier understanding.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Effective Resolution of Core VLA Challenges and Strong Performance: The work successfully addresses major VLA obstacles, namely catastrophic forgetting of multimodal skills and the difficulty of tightly integrating high-level reasoning with precise low-level control. Also itsresults are compelling, validating the model’s efficacy.\n\nNovelty in Training Paradigm and Architecture Design: The core technical strength lies in the Vision-Language-Action Instruction Tuning (VLA-IT) paradigm and its architectural implementation. This approach skillfully unifies autoregressive language generation with flow-based action generation using a Mixture-of-Experts (MoE) adaptation framework within a two-stage training strategy. This design allows for the dynamic alternation between textual reasoning and action execution , providing an efficient solution to task interference while preserving general VLM knowledge. and the diagram of this part is very clear.\n\nThe authors also curate a large-scale, customized 650K-sample VLA-IT dataset featuring detailed annotations for embodied scene understanding and planning, essential for bridging VLM knowledge with embodied scenes.", "weaknesses": "Lack of explaination of Architecture part. The proposed architecture, particularly the reliance on the Mixture-of-Experts (MoE) adaptation and the design of the latent action space, is central to the paper's claims of solving catastrophic forgetting and enabling unified reasoning-guided manipulation. However, key mechanistic details and the representational quality require further clarification to substantiate the claims.\n\nPlease provide a detailed description of the Scale Head (gating network) component of the MoE. Specifically, what is its architecture, what precise input does it take (e.g., is it the hidden state of the VLM?), and how is it trained during the Vision-Language-Action Instruction Tuning (VLA-IT) stage?\n\nAlso, the core challenge in co-training VLM capabilities and action skills is catastrophic interference. Did the training process, besides the standard $\\mathcal{L}_{LM} + \\mathcal{L}_{FM}$ loss3333, include any regularization loss (e.g., sparsity or orthogonality constraints) specifically designed to encourage or enforce the dynamic switching between the Language Adapter and the Action Adapter? Such detail is crucial to validate the claim that MoE effectively enables adaptation while preserving pre-trained knowledge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cL99qhSUqq", "reviewer": "Reviewer_wyF1", "summary": "This paper introduces InstructVLA, which is a VLA model that preserves large vision-language reasoning capabilities while achieving state-of-the-art manipulation performance. To achieve this, it introduces VLA-IT training paradigm with a 650K dataset. To evaluate generalization, this paper also proposes SimplerEnv-Instruct benchmark. InstructVLA achieves good performance on multimodal, Simpler, and SimplerEnv-Instruct benchmarks.", "questions": "**1.** The description of the architecture and training paradigm is quite confusing.\n\n   **1.1** For example, they mention an action lora, a language lora, MoE adaptation, and a scalar head. However, it is hard to find details about them in Figure 2. Are the MoE and language lora only activated in stage 2? \n\n   **1.2** I'm also confused about the training details of stage 1. The authors mention that they use the data of RT-1, and the model needs to predict both action and language, so what is the input of the model in stage 1?  If the task prompt and images are the input, is the output language generated by the authors? \n\n   **1.3** In stage 1, what is the meaning of the action lora? Where exactly is this action lora applied? Do both language tokens and action tokens pass through the action lora? If only action tokens go through it, then what’s the point of the language loss? Is it just used as a simple regularizer?\n\n**2**. The metric of the SimplerEnv-Instruct benchmark is success rate? \n**3.** Table 2 shows that the training stage 2 is harmful to the normal simpler tasks, and there is little explanation about it. Since the necessity of reasoning VLA data is critical, can the authors provide more discussion?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper shows a good example of how to effectively keep the capability of LLM/VLM in a VLA system, which is a very critical problem for current VLA models. It also demonstrates how to further enhance the reasoning ability of VLA. The proposed dataset and benchmark can also be a good reference for later works.", "weaknesses": "Please see the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bfvmBzvHb1", "reviewer": "Reviewer_5dRE", "summary": "This paper introduces InstructVLA , a model designed to solve \"catastrophic forgetting\" in Vision-Language-Action (VLA) models, a common problem where models lose their pre-trained multimodal reasoning abilities after being fine-tuned on manipulation tasks. The authors propose a novel training paradigm called \"Vision-Language-Action Instruction Tuning (VLA-IT)\" , which uses a Mixture-of-Experts (MoE) framework and a new 650K-sample dataset to jointly optimize both textual reasoning and action generation. As a result, InstructVLA successfully preserves its powerful language and reasoning capabilities while delivering state-of-the-art manipulation performance , significantly outperforming baselines like OpenVLA on the new SimplerEnv-Instruct benchmark.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. VLA-IT Paradigm: It introduces a novel training paradigm called \"Vision-Language-Action Instruction Tuning (VLA-IT)\". This paradigm uses a Mixture-of-Experts (MoE) adaptation framework to jointly optimize textual reasoning and action generation, trained on standard VLM corpora and a new 650K-sample VLA-IT dataset.\n2. Two-Stage Training: The training is divided into two stages: (1) Action Pretraining, which trains a VLM-driven \"action expert\" ; and (2) VLA Instruction Tuning, which freezes the action expert and fine-tunes only the VLM backbone via the MoE module to handle complex instructions and multimodal reasoning.\n3. New Dataset and Benchmark: The paper contributes two significant resources: (a) a 650K VLA-IT dataset with diverse instructions, scene captions, and Q&A pairs ; and (b) a new benchmark, SimplerEnv-Instruct, featuring 80 tasks designed to evaluate generalization, closed-loop control, and high-level instruction understanding in VLAs.", "weaknesses": "1. This work is the combination of MoE and the fine-grained instruction finetuning. It seems so incremental. Especially, I can't see any advantages from the design. The biggest contribution is that the author gives the a 650K VLA-IT dataset with diverse instructions, scene captions, and Q&A pairs.\n2. More related works containing the step by step reasoning ability should be added and discussed like CoT-VLA.\n3. Authors should give more attention to the motivation. Particularly when the paper said their problem about ''how to train robotic manipulation skills without suffering \"catastrophic forgetting\" of the powerful multimodal reasoning capabilities inherent in pre-trained Vision-Language Models (VLMs)'', I don't know how instruction tuning can solve this kind of problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HG7oTF50K0", "reviewer": "Reviewer_kYXt", "summary": "The authors argue that prior iterations of VLA models have not placed as much focus on embodied reasoning as they should. By embodied reasoning, we mean the ability of the model to move from being told exactly what manipulation to do, to a robot that can use context clues from the environment to decide what manipulation to do.\n\nTo address this, the authors introduce a new evaluation benchmark, based on augmenting the SimplerEnv setup with multilingual instructions, more indirect descriptions of objects (i.e. \"I need something to clean with\"), etc. They also train the VLA in a 2 stage process. First, an action expert is pretrained on a large manipulation dataset, to predict both learnable latent actions and language descriptions of the actions. This  is tuned against a fixed VLM backbone, using a flow matching loss. This approach was popularized by works like pi_0, but then as a 2nd stage, the authors train LoRAs on the LLM backbone against the fixed action expert. (Some experiments were done with finetuning the action expert too, and results were generally similar / slightly worse). The authors argue this 2nd stage can be viewed as a form of instruction tuning, as mediated by the action expert, to answer more complex questions.\n\nThis model is shown to perform reasonably on two prior benchmarks, a Google Robot and WidowX Robot. On the new SimplerEnv-Instruct benchmark, InstructVLA-Expert (the model trained only with the 1st stage) is only around on par with finetuned OpenVLA, but performs signficantly better after the stage 2 LLM backbone finetuning.", "questions": "When creating SimplerEnv-Instruct, what variations were considered? I see that there are around 80 tasks + 1.1k trials which necessarily limits coverage for practicality reasons, but I am curious what things were decided to not be covered.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Paper does a good job of arguing that existing VLAs perform subpar at more complex queries, and the idea that the LLM backbone needs to be finetuned to exploit the action expert for embodied understanding seems reasonable. The proposed SimplerEnvInstruct benchmark is generally reasonable and gains on this benchmark are pretty significant. Ablations on different parts of the architecture, dataset, etc. are pretty extensive.", "weaknesses": "I think the paper is somewhat overreaching on how much it contributes to embodied reasoning. Even in the pre RT-1 days, there were some papers trying to do embodied reasoning. In my opinion the main difference these days is that the smaller, low latency VLMs are now more capable of answering these questions on their own.\n\nSimilarly, I find it difficult to attribute how much gain comes from the general evolved understanding that MoE backbones are better than dense backbones (many baseline models date to before this time).\n\nLast, there is always some benchmark bias, where in general you expect a paper to do better on its self proposed benchmark compared to other models, the largest gains coming from the self-proposed benchmark gives me some pause.\n\nnit: please label the y-axes in Figure 5, Figure 6. Please make it clearer how many trials were done for each eval number and/or include confidence intervals (why do these only exist in the SimplerEnv-Instruct columns?). In Table 1/2, I think including robot state is a significantly more important variable than the rest and would prefer if that were separated from the other results more in some way. (It is also unclear which baselines methods use robot state and which do not.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)