Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgetting
Authors: Asher James Hancock
Xindi Wu
Lihan Zha
Olga Russakovsky
Anirudha Majumdar
Venue: ICLR 2026 Poster (2026)
Keywords: VLAs
Embodied Reasoning
Action Representation
Primary area: applications to robotics, autonomy, planning

Abstract: Fine-tuning vision-language models (VLMs) on robot teleoperation data to create vision-language-action (VLA) models is a promising paradigm for training generalist policies, but it suffers from a fundamental tradeoff: learning to produce actions often diminishes the VLM's foundational reasoning and multimodal understanding, hindering generalization to novel scenarios, instruction following, and semantic understanding. We argue that this catastrophic forgetting is due to a distribution mismatch between the VLM's internet-scale pretraining corpus and the robotics fine-tuning data. Inspired by this observation, we introduce VLM2VLA: a VLA training paradigm that first resolves this mismatch at the data level by representing low-level actions with natural language. This alignment makes it possible to train VLAs solely with Low-Rank Adaptation (LoRA), thereby minimally modifying the VLM backbone and averting catastrophic forgetting. As a result, the VLM can be fine-tuned on robot teleoperation data without fundamentally altering the underlying architecture and without expensive co-training on internet-scale VLM datasets. Through extensive Visual Question Answering (VQA) studies and over 800 real-world robotics experiments, we demonstrate that VLM2VLA preserves the VLM's core capabilities, enabling zero-shot generalization to novel tasks that require open-world semantic reasoning and multilingual instruction following.

Published as a conference paper at ICLR 2026
ACTIONS AS LANGUAGE: FINE-TUNING VLMs INTO VLAS WITHOUT CATASTROPHIC FORGETTING
Asher J. Hancock, Princeton University
Xindi Wu,
Lihan Zha,
Olga Russakovsky,
Anirudha Majumdar
ABSTRACT
Fine-tuning vision-language models (VLMs) on robot teleoperation data to create lasion-laguaage-cttion (VLA) models is promising paradigm for training general- ist policies, but it suffers from fundamental tradcoff:: learning to produce actions often diminishes the VLM's foundational reasoning and multimodal understanding, hindering generalization to novel scenarios, instruction following, and semantic understanding. We argue that this catastrophic forgetting due to a distribution mismatch between the VLM's internet-ccale pretraining corpus and the robotics fine-tuning data. Inspired by this observation, we introduce VLM2VLA: VLA training paradigm that first resolves this mismatch the data level by representing low-level actions with natural language. This alignment makes it possible to train VLAs solely with Low-Rank Adaptation (LoRA). thereby minimally modifying the VI.M backbone and averting catastrophic forgetting. As result, the VLMM can be fine-tuned on robot teleoperation data without fundamentally altering the underlying architecture and without expensive co-training on internet-scale VLM datasets. Through extensive Visual Question Answering (VQA) studies and over 800 real-world robotics experiments, we demonstrate that VLM2VLA preserves the VLM's s core capabilities, enabling zero-shot generalization to novel tasks that require open- world semantic reasoning and multilingual instruction following.
INTRODUCTION
Language Description of Robot Data
VLM2VLA
Multimodal Understanding
Manipulation
Subtask
Planning
Actions
VEMIVLA
Gernen- 120
VENGVLA
Robot Dataa
LoRA Fine-Tuned VLM
Gemma-3-12B-IT
Per formancc
Grasp Pepper ('Mowe fowerd on wwweeanddllg wnp [10.2 000 1.2.1
New
Movera Plow "Wher erosping Enl the pepper above knetts then move agenaally byreont
Image
Instruction
Data
Image Tokeers
Text Tokera
Figure 1: We present VLM2VLA, data pipeline and training methodology for fine-tuning VLMs into VL.As while preserving their foundational perceptual reasoning capabilities. Our policy retains its pretraining knowledge, enabling strong VQA performance and superior generalization in real robotic manipulation tasks.
The pursuit of generalist robot policies capable of understanding and executing human commands has been significantly advanced by the integration of vision-language models (VLMs) (Team et al.)
2025b Bai al. 2025 Team 2025) throughout the autonomy stack. Trained on internet-scale datasets of image-text pairs, these models have acquired sophisticated capabilities in perception, semantic understanding, and commonsense reasoning. To endow robots with similar capabilities, the prevailing paradigm involves fine-tuning pretrained VLMs on robot demonstration data, transforming them into larion--nngagg--ace models (VLAs) that map from natural language commands and visual observations to robot actions. This approach has yielded impressive results across a wide range
"Correspondence to: jjhancock@pri ncetton.ed.

---

Published as a conference paper at ICLR 2026
of robotic manipulation tasks (Kim al. 2024 Black et al. 2024 Inte Intelligence al. 2025 2025 Brohan et al., 2023b at Padalkar et al., 2024; Team et al. 2025a).
Gao et al.
However, the standard methodology of fundamentally modifying the VLM's architecture, tokenization vocabulary, or a combination thereof, coupled with full parameter fine-tuning on robot imitation learning data, introduces crucial yet often overlooked trade-off. In adapting the VLM for robotic control, we risk overfitting to the robot fine-tuning data, thereby overwriting the general-purpose world knowledge acquired during pretraining (see Fig. 2). The consequences of this trade-off are far-reaching: current VLAs often exhibit diminished ability to generalize to novel objects, handle linguistic variations, be robust distractions, or reason about concepts outside the narrow scope of
their robotic training data (Kim et al. 2024 Brohan et al. 2023a, Hancock et al. 2024 Zhou et al. 2025b).
Preserving the VLM's founda- tional world knowledge during VLA fine-tuning is essential for creating truly generalist robot policies; consequently, numerous techniques have been developed to address this challenge. The most common approach is to co- train with non-robotic data. This training regimen regularizes the VLM against overfitting to robot datasets, thereby mitigating the loss of its foundational capabili-
Nominal VLM
Nominal VLA
VLM2VLA
safe the
safe 'clears (0.10.00 00 0.5 0.1. 0...0....111
No, there significant risk collision with the humans.
A: No, there sigrificant risk collision with the human.
Figure 2: Traditional VLA training procedures often overfit to the robot training data, sacrificing their original reasoning capabilitics for low- level action prediction (center). In contrast, VLM2VLA (right) preserves the world understanding of the nominal VLM (left), allowing the model reason about potential safety risks instead of just motor commands.
ties (Brohan et al. 2023a: Intel
ligence et al., 2025 Team al.
2025a). While these methods can mitigate knowledge loss, co-training with VLM--sale datasets is inherently expensive and requires a carefully constructed dataset mixture for optimal performance.
This paper aims preserve the world knowl- edge of the VLM while adapting for robotic control without co-training. We address this issue by resolving the distribution mismatch be- fween the low-level action spaces needed for robotic control and the image-text distributions of the VLM's s pretraining corpus. This mismatch often compels researchers to use full parame- ter fine-tuning when training VLAs, which con- tributes to catastrophic forgetting by overfitting to the robot teleoperation data.
Actions Language
Actions Tokens
Density
Average Log Probability
Our key insight is that while parameter-efficiee methods like Low-Rank Adaptation (LoRA) (Hu et al. (02T) can avert catastrophic forget- ting, their effectiveness relies on the fine-tuning data being sufficiently close to the model's pre- trained representations. We therefore propose resolving this representational mismatch the data level. Our data-centric approach re- re-represents robot actions as natural language descriptions, thereby aligning the VLA fine-tuning data directly with the VLM's pretrained representation space. This alignment enables LoRA A to effectively adapt the VL.M for robotic control without significantly perturbing its pretrained weights. Fig. F illustrates this idea, showing our language- -basad actions are assigned significantly higher probabilities by the VLM backbone than actions mapped to arbitrary tokens, a common strategy in state-of-the-an VLAs BBrohan et 2023b Kim et 2024 Brohan et al. 2023a: Padalkar et al., 2024 Lee et al. 2025). Our method is model agnostic and simple to implement, obviating the need for sophisticated architectures. complex co-training schemes, or multi-stage training procedures to achieve robust knowledge retention and superior generalization capabilities.
Figure 3: Distribution of action probabilities under Gemma-3-12B-IT before fine-tuning on robot teleop- cration data. The model assigns significantly higher log prebaabbiliiiss to actions represented as language com- pared to those defined by explicit tokenization modifica- tions, e.g., least likely token assignment.
2

---

Published as a conference paper at ICLR 2026
Statement of Contributions. We present VLM2VLA, data pipeline and training methodology for fine-tuning VLMs into VLAs while preserving their foundational perceptual and reasoning capabilities. Our core contributions as follows: 1) Representing actions language: We propose translating low-level robotic imitation data into text, thereby aligning the VLA fine-tuning data with the VLM's pretraining distribution to mitigate catastrophic forgetting. 2) A data re-labeling and training pipeline for knowledge retention: Building on our action representation, we present scalable methodology for re-labeling robot teleoperation datasets for fine-tuning a VLM into VLA through LoRA. 3) Empirical validation of action and reasoning capabilities: We provide extensive empirical validation showing our VLA preserves a suite of crucial capabilities that are often lost in other state-of-the-art models. Specifically, our method's efficacy is demonstrated through extensive real-world evaluation (over 800 robotic experiments), showing generalization to novel tasks with objects and language instructions unseen during training. Moreover, our policy averts catastrophic forgetting, retaining over 85% of the base model's performance across challenging VQA benchmarks.
2 RELATED WORK
Vision-L.anguage Action (VLA) Models: VLA models, which fine-tune pretrained VLM backbones for robotic control, have become the dominant paradigm in training generalist robotic policies (Kim
ct al. 2024 Black et al. 2024 Qu et al.
2025 Li et al. 2024 Driess et al.) 2023 Team et al. 2025a
Lee et al. 2025). The central promise of this approach is knowledge transfer: one can translate the rich semantic knowledge of the VLM, learned from internet- scale pretraining, directly to action prediction. One challenge in adapting transformer- -bssdd VLMs for action prediction is bridging the gap between their discrete, token- based nature and the continuous, vector-valued representations utilized in robotic control. Consequently, many state-of-the-art VLAAs diverge architecturally in how they address the problem of action generation.
NVIDIA et al. 2025 Yang et al. 2025b
Action Representation in VLAs: The first dominant strategy is discretization, where continuous action vectors are explicitly mapped to finite set of tokens (Lee et al. 22025 Pertsch et al. 2025). typically the VLM's least likely tokens (Brohan et al. 2023b a Padalkar et al. 2024]] Kim al.
2024). This approach casts robot control as a standard next-token prediction problem. allowing the VLA to generate outputs autoregressively. The second paradigmatic strategy augments the VLM with separate, lightweight action head (e.g.. using diffusion or flow-matching) to generate continuous
actions directly (Team et al. 2024 Black al., 2024, Intelligence al., 2025 Wen et al. 2025 Zhou et al. 2025b a). While this design choice enables faster action prediction (Pertsch et al. 2025
it introduces new, randomly initialized parameters to the VLM during VLA fine-tuning which can corrupt the original pretrained representations (Driess al. 2025
Zhou et al.
2025b a).
Our work introduces a third approach: representing actions as natural language descriptions all within the VLM's 's existing vocabulary. For instance, the command 'move forward by 4.2 centimeters' is treated as standard text string, allowing our method to bootstrap the VLM's 's intrinsic understanding of numerical magnitude for grounding in physical space. We provide extensive experimental evidence in Section 4.22t this 'actions as language' representation is more effective for policy learning via LoRA fine-tuning than strategies based on least likely token assignment. Unlike the concurrent work of Grover et al. (0025), which use full parameter fine-tuning and co-training, we use only LoRA and our action representation preserve the VLM's foundational reasoning capabilities.
Reasoning Models: To improve performance on complex, long-horizon tasks, a growing body of work has focused on enabling VLAs to perform explicit reasoning. These methods typically use a chain-of-thoughh approach (Wei et al. 2023). where the model first generates high-level plan or reasoning trace that then conditions the prediction of low-level actions. The form of this reasoning varies across the literature. One set of approaches provides task- context, leveraging (vision) language models to generate natural language sub-goals or task plans that guide the policy (Intelligence et al.) 2025 Lin et al.) 2025 Shi et al. [005 Zhao et al. 2025 Ahn et al.) 2022 Silver et al. [223] Shi et 2024, | Clark et al. 20255] Huang et al. 2022 Zeng et al. 2022). A second set of works condition the policy more structured, mid-level representations like trajectory sketches,
code, or spatial affordances (Belkhale et al.) 2024 Nasiriany et al. 2024 Li et al. 2025 Liang et al.
2023 Cheng et al. 2025 Zhi et al. 2025
Zawalski et 2025).
3

---

Published as a conference paper at ICLR 2026
"Put the pat drying rack"
D rab
<Subtask Prediction- (1)) approach the pot.0.)..
O
<Motion Planning> (mn) The robot needs to precisely position
Gemini
Its and offector the the sink by moving downward 1-zz). (in,)..
Action Generation- (1J) [1-0.00, 0.001, 0.001,00.019 1.0 (0.002. 0.002, 0.022, (8,)
a
lan
Figure 4: VLM2VLA's pipeline for annotating existing robot datasets Dau into Dba described via natural language. We use Gemini 2.5 (Team (2025) to decompose each trajectory into sub-trajectories, each with an associated subtask, motion plan, and action chunk.
Our work is distinct from prior VLAs and embodied reasoning models in that (1) we not only represent high-level reasoning in language, but also represent low-level actions (e.g., end-effector movements) using text, (2) we do not utilize any explicit action decoders, and (3) we train the VLA solely with LoRA. We posit that modifications to the VI.M, combined with full-parameter fine-tuning, create distribution shift deleterious to the VLM's world knowledge and reasoning capabilities. We demonstrate evidence of this claim in Section
Co- Training for Knowledge Retention: A critical and often overlooked phenomenon of many VLAs is catastrophic forgetting, where the rich, general-purpos knowledge of the base VLM is lost (Zhou al. 20256). To combat this, the most common effective strategy has been large- co-training (Brohan et al., [023a Intelligence et al.] 2025] Zhho et al. [2025a; [ee et al.] 2025), which mixes robotic and non-robotic datasets (e.g., VQA, image captioning). Co-training effectively regularizes the VI.A fine-tuning process, whereby the VI.M is continuously trained on data drawn from its pretraining distribution to preserve general reasoning and visual understanding. However, co-training requires careful tuning of the mixture ratio between robotic and web data. This hyperparameter is non-trivial and expensive to specify, requiring numerous training runs and real-world evaluations to find an effective setting for both VQA performance and dexterous robotic control.
Sophisticated VLA training procedures in Mixture-of Mfxture--o-Experts (MoE)-style VLAs have recently been developed as alternative means of preventing catastrophic forgetting. For instance, Driess et al. (2025) attempt to shield the VLM from otherwise destructive gradient updates of the action module via stop-gradient operators. Moreover, Zhou et al. (2025a) have explored a sequential training procedure whereby the VLM's weights are frozen during training of the action expert. Our work challenges the necessity of co- training and other sophisticated training schemes to prevent catastrophic forgetting; by representing robot actions as natural language, our VLA can be effectively fine-tuned with only low-rank adaptation, a simple procedure requiring no major architectural changes.
Advanced Training Schemes to Avert Catastrophic Forgetting:
METHODOLOGY
In this section, we discuss our VLA training pipeline. Our approach predicated on a simple principle: to successfully leverage LoRA fine-tuning and avert catastrophic forgetting, we must first align the robot data with the VLM's existing representational space. To this end, we begin by first translating robot teleoperation datasets into natural language descriptions. We posit that this data transformation minimizes distribution shift incurred during LoRA fine-tuning (as evidence by Fig. 3 thereby retaining the VLM's nominal capabilities while enabling effective robot control.
3.1 ACTIONS AS LANGUAGE: A HIERARCHICAL REPRESENTATION
VLM2VLA frames action prediction as a three-stage VQA hierarchical reasoning process, which has been shown to improve generalization to tasks and environments ZZwalski et al. 2025 Intelligence et al. 2025); y the fundamental difference between our action representation and other
VLAs (Belkhale al.] 2024 Kim et al.) 2024| Zawalski et al. 2025 Brohan et al., 2023a Padalkar et al. 2024 Clark et al. 2025) is that levellevel and lewelevel actions are represented directly via language. Let the main task be broken into a series of N steps, indexed by i:
High-Level Subtask Prediction (4)): given observation o; and language instruction L, the model first describes what immediate subtask I is necessary complete the main task.
4

---

Published as a conference paper at ICLR 2026
Mid-Level Motion Planning (m²)): conditioned the current subtask and observation, the model generates a spatially informative motion plan m,, with respect to the robot's 's end- detailing the movements necessary to complete the subtask. Our motion plans describe only directional movements, e.g., 'move left' and 'move down and slightly forward.' This coarse description was specifically chosen benefit from the VLM's latent spatial reasoning, which excels at these types of tasks (Team et al. 2025b; Bai et al. 2025).
Low-Level Action Generation (ā;): conditioned on the current subtask and motion plan, the policy generates a variable-length action-chunk a to execute directly on the robot. In practice, the action- chunk is a list of lists, where each inner list contains the individual commands represented as text for each degree of freedom (DoF) of the robot. In this work, we only considered translational DoFs.
Formally, the distribution we seek to capture with our VLA is pe(a,, mi, 4o L), which we decom- pose as:
po m., Lelos, L)
(o, 1) po ōi) pa ma, 4, o.),
Subtask
3) Action Generation
Prediction
Planning
where parameters 0 are the weights of the robot policy. This parameterization corresponds to vision- saiionlanguage model (transformer) taking as imput a sequence of multimodal image-text input tokens and predicting sequence of output text tokens. All observations reside in RGB image space.
The decomposition given in Equation (T) describes how our VLA operates at test-time. Every action-chunk prediction is conditioned on the current observation, subtask description, and proposed motion-plan; in practice, we generate all N subtasks at once given the initial observation oo and keep this set fixed for the duration of the rollout.
To improve robustness of the policy, we operate in closed-loop fashion with a verifier. At the end of cycle, the verifier determines if the model should re-try the current subtask or proceed onto the next onc, a process which continues until all N subtasks are completed. In this work, utilize Gemini 2.5 Pro (Team 2025) as the verifier (with details in Appendix D.5)
every
3.2 DATA CURATION: TRANSLATING ROBOT TRAJECTORIES INTO A HIERARCHICAL FORMAT
To teach a VLM the aforementioned ppatially-grounder reasoning chain, we must first re-label existing robot datasets in natural language. We assume access dataset of human teleoperated robot trajectories, Dich where each trajectory is a state-action sequence T = {(o,, a. a of length T controlling the relative position of the robot's end-effector (Padalkar et al.) 20244 Khazatsky et al..
2025 Walke et al. (024). Each trajectory comes equipped with a main-task language instruction L. Our goal to automatically describe each trajectory T E Drob at the levels described in Section [3.1 to construct a new robot dataset annotated with natural language, D Herein, we utilize the subset of the Bridgev2 dataset equipped with main-task instructions as Drob (Walke et al. 2024).
Our pipeline for constructing Dian is similar in spirit to the data curation strategies of Clark et al. (2025); Zawalski et al. (2025). We use Gemini (Team) 2025) to automatically annotate trajectories and construct our our natural language dataset (see Fig. for an illustration). Specifically, Gemini will decompose each trajectory into a sequence of N steps. Each step comes equipped with an initial observation o, subtask 4, motion plan m,, and variable-length action chunk ā, allowing us to repre- sent the trajectory described via natural language, F, as the tuple { 44 m,, N.N E D See Appendix B. and B.2 for additional details on dataset curation.
In summary, we transform our original robot dataset of state-action pairs into dataset of image-text pairs, thereby casting robotic control as a standard supervised fine-tuning task. We fine-tune the Gemma-3- 3-BBTIIT model (Team et al., 2025b) using LoRA applied to all its linear modules using the cross-entropy loss. This approach instills action prediction capabilities while preserving the VL.'' world knowledge. Additional training details are outlined in Appendix E
4 EXPERIMENTS
In this section, we evaluate VL.2VILA on suite of tasks answer the following questions:
5

---

Published as a conference paper at ICLR 2026
Table I: Multimodal understanding evaluation. Comparison of VLMs and VLAs across multi- modal understanding benchmarks. We compare against Prismatic VLM (Karamcheti et al. (0024). OpenVLA (Kim al., 2024), ECoT (Zawalski et al. 2025). Gemma-3 (Team et al., 2025b). Molmo
Deitke et al. (2024). MolmoAct {Lee et al.) 2025). PaliGemma Beyer et al. (2024), and mao. (Intelli)
gence et al. 2025). Our models preserve strong performance across diverse multimodal understanding
tasks despite only training on robot data. The best and second best results for each benchmark are
shown in bold and underlined, respectively.
Mediod
TeuVQA
Inl/VQ
ReslWaraaOA
Prismatc VLM Family
Priomatic VLM
7b
35.0
36,8
1456.6
32,0
66.2
55.7
42.5
17.5
19.7
54.6
167
30.5
OpenYLA
26.3
0
43,0
0
BCoT
26.6
0.01
3.7
25.6
Ccummn-3 Family (with VLMEVIA)
Gemman 3-4B-II
4b
39.3
37.1
1205 8
70,2
68.5
64.3
61.5
68.8
40.9
70.5
50.3
44.0
Gemmu-3-1222 3-12B-11
12b
46.0
45.3
1182.1
75.0
76,9
74.7
68.9
30.6
50.1
75.5
55.1
20.5
VEMEYLA-AT
126
45.9
45.2
1082.3 2
65.5
70.9
66.8
69.
74.6
45.8
74,1
41.8
46.5
VLMEVLA (Ouns)
126
42.7
48.0
1391.7
63.9
68.5
67.6
68.9
78.4
46.2
74.0
58.3
43.3
Moluno VLM Funily
Moime MolmoAct
7b
45.3
53.8
1275,3
69.5
65.9
61.3
81.7
923
72.6
93.2
84.1
70.7
7b
28.4
1214.5
52.7
58.1
46.3
57.5
58.7
41.
2.0
55.9
3.5
Paliiaamms Family
PaliGemma
9
35.2
534
20.6
55.2
240
9
10.0
7.
27.0
Q1-Multimodal Understanding: Does VLM2VLA effectively preserve the base VLM's multimodal reasoning ability in standard VQA tasks after fine-tuning on robot data Drmm?
Q2-Robotic Manipulation: Can VLM2VLA achieve level of in-distribution manipulation perfor- mance that is competitive with state-of-the-art VIAAs?
Qe-Reasoning Generalization: Does the preserved knowledge from Q1 directly enable VLM2VLA to generalize to novel, out-of-distribution robotics tasks that require reasoning beyond the fine-tuning data Dian?
Baselines: We contrast our policy against two state-of-the-ard tokenization-base VLAs: Open VLA (Kim et al., 2024), an autoregressive policy constructed from fine-tuning a pretrained 7B Prismatic VLM on the large-scale Open-X-Enin X-Embodimmnn dataset (Karamcheti et al., 2024| Padalkar et al. (2024), and Embodied Chain-of-Thouglh (ECoT), variant of Open trained to produce reasoning traces before action prediction (Zawalski et al. 2025). ECoT, similar to VLM2VLA, is only fine-tuned on the Bridgev2 dataset.
We ablate our action representation by mapping the digits (0-9) in our robot action data to the decoded strings of Gemma-3's ten least likely tokens; therefore, during VLA fine-tuning, the model will learn to autoregressively predict actions via these reserved tokens (see Appendix [B.3). We call this variation of our method, with action tokens, VLM2VLA-AT. To ensure an accurate comparison, both models are trained identically and trained on the same data, except for the action representation.
A blations:
4.1 EVALUATION OF MULTIMODAL UNDERSTANDING
In this section, we quantify the extent to which VLM2VLA, and its variants, preserve their multimodal understanding by thorough evaluation on numerous VQA benchmarks. As demonstrated by Table T both Open VLA and ECoT suffer from catastrophic forgetting relative to the original Prismatic VLM
(Karamcheti et al. 2024 Zhou al., 2025b). In contrast, VLM2VLA experiences only minor losses in performance across all VQA benchmarks, thereby conclusively answering Q1. While the VI.MVILA-AA ablation achieved comparable VQA scores, suggesting that our LoRA training scheme is primarily responsible for mitigating catastrophic forgetting, but not sufficient condition for creating a generalist policy, As we show in Section 4.2 the choice of action representation is important factor affecting generalization in downstream robotic tasks.
argue this is necessary
To compare VLM2VLA against ofate-of-the-an art co- co-tra to-trained VLAs, we also evaluate two recently released open-source models: MolmoAct (Lee et al. 2025) and To.s (Intelligence et al. 2025), along with their VLM backbones Deitke et al.] (2024); Beyer et al. (2024). While MolmoAct is strong in VQA ability, VLM2VLA outperforms it across all reported benchmarks. We acknowledge that

---

Published as a conference paper at ICLR 2026
OPenVLA
ECoT
VLM2VLA-AT
VLM2VLA
100
Out-of- Sitt-f-DDistibuttonn
Performance (X)
77
60
52
53
50-
43
34
30
25
Up
Pick Place
Pick, Place, Lift
Up
Pick
Figure 5: Comparative evaluation of VLA performance in-distribution (ID) and out-of-distribution (OOD) robotic manipulation tasks. VLM2VLA maintains high success rates OOD tasks, highlighting superior generalization capabilities. Each bars corresponds to an average over thirty trials, except for the 'Pick Up- T task, where each bar corresponds to an average over ninety trials.
model size is a confounding variable in the comparison, yet, the results in Table I demonstrate that our methodology is competitive with co-trained VLAs. On the other hand, the significantly lower performance of mo.s across most VQA benchmarks underscores that co-training is not guaranteed solution to catastrophic forgetting.
4.2 EVALUATING ROBOTIC MANIPULATION
Having established that VLM2VLA and its variants retain their foundational pretraining knowledge, we now evaluate if this training procedure translates to robotic control. Specifically, we assess VLM2VLA's 's ability to perform robotic manipulation tasks of varying complexity in in-distribution (ID) and out-of -idt-ii-iisiitttionnn (OOD) scenarios. All real-world policy evaluations receive a total of 30 trials and are performed
D-D WidowX 250S robotic arm in toy kitchen environment, as prescribed in Walke et al.) 2024).
We compare VLM2VLA the baselines and ablations two ID tasks, borderline ID task, and two OOD tasks. The in-distribution experiments probe each policy's 's action prediction capabilities short and long horizons. The nutoof-distribution experiments investigate if the policies can leverage the latent world knowledge of their VLM backbone to generalize beyond the VLA fine- tuning data. All skills tested are 'pick up" or 'pick and place" tasks, which common in policy evaluation [Hancock et al. 2024 Snyder et al. [2025] Kim et al. 2024 Zawalski et al. 2025 Gao
al. 2025). and we describe each in detail below.
Pick Up the Carrot (Pick Up): A baseline task testing the model's fundamental localization and grasping capabilities. This task is considered in-distribution because the task object (carrot) and instruction are within the VLA fine-tuning data.
Put the Carrot On the Yellow Plate (Pick and Place): longer horizon baseline task testing the model's 's ability perform more complex actions. This task is considered in-distribution because the object and instruction are within the VLA fine-tuning data.
Put the Eggplant In the Pan, Then Lift the Fish (Pick, Place, and Lift): A task requiring the robot to perform both pick-and-place and pick-up actions. This task is borderline in-distribution; the objects and instructions, when considered separately, are in-distribution, but not when combined.
multi-step
Pick Up the Carrot (Pick Up- T): A pick-up task with natural language instruction given in of three languages: Spanish, Mandarin, or Hindi. This task is considered out-of-distribution because the instructions translate the task. Two distractor objects are present in each trial to prevent the policy from inferring the task from the observation alone.
not within the VLA fine-tuning dataset, thereby requiring the model to implicitly
Pick Up the Item Above Ash Ketchum (Pick Up A): first locate the pop-culture figure 'Ash Ketchum' and lift the correct object. This task considered
A pick-up task requiring the model to

---

Published as conference paper at ICLR 2026
out-of-distribution because VLAs are not trained to recognize semantic concepts like 'Ash Ketchum," thereby requiring the model to leverage its latent world knowledge to succeed. Distractor logos ("NASA') and objects present. The item located above "Ash Ketchum' varies in-between trials, as does its location the environment.
We report the full task success rates for all models in Fig. 5 (see Appendices D.I to D.6 for additional scoring and experimental details). For the multilingual translation tasks (Pick Up T), we report the average across all languages, which amounts ninety trials per model.
4.2.1 RESULTS OF BASELINES
On the ID pick-up and pick-and-place tasks, Open performs the best. This high performance is likely explained by fine-tuning the much larger Open-X-Embodiment dataset, which provides broader coverage of common grasping skills. Nevertheless, the comparable success rates observed for VLM2VLA answer Q2: the 'action language" approach of VLM2VLA enables competitive action prediction in a specific robot embodiment.
As task complexity increases, the advantage of purely reactive policy, like Open starts to diminish. In the compositional task 'Put the Eggplant the Pan, Then Lift the Fish', we found that Open VLA often successfully completes the first subtask (put the eggplant in the pan), but fails to attempt the second. ECoT, on the other hand, normally generates motion plans to finish both subtasks, but fails execute them accurately Open VLA. We found that of the three policies, for this task, VLM2VLA not only generates correct hierarchical reasoning, but also executes them more frequently (see Fig. and Fig. -
The OOD scenarios most clearly demon-
100
ECOT
VUMEVLA-AT
VLKIVLA
strate the benefits of our knowledge nnowledge-preserving
87
87
Success Rate (%)
pipeline. In the multilingual translation experi- ment (Pick Up- T), our method significantly out- performs both Open VLA and ECoT. As shown
70
50
in Fig. 6 and Fig. 7 VLM2VLA frequently translates the non-English language instruction and identifies the correct object; this a direct application of the multilingual capabilities tained by our model as reported in Section [ task suc- cess rate, the results presented in Fig. suggeett may be relying on heuristics, such as simply picking up the closest or most salient object in the near vicinity, rather than a genuine understanding of the instruction.
20
10
Figure 6: Analysis of task decomposition for OOD ma- nipulation tasks, denoted by the corresponding icon. Points are awarded if the model's task plan correctly identifies task object, task destination, present. See Appendix [D.3 additional details.
While ECoT does achieve
This trend is even more pronounced in the 'Pick Up the Item Above Ash Ketchum' task, requir- ing combination of both open- world semantic knowledge (recognizing pop-culture figure) and spatial reasoning. Here, VLM2VLA is the only model to achieve a meaningful success rate. This result provides positive answer to Q3: the foundational world knowledge preserved by our method can be leveraged to solve novel prob- lems zero-shot.
Human: What subtasks are required to मानरर 3013i
VLM2VLA: ['Move to Carrot', GGaspp Carrot', 'Lift Carrot High']
4.2.2 RESULTS OF ABLATIONS
demonstrated that LoRA fine-tuning can mitigate catastrophic forgetting for both our method and the token-based ablation, but does the 'actions as language' representation actually facilitate learning results demonstrated in Fig. and Fig. show approach consistently outperforms the token-
Section 4.J
Figure 7: A qualitative demonstration of VLM2VLA's zero-shot multilingual capabilities. Given the language instruction in Hindi ('pick up the carrot'), ) our model identifies the correct object amidst distractors (eggplant and banana), demonstrating genuine understanding of the task.
effective policy? The
8

---

Published as a conference paper at ICLR 2026
based ablation across all manipulation tasks. While VLM2VLA-AT performs competitively simple ID tasks, its performance degrades as task complexity increases; this seen most clearly in the OOD setting despite retaining strong VQA capabilities. For instance, Fig. @lshows VLM2VLA-AT struggles with multilingual commands and scores only half as well as VLM2VLA in the Pick Up the Item Above Ash Ketchum' task (achieving success rate of just 30% to model's 60%). These findings are suggestive of a disconnect between the VLM's latent world knowledge and the action token assignment achieved after fine-tuning.
CONCLUSION AND DISCUSSION
In this work, we present VLM2VLA: novel VLA training paradigm that addresses the challenge of catastrophic forgetting often experienced when adapting VLM for robotic control. We demonstrate that by grounding the action prediction problem in natural language, LoRA can be used to train manipulation policies without significantly degrading the backbone VLM's foundational capabilities. Our data-centric approach of treating 'actions as language' obviates the need for major architectural modifications to the VLM or expensive co- co-training schemes. As our experiments show, the retained latent world knowledge of VLM22LA enables zero-shot generalization to novel tasks that require open-world semantic reasoning. Ultimately, this work offers one path toward building generalist policies by closing the gap between large-scale foundation models and real-world robotic control.
5.1 LIMITATIONS AND FUTURE WORK
Our approach has several limitations that present clear avenues for future work.
Inference latency. VLM2VLA generates actions autoregressively, which is inherently slow; in our experiments, the median run-time required for one cycle of action generation was 6. seconds, though subject to high variance (see Appendix D.6 for additional inference details). Future work could explore more advanced decoding techniques to accelerate our hierarchical prediction scheme without compromising the VLM's pretrained representations.
Dexterous tasks. Our work has focused on translational end-effector control, which is suitable for many simple pick-and-place tasks. While this design choice afforded strong action labeling from Gemini, it precludes dexterous manipulation tasks which require nuanced action prediction, e.g., rotation. Moreover, the granularity at which we predict motion plans is coarse, and logical next step is to obtain fine-grained language annotations when constructing VLA fine-tuning dataset. As vision- languuee models become more adept at spatial reasoning, we anticipate this re-labeling scheme will enable precise robotic control.
Cruss-embodiment learning. The scope of our work limited training to a specific robot embodiment, thereby preventing the same policy from readily adapting other robots, especially those which utilize other means of low-level control, such as joint angles, which do not map easily to spatial affordances. Yet, we believe Using language as a common medium, it ought to be possible to describe all robot actions regardless of embodiment. Therefore, performing VLM2VLA's relabeling scheme could potentially be used to train a cross- croos-embodiment policy.
'actions language' relabeling scheme is promising solution.
Improving or eliminating the verifier. Our method currently relies on a separate verifier module to transition between subtasks, further slowing down inference speed. Future work could explore how to better train the base VL.M as a verifier, thereby averting this step altogether.
Larger-scale training. We are working towards scaling our training pipeline with larger robotics datasets. We hypothesize that combining our language-base based action representation with large- scale training may unlock zero-shot generalization, open-world semantic reasoning, and instruction following capabilities that go well beyond the state-of-the-art.
REPRODUCIBILITY STATEMENT
All models reported in this work open-source; work. VLM2VLA, is based the publicly available Gemma model (Team et al., 20256). We have attached an extensive Appendix providing additional details of our approach. Information regarding data curation procedure is available in

---

Published as a conference paper at ICLR 2026
Appendices B.1 B.3 Our VQA analysis is listed in Appendix C Our hardware experiments described in Appendices D.I to D.6. Full training details and the computational resources used are provided in Appendix E
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho. Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffire,, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huel Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as I can, not as I say: Grounding language in robotic affordances, 2022. URL https: /ttp::///xxiv..////////// org/ abs 2204.040000 01691
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng. Hang Zhang. Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL technical report, 2025.
URL https: /ttp:///aaavvvvvea org/ abs/2502. 13923
Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. RT-H: Action hicrarchies using language, 2024.
URL https::://atttiii org/ abs / 2403. 01823
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai Paligemma: A versatile 3b vlm for transfer, 2024. URL https: http:i:.//axxv.oooo
abs 0t5/2407.000 07726
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak. Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. no A vision-language- action flow model for general robot control, 2024. URL https: arxiv. .rgg /algg/////// 2410 24164
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wa Wei Edward Lee, Sergey Levine, Yao Lu, Hen- ryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT- 2: Visionnlanguaage-action models transfer web knowledge to robotic control, 2023a. URL settps:/arxiv..or
15818
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang- Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Decksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin
10

---

Published as a conference paper at ICLR 2026
Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Van- houcke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: Robotics transformer for real-world control at scale, 2023b. URL htttps://arxivv http:///rxivv.rrrgaaaeee abs/ http://a///22111./222
An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem Bıyık, Hongxu Yin, Sifei Liu, and Xiaolong Wang. NaVILA: lgged robot vision- dasiionlanguage-cttion model for navigation, 2025. URL pestps://arxte org/ abs /212 04453
Jaden Clark, Suvir Mirchandani, Dorsa Sadigh, and Suneel Belkhale. Action-free free reasoning for policy generalization, 2025, Utt.tttts:///rxiiv w5wp://wwww.gg/////2..22222 03729
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang. Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison- Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Crystal Nam. Sophie Lebrecht, Caitlin Wittlif, Carissa Schocnick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi. Molmo and pixmo: Open weights and open data for ofte-oof-the-art vision-language models, 2024. URL https: org/ abs 2499
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. PaLM-E: An embodied multimodal language model, 2023. URL https: wtwpiviivvv..////i///...///aaava org/ /tab////3333 03378
Danny Driess, Jost Tobias Springenberg. Brian Ichter, Lili Yu, Adrian Li-Bell, Karl Pertsch, Allen Z. Ren, Homer Walke, Quan Vuong, Lucy Xiaoyang Shi, and Sergey Levine. Knowledge insulating vision- Larion-languge--acce action models: Train fast, rum fast, generalize better, 2025. URL https://
arxiv. org. abs 2505. 23705
Jensen Gao, Suneel Belkhale, Sudeep Dasari, Ashwin Balakrishna, Dhruv Shah, and Dorsa Sadigh.
A taxonomy for evaluating generalist robot policies, 2025. URL
2503.01230
Shresth Grover, Akshay Gopalkrishnan, Bo Ai, Henrik I. Christensen, Hao Su, and Xuanlin Li. En- hancing generalization in vision-language-aue models by preserving pretrained representations, 2025.
Asher J. Hancock, Allen Z. Ren, and Anirudha Majumdar. Run-time observation interventions make vision-language-acte models more visually robust, 2024.
2410.01977
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. LoRA: Low-rank adaptation of large language models, 2021. URL https
09685
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models, 2022. URL https: /ttp:////rivvvv www.wiiiw....oooo.oooooo..o.oort hal//////// 2207 05608
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz. James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. mo.s:
11

---

Published as a conference paper at ICLR 2026
vision-languagg-acttion model with open- open-world generalization, 2025. URL
2..5/2504..6500 16054
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Perey Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic VLMMs: Investigating the design space of visually-conditioned language models, abs////////// 24/2244222 07865
2024. httptttsss//axiv..tot/
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sungjae Park, Iliaa Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang. Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung. Jaimyn Drake, Ethan Paul Foster, Jensen Gao, Vitor Guizilini, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Muhammad Zubair Irshad, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O'Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martin-Martin, Subramanian Ramamoorthy. Dorsa Sadigh, Shuran Song. Jiajun Wu, Michael C. Yip. Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. DROID: A lcage-scale in-the-wile robot manipulation dataset, 2025. URL htt.h::///rr..: org bog//bb// 2403. 12945
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burch- fiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Open An open-source vision- langungg-nugggateeee action model, 2024. URL BtRhttts: /tttpp:://axivv../ /awpiv.vvv...///.a/ http:/aaxxivvaaaaaaaee hbtp//bb/2400 69246
Jason Lee, Jiafei Duan, Haoquan Fang, Yuquan Deng, Shuo Liu, Boyang Li, Bohan Fang, Jieyu Zhang, Yi Ru Wang, Sangho Lee, Winson Han, Wilbert Pumacay, Angelica Wu, Rose Hendrix, Karen Farley, Eli VanderBilt, Ali Farhadi, Dieter Fox, and Ranjay Krishna. MolmoAct: Action reasoning models that can reason in space, 2025. URL https: //axsiv//arivv..a/aaa org. /2508. 07917
Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, Xiaofan Wang, Bei Liu, Jianlong Fu, Jianmin Bao, Dong Chen, Yuanchun Shi, Jiaolong Yang, and Baining Guo. CogACT: A foundational vision-language- action model for synergizing cognition and action in robotic manipulation, 2024. URL https
7/arxiv. org abs 2211 9650
Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Raymond Yu, Caelan Reed Garrett, Fabio Ramos, Dieter Fox, Anqi Li, Abhishek Gupta, and Ankit Goyal. HAMSTER: Hicrarchical action models for open-world robot manipulation, 2025. URL https: wwtpaaxxvvvvv...... org. org/abss
2502. 05485
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control, 2023. URL
Fanqi Lin, Ruiqian Nai, Yingdong Hu, Jiacheng You, Junming Zhao, and Yang Gao. OneTwoVLA: A unified vision-language-actionn model with adaptive reasoning, 2025. URL https://axxiv
orq/abs/2505. 11917
Soroush Nasiriany, Sean Kirmani, Tianli Ding, Laura Smith, Yuke Zhu, Danny Driess, Dorsa Sadigh, and Ted Xiao. RT-Affordance: Affordances are versatile intermediate representations for robot manipulation, 2024. URL htt.httts////xxi..orrr///o// http:///aaaii...rr/////////.../. 2241000004421
12

---

Published as a conference paper at ICLR 2026
NVIDIA, :, Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da. Runyu Ding, Linxi "Jim" Fan. Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop, Loie Magne, Ajay Mandlekar, Avnish Narayan, Soroush Nasiriany, Scott Reed, You Liang Tan, Guanzhi Wang. Zu Wang, Jing Wang, Qi Wang, Jiannan Xiang. Yuqi Xie, Yinzhen Xu, Zhen- jia Xu, Seonghyeon Ye, Zhiding Yu, Ao Zhang, Hao Zhang, Yizhou Zhao, Ruijie Zheng, and Yuke Zhu. GROOT NI: An open foundation model for generalist humanoid robots, 2025. URL https://axxia org/ abs /tt.////50.. 14734
Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open X-Embodiment: Robotic learning datasets and RT-X models. In Proceedings of IEEE International Conference on Robotics and Automation (ICRA), 2024.
Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. FAST: Efficient action tokenization for vision-language-action models, 2025. URL http:t::::/aaxxi..ooto org/ abs/ /ttp//aaww...r//p////.11..1111 09747
Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang. Jia Yuan Gu, Bin Zhao, Dong Wang, and Xuelong Li. SpatialVLA: Exploring spatial representations for visual-language-aattee
action action model, 2025. URL Intpps: / org/ abs/ /20.//251.... 15830
Lucy Xiaoyang Shi, Zheyuan Hu, Tony Z. Zhao, Archit Sharma, Karl Pertsch, Jianlan Luo, Sergey Levine, and Chelsea Finn. Yell at your robot: Improving on-the-fly fly from language corrections, 2024. URL https::///// / a arxiv. /abr/bbb/// /203. 12910
Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang. Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, and Chelsea Finn. Hi robot: Open-ended instruction following with hierarchical vision-language-action models, 2025. URL httph:tttt///aee httt:p::/axxivv.orr/ab////// 2502. 19417
Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B. Tenenbaum, Leslie Pack Kaclbling, and Michael Katz. Generalized planning in PDDL domains with pretrained large language models, 2023. URL
https: org/ abs 2305 11014
David Snyder, Asher James Hancock, Apurva Badithela, Emma Dixon, Patrick Miller, Rares Andrei Ambrus, Anirudha Majumdar, Masha Itkina, and Haruki Nishimura. Is your imitation learning policy better than mine? policy comparison with near-optimal stopping, 2025. URL https
//arxiv. / abs/ / 2503. 10966
Gemini Model Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL
0557.06265
Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montser- rat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, Steven Bohez, Konstantinos Bousmalis, Anthony Brohan, Thomas Buschmann, Arunkumar Byravan, Serkan Cabi, Ken Caluwaerts, Federico Casarini, Oscar Chang, Jose Enrique Chen, Xi Chen, Hao-Tien Lewis Chiang, Krzysztof Choromanski, David D' Ambrosio, Sudeep Dasari, Todor Davchev, Coline Devin, Norman Di Palo, Tianli Ding, Adil Dostmohamed, Danny Driess, Yilun Du Debidatta Dwibedi, Michael Elabd, Claudio Fantacci, Cody Fong, Erik Frey, Chuyuan Fu, Marissa Giustina, Keerthana Gopalakrishnan, Laura Graesser, Leonard Hasenclever, Nicolas Heess, Brandon Hernaez, Alexander Herzog, R. Alex Hofer, Jan Humplik, Atil Iscen, Mithun George Jacob, Decpali Jain, Ryan Julian, Dmitry Kalashnikov, M. Emre Karagozler, Stefani Karp. Chase Kew, Jerad Kirkland, Sean Kirmani, Yuheng Kuang, Thomas Lampe, Antoine Laurens, Isabel Leal, Alex X. Lee, Tsang- Wei Edward Lee, Jacky Liang, Yixin Lin, Sharath Maddineni, Anirudha Majumdar, Assaf Hurwitz Michaely, Robert Moreno, Michael Neunert, Francesco Nori, Carolina Parada, Emilio Parisotto, Peter Pastor, Acom Pooley, Kanishka Rao, Krista Reymann, Dorsa Sadigh. Stefano Saliceti, Pannag Sanketi, Pierre Sermanet, Dhruv Shah, Mohit Sharma, Kathryn Shea, Charles Shu, Vikas Sindhwani, Sumeet Singh, Radu Soricut, Jost To- bias Springenberg, Rachel Sterneck, Razvan Surdulescu, Jie Tan, Jonathan Tompson, Vincent Vanhoucke, Jake Varley, Grace Vesom, Giulia Vezzani, Oriol Vinyals, Ayzaan Wahid, Stefan
13

---

Published as a conference paper at ICLR 2026
Welker, Paul Wohlhart, Fei Xia, Ted Xiao, Annie Xie, Jinyu Xie, Peng Xu, Sichun Xu, Ying Xu, Zhuo Xu, Yuxiang Yang, Rui Yao, Sergey Yaroshenko, Wenhao Yu, Wentao Yuan, Jingwei Zhang, Tingnan Zhang, Allan Zhou, and Yuxiang Zhou. Gemini robotics: Bringing AI into the physical world, 2025a. URL www.thttt//.//..mm ptatpa://arxisn org/ www./rgg///////.. ab///5b//223 20020
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaöl Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Buketee Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan- Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky. Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng. Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic. Amit Vadi. András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Seoouuetee-Choo CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimezak- Plucińska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, lan Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay. Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Poder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg. Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu Zichuan Wei. Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean. Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean- -aptiste Alayrac. Rohan Anil, Dmitry, Lepikhin, Sebastian Borgcaud, Olivier Bachem, Armand Joulin, Alck Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025b. URL htt.::ttt///eett httptpp://aaxxv.coo wwtp:vvoog....//./e./..ee 2503. 9786
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh. Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy, 2024. URL https: .rw.vv.rrr//ee wttp:rggg.b.......//// 12213
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lam- bert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. TRL: Transformer Reinforcement Learning, 2025. URL / conm/uuggingfac//tr//
Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Esstruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang. Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale, 2024. URL https
12952
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi. Quoe Le, and Denny Zhou. Ch prompting elicits reasoning in large language models, 2023. 11903
14

---

Published as a conference paper at ICLR 2026
Junjie Wen, Yichen Zhu, Jinming Li, Zhibin Tang, Chaomin Shen, and Feifei Feng. DexVLA: Vision-language model with plug-in in diffusion expert for general robot control, 2025. URL
httpss//arxiv org/ a 2502. 05855
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang. Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su. Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025a. URL http:: www.v..ivvvv..///iv///.../aa.a.. otg 250. 09388
Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng. Yongyuan Liang, Yu Gu, Mu Cai, Sconghyeon Ye. Joel Jang, Yuquan Deng, Lars Liden, and Jianfeng Gao. Magma: A foundation model for multimodal ai agents, 2025b.
org abs 2502 13130
Michal Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied -fthoht reasoning. 2025. URL htttp:://axxi..orpppppp
2407. 08693
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong. Stefan Welker, Fed- erico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with language, 2022.
http://bba//aaa/////////// 2204. 0
Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma. Song Han. Chelsea Finn, Ankur Handa, Ming-Yi Yu Liu, Donglai Xiang, Gordon Wetzstein, and Tsung- Yi Lin. CoT-VLA: Visual chain-of-thought reasoning for vision-language- action models, 2025.
22020
Peiyuan Zhi, Zhiyuan Zhang, Yu Zhao, Muzhi Han, Zeyu Zhang, Zhitian Li, Ziyuan Jiao, Baoxiong Jia, and Siyuan Huang. Closed-loop vocabulay mobile manipulation with gpt-4v, 2025. URL httpp:::/aaaii..rrr/aa//////////
Zhongyi Zhou, Yichen Zhu, Junjie Wen, Chaomin Shen, and Yi Xu. ChatVLA-2: Vision-language- action model with open- embodied reasoning from pretrained knowledge, 2025a. URL
https:// http:://aaxxv...oo owg.//i..rrreeo
wwtp:////////ooo.....oo..oosss/t abs 2505. 21906
Zhongyi Zhou, Yichen Zhu, Minjie Zhu, Junjie Wen, Ning Liu, Zhiyuan Xu, Weibin Meng, Ran Cheng, Yaxin Peng, Chaomin Shen, and Feifei Feng. ChatVL.: Unified multimodal understanding and robot control with vision vision-language-cttin model, 2025b. URL https: // /aapaaaxivvo.// orxiv.org/ abss25002.440
15

---

Published as a conference paper at ICLR 2026
VLM2VLA
VLM2VLA-AT
VLM2VLA-NR
VLM2VLA-Q
SASA
100
Out-of- Dift----iiiiittion
Performance (96
75
62
53
53
53
50
37
28
28
25
Pick Up
Pick and Place
Pick, Place, and Lift
Pick Up-1
Pick
Figure 8: Comparative evaluation of VLM2VLA ablations on in-distribution (ID) and out-of-distribttion (OOD) robotic manipulation tasks. VLM2VLA maintains high success rates on OOD tasks relative to ablated variants, highlighting the importance of reasoning and the robustness of methodology across different model architectures. Each bar correspouds to an average over thirty trials, except for the 'Pick Up- T" task, where each corresponds to an average over ninety trials.
A ADDITIONAL EXPERIMENTS
To further investigate the core components our methodology. we conduct two additional ablation studies examining: (1) how test-time reasoning impacts manipulation performance, and (2) if VLM2VLA generalizes to different VLM architectures. For the first ablation, we train VLM2VLA without the reasoning components (i.e., high-level subtask prediction and mid-level motion plamning), denoting this variant as VLM2VLA-NER For the second ablation, we apply our methodology to Qwen- 3-8B-IT IT Yang et al. (2025a), denoting this model VLM2VLA-Q. Both ablations evaluated on the same suite of manipulation tasks described in Section 4.2 with results reported in Fig. g We additionally include the action-token ablation (VLM2VLA-AT from Section 4.2
for completeness.
Does Reasoning Impact
VLM2VLA
VLM2VLA-AT
VLM2VLA-Q
Robotic Manipulation? We
100
100
100
investigate the contribution
Success Rate (9))
37
of test-time reasoning VLM2VLA's performance by comparing against VLM2VLA- NR, an ablation trained without reasoning capabilities (i.e., subtask prediction and motion planning). As demonstrated in Fig. 8 the absence of rea- soning substantially degrades performance longer-horizon tasks. Notably, VLM2VLA-NNR exhibits reduced success rates on pick-and-place tasks compared the full VLM2VLA model. This performance gap is further amplified in OOD scenarios that require leveraging latent world knowledge, where VLM2VLA-NR achieves markedly lower that the three-step reasoning procedure proposed in our method is critical for enabling effective manipulation, particularly in complex and novel task settings.
75
77
77
63
53
53
47
%5
Figure 9: Analysis of task decomposition OOD manipulation tasks, denoted by the corresponding icon. Points are awarded if the model's 's task plan correctly identifies the task object, and task destination, if present. See Appendix for additional details.
rates. These results underscore
Is VLM2VLA Model Agnostic? To assess whether our 'actions as language' approach generalizes across different VLM architectures, we train VLM2VLA-Q using identical data and hyperparameters as our primary model, but with Qwen-3-8B-B shown Fig. VLM2VLA-Q achieves competitive success rates across all tasks, demonstrating that methodology is indeed model agnostic. While VLM2VLA-0 exhibits slightly lower overall task success rates compared to the full VLM2YLA model, this difference can be partially attributed to the reduced model capacity (8B vs. 12B parameters). Interestingly, Fig. reveals that VLM2VLA-6 demonstrates comparable planning capabilities to VLM2VLA, successfully translating multilingual commands and correctly identifying task objects in scenarios requiring latent world knowledge. This
as the backbone instead of Gemma-3-12B-M As
16

---

Published as a conference paper at ICLR 2026
finding suggests that the gap in task success rates may stem from action execution precision rather than reasoning deficiencies. Nevertheless, the consistent performance VLM2VLA-Q validates that approach can be successfully applied to different VLM architectures.
B DATA CURATION
PROMPTING
In this section, we showcase Gemini prompting strategy to generate language-annotated robot trajectories scale. Our example specific to the WidowX 250S robot embodiment, focusing solely on translational degrees of fireedo. All text within the prompt is invariant across robot trajectories, except for words/phrases designated in color, which are trajectory-dependmn..
Gemini Inputs and Prompt Template
Inputs:
Gemini is provided with the robot trajectory data r = {(o, a
consisting of RGB observations and actions a, controlling the relative position of the robot's end-effector. Additionally, Gemini is provided with the main-task instruction L. We combine all actions and instruction into text file called trajectory.lo.contee
Gemini Prompt
Attached images of robot performing task, and the real actions in text file called trajectory log content. You are programming a 4-DoF robot arm (x.y.z.gripper). Positive points forward (negative points backward) with respect robot's end-effector. Positive points left (negative points right) with respect to the robot's 's end effector. Positive z points up (negative z points down) with respect to the robot's end-effecctrr. The gripper takes binary value of for open and 0 for closed.
Your task is to analyze the language instruction found in the trajectory log. Crucially, first think step-by-step and explain your reasoning for each action phase necessary to complete the instruction given all the images and their ordering during the trajectory. Be short and concise. Output your reasoning as structured JSON file.
Then, after you've ve explained your reasoning, give the actions with respect to (dx, dy, dz, gripper) that robot must execute achieve this goal given orientation described previously, associating each action group of actions with the natural language decom- position you decided on. Store these actions in JSON format as a list of dictionaries, where each dictionary has 'step deseription' (string) and "actions" (list of lists represent- ing [dx, dy, dz, gripper)). Make sure your actions match what is provided in the attached .xtt file. For example:
{{
sstep_dsscr inteescrppiion: "Grasp the Pepper", actions":
[0.010, 0.026, 0. 0.002, 1.0),
[0.017, 0.033, 0.008, 1.0],
[0.003, 0.007, -0.018, 1.0),
[-0.006, 0.0, 00.015, 1.0),
[0.003, 0.002, 0.001, 0.0]
}}
Here is the trajectory log content: trajectory lontent
17

---

Published as a conference paper at ICLR 2026
In summary, Gemini 2.5 Team 2025) is given the entire trajectory {(o,, (,)]f E Deeb and main-task instruction L as context, as well as the robot's coordinate system and sign convention in the base frame. For example, the Bridgev2 dataset Walke et al. (2024) utilizes the 6-DoF WidowX 250S robotic arm, where movement in the positive "x" direction corresponds to 'forward', negative 'right,' and forth. Since all action sequences
movement in the 'y' directions corresponds are given as relative movements of the robot's end-effector, the aforementioned coordinate system provides strong spatial grounding for Gemini construct high-quality language annotations and determine the most appropriate timesteps to carve the trajectory into sub- aob-trajectories. To address potential concerns about auto-labeling errors, we manually inspected random subset of Gemini's generated data for accuracy in subtask decomposition, motion-planning. and action prediction.
Initially, we solely used Gemini 2.5 Pro, but later switched to Gemini 2.5 Flash to save on costs. In total, generating the entire dataset costed roughly $900. Finally, we provide an example of single language-annotts trajectory from the Bridgev2 dataset. We additionally include the first and last observations of the robot trajectory for reference.
Figure 10: Key observations throughout the original, robot trajectory, which are provided to Gemini as context.
(a) Observation: (Initial)
(b) Observation: 10
(c) Observation: 20 (d) Observation: 28 (Final)
Gemini Language Annotation Example
Output:
Below is the structured JSON output generated by Gemini for the task of L = "open the silver pot'. Gemini first decomposes the trajectory into sequence of steps, then performs a chain-of-thouuht reasoning to generate motion plan. Finally, it copies the corresponding actions to each step.
"STEF_ DEEEEE_DESRIPPION "Move to Lid Handle", "REASONING"; "The robot must first navigate its
end-effector to be directly above the lid handle. This requires moving downwards (-dz) and adjusting the horizontal position (likely backward -dx and right -dy based on the initial position and target). The gripper must remain open (1) for grasping.".
"MAIN_MOVEMERR
"Primarily backward (-dx), right (-dy), and downward (-dz) movements. Gripper stays open (1).",
"ACTIONS":
0.0, 0.0, 0.0, 1,01,
-0.002,
-0.002,
-0.013,
1.0 },
-0.023,
-0.007,
-0.005,
1.0
-0.018,
-0.006,
0.022,
1.0
-0.018,
-0.005,
-0.003,
1.0 },
-0.018,
-0.006,
-0.005,
1.0 },
-0.015,
-0.014,
-0.014,
1.0
18

---

Published as a conference paper at ICLR 2026
-0.004, 00.0 -0.007, 1.0 ), C.O, -0.005, -0.003, 1.0
}
"STEP DESCRIPTION": "Grasp Lid Handle", "REASONING": "Once positioned correctly, the robot needs to grasp the handle. The robot closes the gripper (0) There might be a final small downward adjustment. (-dz) just before or during the grasp.".
"MAIN_MOVEMENTS": "Close gripper (0), minor downward adjustment (-dz). Gripper changes 0.",
"ACTIONS": C.00, -0.007, -0.010, 0.0
"STEP_ DESCREPTISR:::NN "Lift Lid",
"REASONING": "After securing the handle, the robot must lift the lid vertically (+dz) to clear the base of the pot. It also needs to move it slightly away from the pot base, involving some horizontal movement (likely forward +dx and left +dy). The gripper remains closed (0) to hold the lid.",
"MAIN_ MOVEENNNVEM:::"": "Primarily upward (+dz), forward (+dx), nd left (+dy). Gripper remains closed (0).",
"ACTIONS":
0.007, 0.0, -0.001, 0.0 1,
[ 0.002, 0.0, 0.0, 0.0 },
-0.002, -0.002, 0.0, 0.0 ].
-0.004, 0.0, 0.002, 0.0 ),
0.0, 0.004, 0.004, 0.0
[ C.015, 0.014, 0.017,
0.0 ),
0.023,
0.007, 0.020, 0.0 ]
},
"STEP DESCRIPTION": "Move Lid to Placement Spot & Lower", "REASONING": "The goal is to open the pot, implying the lid needs to be placed somewhere else. This involves horizontal movement (forward +dx, left +dy) towards the placement spot and then lowering the lid (-dz). The gripper stays closed (0).",
"MAIN_MOVEMENT
"Primarily forward (+dx), left (+dy), and downward (-dz). Gripper remains closed (0).",
"ACTIONS":
0.027, 0.008,
-0.003,
0.0 ]
0.021,
0.017,
-0.007,
0.0 1
0.018,
0.007,
-0.011,
0.0 1,
0.015,
0.007,
-0.010,
0.0 1,
0.010,
0.004,
-0.014,
0.0 ]
WETEE_DDESRIPTIONN. "Release Lid", "REASONING"; "Once the lid is resting the surface, the robor must release it by opening the gripper (1).", : "Open gripper (1).",
19

---

Published as a conference paper at ICLR 2026
"ACTIONS": 00003, 0.001, -0.002, 1.0
"STEP DESCRIPTION": "Retract", "REASONING": "Finally, the robot should move its arm away from the lid and the pot to a safe, neutral position, typically upwards (+dz) and potentially backward (-dx) The gripper remains open (1).", "MAIN_MOVEMENTSS: "Primarily upward (tdz) and backward (-dx). Gripper remains open (1).",
"ACTIONS":
-0.002, -0.002, 0.0, 1.0 },
-0.005,
-0.005,
0.006, 1.0 l,
C.0,0
-0.003, 0.022, 1.0 }
-0.010, 0.002,
0.022,
1.01,
-0.010, -0.001,
0.017,
1.0 1,
0.0.0 0.0, 0.001, 1.0
B.2 DATA PREPARATION
After Gemini splits each trajectory into sub-trajectories, found it helpful in policy training to chunk the actions along a single dimension to have larger magnitudes; in particular, we found that without this step, VLM2VLA often provided negligible action predictions, even though its predicted notion-plans appears sensible.
In this work, used threshold along single dimension, without sign change, of 2.5 centimeters and an absolute threshold of 5 centimeters. We include an example of the data for training Gemma-3 before and after this post-processing step below:
Original Sub- Sub-Trajectory Action Data
Inputs:
The model is provided with the subtask 'Grasp the Yellow Pepper' and image file: obs obpg..jpg
Human Prompt
You are programming 4-DoF robot arm. Based on the image (obs 0.jpg) and subtask 'Grasp the Yellow Pepper", output the sequence of low-level [x, y, 7, gripper] actions needed to accomplish the task.
20

---

Published as a conference paper at ICLR 2026
Model Output
[[ 000,
0.000,
0.000, 1.01,
[ [-0.002,
0.000,
-0.007, 1.01,
[ 0.000,
-0.004,
-0.016, 1.01,
[ 0.002,
-0.002,
-0.014, 1.01,
[ 0.003,
0.000, -0.008,
1.0],
[ 0.002,
0.000, -0.011, 1.0],
[ 0.000,
0.000, -0.005,
1.01,
[ 0.000, [ 0.000,
0.000,
-0.007, 1.01,
0.000,
0.0006
1.01,
0.001, 0.,03,, -0. 003, 0.011
Post-Processed Sub-Trajectory Action Data
Inputs:
The model is provided with the subtask 'Grasp the Yellow Pepper' and image file: obs_0.jpg
Human Prompt
You programming 4-DoF robot Based the image (obs 0.j.p) and subtask "Grasp the Yellow Pepper', output the sequence of low-level [x, y. z, gripper] actions needed to accomplish the task.
Model Output
002, -0.004, -0.023, 1.0), [0.007, -0.002, -0. 045, 1.0), [0.0, 0.0, -0.006, 1.0), 10.001, -0.003, -0.003, 0.0]]
We further augment our dataset with two additional sets of data. First we construct positive and negative examples subtask completion, each consisting i: the robot successfully completed subtask if and only if = + 1. In principle, this data could train the VLM as the verifier, but we found our model unable to reliably perform this step. We leave addressing this shortcoming future work.
tuple (ō,, 44,4) with j distinct from
Secondly, we train the model to predict which direction (e.g., left, right, or none) the robot should move to complete given subtask, using the same movement threshold of 2.5 centimeters. While we do not query our VLA this manner at run-time, we found this auxiliary training signal beneficial for downstream performance.
B.3 ACTIONS AS TOKENS
For the action representation ablation, we cast the digits 0-9 to the tenth least likely tokens under the Gemma-3. IT model. We describe this mapping the table provided in Table In addition, we provide an example of decoded model output under this mapping, identical to the one provided above.
21

---

Published as a conference paper at ICLR 2026
Table 2: Mapping of digits the least likely tokens under the Gemma-3 model.
Digit
Token
Token ID
<unused6142>
262044
<unused6141>
262043
7
<unused6140>
262042
<unused6139>
262041
5
<unused6138> 262040
<unused6137> 262039
<unused6136> 262038
<unused6135>
262037
<unused6134> 262036
<unused6133>
262035
Post-Processed Sub-Trajectory Action Data Under Action Tokenization
Model Output
<<u
3 -Kunused
6133>
1113 <unused6134> (umssee <unused6133+),
-anneeeeccccccccaa3erraaaaaeaaas <cun
(err:>>uunsee -<unused6
[33_.caaeeedii33nnnaddd333m.a139 <unused61343..
133>
unused6133>1,
<unuse
[rangpeccc.cccccd33333mmm333s3aa <un
iaa.gl3heeccuuuddc333mmmme3333et -Kunused6
1.5.2xmuaedd1133xmmaadd33aaaeea3 <anp unused6133>1
<unused6133>.<
C VQA EXPERIMENT DETAILS
We build our evaluation pipeline using the open-source framework I evaaevval which provides standardized implementations of multimodal benchmark datasets, evaluation metrics, and inference wrappers for wide range of models. We report results for our VLM2VLA model (finetuned with Gemma-3-128 backhone), two strong baselines (OpenVLLA and ECoT, both Prismatic-7B backbone), and the original instruction-tuned Gemma-3-4B and Gemma-3-11BB models.
MolmoAct and No.s are designed handle inputs from multiple cameras (e.g., exterior and wrist views). However, since standard VQA benchmarks provide only single image per sample, use multi-view vision-language model to process single-image VQA benchmarks by utilizing only the primary image input from each sample and masking the other image inputs. We use the To.5 b-bse checkpoint for To.5 experiments.
D ROBOT MANIPULATION TASKS AND DETAILED RESULTS
D.1 EXPERIMENTS AND MANIPULATION EVALUATION METRICS
As described in Section 4.2 we evaluate all generalist robot policies on two in-distribution tasks, one borderline in-distribution task, and two out-of-distribution tasks. Each policy received total of 30
https://gi Nvol ng MMMs l.ab/1 mns-eva eval
22

---

Published as a conference paper at ICLR 2026
roll-outs for each task. In this section, provide additional details the scoring rubric used to evaluate each method, giving special care to fairly evaluate planning and non pon-nlanning policies.
Pick Up the Carrot: The robot's goal is to navigate and lift up the carrot. There are no distractor objects present. This experiment is considered within distribution because it is part of the Bridgev2 dataset.
Between trials, the carrot's initial position varied between three possibilities: (i) directly below the robot's gripper, (ii) directly in front of the robot, (iii), in front and to the left right of the robot. The robot's initial position was constant between trials.
Scoring: Partial credit (1 out of 2) is assigned if the robot makes contact with the carrot.
Put the Carrot On the Yellow Plate: The robot's goal is to grasp the carrot, move to the plate, and successfully release it on the plate. There are distractor objects present. This experiment is considered within distribution because it is part of the Bridgev2 dataset.
Between trials, the carrot's initial position varied between two possibilities: (i) directly in front of the robot and (ii) in front and to the left or right of the robot. The robot's initial position was constant between trials.
Scoring: Partial credit (I out of 2) is assigned if the robot makes contact with the carrot.
Put the Eggplant In the Pan, Then Lift the Fish: The robot's goal is to grasp the eggplant, move and release it in the pan, then navigate to and successfully lift the fish. This task is considered borderline in-distribution because the individual subtasks are within distribution, but the combination is not.
The eggplant is located to the right of the fish between all trials, which is located in front of and to the right of the robot's 's initial condition; all starting positions for the eggplant, pan, fish, and robot are invariant across trials.
Scoring:
For non- planning policies (Open makes contact with the eggplant, (2 out of 5) if the robot places the eggplant the pan, (3 out of 5) the robot moved toward the fish, (4 out of 5) if the robot contacts the fish, and (5 out of 5) if the robot lifts the fish.
partial credit (1 out of 5) is awarded if the robot
For planning policies (ECoT, VLM2VLA), partial credit (1 out of 5) is awarded if the policy generates a subtask decomposition or motion-plan -laan with the correct primitives: grasping the eggplant, moving to the pan, lifting the fish. The remaining partial credit is awarded the same as for non-plannign policies: (2 out of 5) if the robot makes contact with the eggplant, (3 out of 5) if the robot places the eggplant in the pan, (4 out of 5) if the robot contacts the fish, and (5 out of 5) if the robot lifts the fish.
Pick Up the Carrot (Multilingual Translation): The robot's goal is to navigate to and lift up the carrot. Instructions are given in one of three languages: Spanish, Mandarin, and Hindi This task is considered out of distribution because the language commands are not within the Bridgev2 dataset.
Between trials, the carrot's 's initial position varied between three possibilities: (i) directly below the robot's gripper, (ii) directly in front of the robot, (iii). in front and to the left or right of the robot. The robot's initial position was constant between trials. Two distractor objects (banana and eggplant) present in each trial in an attempt to prevent the policy from inferring the task from the observation alone.
For non-planning policies (OpenVLA), partial credit (1 out of 2) is awarded if the robot makes contact with the carrot. For planning policies (ECoT, VLM2VLA), partial credit out of 2) is awarded if the policy's 's planning included the correct item 'carrot' and the robot makes contact with the carrot.
Scoring:
The robot's goal is to navigate to and lift up the object situated above the pop-culture figure 'Ash Ketchum." This task is considered out of distribution because the picture of Ash Ketchum" is not present in the Bridgev2 dataset.
Pick Up the Item Above Ash Ketchum:
Between trials, the picture of Ash Ketchum' varied between being either to the left or right of the robot's ss starting position, which remained invariant across trials. The target object is placed a few centimeters above the picture to preclude partial observability.
23

---

Published as a conference paper at ICLR 2026
Scoring: For non non-planninn planning policies (OpenVLA), partial credit (1 out of 2) is awarded if the robot makes contact with the correct object. For planning policies (ECoT, VLM2VLA), partial credit (1 out of 2) is awarded if the policy's planning included the correct item and the robot makes contact with the item.
D.2 HARDWARE DETAILS
All real-world policy evaluations were performed on a 6-DoF WidowX 250S robotic arm in a toy kitchen environment, as prescribed in (Walke et al. 2024). Images were captured with a Realsense D435 camera mounted on the right side of the robot to provide initial conditions of task and distractor objects between trials were varied ensure an accurate appraisal of policy performance. Like the environment, all objects are drawn from the Bridgev2 dataset minimize aut-of-distribution failures. To prevent damage to our setup, safety filter was applied to all policies evaluated in this work. The filter prevented further execution of actions which would drive the robot into the environment, c.g., zero out downward or forward commands the end-effector contacting the counter wall.
point of view. The
D.3 TASK DECOMPOSITION EVALUATION METRICS
In this section, we describe the scoring procedure used generate Fig. The objective quantify each policy's 's ability to interpret and break down a variety of user instructions. Performance was measured by analyzing the textual output of each model's thain-of-thoughle reasoning, or subtask decomposition, whenever available.
For each task considered from Appendix D.I
model's reasoning was considered successful if the model's 's textual output contained specific set of required keywords. The evaluation was case- insensitive and manually checked to ensure all policies received fair appraisal. The keywords and synonyms are defined as follows:
eggplant: purple, aubergine
carrot: orange, gajar
pan: (no synonyms defined)
fish: (no synonyms defined)
We now specify the scoring rubric for each experiment which required analysis:
Put the Eggplant In the Pan, Then Lift the Fish: success required the simultaneous presence of synonyms for "eggplant," "pan,' and 'fish.'
Pick Up the Carrot (Multilingual Translation): success required the presence of a carrot synonym in the model's output. Interestingly, some subtask decompositions from VLM2VLA occasionally contained the word 'ajar,' the Hindi synonym for carrot, demonstrating direct translation abilities of our method. This phenomena was not observed for Spanish nor Mandarin inputs.
Pick Up the Item Above Ash Ketchum: success required identifying the correct object for that specific trial. The required keyword was either a synonym of carrot or eggplant, depending on the ground truth object for that trial.
D.4 TEST-TIME PROMPTING
In this section, we describe the prompting strategies used in VLM2VLA to elicit grounded subtask predictions, spatial reasoning for motion-pla paaionn, and translational action prediction.
First, given the initial image, prompt VLM2VLA to break the main language instruction into series of subtasks.
24

---

Published as a conference paper at ICLR 2026
Subtask Prediction Prompt
Describe the sequence of remaining high-level steps required to complete the overall task 'main task', starting from the current state. Give your output list. Be specific and do not skip steps. Here is an example for put the pot in the sink': ["Move Down to Pot', 'Grasp Pot', 'Lift Pot High', "Move Pot Left the Sink', 'Lower Pot to Sink', 'Release".. Start with 'Move to'
For given subtask, we prompt VLM2VLA with the current observation and subtask.
Motion-Planning Prompt
Given the image, reason about what high-level actions the robot arm should take to complete the task "subtask". +dx is forward, -dx is backward, +dy left, -dy right, +dz up, -dz is down, 1 is gripper open, 0 is gripper closed. Provide concise, accurate spatial reasoning.
Once motion-plan is generated, VLM2VLA is prompted with that reasoning, current observation, and subtask.
Action Generation Prompt
You are programming a 4-DoF robot arm (x.y.z.gripper). +dx is forward, -dx is backward, +dy is left, -dy is right, +dz is up, -dz is down, 1 is gripper open, 0 is gripper closed. Based on the sub-task "subtask' and the reasoning "motion-pla" reasoning' output the sequence of low-level [dx, dy, dz, gripper] actions needed as python list of lists. Be concise in direction of movement. Example format: [[dx1,dyl,de] 1,z1, gripl]. [dx2,dy2,d22, grip2]]. PROVIDE ONLY THE PYTHON LIST.
All outputs from VLM2VLA are generated using nucleus (top-p) sampling with p=0.95. We use distinct temperature settings for each generation stage: 0.1 for motion planning and 0.5 for action prediction. For the initial sub-task decomposition stage, the temperature set 0.5 for in-distribution tasks and 1.0 for all other scenarios. The aforementioned hyperparameters were empirically found to work well.
We use the default setting of greedy decoding for both the OpenVLA and ECoT baselines. The task prompts varied slightly between policies to maximize performance each model on a given task. While all models received near identical commands, we followed the prompt formats suggested by Kim et al. 2024| Zawalski et al. 2025) for the baselines, and used our own for VLM2VLA and VLM2VLA-AVE AT (see Appendix D.4)
D.5 VERIFIER PROMPTING
To improve robustness of the policy, we operate in closed-loop fashion with verifier V O O x L x L - L, where O is the RGB image space and L is the language space; at the end of each action-gence sirionngnneratinn cycle, we query the verifier with the observations before (ō:) and after (ō++1) action execution (ā:), as well as the current (4)) and next (4+1) subtask. The verifier V V(o,, 0.+1, 4, 4++) will reason if the current subtask was completed successfully, given the observations and the next subtask. The output from the verifier dictates the subtask used in the next action-generann which continues until all N subtasks are completed. In this work, we utilize Gemini 2.5 Pro (Team 2025) as the verifier, albeit one may train the model itself to do this step.
cycle,
In this section, we describe the prompting strategies used in this work to query the external verifier the subtask was successfully completed.
25

---

Published as a conference paper at ICLR 2026
Verifier Prompting Strategy
Inputs:
After having executed action chunk ag Gemini provided with two observations, (Ø; 0:+1) and the current and next subtask (4,, l4+a).
Verifier Instructions
You are a precision- oriented robot inspector. Your primary job is to evaluate the two images (Øi, 0:-1) if the "Current Subtask' (4:) executed with enough precision guarantee the success of the "Next Subtask' (1++1).
CRITICAL RULE: Judge the Precondition.
A 'move to' action is ONLY successful the final position is perfectly aligned for the subsequent action, i.e., executing grasp as the step will result in successful grasp.
A "close enough' position is FAILURE if it jeopardizes the next step.
For a 'grasp' subtask succeed, the gripper MUST be centered directly above the object's grasp point in the preceding 'move' step. Any significant offset or misalignment in the "after" image a failure.
Provide your analysis in the specified JSON format, following the turn-based examples.
Expected Model Output (JSON)
"success"; true, "confidence": "High", "reasoning": "This is a successful grasp because the second image clearly shows the robot has successfully grasped the eggplant."
D.6 INFERENCE LATENCY
We conducted series of rollouts and measured the wall-clock time for single inference eycle, which we defined as the generation of both the mid-level reasoning trace and low-level action prediction. All experiments were conducted on an NVIDIA A100 GPU, We collected data across 30 evaluation runs and report the summary statistics in Table 3
Table 3: Inference Latency Statistics (N=30 trials)
Statistic
Value
Median
6.1 [s]
Mean (Average)
10.5 [s]
Standard Deviation
14.3 [s]
Interquartile Range (IQR)
5.0 6.7 [s]
Minimum
3.8 [s]
Maximum
48.8 [s]
The median of 6.1 seconds and small interquartile range demonstrate VLM2VLA is capable of fast inference. However, the high standard deviation of 14.3 seconds warrants discussion. Our analysis of
the output logs revealed that 1) approximately 10% of cases triggered retry mechanism due
26

---

Published as a conference paper at ICLR 2026
bad output formatting from model, causing additional run-time and 2) small subset of trials exhibited unusually long run-times (> 45 seconds), suggesting a computational bottleneck.
These results highlight the importance of developing fast decoding schemes for low-latency perfor- mance in real-world scenarios, which we reserve for future work.
E TRAINING DETAILS
We fine-tune the Gemma-3- 3-12B-IT (Team et al. 2025b) model using the TRL library (von Werra et al. 2025). Training is managed with Hugging Face's Accelerate framework, utilizing DeepSpeed ZeRO Stage configuration for efficient memory usage across multiple GPUs. We employ parameter- efficient fine-tuning (PEFT) using low-rank adaptation (LoRA) to update the model weights.
To obtain the final checkpoint, train the model single node with 4 NVIDIA A100 GPUs. The final model was fine-tuned for one epoch, which we found sufficient for this work. In total, our model required approximately 300 GPU hours. The key hyper-parameters used for this training run are described in Table 4 notably, we found that using a small effective batch size worked well.
Table 4: VLM2VLA Training Saper--pramettrs
Hyperparameter
Value
Training Strategy
Base Model
Cemma-33-1B-II
Frameworks
TRL, Accelerate, DeepSpeed (ZeRO Stage PEFT (LoRA)
Fine-Tuning Method
Precision
bfloat16 (BF16)
LoRA (PEFT) Configuration
Rank (r)
Alpha (a)
Target Modules
q.rroj, k.proj, v.proj, o.prod, up_proj, down_proj, gate.proj
Optimization
Optimizer
AdamW
Learning Rate
5e-5
IR Scheduler
Linear Decay
Adam Betal
Adam Beta2
0.999
Adam Epsilon
1c-8
Dataloader Configuration
Global Batch Size
Per-Device Batch Size
Gradient Accumulation Steps
Effective Global Batch Size
Max Sequence Length
1024
F USAGE OF LARGE LANGUAGE MODELS (LLMS)
During the preparation of this work, the authors utilized Large Language Models (LLMs) to support various aspects of the research and writing process. The specific applications LLMs in this paper outlined below:
Code Generation and Debugging: LLMs were employed to generate and debug code in the preliminary stages of this work. This included creating Python scripts to generate the plots presented
27

---

Published as a conference paper at ICLR 2026
herein. Moreover, LLMs assisted in debugging error logs when training VLM2VLA and proposing the corresponding fixes.
Manuscript Preparation: LLMs were utilized to format tables in LaTeX code for use in this manuscript. Additionally, LLMs aided in creating the template used the Appendix to designate the 'Human Prompt' and the 'Model Output.'
Writing Assistance: LLMs were queried to enhance the quality and readability of the manuscript. This involved checking grammar, and suggesting edits to improve sentence structure, syntax, and clarity, thereby helping to polish the final version.
28


Reviews:
[{"review_id": "paMJb6ipQF", "reviewer": "Reviewer_9qWQ", "summary": "The paper focus on a reasonable issue, finetuning VLM to VLA will cause the VLM lost its memory for QA but overfit to robot control. The paper proposes to solve this issue by convert the low level action to natural language format, then use lora finetuning.", "questions": "See Weakness for big concern. Besides those:\n1. Seems in section 4.2, it shows some results on robotic control task, but there is some conerns:\na. it deos not compare with a strong baseline, like pi0\nb. the task is too simple, mainly pick and place, which i think should focus more on long horizon task, which might better show the method can keep more reasoning or planning ability of the pretrianed VLM.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The scope of this paper makes sense, which i agree that fine-tuning the VLM to VLA wil let the VLA lost its QA reasoning ability and overfit to specific control actions prediction.\n\n2. The paper is well organized and easy to follow.", "weaknesses": "1. Even the paper throws a meaningful problem, which is how to fine-tune a vlm to vla without degrade its ablity for reasoning, but the paper's solution is not cool enough, since just format low level actions as language then train it using formal vlm training, this is not novel and used by many papers like [1][2]\n\n2. Not sure what is the paper's main contribution, seems Table 1 want to claim VLM2VLA performs better in series VQA task, but, do the VQA person will use VLA for VQA task, or Robotics ppl care these VQA task? So not sure if evaluate on these VQA task really make sense if this work focus on VLA.\n\n\n[1] Niu, D., Sharma, Y., Biamby, G., Quenum, J., Bai, Y., Shi, B., ... & Herzig, R. (2024). Llarva: Vision-action instruction tuning enhances robot learning.\n[2] Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., ... & Han, K. (2023, December). Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning (pp. 2165-2183). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nBsGBI8Y64", "reviewer": "Reviewer_75Ka", "summary": "This paper proposes VLM2VLA, a data-centric method for fine-tuning Vision-Language Models (VLMs) into Vision-Language-Action (VLA) models while minimizing catastrophic forgetting. The key idea is to represent robot actions as natural language phrases rather than discrete tokens or continuous vectors, allowing fine-tuning via LoRA without modifying the base VLM architecture.", "questions": "1. What’s the X axis of figure 6 and does the images of fish and country flag correspond to the plot?\n2. Why only are translational dofs considered?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Casting low-level actions as text tokens is conceptually elegant and well aligned with the VLM’s pretraining distribution, avoiding token-level mismatches common in prior VLAs.\n2. Experiments show that multimodal reasoning and multilingual understanding are preserved, as in across diverse VQA benchmarks and real-world robotic tasks.\n3. The proposed pipeline of using LoRA fine-tuning only avoids costly co-training or architecture modifications.", "weaknesses": "1. The comparison with baselines on robotics tasks are limited. MolmoAct and π0.5 are mentioned in Table 1 but not reported in robot experiments. Other VLA works that address the catastrophic forgetting problems[2] or unifying the architectures[1] are not compared.\n2. Robot experiments are relatively simple and there are no simulation experiments, which could have provided scalable quantitative metrics.\n3. While compared with an action expert ablation with discrete tokens, there is no comparison with flow-matching or diffusion-based action experts such as π0.5 and MolmoAct, which provides state-of-the-art continuous control.\n4. The proposed three-stage hierarchy (subtask → motion plan → action chunk) is promising but conceptually overlaps with system-1/system-2 reasoning paradigms (planning vs. acting). A comparison with a 2 system VLA is needed to validate the proposed method, where a system2 is responsible for planning and translating and a system1 is responsible for action generation.\n\n[1] Lin, Fanqi, et al. \"OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoning.\" arXiv preprint arXiv:2505.11917 (2025).\n\n[2] Huang, Huang, et al. \"Otter: A vision-language-action model with text-aware visual feature extraction.\" arXiv preprint arXiv:2503.03734 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AiLymeqX1o", "reviewer": "Reviewer_89io", "summary": "This paper presents the VLM2VLA pipeline, which turns VLMs into VLAs while maintaining their generalization performance for OOD settings. It addresses the critical issue of VLA pretraining degrading VLM features over time, due to the large domain shift between VLM training and action generation. The concept is based on addressing the distribution mismatch between actions and text, which typically undermines the knowledge gained from VLM pretraining.\n\nThe method involves modeling actions using words and then fine-tuning the VLM on that action word data. Since the VLM has encountered many instances of numbers in the context of science texts, etc., it already has a good understanding of them and can easily grasp the concept of action logic. Crucially, it's a VLA training pipeline that uses LoRA to maintain the VLM features. It also uses a hierarchical reasoning setup in combination with the text actions: high-level subtask -> mid-level motion planning -> low-level action. Results on VLM benchmarks and various real-world manipulation setups demonstrate that the proposed method is able to preserve VLM knowledge while serving as a good VLA with improved OOD performance.", "questions": "- Can you add ablations for the hierarchical word action setup to study the impact of each hierarchy level on the performance? Especially since all are generated by Gemini and not humans. How important are good plans? Would be adoption work also with false traces? More insights here would strengthen the paper\n- Additional tests with other VLMs would also be great to see how the pipeline performs across this axis\n- Is LORA for all training stages a consequence of limited compute? How would it behave with smaller VLMs and full finetuning? Any insights on LORA rank and downstream performance?\n- Does it scale to SIMPLER benchmark evaluation with 7 DoF? If so, what are the results?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This is an overall very insightful paper with good experiments and results, which is helpful for the VLA community.\n- It also tackles a critical issue in VLA pretraining, where the VLA training process destroys the VLM's foundational features\n- The Action as Language Representation is a novel with good results but limited ablations\n- Demonstrating that this method can work with VLA training solely using LoRA is a significant advantage for achieving efficient VLA adoption from VLMs.\n- The OOD experiments and results are promising and demonstrate that the method effectively maintains VLM generalization.", "weaknesses": "- The current main weakness of the paper is the weak ablations of their key contributions: no ablation of how the 3-stage word actions impact the training, with baselines that only predict low-level actions, mid-level ones, etc\n- Low-level action has translational DoF only, without any more complicated movements\n- Very low inference speed, with several seconds for a single action prediction\n- No VLM-centric ablations. It would have been interesting to study how different VLM backbones affect the performance of this pipeline", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qr0SZWgnPu", "reviewer": "Reviewer_HTUd", "summary": "The paper introduces *VLM2VLA* to overcome the trade-off where training a powerful VLM to control a robot leads to a loss of its general understanding. By first translating robot actions into natural language, VLM2VLA bridges the gap between the VLM's pretraining and the robot data. This technique allows the model to be efficiently fine-tuned with LoRA, keeping its original knowledge intact. This successful preservation is experimentally validated by the robot's ability to demonstrate advanced semantic reasoning and zero-shot generalization on novel, complex tasks.", "questions": "I have a couple of questions regarding the experimental methodology:\n- Have you ablated different VLM models for the purpose of task curation?\n- Was the robot learning fine-tuning based exclusively on the Bridgev2 dataset, or did it also include real demonstrations collected from the exact kitchen setup depicted in Figure 5? \n- I am also interested in whether the In-Distribution (ID) tasks are considered ID solely based on the instruction and objects, or if they also rely on the identical physical kitchen setup and object locations.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper is well-written and structured.\n- The problem is clearly stated and is highly interesting and relevant to the robot learning community.\n- The solution is quite novel. The idea of using natural language to describe actions is interesting and exceptionally well-presented and motivated in the paper, particularly in Figure 3.\n- The experimental setup is strong; however, the results are most fascinating in the Out-of-Distribution (OOD) tasks, especially those where instructions were given in different languages. This particular study provides strong evidence that the *VLM2VLA* method effectively preserves the VLM's capabilities. \n- The limitations and future work are clearly stated.", "weaknesses": "While this work presents many positive aspects, I offer a few suggestions that could further strengthen its quality and experimental validation.\n- The baselines $\\pi_{0.5}$ [1] and MolmoAct  [2] should also be evaluated on the real-world tasks for completeness, consistent with the other baselines in Section 4.2. Including these results would significantly strengthen the experimental validation and make the overall conclusions more robust.\n- I believe the performance on the VQA tasks for the corresponding reference VLM models for $\\pi_{0.5}$ [1] and MolmoAct [2] should be added as done with OpenVLA by including Prismatic VLM and VLM2VLA by including Gemma-3.\n- While the chosen tasks are compelling and relevant, the overall number of experimental tasks discussed in the paper remains limited.\n\n[1] Intelligence, Physical, et al. \"$\\pi_ {0.5} $: a Vision-Language-Action Model with Open-World Generalization.\" arXiv preprint arXiv:2504.16054 (2025).\n\n[2] Lee, Jason, et al. \"Molmoact: Action reasoning models that can reason in space.\" arXiv preprint arXiv:2508.07917 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)