VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models
Authors: Jianke Zhang
Xiaoyu Chen
Yanjiang Guo
Yucheng Hu
Jianyu Chen
Venue: ICLR 2026 Poster (2026)
Keywords: VLM
VLA
Empirical Study
Primary area: applications to robotics, autonomy, planning
TL;DR: We present a simple and effective framework to fairly benchmark different Vision-Language Models (VLMs) as backbones for robotic policies, revealing a notable performance gap that highlights a disconnect between the VLM and VLA domains.

Abstract: Vision-Language-Action (VLA) models, which integrate pretrained large Vision-Language Models (VLMs) into their policy backbone, are gaining significant attention for their promising generalization capabilities. This paper revisits a fundamental yet seldom systematically studied question: how VLM choice and competence translate to downstream VLA policies performance? We introduce \textbf{VLM4VLA}, a minimal adaptation pipeline that converts general-purpose VLMs into VLA policies using only a small set of new learnable parameters for fair and efficient comparison. Despite its simplicity, VLM4VLA proves surprisingly competitive with more sophisticated network designs.
Through extensive empirical studies on various downstream tasks across three benchmarks, we find that while VLM initialization offers a consistent benefit over training from scratch, a VLM's general capabilities are poor predictors of its downstream task performance. This challenges common assumptions, indicating that standard VLM competence is necessary but insufficient for effective embodied control.
We further investigate the impact of specific embodied capabilities by fine-tuning VLMs on seven auxiliary embodied tasks (e.g., embodied QA, visual pointing, depth estimation). Contrary to intuition, improving a VLM's performance on specific embodied skills does not guarantee better downstream control performance.
Finally, modality-level ablations identify the visual module in VLM, rather than the language component, as the primary performance bottleneck. We demonstrate that injecting control-relevant supervision into the vision encoder of the VLM yields consistent gains, even when the encoder remains frozen during downstream fine-tuning. This isolates a persistent domain gap between current VLM pretraining objectives and the requirements of embodied action-planning. 
Project Page: \href{https://cladernyjorn.github.io/VLM4VLA.github.io/}{https://cladernyjorn.github.io/VLM4VLA.github.io}.

Published as a conference paper at ICLR 2026
VLM4VLA: REVISITING
VISION-LANGUAGE. MODELS IN VISION-LANGUAGE-ACTIONN MODELS
Jianke Zhang', Xiaoyu Chen', Yanjiang Guo', Yucheng Hu', Jianyu Chen' Institute for Interdisciplinary Information Sciences, Tsinghua University
ABSTRACT
Msion-Languugg-AAction (VLA) models, which integrate pretrained large Vision- Language Models (VLMs) into their policy backhone, are gaining significant attention for their promising generalization capabilities. This paper revisits fundamental yet seldom systematically studied question: how VLM choice and competence translate to downstream VLA policies performance? We introduce VLMs into VLA policies using only small set of new learnable parameters for fair and efficient comparison. Despite its simplicity. VLM4VLA proves surprisingly competitive with more sophisticated network designs. Through extensive empir- ical studies on various downstream tasks across three benchmarks, we find that while VLM initialization offers a consistent benefit over training from scratch, VLM's general capabilities are poor predictors of its downstream task perfor- mance. This challenges common assumptions, indicating that standard VLM competence is necessary but insufficient for effective embodied control. We fur- ther investigate the impact of specific embodied capabilities by fine-tuning VLMs seven auxiliary embodied tasks (e.g.. embodied QA, visual pointing, depth estimation). Contrary to intuition, improving a VLM's performance on specific embodied skills does not guarantee better downstream control performance. Fi- nally. modality-leved ablations identify the visual module in VLM, rather than the language component, as the primary performance bottleneck. We demonstrate that injecting control-relevant supervision into the vision encoder of the VLM yields consistent gains, even when the encoder remains frozen during downstream fine-tuning. This isolates persistent domain gap between current VLM pretrain- ing objectives and the requirements of embodied action- ection-planning. Project Page: http:///deeeeooooouuouuuuu/o////
VLM4VLA, minimal adaptation pipeline that converts
INTRODUCTION
Lirion-Languuge-Atttee (VLA) models (Brohan et al., 2023) have recently emerged as a central
research focus, they leverage the extensive vasual-language knowledge from Vision-Language Models (VLMs) as a prior for enhancing the generalization of robotic strategies (Kim et al. 2024
Black et al. 2024 Zhang al. [0244] Chen al., (025c). The majority of existing VLA methods
have focused on developing more advanced network architectures (Li et al.) 2023 Shi et al. 2025).
incorporating additional training paradigms or modalities (Zheng al. [024]]] Chen et al. 2024a
Zhang et al. 2025). ). and refining action decoding schemes (Zhao et al., 2023 Pertsch et al. 2025
Wen al. 2025). However, limited attention (Liu al. (025) has been given fundamental
question at the core of VI.A: how do the choice and specific capabilities of the underlying VLM
affect the performance of VLA policies?
In this paper, we revisit this crucial problem. To provide fair and clean test interface that evaluates the capabilities of VLMs without introducing extraneous variables, we first build the generic VLM4VLA pipeline convert general-purpose VLMs into VLAs, shown in Figure 2 VLM4VLA is carefully designed network plug-in, introducing fewer than 1 1% new parameters. To enhance the stability of inference and the robustness of evaluation, we simple MLP head rather than diffusion- based approach, thus controlling stochasticity and reducing tuning complexity. This network allows us to train the modified VLMMs directly using downstream robot data, facilitating alignment between

---

Published as a conference paper at ICLR 2026
"Slide Window
"Open Drawer"
*Pick Block'
Cube SimplerEnv Bridge
Spoon
paligemmat
paligemmaz
Kosmood
Task-1
qwen2 SwfBb
42 qwen2.554 Swthb
qwenšvi 4b
Thsk-2
Carrot
Action Policy
Vision Language Model
Teak-3
Linero-Long
Calvin ABC-D
Task-
Length
Head
Visual Grounding & Pointing
Trajactory & Action Planning
Segmentation
Head
Head
LLM
VLM
VLM
Vision
VLM SFT
Tasks
Robot Data
Robot
Robot Data
(a)
(b)
(c)
Depth Estimation
Videe Spatial Understanding
An overview of our VLM4VLA framework. (Left)) The evaluation pipeline for testing different VLM backbones, which are evaluated on downstream tasks after an optional fine- tuni-tuning stage auxiliary embodied tasks. (Bottom Right) We systematically investigate three factors influencing VLM-to-VLA transfer: the choice of VLM backbone, the impact of fine-tuning on auxiliary embodied tasks, and the vision encoder's training strategy (frozen fine-tuned). (Top Right) A visualization of inconsistent performance of various VLM backbones across downstream tasks.
Figure 1:
the VLM's capabilities and the demands of robotic tasks. Despite its simplicity, VLM4VLA proves effective, demonstrating competitive performance against more advanced network designs, such as the flow-matching action expert (Black al., 2024), on benchmark tests. This provides solid foundation for conducting fair and scalable experiments across various VLMs, aligning their capabilities with demands of embodied tasks.
Based on the VLM4VLA pipeline, we conduct large-scale empirical study across various down- stream tasks on three commonly used benchmarks, evaluating 17 different VLMs that are either zero-shot fine-tuned. Specifically, we investigate the VLM capabilities across three dimensions:
General capability. Given that the VLM backbone provides superior generalization in VLAs, we first explore the relationship between the general capabilities of VLMs and their performance on downstream control tasks.
Embodied-speciffc capability. Recent works have sought to enhance VLAs by improving the VLM backbone on a series of VLM tasks. examine how the embodied-speciff capabilities of VLMS correlate with performance downstream control tasks.
Vision encoder. Finally, we investigate the influence of the vision encoder in VLMs on downstream control task performance.
For general capability, we select seven open-sourced VLMMs as the backbone. shown in Figure T (a), we apply the VLM4VLA pipeline directly to these VLMSs and fine-tune them with robot data. For embodied-speciffe capability, we collect seven commonly used embodied tasks or pretrained models based on the Qwen2 VL (Bai et al.] (025) backbone to conduct ablation studies. As shown in Figure [ (b), we either fine-tune the VLM using auxiliary datasets or directly use the released fine-tuned models. We then apply the VLM4VLÁ pipeline adapt these models to the embodied policies. Lastly, for the vision encoder, we test three VLMs with either frozen or fine-tuned vision encoders (Figure (c)). We evaluate the fine-tuned models range of tasks across three
2

---

Published as a conference paper at ICLR 2026
benchmarks including Calvin (Mees et al. 2022). SimplerEnv 2023) to assess their embodied capabilities.
(Li et al. 2024). and Libero (Liu et al.
To our suprise, our findings reveal that VLM's general capabilities are poor predictors of its downstream task performance, contrary to common assumptions. For instance, we observe that Kosmos (Peng et al., 2023) outperforms Qwen-2 5VL (Bai et al. 2025) and Pa gemma (Steiner et al. 2024) across multiple environments. Inconsistencies across benchmarks suggest that VLA policies require capabilities beyond those currently pursued by VLMs. Furthermore, the gains by fine-tuning VLMs on specific auxiliary embodied tasks do not transfer to the downstream control tasks. Lastly, we identify the vision encoder as critical bottleneck, with fine-tuning the vision encoder proving crucial for strong control performance.
Taken together, these results indicate a significant gap between current VLM research and the practical demands of VLA models. We believe this work highlights the fundamental question of the VLM's role embodied agents and can help guide future exploration in this area.
2 RELATED WORKS
Vision-Language-AAchie Models Recent works are increasingly focusing on introducing Vision-
Language-Models (VLMs) (Dai et al. 2024 Touvron et al. 2023; Wang et al.
2025
Bai al.
2025) into robot policies to enhance their generalization capabilities (Brohan et al. 2023
Guo et al.
2025 Hu et al., 2024| Chen et al. 2024b), which are known Vision-
Action (VLA)
models. Early methods like RT-2 (Brohan et al. 2023) and OpenVLA (Kim et al. 2024) discretized
actions into language tokens, enabling action learning through the VLM's autoregressive framework. Subsequent works have utilized policy heads to decode continuous actions, gradually evolving the VLA design into a hierarchical structure that combines VLM with policy head (Black al.) 2024; Zhang et al. 20244] Cui et al.) 2025). However, most prior works focused on constructing complex policy networks, while overlooking the impact of the VLM backbone itself on the overall VLA performance. Though RoboVLMs (Liu et al. 2025 compared the influence of few early VLM backbones, did not ensure consistency in its implementations. In contrast, we design fairer experimental framework to comprehensively test the impact of various advanced VLMs across multiple environments. By employing the simplest and a consistent additional action policy, we minimize the influence of the policy head component the experimental results.
Embodied Tasks for VLM Many prior works have explored introducing embodiment--elateé tasks into Vision-Language Models to enhance their spatial understanding Yang et al.
2025 Feng 2025)
and task execution capabilities (Intelligence et al. 2025 Zhhou al. 20056b). These tasks include
dense prediction (Zhen et al. 2024) and autoregressive VQA tasks (Rana et al. 2023 Yuan et al.
2024 (ermmant et al.] 2024). Recently, inspired by the hierarchical System and System 2 design
paradigm (Zhang al. 2024 Shentu et al. 2024), much research focused on using VLMs to build general-purpose "robot brain a large language model capable of planning or completing real- world tasks via language. For instance, Robo2v (Chen al. 2025b) annotated datasets like Open-X for VQA tasks, while Robobrais in (Team] 2025) constructed affordance and visual trace tasks from RoboVQA (Sermanet al. 2024) and video datasets. Additionally, many VLA studies leverage auxiliary post-training or co- co- training tasks of VLM (Intelligence et al. 2025) Zhang et al.) (025)) to obtain better backbone networks.
our subsequent experiments, we fine-tune VLM on various post-training tasks and compare its performance as VLA backbone against the corresponding VLM baselines, thereby validating the effectiveness of these auxiliary tasks for VLA performance.
STUDY DESIGN
To create a comprehensive and fair benchmark of VLM performance on manipulation tasks, we designed the VLM4VLA framework around three core principles:
Fairness and Reproducibility We employ a consistent model architecture and training/testing settings across multiple simulation environments to ensure fair and reproducible comparisons. Minimalist Design We encapsulate VLMs within a simple yet effective VLA framework, thereby minimizing the influence of complex, extraneous policy designs the comparison.
3

---

Published as a conference paper at ICLR 2026
Isolating Core VLM Capabilities To directly evaluate the VLM's intrinsic knowledge, framework relies exclusively on visual-language inputs. We format input sequences to match each VLM's native instruction-tuning format and deliberately exclude other modalities like proprioceptive state or tactile feedback. This isolates the VL.'s contribution and directly tests how its intrinsic capabilities translate to manipulation.
In the following sections, we will detail the VLM4YLA research framework. In Sec 3.1 we introduce the basic experimental design. We then describe the model architecture within VLM4VLA Sec 3.2 and Sec 3.3 explains the experiment setup and evaluation protocol.
3.1 BASIC EXPERIMENT DESIGN
As shown in Figure I our objective to investigate the impact of different VLMs, as well as VLM-specific fine-tuning tasks, on the performance of the resulting VLA. Our experiments primarily focus on VLMs ranging from IB to 1OB parameters and various embodiment-relatedd auxiliary tasks. Beyond fundamental vision- to embodied tasks include visual grounding, trajectory and action prediction, task planning, video understanding, and spatial reasoning. Some models also possess image rendering capabilities, such as depth prediction and semantic segmentation, as seen in works like |eyer et al.] (2024); Chen et al. (2025a); Deng et al.] (2025). To compare the effectiveness of these models and tasks, encapsulate each VLM- in its original form and after being fine-tuned on a specific auxiliary task-into a VLA, following the methodology described in Sec 3.2 All VLAs, despite their different backbones, share an identical action encoding and decoding scheme and introduce a minimal number of additional trainable parameters (less than 1%).
alignment, the general capabilities of VLMs relevant
Subsequently, these VLM4VLA models are trained on robotic datasets using a consistent training setup. During this process, we fine-tune all model parameters, as our studys in Sec demonstrate that freezing parts of the model leads to significant performance degradation. The fine-tuned models are then deployed for rollouts in the target environments, where we measure the success rates on various tasks. The VLA performance is assessed according the protocol detailed in Sect 3.3
3.2 VLM4VLA NETWORK DESIGN
In this section, we detail the method for con- structing a consistent VLA from various VLMs within the VLM4VLA framework, as illustrated in Figure 2 Our objective is to build VLA ar- chitecture that is generic across different VLMs, lightweight, and capable of fully leveraging the VL.M's intrinsic knowledge.
Action Chunk
Finetuned Auxiliary Tasks
Vision Token
1
MLP
Text Token
Learnable Action Query
We introduce a learnable action query token to extract embodiment-rellted knowledge from the VLM. The representation of this token then decoded into action chunk. To align with the pre-training input format of each model, adapt unique token concatenation scheme for each VLM4VLA instance. We take the last hi hidden state of the (ActionQuery) token, encoded by the VL.,, and decode it into an action chunk using a small MLP--ased policy head. The overall action output can be formulated as:
VLM Backbone
All parameters are tuned
...
Figure 2: VLA Network in VLM4VLA
action MLP
(tex) ... (text) (ActionQuery))))
where (img) represents the visual embeddings from the vision encoder, (text) corresponds to the language embeddings containing the instruction and any additional prompts, and (ActionQuery) is the learnable action query token. The token sequence above omits VI.M-specific special tokens (e.g. (PAD) or (EOS) ...) for brevity. Further details can be found in Appendix A.2
Training Objective During training, finetune all parameters of the VLM, including the LLM, the vision encoder, and the word embeddings. We deliberately avoid widely-used objectives like diffusion loss and flow-matching loss. Our preliminary experiments revealed that these losses
4

---

Published as a conference paper at ICLR 2026
introduce significant stochasticity during inference, requiring much larger number of rollouts for accurate evaluation. They also cause substantial performance fluctuations between different checkpoints late in training. which is not conducive to the fair comparison we aim to achieve. As result, we utilize a maximum likelihood imitation learning objective. The desired relative position a of the end-effector (or continuous joint action) is optimized via MSE loss. The discrete status and of the end-effector is optimized with a binary cross-entropy loss:
L
=
Σ B
(11ar -a
BCB(a (
where â and àend denote the demonstration data for the relative position and status of the end- effector in a sampled mini-batch B.
3.3 EXPERIMENT SETTINGS AND EVALUATION PROTOCOL
We fine-tune the VLM and the action policy for each model using identical hyperparameters and model configurations for fair comparisons. We conduct a learning-rate sweep and select unified hyperparameter configuration to ensure that all models reached convergence at evaluation time (details can be found in Appendix A.2 Specifically, the model uses single-view image the current frame as its visual input and does not take proprioceptive information; this prevents the model from directly learning actions from the state. To handle inconsistent input image sizes across different VLMs, we standardize the input to 224 x 224 resolution for all training. If a VLM requires a different input size, we first process the image at 224 x 224 and then resize to the model-specific dimensions. During training, all parameters of VLM are trained, including the vision encoder, word embeddings, LLM, and small policy head. For each VLM, we use a simple instruction prompt consistent with its pre-training format (detailed prompts format for each model can be found in Appendix (A.2.) (A.2.3).
To ensure the reproducibility and fairness of our experiments, we do not use real-world tasks for evaluation. A direct consequence of this choice is that it prevents other researchers from directly comparing their own models against our findings on physical robots. We test different models in three simulation environments: Calvin (Mees et al., 2022). SimplerEnv (Li et al., 2024), and Libero (Liu et al. 2023). Since each environment requires separate training and testing, to improve the efficiency and rigor of evaluation, During tests, we experiment with executing the full action chunk, half of the action chunk, and a single step over all validation checkpoints. We report the best-performing result. More details about the settings of different environments can be found in the Appendix A.I
select the most challenging scenarios as evaluation benchmarks.
Calvin ABC-D We evaluate on the Calvin ABC-D task. We train the model for 30k steps on the ABC splits and evaluate 1000 task sequences, each with length of 5. During testing, the policy is required to complete a sequence of 1-5 tasks. This setup challenges the VLM's ability to generalize novel visual scenes. We report the average number of successfully completed tasks per sequence.
SimplerEnv Bridge To better differentiate the performance of various VL.M-based policies, choose the WindowX (Bridge V2) task suite, which is more challenging than the Fractal suite. We train for 50k steps Bri Bridge-v2 (Walke et al., 2023). During evaluation, 24 trials with random initializations for each of the four scenes (Pick Carrot, Pick Eggplant, Pick Spoon, and Stack Cube) and calculate the success rate.
Libero-Long (-10) Among the five task suites in Libero, we evaluate different models the most challenging suite L.i be ro-gg, which consists of 10 tasks involving a variety of objects and skills. All models are trained for 50k steps on the training split and evaluated with 50 trials with random initializations for each task.
4 EXPERIMENT AND RESULTS
To comprchensively evaluate how VLM capabilities transfer to robot manipulation, our experimental analysis is three-fold. First, in 4.c we benchmark various open-source VLMs on a challenging set of tasks across three simulators to assess the relationship between general capability and the
5

---

Published as a conference paper at ICLR 2026
downstream performance. Second, in Sec 4.2
we investigate whether improvements gained from fine-tuning on specific auxiliary embodied tasks translate to better final performance. Finally, in Sec we analyze the importance of the vision encoder by comparing the outcomes of freezing versus fine-tuning it during VLA adaptation.
4.1
PERFORMANCE OF DIFFERENT VLMS ON VLM4VLA
4.1.1 BASELINES
VLM Baselines We evaluate several Vision-Language Models commonly used in open-source VLAs, with model sizes generally ranging from 1B to 10B parameters, which ensures the feasibility of extensive action-learning finetuning and rollout testing. We test a variety of VLM models from the VLA domain, including the Paligemma series (paligemma-1 1 (Beyer et al., 2024) and paligemma--2 (Steiner et al. 2024)), the QwenVL series (Qwen2 5VL-3B, Owen2. 5VL-77 7B and 3VLn3VL-4B) (Bai et al. 2025), VL3.55L33.5-BB (Wang et al. 2025) and Kosmos-2 (Peng ct al. 2023). Among these, the QwenVL and source VLMs, Pal i gemma is designed for better adaptability to downstream finetuning, and the Kosmos series excels at grounding tasks. These models cover a range of multimodal LLMs with diverse architectures and strengths, allowing us comprehensively compare their action-learning capabilities.
series are top-tier ceneral-purpose open-
VLA Baselines For comparison, we select several expert VLAs reference baselines, which use different VLMs as their backbones:
OpenVLA (Kim et al. 2024): Uses Llama2-7P with DINOV2/SigLIP as its VLM backbone. It differs from our VLM4VLA framework by decoding actions into a discrete space. The total model size approximately 7.7B. For the Calvin environment, we report our reproduced OLe results, while for the Simpler and Libero environments, we report the official results. Similar to our VL.4VVLA setup, this model uses only a single image as input and does not leverage proprioceptive state.
pi0 (Blckk et al. 2024y: Based on P Paligemma-1, with total size of approximately 3.IB. We modify the code provided by ooen-pi-zeroj train and test within setups, ensuring consistent settings with other backbones. We remove the proprioceptive expert and use a single image input. More details can be found in Appendix AA.3
ThinkAct (Huang et al. 2025a): A recent VLA model enhanced with reinforcement learning, based on Qwen2 5VL-7B. We report its official results on Simpler and Libero for comparison against our Qwen2. 5VL-7B baseline. Unlike other baselines, ThinkAct takes use of proprioceptive state as an input.
4.1.2 MAIN RESULTTS
The performance of different VLMs robotics tasks is presented in Tables and In addition to success rates, we report the number finetuned parameters for each model in the size column.
As shown in Table for the Calvin ABC-D task, QwenVL and InternVL significantly outperform other VLMs. Notably, the average of 4.057 completed tasks by Qwen2. 5VL-7B is nearly on par with -fate-of-the-ar art VLAs. Furthermore, VLMs that perform better QA-benchmarks, such as the Qwen.VL series, also exhibit superior performance on Calvin, suggesting a correlation between the capabilities tested by Calvin and other VQA benchmarks. Concurrently, we observe that pi which is based on Pal. welliggmm----mmmmmmmmmmmm and does not use state information, performs similarly to the base Pali illigemaa11 model. This indicates that the additional action expert constrained by the inherent capabilities of VLM backbone itself and fails to yield performance improvements in the Calvin environment. Table disslay the performance of the models on the Simpler-Bre Bridge and Libero- 10 (Iong) tasks. We find that KosMos-2, the smallest model, achieves the highest success rate on both tasks. On the Simpler-Bridge task, the Pali gemma series models outperform the Qwen2. 5VL series, whereas on Libero-10, 10, the performance across different VLM categories is comparable. It worth noting that ThinkAct, finetuned from Qwen2. 5VL-73. performs similarly to the base Owen2. 5VL-7B on Simpler-Bridge. However, its performance Libero- 10 is substantially better than all other models. This may be caused by the proprioceptive state information utilized by ThinkAct, which can be useful input for Libero environment.

---

Published as a conference paper at ICLR 2026
Table Results on Calvin ABC-D, Entries marked with are expert VLAs modified and reproduced with our training and test settings.
Model (VLM Backbone)
Size
Task-1
Task-2
Task-3
Task-4
Task-5
Calvin
Expert Vision- Lannon-Lagggge-eee Models
Opt (Llama-2)
7.7B
0.792
0.644
0.499
0.368
0.245
2.548
pio* (Paligemma-l)
3.1B
0.896
0.785
0.786
0.610
0.532
3.509
VLM with VLM4VLA Models
Qwen2.5VL-33
3.8B
0.922
0.842
0.766
0.700
0.626
3,856
Qwen2.5V1.-77
8.3B
0.935
0.864
0.807
0.758
0.693
4.057
Qwen3VL-44
4.4B
0.933
0.857
0.790
0.719
0.644
3,943
Paligemma-1
2.9B
0.914
0.813
0.692
0.599
0)4888
3,506
Paligemma-2
3.0B
0.901
0.775
0.669
0.575
0.486
3,406
KosMos-2
1.7B
0.878
0.721
0.591
0.498
0.408
3.096
Intem VL3.5-4B
4.7B
0.934
0.864
0.795
0.728
0.656
3,977
Table 2: Results on SimplerEnv-Brice and Libero-10. Entries marked with * are expert VLAs modified and reproduced with our training and test settings.
Model (VLM Backbone)
Size
Carrot
Eggplant
Spoon
Cube
Simplert
Libero
Expert Vision Lanugoge Action Models
OpenVLA (Llama-2)
7.7B
4.2
0.0
0.0
12.5
4.2
53.7
pi0* (Paligemma-l) 1)
3.1B
62.5
100.0
54.2
25.0
60.4
46,0
ThinkAct (Qwen2.5VI-77)
7.4B
37.5
70.8
58.3
8.7
43.8
70.9
VLM with VLM4VLA Models
Qwen2.5VL-3B
3.8B
20.8
91.7
79.2
0.0
47.9
43.0
Qwen2.5VL-7B
8.3B
12.5
100.0
75.0
45.0
Qwen3V1.-4B
4.4B
54.2
95.8
75.0
0.0
56.3
44.4
Paligemma-1
2.9B
50.0
91.7
75.0
4.2
55.2
44.2
Paligemma-2
3.0B
75.0
75.0
79.2
0.0
57.3
46,2
KusMos-2
1.7B
37.5
100.0
75.0
29.2
60.4
55.0
Inte5nVL3..544B
4.7B
12.5
100.0
62.5
54.2
57.3
62.8
As illustrated in Figure 3 conducted linear regression analysis to examine the relationship between VLA performance and the general capa- bilities of the underlying VLMs. We plot the per- formance of various VLMS on several general- purpose VQA benchmarks against their perfor- mance on VLA tasks. For Pali Paligemma gemma and Kosmos, s, approximated their general capa- bilities using results from their proprietary tasks. For Qwen-VL and InternVL, average scores from multiple general-purpose VQA benchmarks. A linear regression line is fitted to visualize the correlation between these two sets of metrics. The results indicate that dif- ferent evaluation environments exhibit varying degrees linear correlation with these general VLM capabilities. We found that the results the Calvin benchmark exhibit high correlation
Strong Correlation
Simpler: Weak Negative Correlation
Performancce
Libero: Weak Positive Correiation - SimplerEnv
Liber-LLong
used the
VLM General Capability
Figure 3: Comparison of the linear relationship between general VLM capabilities and VLA per- formance.
with performance on VQA benchmarks. In contrast, for the Simpler and Libero environments, there is no apparent correlation between a VLM's s QA performance and its VLA performance. This suggests significant gap exists between the capabilities required for VLA manipulation tasks and those measured by existing VQA benchmarks. More details about constructing Figure can be found in Appendix A.4
7

---

Published as a conference paper at ICLR 2026
4.2 IMPACT OF DIFFERENT VLM AUXILIARY TASKS ON VLA PERFORMANCE
Recent works have proposed using robotic data to construct VQA datasets for improving VLM backbones, e.g., Robobrain (Team 2025), Robix (Huang et al.) 2025b). However, few studies have investigated whether this additional continual finetuning actually benefits VLAs in downstream tasks. In this section, we construct or collect several SFT tasks for VLM, including VQA datasets and generation task. We first finetune the 2wen2.5V 5VL model and subsequently, we employ each finetuned VLM as the backhone for our VLM4VLA framework and evaluate its performance on the Calvin benchmark. Specifically, we compare the following finetuned VLMs (more details about SFT datasets can be found in Appendix )
Robopoint (Yuan et al.) 2024 A pointing task dataset collected in simulator. Given an image and a target location, the model is required to output the 2D coordinates that satisfy the target requirement. This finetuning dataset contains 1.432M samples. After finetuning, the performance of Qwen2. 5VL-3E 3B on the pointing task improved by ~20% on the test split.
Vica-332k (Feng 2025) A spatial understanding dataset constructed from RGB-D datasets. It covers a wide range of capabilities, including size estimation, position understanding, distance estimation, and improvement of 155% on VSI-Bench Yang et al. (2025).
After finetuning Qwen2. 5VL-3B this dataset, achieve performance
Bridgevqa We used VQASynth annotate data from Bridge-v2, Fractal, and Calvin ABC, combining them into a spatial understanding question-answering dataset. The main QA content includes judging the distance, size, and depth between objects.
Robo2v1m (Chen et al. 2025b) An action-oriented guestion-answert dataset built from 176k real robot trajectories, containing 667k VQA pairs. It involves tasks such as multi-view understanding, suc- cess prediction, position management, and trajectory judgment. After finetuning Owen2. 5VL-3B on this dataset, we achieved a 60% performance improvement on test set.
Robobrain2 Team 2025) A large-scale embodied VQA dataset and VLM finetuned on Qwen2. 5VL-7B, The tasks include pointing, planning, and marking trajectory. We directly use their finetuned VLM as the backbone for our VLM4VLA.
Omni-Generattion We integrate a diffusion model into OwenVL-73 and train on both image generation, depth map generation, and semantic segmentation map generation tasks together with general VQA tasks. We use the resulting VL.M part as the backbone for VLM4VLA.
VOA-Mix We mix the aforementioned VQA datasets with other large-scale general data to finetune Qwen2. 5BL-7B,
Qwen2.5VL-78 Methods
Qwen2.5VL-38 Methods
0.5
Langt
Baseline-38
Vica332k
Figure 4: Performance of different auxiliary VLM finetune tasks. The 'Length' dimension is scaled by a factor 5 to normalize it to the range [0, 1]. The results for the VLAs trained under each task and for each gradient steps (10k, 15k, 20k, 25k and 30k) rendered box plots provide view of the impact of different tasks on the VLA's performance and stability.
Figure 4 presents the performance of the various finetuned VLMs. Overall, all models underperform the original baseline, with most exhibiting slight degradation in performance and an obvious increase variance. For Qwen2 .VI.-BR. the model finetuned on Vi ca3233 performs better
8

---

Published as a conference paper at ICLR 2026
than those finetuned on other datasets. This could be attributed to the dataset's broad data coverage and diverse task types, which may prevent the model from overfitting to narrow set capabilities and consequently degrading others. For Qwen2. 5VL-73, the VQA-Mix model shows the least performance degradation, achieving results nearly identical to the baseline. This suggests that general- purpose VQA data plays crucial role during imbodiment-focusee finetuning, further implying that VLAs may require broad, general capabilities, beyond just embodied skills, to perform well on downstream tasks.
It is also worth noting that finetuning with generation tasks (i.e., Omni Generation on Owen2. 5VL-7B), such as depth and semantic map prediction, did not yield performance ben- efits. This may indicate that simply introducing generation tasks or dense 3D-aware tasks into VLM finetuning process does not provide a tangible advantage for the VL.A Similarly, Robobraing in, positioned as general-purpose embodied brain, also underperformed the baseline in our VLM4VLA tests. This suggest that existing embodied VQA-style tasks do not offer clear benefit for training end-to-end VLAs to execute downstream manipulation tasks.
4.3 IMPORTANCE OF VISION ENCODER
Table Influence of freezing vision encoder of VLMs
Size
Calvin ABC-D1
SimplerBridget
Swen2.5VLL33B
3.8B
3.856
48.00
+ freeze vision encoder
3.1B
2.855 (-1.001)
23.95 (-24.05)
Qwen2.5VL. VL
8.3B
4.057
46.75
freeze vision encoder
7.6B
2.823 (-1.234)
25.50 (-21.25)
Paligemma-1 1
2.9B
3.506
55.25
+ freeze vision encoder
2.5B
0.495
(-3.01
13.25 (-42.00)
Table shows the performance three models when the vision encoder is frozen during VLM4VLA training. We observe significant performance degradation for all models on both the Calvin and Simpler benchmarks after freezing the vision encoder, with this drop being particularly pronounced for the Paligemma models. In the case of Qwen2. 5VL-73 73, the vision encoder accounts for 0.7B parameters. When 7.6B, which is much larger than 3.8B in Qwen 5VL-3B. However, the performance of the frozen Qwen2. 5VL-7B B is not only significantly worse than its fully finetuned counterpart but also sub- stantially underperforms the fully finetuned Qwen2 5VL-3B. This observation is consistent with findings in prior work (Kim et al. 2024 Huang et al., 2025c).
frozen, the total number of tunable parameters is still a substantial
This finding strongly suggests that finetuning the vision encoder is crucial when adapting a VLM into a VLA, and that the impact of this module can be more significant than merely increasing the number of trainable parameters in the language model. A potential reason for this phenomenon that the VLM's 's pre-trained vision encoder is not well-aligned with the visual domain of embodied scenes. The model likely requires significant adaptation of its vision module to effectively process and align with the visual signals from these novel environments. This highlights a promising direction for future research.
4.4 INFLUENCE OF PRETRAINED VLM
To isolate the contribution of the VLM backbone, we establish lower-bound baseline by training VLM4VLA variants entirely from random initialization. This clarifies whether the observed gener- alization stems from the architecture alone from VLM pre-training. We train Qwen2.5-VI 5-VL (3B and 7B) and PaliGemma from scratch on the Calvin ABC-D and Simpler-Bride benchmarks. We strictly match the training hyperparameters and evaluation protocols reported in the main paper for all from-scratch runs.
Models trained from scratch exhibit substantial degradation across both benchmarks and all model sizes. This gap indicates that VLM pre-training is fundamental to the generalization ability of the VLA model, beyond architectural design. Training from scratch yields markedly lower perfor-

---

Published as a conference paper at ICLR 2026
Table 4: Performance of training from scratch compared to pretrained initialization. We report absolute scores and the change (in parentheses) relative to pretrained models on Calvin ABC-D and Simpler-Bridge.
Model
Calvin ABC-D
Simpler-Bridge
Qwen2.5-VI-33B (pretrained)
3.856
48.00
Qwen2.5-VL-3B (from scratch)
1.381
(-2.475)
15.75 (-32.25)
Qwen2,5-VL (pretrained)
4.057
46.75
Qwen2.5-VL-7B (from scratch)
1.769 (-2.288)
18.20 (-28.75)
PaliGemma-l (pretrained)
3.506
55.25
PaliGemma-1 (from scratch)
1.129 (-2.377)
14.50 (-40.75)
mance, confirming that VLM pre-training is a prerequisite for strong downstream generalization in VLM4VLA.
CONCLUSIONS
In this paper, we investigated the impact of various VLMs, including the effect of auxiliary fine-tuning tasks, on the performance of VLA models. Through extensive training and evaluation experiments conducted across three distinet environments, we assessed the capabilities of seven models and seven categories of auxiliary data for executing manipulation tasks. A core insight from our study is the significant gap between the capabilities of current VLMs and the demands of VLA embodied tasks. Specifically, observe notable discrepancy between general VLM capabilities and its VLA effectiveness. While comparisons with necessary foundation, their general capabilities poor predictors of downstream success. Furthermore, even improving specific embodied skills does not guarantee better control. These findings challenge the intuition that stronger VLMs automatically yield better agents, highlighting disconnect between current pre-training paradigms and embodied needs. The mechanisms driving this gap remain an open question for future research. Finally, limitation of our work is the absence of experiments on physical robots. This decision was motivated by challenges related to reproducibility, as well as difficulties in ensuring test efficiency and fairness across physical hardware setups. While real-world deployment remains the ultimate goal, we believe that our comprehensive results across multiple, diverse simulation benchmarks provide valuable insights that can inspire and guide future research in this area.
from-scratch baselines confirm that pre-trained VLMs
ETHICS STATEMENT
The data used in the VLM4VLA is sourced exclusively from public repositories, Our contributions fully adhere the terms of these licenses. We did not use any data beyond what is publicly available and downloadable.
REPRODUCIBILITY STATEMENT
All results and experimental conclusions in this paper are reproducible. To facilitate reproducibility, have included source code in the supplementary material. We committed to open science and plan to publicly release the complete codebase, trained models, datasets, and evaluation logs upon publication of this work.
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 2502.
2025.
10

---

Published as a conference paper at ICLR 2026
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang. Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXivv2407.07726, 2024.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian lchter. et al. pi_O: vision- vision-language- language-actionn flow model for general robot control. arXiv preprint arXiv:2410.24166 2024.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-actinn models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15816 2023,
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455-14465, 2024a.
Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, et al. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset. arXiv preprint arXiv:2505.088 09568, 2025a.
Kaiyuan Chen, Shuangyu Xie, Zehan Ma, and Ken Goldberg. Robo2vim: Visual question answering from large-scale in-the-wild robot manipulation datasets, 2025b. URL http:/ttts///aaii.ooto/
abs/ 2505. 15517
Xiaoyu Chen, Junliang Guo, Tianyu Hc. Chubeng Zhang. Pushi Zhang, Derek Cathera Yang, Li Zhao, and Jiang Bian. Igor: Image-goal representations are the atomic control units for foundation models in embodied ai, 2024b. UtRps:tt:://art org/ abs 2411 1.0085
Xiaoyu Chen, Hangxing Wei, Pushi Zhang, Chuheng Zhang, Kaixin Wang. Yanjiang Guo, Rushuai Yang, Yucen Wang, Xinquan Xiao, Li Zhao, et al. Villa-x: enhancing latent action modeling in Dision-language-actio models. arXiv preprint arXiv:2507.23682, 2025c.
Can Cui, Pengxiang Ding, Wenxuan Song, Shuanghao Bai, Xinyang Tong, Zirui Ge, Runze Suo, Wanqi Zhou, Yang Liu, Bofang Jia, et al. Openhelix: A short survey, empirical analysis. and open-source dusl-system vla model for robotic manipulation. arXiv preprint arXiv:2505.03912. 2025.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
Chaorui Deng, Deyao Zhu, Kunchang Li. Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu. Xiaonan Nie. Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505. 1022055..6660 14683, 2025.
Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. Embspatial-benchh Bench- marking spatial understanding for embodied tasks with large vision-language models. arXiv preprint 0rXiv:2406.05756, 2024.
Qi Feng. Towards visuospatial cognition via hierarchical fusion of visual experts. arXiv preprint arXiv:2505. 123663 2025.
Yanjiang Guo, Jianke Zhang, Xiaoyu Chen, Xiang Ji, Yen-Jen Wang, Yucheng Hu, and Jianyu Chen. Improving vision-language-aatie actionnee-aiinnnnoooo model with online reinforcement learning. arXiv preprint arXiv:2501. 16664, 2025.
Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. arXiv preprint ar /4803, 2024.
11

---

Published as a conference paper at ICLR 2026
Chi-Pin Huang, Yuch-Hua Wu, Min-Hung Chen, Yu-Chiang Frank Wang, and Fu-En Yang. Thinkact: lisioo-lnngugo-acttion reasoning via reinforced visual latent planning. arXiv preprint arXiv:2507.16815 2025a.
Fang Huang, Zhang Mengxi, Dong Heng, and Li Wei. Robix: A unified model for robot interaction, reasoning and planning. arXiv preprint arXiv:2509.00 106, 2025b.
Huang Huang, Fangchen Liu, Letian Fu, Tingfan Wu, Mustafa Mukadam, Jitendra Malik, Ken Goldberg, and Pieter Abbeel. Otter: A vision-language-achiie model with text-aware visual feature extraction. arXiv preprint arXiv:2503.03734 2025c.
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. pi0.5: vision-language-action model with open-world generalization. arXiv preprint arXiv:2504. 16054, 2025.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-language-actinnn model. arXiv preprint (tt:::20060092446 2024.
Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators. arXiv preprint arXiv:2311.01377 2023.
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong. and Ted Xiao. Evaluating real-world robot manipulation policies in simulation. arXiv preprint 2xXiv:2405005941 2024.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03311, 2023.
Huaping Liu, Xinghang Li, Peiyan Li, Minghuan Liu, Dong Wang. Jirong Liu, Bingyi Kang. Xiao Ma, Tao Kong, and Hanbo Zhang. Towards generalist robot policies: What matters in building saiion-languge--are models. arXiv preprint 142 14058, 2025.
Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditionee policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L), 7(3):7327-7344 2022.
Kun Ouyang, Yuanxin Liu, Haoning Wa, Yi Liu, Hao Zhou, Jie Zhou, Fandong Meng. and Xu Sun. Spacer: Reinforcing milms video spatial reasoning. arXiv preprint arXiv:2504.00 18005 2025.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-languuge--ction models. arXiv preprint 27X1v22501.09747, 2025.
Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable task planning. arXiv preprint STTV:22077001333 2023.
Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrishnan, Christine Chan, Gabriel Aulac-Amold. Sharath Maddineni, Nikhil J Joshi, et al. Robovga: Multimodal long-horizon reasoning for robotics. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 645-652 IEEE, 2024.
Yide Shentu, Philipp Wu, Aravind Rajeswaran, and Pieter Abbeel. From Ilms to actions: Latent codes as bridges in hierarchical robot control. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3539-8546 IEEE, 2024.
12

---

Published as a conference paper at ICLR 2026
Hao Shi, Bin Xie, Yingfei Liu, Lin Sun, Fengrong Liu, Tiancai Wang, Erjin Zhou, Haoqiang Fan, Xiangyu Zhang, and Gao Huang. Memoryvla: Perceptual coggiitie memory in vision -ainugla action models for robotic manipulation. arXiv preprint arXiv:2508. 19236, 2025.
Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bit- ton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, et al. Paligemma 2: family of versatile vlms for transfer. arXiv preprint arXiv:2412.03555 2024.
BAAI RoboBrain Team. Robobrain 2.0 technical report. arXiv preprint a<Xiv:2507.02023 2025.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302..0 13971, 2023.
Homer Walke, Kevin Black, Abraham Lee. Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong. Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Intern133. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508. 18265,, 2025.
Junjie Wen, Yichen Zhu, Jinming Li, Zhibin Tang, Chaomin Shen, and Feifei Feng. Dexvla: Vision-language model with plug-in diffusion expert for general robot control. arXiv preprint arXiv:2502.05855, 2025.
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 16632-10643, 2025.
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vision- languaee model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406. 10721, 2024.
Jianke Zhang, Yanjiang Guo, Xiaoyu Chen, Yen-Jen Wang, Yucheng Hu, Chengming Shi, and Jianyu Chen. Hirt: Enhancing robotic control with hierarchical robot transformers. arXiv preprint arXiv:2410.05223. 2024.
Jianke Zhang, Yanjiang Guo, Yucheng Hu, Xiaoyu Chen, Xiang Zhu, and Jianyu Chen. Up-vla: A unified understanding and prediction model for embodied agent. arXiv preprint arXiv:2501.18867 2025.
Tony 2 Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13700 2023.
Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A 3d vision-language-actioo generative world model. arXiv preprint 103.:2403.00631,, 2024.
Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé III, Andrey Kolobov, Furong Huang, and Jianwei Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. arXiv preprint 2rXiv:2412.103444 2024.
Enshen Zhou, Jingkun An, Cheng Chi, Yi Han, Shanyu Rong, Chi Zhang, Pengwei Wang, Zhongyuan Wang, Tiejun Huang, Lu Sheng, et al. Roborefer: Towards spatial referring with reasoning in vision-language models for robotics. arXiv preprint arXiv:2506.0445 04308, 2025a.
Zhongyi Zhou, Yichen Zhu, Minjie Zhu, Junjie Wen, Ning Liu, Zhiyuan Xu, Weibin Meng, Ran Cheng, Yaxin Peng, Chaomin Shen, et al. Chatvla: Unified multimodal understanding and robot control with vision-language-aaee model. arXiv preprint 2rXhv:2502.14420, 2025b.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 MORE DETAILS ABOUT TESING ENVIRONMENTS
Calvin ABC-D Calvin is a simulation benchmark for various tabletop manipulation tasks. Its dataset consists of four splits (A, B, C, and D), each with different scene configurations (primarily changes in object and scene colors). During testing, the policy is required to complete sequence of 1-5 tasks. To prevent larger models from merely overfitting to seen color schemes, we adopt the ABC-D setup, where the model is trained on scenes A, B, and C, and tested scene D. This setup challenges the VLM's ability to generalize to novel visual scenes. We train the model for 30k steps on the ABC splits and evaluate it on 1000 task sequences, each with length of 5. We report the average number of successfully completed tasks per sequence.
SimplerEny Bridge SimplerEnv is suite of real-to-sim evaluation environments where the policy is trained on real-world data and tested in simulation. The test scenarios are divided into two categories based the data source: Google Robot (Fractal) and WindowX (Bridge V2). The former has smaller visual gap between the training and testing domains, where most policies perform well. To better differentiate the performance of various VLM-based policies, we choose the more challenging Bridge split as our evaluation metric. We train for 50k steps on B dgeV2, with validation every 5k steps (for total of 10 checkpoints). During evaluation, we run 24 trials with random initializations for each of the four scenes (P (Pick Carrot, Pick Eggplant, Pick Spoon, and Stack Cube) and calculate the success rate. We report the best result from the validation checkpoints.
Libero-Long (-10) Libero is simulation benchmark for multi-category manipulation tasks on fixed tabletop, which contains five task suites. We select the most challenging suite, Libero- -oogg (also known as Libero-100 which consists of 10 long-horizon tasks involving variety of objects, scenes, and manipulation types.
A.2 MORE IMPLEMENTATION DETAILS
A.2.1 TRAINING SETUPS
To ensure a fair comparison, we fine-tune the Vision-Language Model (VLM) and the action policy with identical hyperparameters and model configurations in each test environment. Specifically, all our experiments are conducted on NVIDIA A100 GPUs. The model uses a single-view image the current frame its visual input and does take state information as input, which prevents the model from directly learning actions from the state. We fine-tune all parameters of the VLM, including the vision encoder, token embeddings, and the LLM part. The detailed training hyperparameters are as follows:
Calvin: We use a batch size of 256, learning rate of 2 x 10-5 for all models, and an action chunk size of 10.
Simpler & Libero: We use batch size of 512, a uniform learning rate of 5 x 10-5, and an action chunk size of 4.
A.2.2 ABOUT CHOICE OF HYPERPARAMETERS
Here, we provide additional details on hyperparameter choices (action chunk, learning rate, and batch size) and discuss the implications of using exactly the same hyperparameters across methods.
For the results reported in the paper, we have performed a sweep on different action chunk sizes during evaluation. We also use different checkpoints to mitigate the training instability. We reported the best performance among these runs. In most cases, the best performance the final checkpoint, indicating that the models had already converged. The protocol that using the same hyperparameters ensures that our results reflect each model's 's peak capability (convergence). effectively decoupling performance from training proficiency or randomness.
achieved before
To verify the robustness of our Learning Rate (LR) choice, performed sweep using values {le 2. - 5, 5e 5, le - 4} on representative models (Qwen2.5-VL-31) Sive2.55VL-77 and PaliGemma-l). As shown in the table below, while minor fluctuations exist, the downstream
14

---

Published as a conference paper at ICLR 2026
performance remains highly stable across this range, and the relative ranking of models remains consistent.
Learning Rate
1e-5
2e-5
5e-5
1e-4
Owen2.5VL-B3
3.844
3,856
3.832
3.792
Qwen2.5VL-7B
4.050
4.057
4.027
3,986
Paligemma-1
3.506
3.506
3.487
3.492
Table 5: Performance across learning rates Calvin ABC-D.
Regarding batch size, maximized this parameter uniformly across all models and make sure this stays the same. We believe that larger batch sizes benefit all architectures by stabilizing gradients; therefore, enforcing large, identical batch size prevents any model from being disadvantaged by gradient noise.
A.2.3 LANGUAGE PROMPTS AND FORMAT
For fair comparison between different VLMs, we prompt all models with the simplest format of prompts while keeping consistent with the VLM's pretraining prompt format. The detailed prompts for the different VLMs used VLM4VLA are as follows (including special tokens). For convenience, here we use the robotic instruction pick up cube example of input encoded in this manner.
Paligemma-1 & Paligemma-2
The Paligemma model employs a prefix-suffie encoding scheme during its pre- pre- training.. In this scheme, the prefix part is processed with bidirectional attention, while the suffix part uses unidirectional (causal) attention for generative tasks. The two parts are demarcated by a <bos> token. For implementation, encoding scheme for text tokens, as this aligns with the model's generative training objective.
adopt the prefix encoding scheme for image tokens and the suffix
{img)... ... (img) (bos)pick up cube cubel, |(Acttionuerr))
Kosmos-2
Kasmos is a VLM optimized for grounding tasks, where its training sequences contain multiple task-specific tokens. Since the action output of VLA is not a grounding task, we do not include these additional task tokens in our input. The expectation is that the model will automatically learn to leverage its skills acquired from other tasks during the fine-tuning process.
(s)mige)) (img) (img) (/image) pick up cube /
Qwen2.5VL-3B & Qwen2.5VL-7B & Qwen3VL.-4B
For the Qwen series models, which are cestruction-fine-tuuee general-purpose conversational models, an input format that closely mirrors their training would necessitate the inclusion of both system and user prompts. We therefore compared two distinct token concatenation schemes:
Scheme 1 (Full Compliance): This approach strictly adheres to the instruction-Se -FFT format.
While this ensures maximum alignment with the model's pre-training, it results in longer input token sequence:
(Jim_start))systet You are helpful assistant. (artaattt.iimmmnllll (lim_start (aartt)Jusrrr Jser'' | -ttooo)stttt (img) ... (img) [|visionneed) What action should the robotic arm take to pick up cube(Jim end)) An
{lim-start) (asssatrrtt)sssstaattt (n (AActionQuery)
Scheme 2 (Simplified Consistency): This scheme aligns with the simplified approach used for Kosmos and Palice Paligemma. It omits the extraneous conversational prompts (e.g.. system and user roles) and includes only the most essential special tokens required for sequence construction:
(lim start]) (vvision (art))(img) ... (img) (Ivision endl)pick up cube c n(AActionuuer)
15

---

Published as a conference paper at ICLR 2026
Prompt
Calvin ABC-D1
SimplerBridget
Qwen2.5VL-3B
Long
3.856
48.00
Qwen2.5VL-3B
Short
3.844 (-0.012)
46.75 (-1.25)
Qwen2.5VL-7B Qwen2.5VL-7B
Long
4.057
46,75
Short
3.947 (-0.110)
44.00 (-2.75)
Table 6: Comparison between different prompt formats. Long prompt denotes Scheme 1, while the Short prompt denote the Scheme
The comparison in Table indicates that prompts perfectly matching the Supervised Fine-Tuning (SFT) format perform marginally better than their minimal counterparts. Consequently, we con- sistently apply the Scheme I prompt style the QwenVL and InternVI series in our remaining experiments to maintain alignment with their original training formats.
Intern V L3.5-4B
The training format of Intern VL is highly similar to that of Qwen- VL, as both models utilize the same Qwen-LLM backbone. Accordingly, adopt the same prompt format as the Qwen series, with modifications made to certain differing special tokens:
(im_sstrr))systeen (nous Ynuu are helpful assistant. < (lim_end])
(lim_start] Juser', n ([lmgl) (img)... ... (img) (//img) What action should the robotic arm take pick up cube(/im_ent An (lim_start)
assistant' atassstaa)assssttaatt (atissttttt((Attoonn(oto ((AActionuerry)
A.2.4 IMPLEMENTATION DETAILS ABOUT VLM4VLA ARCHITECHTURE
We provide the detailed model architechture in Table 7 We maintain consistent architecture for the action-specific modules across all models, including a fixed number of Icarnablc tokens and a uniform hidden size for the action head. Consequently, the variation in the number of additional trainable parameters is solely dependent on the VLM's native h hidden. size. The Policy Head architecture presented here omits the final linear layer, which maps the features the action space (i.e., Linear (1024, 7))
A.3 IMPLEMENTATION AND EVALUATION DETAILS FOR THE PIO
Reproducing pi0 In experiments, reproduced the pi model to serve strong compar- ative baseline. We leveraged the model implementation from the official code released by Allen but excluded its proprioceptive input module. Specifically, we integrated the pi0 model architecture into the VLM4VLA data-loading and training framework for fine-tuning and evaluation. This model-only migration deliberate choice, we observed that the data preprocessing strategies used in the original p 0 implementation for the Simpler environment were inconsistent with those of Open and our VLM4VLA framework, thus ensuring fairer comparison.
Modification of the Proprioceptive State Expert We modified the state expert in pi0 by directly concatenating the outputs of the VLM expert and the action expert. Since the state and action experts in the original pi architecture share the same parameters, our modification only alters the model's input pathway without changing its underlying structure or parameter count.
During our experiments, we discovered that the Flow-matee now-maachhig loss used in pi0 leads to significant performance instability. Specifically, multiple rollouts of the same pi0 policy in identical starting environment could yield success rate fluctuations as high +20%. In stark contrast, our VLM4VLA approach is deterministic under the same conditions, producing identical outcomes for each without stochasticity. This high variance necessitated more extensive evaluation rollouts for pi0 0 to obtain a reliable estimate of its true performance. Consequently, when testing ç 0, performed greater number of rollouts for each checkpoint under various settings:
Evaluation Protocol
16

---

Published as a conference paper at ICLR 2026
Backbone Size
Policy Head Size
Learnable Tokens
LLM Hidden
VLM
Policy Head
Size
Linear (2048, 1024)
Qwen2.5VL-3B
3755M
6.58M
2048
ReLU
Linear (1024, 1024)
Linear (3584, 1792)
Qwen2.5VL-7B
8292M
11.69M
3584
ReIU
Linear (1792, 1024)
Linear (2560, 1280)
Qwen3VL-4B
4437M
8.02M
2560
ReIU
Linear (1280, 1024)
Linear (2048, 1024)
Paligemma-1
2923M
6.58M
2048
ReLU
Linear (1024, 1024)
Linear (2304, 1502)
Paligemma-2
3032M
7.27M
2304
ReLU
Linear (1502, 1024)
Linear (2048, 1024)
KosMos-2
1664M
6.58M
2048
ReLU
Linear (1024, 1024)
Linear (2560, 1280)
Intern VI.3.5-4B
4732M
8.02M
2560
ReLU
Linear (1280, 1024)
Table 7: Detailed architecture different VLM4VLA backbones.
Calvin: We averaged the results over 3 evaluation per model.
Simpler: We performed 240 evaluation runs for each model on every task.
Libero: We performed 250 evaluation runs for each model on every task.
A.4 CORRELATION ANALYSIS BETWEEN VLM CAPABILITY AND VLA PERFORMANCE
Here we provide a detailed explanation of how the data points in Figure are obtained and what each graphical element represents.
The x-axis represents the general capability of different VLMs.
Each set of three vertically aligned points with the same color corresponds to the results of the same VIM used as the VI.MVVLA backbone across three benchmarks.
Marker shapes denote benchmarks: circles for SimplerEnv, triangles for LIBERO, and squares for CALVIN (see the legend on the right).
The three colored lines linear fits to the points from each benchmark: red for Simpler (fit over circle markers), blue for LIBERO (fit over triangle markers), and green for CALVIN (fit over square markers).
Figure 3 uses VLMs from data sourced from the experiments reported in Tables and 2. We elaborate how we evaluate different VLMs in terms their general capabilities. Intern and QwenVL are general-purpose VLMs that have been evaluated across wide range of multimodal understanding benchmarks. We take the average of their officially reported results on a set of multimodal general, reasoning, text, grounding, agentic, and embodied benchmarks as the x-axis value. This set includes: MMBench v1.1 (en), MMStar, BLINK, HallusionBenchh, AI2D, OCRBench, MVBench. VideoMME. MMMU, MathVista Math Vision, CountBenchQA, RefCOCO (all), GPQA, OmniSpatial, RealWorldQA, ERQA, and VSI-Bench. For PaliGemma and Kosmos, which are not trained for gerpral-purpose tasks,
their performance their respective specialized
17

---

Published as a conference paper at ICLR 2026
VLM4VLA Baseline
Task-1 I
Task-2
Task-3
Task-4
Task-5
ALL1
Qwen2.5VL-3B
0,922
0.842
0.766
0.700
0.626
3.856
Rohopoint
0,924
0.829
0.749
0,683
0.598
3.783 (-0.073)
Vica332k
0.940
0.842
0.758
0.692
0.615
3.847 (-0.009)
Bridgeqva
0,945
0.843
0.744
0,657
0.577
3.765 (-0.091)
Robo2vlm
0.925
0.833
0.737
0.668
0.597
3,760 (-0.096)
Owen2.5VL-77
0.935
0.864
0.807
0.758
0.693
4.057
Rohohrain2
0.938
0.849
0.767
0.704
0.629
3.887 (-0.170)
Omni-CGenraation
0.928
0.833
0.760
0.711
0.654
3.876 187)
VQA-Mix
0.938
0.856
0.794
0.732
0.658
3.978 (-0.079)
Table 8: Comparison of different Embodied VQA finetuning on VLM4VLA (best results on each settings)
evaluation suites as the metric. Specifically, PaliGemma is evaluated on RefCOCO (all), AI2D, CountBenchQA, and GQA; Kosmos is grounding- focused VLM, which uses only RefCOCO results as its metric. Because the latter two models are evaluated on subset of the tasks used for the former two (reflecting their lack of general-task capability), we apply downweighting to their scores. This yields an approximate assessment of the "general capability of different types of VLMS" as reference.
To quantify the strength of the linear relationship between each benchmark and VLM capability, we report the correlation metrics for the three fitted lines (Pearson correlation coefficient and R());
Calvin: r = 0.919, R 0.844
SimplerEnv: 0..221, 1, R2 0.103
Libero: 0.381, R 0.145
A.5 DETAILED RESULTS OF AUXILIARY TASKS
The results on each finetuned VLM on VLA tasks (Calvin ABC-D) are shown in Table 8 (correspond-
ing to Chart
A.6 DETAILS ABOUT DATASETS USED FOR FINETUNING VLM
Robopoint Yuan et al. 2024) pointing task dataset collected simulator. Given an image and a target location, the model is required to output the 2D coordinates that satisfy the target requirement. This finetuning dataset contains 1.432M samples. As shown in the table, after finetuning. the performance of Qwen2. 5VL-3B on the pointing task improved by ~20 on the test split.
Vica-332k (Feng 2025) A spatial understanding dataset constructed from RGB-D datasets. covers a wide range of capabilities, including size estimation, position understanding, distance estimation, and so on. After finetuning Qwen2. 5VL-3B on this dataset, we achieve performance improvement of ~ 15% VSI-Bench Yang et al. (2025).
Bridgevqa We used [QASynth to annotate data from Bridge-v2, Fractal, and Calvin ABC, combining them into a spatial understanding question-answeringg dataset. The main QA content includes judging the distance, size, and depth between two objects.
Robo2v1m (Chen et al. 2025b) An action-oriented question question-answering answering dataset built from 176k real robot trajectories, containing 667k VQA pairs. It involves tasks such as multi-view understanding, suc- cess prediction, position management, and trajectory judgment. After finetuning Qwen2. 5VL-3B on this dataset, we achieved a 60% performance improvement on its test set.
Robobrain2 (Team 2025) A large-scale embodied VQA dataset and VLM finetuned
Owen2. 5VL-7B 7B. The tasks include pointing, planning, and marking trajectory. We directly use their finetuned VLM the backbone for our VI.44VI.A
18

---

Published as a conference paper at ICLR 2026
Omni-Generation Qwen-vlo integrates diffusion model into VLM and trains both image generation, depth map generation, and semantic segmentation map generation tasks. We use the resulting VLM part (Owen2.. 5VL-7B
7B the backbone for VLM4VLA.
VQA-Mix We mix the aforementioned VQA datasets with other large-scale general VQA data to finetune Qwen2. 7B. The embodied related tasks include approximately 620k image-text pairs, including Vica Feng 2025), SpaceR Ouyang et al. (2025), Embspatial Du et al. (2024) and Refspatial
Zhou et al. (2025a). The general VQA data consists of 360k QA pairs.
Example of Robopoint Data
Question: Locate several points on an item situated beneath the bordered item.
Answer: [(0.56, 0.69), (0.53, 0.76), (0.45, 0.72), (0.43, 0.67)]
Example of Vica Data
Question: How large is this room in terms of square meters? more than one room shown, approximate the overall space.
Answer: 18.33
Example of Bridgevqa Data
Question: Who is positioned more to the left, the red spoon or the robot arm?
Answer: Red spoon is more to the left.
Example of Robo2VLM Data
Question:
The robot task is to flip the switch of the extension cord. Which colored arrow correctly shows the direction the robot will move next? Choices: A. Yellow. B. Blue. C. None of the above. D. Purple. E. Green. Please answer directly with only the letter of the correct option and nothing else.
Answer: C
19

---

Published as a conference paper at ICLR 2026
Example of Robobrain Data (ShareRobot)
Question: reach for the ketchup bottle
Answer: [[113.66.106.1 11....0000..1111151111111.11..1.
Example of Omni-Generation Tasks
Question: Use red mask to segment the edges of the banana in the image.
Answer: Generated mask in the image.
B USAGE OF LLMs
In the final stages of preparing this manuscript, the authors used Large Language Model (LLM) solely for grammar checking and language polishing. The model assisted in improving sentence structure and correcting grammatical errors enhance readability.
20


Reviews:
[{"review_id": "7cyq5WvhRJ", "reviewer": "Reviewer_C7oU", "summary": "This paper studies the role of VLMs as backbones for VLA models. The authors fine-tune 7 open VLMs on robot simulation data and evaluate them across three benchmarks (Calvin, SimplerEnv, Libero). Their results show that a simple architecture and training objective can achieve competitive performance against recent models with specific VLMs. The authors also show that performance on Calvin is correlated with the VLM’s VQA performance, and that fine-tuning on VQA data from robot tasks does not help improve performance. Finally, the authors show the importance of fine-tuning the vision encoder of the VLM.", "questions": "1. Which VQA benchmarks were used for the correlation analysis in Fig 3?\n2. Have you tried correlating robotic task performance with other downstream tasks besides VQA?\n3. What is the (lower-bound) performance of a VLM4VLA initialized from scratch?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A meta-analysis of the role of VLMs in VLA models, showing competitive performance with a simplified training framework.\n2. A study of the relationship between robot task performance and generic VQA performance.\n3. A study of the (lack of) usefulness of VQA data extracted from robot data.\n4. An analysis of the importance of fine-tuning the vision encoder, likely due to the domain mismatch between pretraining and robot data.", "weaknesses": "1. While the experiments show that some VLMs can achieve competitive performance in the VLM4VLA framework, the paper lacks specific guidelines and insights into why specific models perform better. While the authors found VQA performance to be predictive of Calvin performance, this was not the case for the other benchmarks. This left me wondering how I would choose the next VLM to initialize my VLA from.\n2. I am also concerned about the decision of using the same hyperparameters for all the models. While this drastically lowers the number of experiments, models of different sizes would likely require different hyperparameters to perform best. This can affect all the results and conclusions that the authors derive from their experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xh4URJKozg", "reviewer": "Reviewer_n8E9", "summary": "This paper studies the impact of the underlying VLM on VLA policy performance through the VLM4VLA architecture. VLM4VLA trains a continuous regression head on top of the VLM for the robot action prediction task. The paper trains and evaluates VLAs based on several different VLM sizes across the CALVIN, SimplerEnv, and Libero simulated environments. The paper finds that the impact of base VLM capabilities varies greatly by environment. Furthermore, continuing to train the VLM on embodiment and 3D-specific tasks produces a slight degradation in VLA performance.", "questions": "1. Is it still important to train the visual encoder if the image resolution is increased, as discussed in Section 4.3 (see Weakness 1)?\n2. What do the colors of the lines and shapes represent in Figure 3? Likewise, which points are used to fit the lines?\n3. How does the paper compare the VLM performance of the models between “proprietary tasks” and “general-purpose VQA benchmarks” (L357–368)? Shouldn’t VLM general capability be assessed using the same benchmarks for all models?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses the important problem of the impact of VLMs on VLA performance, which has been relatively understudied in prior work.\n1. The paper shows that general VLM capability does not necessarily correlate with VLA capability. This is an important finding since it contradicts the common intuition that a stronger VLM model is always better for VLAs. For example, recent VLA works use newer VLM bases, which this study shows is not necessarily a good decision.\n1. The paper supports its claims with comprehensive empirical analysis across many different base VLMs and simulated environments.\n1. The result showing that further training on VLM auxiliary tasks does not improve downstream VLA performance, as presented in Figure 4, is a novel insight. The paper supports this claim with comprehensive results that test a wide variety of auxiliary tasks. Like the VLM-to-VLA transfer result, this finding is also surprising and contrary to prior efforts that design auxiliary tasks to improve VLM embodied capabilities.\n1. The results in Section 4.3 provide an important lesson by comprehensively demonstrating the importance of training the visual encoder, where freezing the visual encoder leads to a large performance drop.\n1. The VLM4VLA architecture provides a consistent and simple way to adapt VLMs to VLAs. The paper also shows that this architecture produces results superior to prior works that use more complicated designs.\n1. The paper includes sufficient reproduction details in the appendix.", "weaknesses": "1. The importance of the visual encoder can also be explained by several other factors beyond the need to finetune it. (1) The Qwen2.5-VL model is sensitive to image resolution, with higher resolutions using more visual tokens per image and typically producing better performance. It is possible that the visual encoder could be frozen if the image resolution were increased. (2) The VLMs are trained primarily on real images, while the selected benchmarks are not photorealistic and use only simple rendering (see the top of Figure 1). Thus, the visual encoder may need to be finetuned to overcome this domain gap, which would likely not be an issue with real robot data.\n2. The linear correlation plot in Figure 3 is hard to interpret (see Question 2). The paper should report the strength of the correlation.\n3. The paper appears to rely on VQA benchmarks as a measure of VLM capability in Figure 3. Are there more specific VLM capabilities, such as spatial understanding, that have a direct relationship with VLA performance?\n4. The paper does not provide analysis of what drives VLA performance based on the base VLM. For example, why does Kosmos-2 achieve the highest success rate in Table 2? While showing that VLM capabilities do not necessarily correspond to VLA capabilities is a valuable contribution, providing initial evidence for why this occurs would strengthen the paper.\n5. The paper does not experiment with larger VLM sizes. It is possible that larger VLMs learn more general features that improve VLA capabilities, which this paper does not rule out.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZPlgvjRHId", "reviewer": "Reviewer_KoNy", "summary": "The authors propose VLM4VLA, a general framework for adapting arbitrary Vision-Language-Models (VLMs) into Vision-Language-Action (VLA) models, requiring only a minimal number of new parameters. The work also investigates how the choice of the base VLM affects the downstream VLA's performance.", "questions": "1. Could you provide a performance comparison and analysis for a model using both a diffusion action head and an MLP action head? Or, perhaps more directly, a comparison between using embeddings vs. using the KV-cache for information interaction?\n2. Are there any additional experiments that could demonstrate the difference in generalization capabilities among the various VLMs when applied to VLA tasks?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper presents a framework for fairly comparing the performance of different VLMs on VLA tasks and provides an in-depth study into the reasons for performance discrepancies.\n2. By using an MLP action head instead of a more complex diffusion-based one, the framework avoids introducing stochasticity. This ensures a \"fair and reproducible\" comparison across the different VLMs.\n3. It systematically proposes three benchmarks for evaluating VLM capabilities: general capability, embodied-specific capability, and the vision encoder.\n4. The experiments cover a wide range of models and test tasks, providing a strong empirical baseline for future work.", "weaknesses": "1. The study lacks real-robot experiments. The sim-to-real gap is a major concern in the VLA field, and this work doesn't clarify how different VLMs might affect the model's final generalization to real-world scenarios.\n2. Diffusion action heads and MLP action heads may leverage VLM capabilities differently (e.g., many diffusion heads use the VLM's KV-cache for information interaction). The paper does not directly compare the impact of these two approaches on VLA performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MrHO1EHTIa", "reviewer": "Reviewer_ZXC9", "summary": "This work conducts an empirical study (in simulation) to investigate the relationship between the capabilities of the VLM backbone and its downstream performance in a VLA. The authors propose VLM4VLA, an adaptation pipeline that adds a MLP head to the VLM backbone to ensure a fair comparison across models. Through experiments in simulation, the authors arrive at three findings: 1) a VLM's performance on general VQA benchmarks is often a poor predictor of performance on robotic manipulation tasks, 2) fine-tuning a VLM on auxiliary embodied VQA tasks doesn't reliably improve performance, and 3) fine-tuning the VLM's vision encoder is necessary.", "questions": "Clarification of Pi0 baseline: could you elaborate a bit more on the specifics of why the official implementation of Pi0 was not used? In particular, were the official weights of the Physical Intelligence model used? \n\nDefining \"VLM General Capability:\" Figure 3 aggregates 'vlm capability' on the x-axis, but it is not precisely defined. What exactly does this mean?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Large-scale study: the primary strength of this work is the sheer number of models (7) and simulation environments (3) studied. This comprehensive evaluation provides a valuable perspective\n\nVLM4VLA Framework: by using an MLP (non-stochastic), the authors maintain a simple adaptation module which enables isolating the VLM backbone as one of the only independent variables. Ensuring prompts are adjusted per model is further evidence of clean experimentation.\n\nFindings: the findings that VQA performance is not predictive of manipulation performance, in simulation, is good to know for future work. However, I would like to add that I think the main reason researchers leverage a VLM backbone is to capitalize on the potential of generalization, not in-distribution robot manipulation performance (this must be taught via imitation learning).", "weaknesses": "Reliance on simulation: the paper's most significant weakness, which the authors acknowledge, is exclusive use of simulated environments. While this choice was made for reproducibility and scale, it limits whether or not the paper's main claims generalize to the real-world, which is ultimately what is needed in robotics. The sim-to-real gap is a well-known challenge in robotics and care must be taken to extrapolate findings from simulation to the real world. Moreover, Figure 3 supports this statement: the findings of VLA performance and VLM capability are not consistent across simulation environments. \n\nAs another example of the potential pitfalls of simulation, as vision encoders are often pre-trained on real-world images, evaluating them purely in the simulated domain may add a confounding variable to the analysis, as most VLAs are only fine-tuned on real-world data. Therefore, I suggest that all main claims in the paper add the qualifier that they hold only in the simulation benchmarks tested. \n\nNovelty of vision encoder finding: while the result that fine-tuning the vision encoder is crucial, this observation is not entirely new. Other works such as OpenVLA [1] and Otter [2] have described similar findings. \n\n[1] Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., ... & Finn, C. (2024). Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246.\n\n[2] Huang, H., Liu, F., Fu, L., Wu, T., Mukadam, M., Malik, J., ... & Abbeel, P. (2025). Otter: A vision-language-action model with text-aware visual feature extraction. arXiv preprint arXiv:2503.03734.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)