UP-VLA:  A Unified Understanding and Prediction Model for Embodied Agent
Authors: Jianke Zhang
Yanjiang Guo
Yucheng Hu
Xiaoyu Chen
Xiang Zhu
Jianyu Chen
Venue: ICML 2025 poster (2025)
Keywords: VLA
VLM
Embodied Agent
Primary area: applications->robotics
TL;DR: We propose UP-VLA，a unified understanding and prediction model for embodied agent.

Abstract: Recent advancements in Vision-Language-Action (VLA) models have leveraged pre-trained Vision-Language Models (VLMs) to improve the generalization capabilities.
VLMs, typically pre-trained on vision-language understanding tasks, provide rich semantic knowledge and reasoning abilities. 
However, prior research has shown that VLMs often focus on
high-level semantic content and neglect low-level features, 
limiting their ability to capture detailed spatial information and understand physical dynamics.
These aspects, which are crucial for embodied control tasks, remain underexplored in existing pre-training paradigms.
In this paper, we investigate the training paradigm for VLAs, and 
introduce \textbf{UP-VLA}, a \textbf{U}nified VLA model training with both multi-modal \textbf{U}nderstanding and future \textbf{P}rediction objectives, enhancing both high-level semantic comprehension and low-level spatial understanding. Experimental results show that UP-VLA achieves a 33\% improvement on the Calvin ABC-D benchmark compared to the previous state-of-the-art method. Additionally, UP-VLA demonstrates improved success rates in real-world manipulation tasks, particularly those requiring precise spatial information.

UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent
Jianke Zhang
Yanjiang Guo
12
Yucheng Hu"
Xiaoyu Chen 12 Xiang Zhu
Jianyu Chen
Abstract
"What the boy holdiing his hand?"
Recent advancements in VisionLanguaae-Acttinn (VLA) models have leveraged pre-trained Vision- Language Models (VLMs) to improve the gen- eralization capabilities. VLMMs, typically pre- trained on vision-language understanding tasks, provide rich semantic knowledge and reasoning abilities. However, prior research has shown that VLMs often focus on high-level semantic con- tent and neglect low-level features, limiting their ability to capture detailed visual and spatial in- formation. These aspects, which are crucial for robotic control tasks, remain underexplored in existing pre-training paradigms. In this paper. investigate the training paradigm for VLAs, and introduce UP-VLA, a Unified VLA model training with both multi-modal Understanding and future Prediction objectives, enhancing both high-level semantic comprehension and low- level spatial understanding. Experimental results show that UP-VLA achieves 33% improvement the Calvin ABC-D benchmark compared to the previous state-of-the-an method. Addition- ally, UP-VLA demonstrates improved success rates real-world manipulation tasks, particu- larly those requiring precise spatial information. Code can be found https:/ //tp//giiie /github... con/ C1 adernyJorn UP-
Multi-Model Understanding
blue watch.
Internet Vision- language Datasets
"Pick
Future Prediction
Internet Video
"Pick the blue block."
UP-VLA
Action Generation
Datasets
Figure I. UP-VLA is pre-trained with both multi-modal under- standing objective and future prediction objective to better capture both high-level semantic information and low-level spatial details, enhancing embodied pocision-making tasks.
2022; Dai et al., 2024; Driess et al., 2023) robotic action datasets, incorporating appropriate action modeling compo- nents (Jang et al., 2022; Li et al., 2023b; Wu et al., 2023; Zhang et al., 2024; Kim et al., 2024; Zheng et al., 2024b; Cui et al., 2025). This method enables VLA models to inherit the semantic knowledge and reasoning capabilities encoded in powerful VLMs, thereby enhancing decision-making in unknown environments.
However, previous works have identified certain weaknesses in VLMs, particularly in capturing low-level information and understanding physical dynamics (Zheng et al., 2024a; Chen et al., 2024a). Zheng al. (2024a) highlighted that VLMM weak in low-level vision tasks without additional training. Chen et al. (2024a); Wen et al. (2024) pointed out that pretrained VLMs lack spatial understanding and fail to capture low-level details such as distance and size differences. Furthermore, studies (Balazadeh et al., 2024; Ghaffari & Krishnaswamy, 2024; et al., 2024) have re- vealed significant challenges in VLMs' ability understand physical dynamics. These limitations are largely attributed to the pre-training paradigm of VLMs (Wen et al., 2024; Chen et al., 2024a), which prioritizes multi-modal under- standing tasks, such as Visual Question Answering (VQA), that enhance semantic reasoning but may overlook low-level details that are crucial for embodied decision makionn tasks. While the generalization advantages offered by current pre-
1. Introduction
Constructing Vision-Language-Actionn (VLA) models (Bro- han et al., 2023; Li et al., 2023b) capable of solving multiple tasks in open environments has become central focus of research in robotics. A promising approach for VLA models involves fine-tuning large-scale pre-trained Vision- Language Models (VLMs) (Li et al., 2023a; Wang et al.,
'Equal contribution 'Institute for Interdisciplinary Informa- tion Sciences. Tsinghua University.Beijing China. 'Shanghai Qi Zhi Institute, Shanghai, China. Correspondence to: Jianyu Chen <jianyuchen @ asinghua.edu..c...
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Unified Understanding and Prediction Model for Embodied Agent
training approaches are desirable, they raise an important question: better training pipeline be developed to combine the strengths of both worlds, retaining semantic understanding while also emphasizing low-level features critical for control?
Image
Instruction
Instruction
Image
Vision Language Model
Visual Prediction Model
Action
Future Action
In this paper, we re-consider the pre-training approach for VLA models. Rather than focusing exclusively on high-level semantic information like in vision-language pre- training. we propose a training paradigm that emphasizes both high-level semantic understanding and low-level visual patterns. Inspired by prior papers on visual pre- pre-training (Wu et al., 2023; Guo et al., 2024), we introduce a novel training paradigm for VLA models that aligns representa- tions with both high-level features using the multi-modal understanding dataset and low-level features through future predictive generation. Specifically, we co-train an auto- regressive model with a flexible attention mask on three types of datasets, as illustrated in Figure 1.
VLM-based VLA model
Prediction-be baseee--eeddde model
Image Instruction
ABCD
Real Unseen
Calvi ABC-D
Unified UP-VLA Model
Text
Bral
Real
Unitled UP UP-VLA [Ours)
Prechee
Seen
Figure 2. Comparison between ULL, VI.M-hased VIA mod- els and prediction- models. The hottom-right chart illustrates the performance across multiple tasks in both simulated and real- world environments. We select the best model from each type of method
Our experiments validate the effectiveness of our new train- ing paradigm for VLA models. As summarized in Figure 2, we tested three clusters of tasks ranging from simulation to real world settings to assess different abilities of the al- gorithms. ABCD-D and real-seen focus evaluating the model's 's in-distribution multitask learning capabilities, real- unseen focus on real-world semantic generalization, while the remaining two tasks measure the methods' adaptation and precise control abilities. In aligmment with our previous analysis, the VLM- VLA model demonstrates rela- tively strong performance in both in-distribution multitask settings and the real-world generalization task (real-unseen). Conversely, visual prediction-base based pre-training achieves relatively better scores in tasks requiring adaptation and pre- cise control (real-precise and ABC-D). Notably, UP-VLA achieves a 33% improvement on the Calvin ABC-DD gener- alization benchmark and shows significant improvement in real-world task. These results highlight the effectiveness of UP-VLA method in retaining both semantic and low-level features. Our contributions can be summarized as follows.
2. Related Works
VLA Models for Generalist Robot Policies Recent studies have explored the application of VLMs (Li et al., 2023a; Wang et al., 2022; Dai et al., 2024; Driess et al., 2023; Wang et al., 2023; Zhao et al., 2025; Ding et al., 2025; Song et al., 2025; Ding et al., 2024) in robotics, leveraging their strong understanding of línguistic instructions and visual scenes, A notable example is RT-2 (Brohan et al., 2023), which directly utilizes VLMs to generate discrete action tokens autoregressively, demonstrating the semantic grounding ca- pabilities of VLA methods. Recent works aim to enhance VLA models with better generalization performance (Kim et al., 2024; O' O'Neill et al., 2023), gros--embodiment con- trol capabilities (Black al., 2024), and improved reason- ing efficiency (Zhang et al., 2024). The previous work, 3D-VLA (Zhen et al., 2024), also explored co-training for multi-modae understanding and generation, but focused on introducing 3d information and employed a separate diffu- sion model for generation. In contrast, our approach uses a unified model to handle multi-modal input and mainly focuses addressing the limitations of VLA models in visual perception and physical dynamics.
1. Motivated by recent insights into the limitations of VI.Ms, we integrate video datasets rich in detailed information and dynamic contexts into the pre-training of VLA models to enhance their capabilities.
Visual Pretraining Methods for Robotics Leveraging pre- trained vision models for robotic perception has become a crucial area of research in robot control. Early works (Brohan et al., 2022: Jang et al., 2022) employed pretrained vision encoders like ViT (Dosovitskiy et al., 2020) and Ef- ficientNet (Tan & Le, 2019) to encode visual observations. Recently, numerous studies have incorporated generative models (Ho et al., 2020; Blattmann et al., 2023) for train- ing policies via future frame prediction (Guo et al., 2024; Ding et al., 2024) and video generation (Du et al., 2024). For example, SuSIE (Black et al., 2023) learns robot ac- tions by predicting keyframes, while GR-1 (Wu et al., 2023)
2. We introduce a novel training paradigm for VLA mod- els that combines both vision-language understanding and future prediction objectives, enabling the capture of both high-level semantic and low-level visual pat- terns essential for embodied agents.
3. We achieve significant improvements in success rates across both simulated and real-world manipulation tasks. Additionally, we conduct an ablation study to validate the effectiveness of two types of pre-training.
2

---

A Unified Understanding and Prediction Model for Embodied Agent
directly pretrains policies through video generation. PAD (Guo et al., 2024) employs diffusion models to predict fu- ture images and multi-step actions simultaneously. IGOR (Chen et al., 2024b) uses latent actions that compress vi- sual changes as the intermediate goal for low-level actions. These studies highlight that visual prediction tasks can ben- efit models' visual generalization to unseen scenes, Our approach leverages autoregressive VLMs to predict future images, capturing physical dynamics with rich visual infor- mation.
derstanding prompting techniques.
4.1. Backbone
As illustrated in Figure 3, we employ Phi-1.5 (Li et al., 2023c) as the underlying large language model. For multi- modal understanding tasks, we follow the standard VLM encoding approach, projecting images into the language embedding space via a CLIP-ViT (Radford et al., 2021) en- coder. These projected image features are then concatenated with language embeddings and fed into the large language model. For image prediction tasks, we encode the currently observed image into discrete tokens using VQ-GAN (Esser et al., 2021). Instead of using noise prediction or masked re- construction, we directly predict future image tokens in the corresponding position of output tokens, which encourages the model to focus on the visual information in the current frame and predict future changes conditioned on language.
3. Preliminaries
VLA for Language Conditioned Robot Control The language-conditioned manipulation problem is considered decision sequence under the environment modeled by free-form language instruction specifying certain task and the initial observation 01. For demonstrations DD = where each frame = {(o,, )== con- sists visual observations o and actions a. Vision-Language- Action (VLA) models typically train VLMMT as a robotic action policy by minimizing the error between à ~ #o l). Leveraging the multi-modal understanding capability of VLM, VIA has better generalization across tasks, espe- cially enhanced semantic understanding of unseen objects and improved ability to understand or reason complex natu- ral language instructions.
(TI T2,-.. ; Jn
4.2. Bridging Visual Prediction and Multi-modal Understanding
To enable L.LMs to possess both visual prediction and multi- modal understanding capabilities, we incorporate both fu- ture prediction tasks from robotics data and image-text pairs during training. These two types of tasks into unified format so they can be mixed and processed in parallel through the LLM backbone. Therefore, we extend the multitasking approach described in see 3.
be encoded
Unified Vision-Language Pretraining via Autoregressive Modelling Unified multi-moda, language models are ca- pable of understanding and generation. An effective and scalable approach is improve the VLMs with additional image generation task, as demonstrated in works like SeeD- X (Ge et al., 2024) and Showo (Xie et al., 2024). Following these approaches, we utilize a discrete image encoder to handle encoding and decoding for image-generation tasks, while employing a continuous vision encoder for multi- modal understanding and reasoning tasks. During training, the LLM input is prompted based on the task type in the following format:
Multi-modal Understanding Given a paired image-text question-answeriig set (1, L). we encode the image into the language embedding space via a continuous encoder and a connection layer F,, resulting These embeddings are concatenated with text embeddings (4) to form the multi-modal input. To generate a text sequence that can focus on the image while com- prehending the language, we modify the causal attention mechanism so that image tokens can attend to each other, as shown in Figure 4(a). Finally, we use an autoregressive manner predict the next language token. This task can be briefly described L = T. MUM (I.
{u4 ===0 F1
(MMU), (ui, "2. un)) (1, ... } ,)m)) [T22], (41, la..,,,,,,, (v1, -2. bn)}
Future Visual Prediction For image prediction, given an image and instruction pair (Ot, L) at time I, we encode the current visual observation using a discrete encoder Eg: V, = (w:) = (O.). Unlike multi-modal understanding tasks, the objective in visual prediction to encode future visual observation by focusing on the instruction prompts. Thus, as depicted in Figure 4(b), we place the image tokens after the language tokens, enabling the image to attend to all input information. Meanwhile, we introduce a new special token PRE to denote this new task. Instead of using next token prediction, we model the future image tokens at the
where represents language tokens, and u, v correspond to continuous and discrete image tokens for different tasks.
4. Methodology
Our goal is to develop better training schema for VLAs. In this section, we describe the details of UP-VILA We first build our backbone on top of a unified VLM. Then, we design a unified mechanism to bridge the gap between visual prediction with multi-modal understanding. Finally, we enhance action learning with unified prediction and un-
3

---

A Unified Understanding and Prediction Model for Embodied Agent
"Whut
"Whet's & the boy holding in his hand?
CU
Projector
Sakuniar
Internet Vision-
UP -L- Model
Autoregressive Generate
Model
Autorogressvve
Language
NONANN
Inatrsttoo
londe
Cops Longuage Anownr
up
UP VLA Model
Direct Generation
Internet Video
UP Model
Direct Generate
Langugge undennaandingg
Futurra Predictonn
Candinaous image tokanu
Toxt tolena Acrice Takan
Mhic und luy
Imrsges
Futtan
Robet
(a) Pretralning Stage
(b) Finetuning Stage
Figure 3. Overview of UP-VLA. Our model unifies visual-language understanding, future image generation, and action learning in an autoregressive manner. It takes the current visual scene and language instructions as inputs, produces high-level understanding of the scene, and subsequently predicts future images and robotic actions based on these understanding tokens.
same positions of the image tokens:
Finally, we generate actions via joint prediction:
P(Ortar II polui+aelve
PRE (O,, T.) (O+tse, Acata)= =
and then use a discrete decoder to reconstruct the predicted future observation image Ö.+ This task can be succinctly described as Ó.+AI= = xg
PRE L).
We use small policy head to output low-level actions, con- sisting of MAP module single-layer attention module) and a linear layer: d,++A MLP(MMAPAAA)))
4.3. Enhancing Action Learning with Joint Prediction and Understanding
While previous VLA method leverages the muli-modal un- derstanding knowledge of pre-trained VI.M, it fails to ex- ploit the rich visual information and the physical dynamics. To address this limitation, we propose joint prediction- und-understanding action learning mechanism. We integrate action output with image prediction tasks. Given the cur- rent observation-instructioo pair (O,, L), our model predicts both future observations and a sequence of actions at each time step: (Ö.+at, A = T
4.4. Training Strategy
We initialize the backbone of UP-VLA using Show-o (Xie et al., 2024). During training, fully fine-tune the pa- rameters of the LLM and freeze all encoders. The training process can be divided into two stages. In the first stage, we aim to endow the VLM with both visual prediction and modal understanding capabilities. In the second stage, we focus on learning actions using robot data. We apply different sampling ratios for different tasks.
_PR (O,, L). where Å cor- responds the final layer features the positions of the action tokens.
1.4.1. TRAINING PIPELINE
In addition, as shown in Figure 4(c), extend the language instruction input with scene descriptions generated by the model itself. The expanded input prompt
Prediction and Understanding Pretraining Stage. We mix training data across two domains: one part is from Bridge (Walke et al., 2023), which includes 25k robotic arm demonstrations. We use this data for future prediction. Another part is from LLLav--uniig--666 (Liu et al., 2024), which includes 665k image-text pairs for enhancing high- level understanding capability.
L' (O'), m8
M MU (Ot, Lpromp)) L]
where L is the language instruction and O represents vari- ous visual information at the current time step. The obser- vation O,, after processing through the continuous vision encoder E = MLP(VIT), is mapped into the language embedding space E1 (Ot) which can be directly used as lan- guage tokens. The final component Tg
Prediction with Action Tuning Stage.The model is fine- tuned on downstream embodied tasks. We train UP-VLA using the joint prediction iiniictioonnnnduune wareiii--nnndnnddsttatat unddrdsdtttnnnnngg action learning approach in see 4.3. We continue to co-train with the image- text pairs to preserve multi-modal understanding ability.
MM (O,, Ipromp)) is the generated description of the current scene, Lh a specific prompt, such as "describe this image".

---

A Unified Understanding and Prediction Model for Embodied Agent
EOI
Start/End Continuous Imng
SOT
EOT
Start/End of Text
abtu
SOI
EO
Start/End Discrete Imn Tokens
720
SOA
F0A
Start End of Action
Future img Tokens
Image
Action
(a) MMU Datasets
(b) Future Prediction Datasets.
(c) Robotic datasets
SOI
EOI
SOT
EOT
SOT
501
EO1
Multi-model Understanding
Future Prediction
(c)
EDI SOT
EOT SOI
Eoll
SOA
EDA
(LP(VIT(O,))
KMI
(prrompo))))
Figure Illustration the unified prompting and attention mechanism. use special tokens to segment imput sequences identify task types. For MMU tasks, consecutive image tokens placed before the language tokens, allowing image tokens to attend to one another. For image prediction, images are positioned after language tokens, enabling them to attend to all prior information and predict future images that align with the language instructions. For action learning, which combines understanding and prediction, tokens from both tasks are concatenated, allowing actions to attend to both high-level scene understanding and low-level visual information.
Env A
Env
Seen Tasks
Unseen Tasks
Precise Tasks
Env
Env
into green plate
Pick the pepper
the pen
Catch blue
Press the button
Open the drawer
Pick the orange lego
Pick the arrow
Route the cable
(1) Calvin ABC->D
(2) Real-world Panda Manipulation
Figure 5. Visualization of our evaluation environments, The left is Calvin (Mees al., 2022) in which we test on both ABC-D and ABCD-D settings. For real world, we train our model on simple tasks and test on more complex scenarios.
4.4.2. TRAINING OBJECTIVE
entropy to predict future image tokens Vi+At {u' J.M
The UP-VLA method involves three modeling targets: lan- guage modeling for multi-modal understanding, image mod- cling for visual prediction, and action modeling for embod- ied tasks.
LPRE = Σ (1 01, "1. "8,''. "
Action Modeling for Embodied Tasks. We minimize the mean squared eттor (MSE) between the predicted relative position apos and the ground actions apos The discrete status Gend of the end-effector is optimized with binary cross-entropy loss (BCE):
Language Modeling for multi-modal Understanding. Given M visual tokens M and N text tokens I == (4 maximize the likelihood of the next token using cross-entropy loss:
LMMU = Σ log po (4-1)
LACT Σ |lāpos Apos 11 +
BCE(Qend aend)
Image Modeling for Visual Prediction. For the future image prediction task, given M current image tokens v. = (v.) N and instruction tokens I (4) we use cross-
We use varying weights combine these three losses:
AIMMMU + >2L AsLACT
5

---

A Unified Understanding and Prediction Model for Embodied Agent
5. Experiments
RT-2(Brohha et al., 2023) or Open VLA (Kim al., 2024)).
UP-VLA a versatile iisionllanguage-action model that can perform multi-modal understanding and future predic- tion while generating robot actions. In this section, we evaluate UP-VLA in two domains including the simulation CALVIN benchmark (Mees et al., 2022) and a real-world panda manipulation environment to verify the effectiveness of our UP-VLA framework.
Uni-Pi (Du et al., 2024): learns to generate future se- quences and then output actions with an inverse kine- matics model.
Susie (Black et al., 2023): first generates goal image and then learns a goal-conditioned diffusion policy.
GR-1 (Wu et al., 2023): pretrains a transformer on video prediction task and then finetune on robot data to learn multi-task robot manipulation.
5.1. Experiment Setup and baseline
Setups. For simulation evaluation, we utilize CALVIN (Mees et al., 2022), an open-source benchmark to learn long- tasks. As shown in Figure 5(a), the CALVIN environment comprises 4 different scenes denoted ABCD. We evaluate UP-VLA on both ABCD-D and ABC-D settings.
UP- DP-VLL-phhi-wo-mmme UP-VLA initialized from Phi- 1.5 (Li et al., 2023c) and is trained without multi-modal understanding tasks, serves as a reimplemented GR-1 under the same setting of our method.
horizon language-conditiong
3D Diffuser Actor (Ke et al., 2024): learn a 3D diffu- sion policy using depth image with camera pose.
Our real-world experiments involve multiple table-top ma- nipulation tasks on the Franka-Emiks, Panda robot, including picking and placing, routing cables, pressing buttons, and opening drawers. Specifically, we collect over 2k demon- strations above 6 skills. As shown in Figure 5(b), we train UP-VLA on simple scenes while testing on more complex settings. We place several seen and unseen objects the table to introduce distractions and test whether the model can grasp entirely new objects to verify its semantic ground- ing capabilities. Meanwhile, we evaluate the model's ability to perform more fine-grainet tasks, such as routing cables, grasping smaller unseen blocks, or picking up a pen. More details of dataset can be found in Appendix B.
5.2. Simulation Evaluation
Table 1 and Table 2 presents the experimental results in simulation environments. UP-VLA achieves the highest performance on both ABC-D and ABCD-D tasks. Com- pared to other baselines, which perform significantly worse on ABC-D than on ABCD DD UP-VLA achieves higher completion lengths in both scenarios, indicating that our method has better multitask learning and generalization ca- pabilities in simulation tasks.
Effectiveness of Visual Prediction. Comparing VLAA-based methods and prediction-based methods in Table 1, it is ev- ident that previous VLA approaches underperform in sim- ulation tasks relative to prediction-based methods. For ex- ample, RoboFlamingo, achieves a length of 2.47, which is less than GR-1's length of 3.06. This suggests that relying solely on dision-language understanding pretraining can be limiting in tasks that emphasize visual generalization. Our method addresses this limitation by incorporating visual pre- diction into the original VI.A framework. Compared to UP- VLA-RT-2, which uses only action learning and achieves a completion length of 1.44, UP-VLA with visual prediction significantly improves the length to 4.08. This demonstrates that integrating visual prediction can substantially enhance the performance of original VLA methods.
Baselines. We mainly compare UP-VLA with two types of baselines: VLM-based VLA methods and future- predictrediition- based methods (mainly including future image generation, goal generation and video generation). All baselines in our experiment are listed as below:
RT-1 (Brohan et al., 2022): a small robot action trans- former using pretrained Efficient-Net (Tan & Le, 2019) as vision encoder.
Diffusion Policy (Chi et al., 2023): a small action model using diffusion model.
Robo-Flamingo (Li et al., 2023b): a typical VLA model consists of a pretrained VLM and an LSTM policy head.
Effectiveness of multi-modal Understanding, Compared to prediction- methods, the UP-VLA method demon- strates superior performance. To further investigate the factors contributing to the performance improvement, we design HI-VLA--pi-wia variable of model backbone differences. This method ini- tializes UP-VIA using a pure LLM,, phil.5 .5 (Li et al., 2023c) and performs pretraining on the Bridge dataset for future prediction and is then finetuned with downstream robot task. Unlike VLA, UP-VLA-phh -hi--/hhiooooo will-w/wommmmmmm mom does not include multi-modal understanding training, nor does incorporate
3D-VI. (Zhen et al., 2024): a unified VI.A model that enable 3D reasoning, multi-modal goal generation, and action planning. Different from VLA, 3D- VLA mainly focuses on introducing 3D knowledge into VLM (LLM).
as baseline to climinate the
UP-VLA-RT-2: an apple-to-apple baseline that makes use of the same backbone with UP-VLA and direct output actions autoregressively (reimplementation of
6

---

A Unified Understanding and Prediction Model for Embodied Agent
Tasks completed in row
Method
Type
Task
2
3
4
5
Avg. Len 1
RT-1
other
ABC-D
0.533
0.222
0.094
0.038
0.013
0.90
Diffusion Policy*
other
ABC-D
0.402
0.123
0.026
0.008
0.000
0.56
3D Diffuser Actor
other
ABC-D
0.938
0.803
0.662
0.533
0.412
3.35
3D-VLA
VLA
ABC-D
0.447
0.163
0.081
0.016
0.000
0.71
UP-VLA-RT-2*
VLA
ABC-D
0,612
0.389
0.236
0.138
0.062
1.44
Robo-Flamingo
VLA
ABC-D
0.824
0.619
0.466
0.331
0.235
2.47
Uni-Pi
Prediction
ABC-D
0.560
0.160
0.080
0.080
0.040
0.92
Susie
Prediction
ABC-D
0.870
0.690
0.490
0,380
0.260
2.69
GR-1
Prediction
ABC-D
0.854
0.712
0.596
0.497
0.401
3,06
UP-VLA- ph---m-ph-woommmu UP-VLA
Prediction
ABC-D
0.844
0.705
0.604
0.520
0.430
3.13
VLA
ABC-D
0.928
0.865
0.815
0.769
0.699
4.08
Table
Zero-shot long-horizon evaluation on the Calvin benchmark where agent is asked to complete five chained tasks sequentially. Results marked with asterisk (+) indicate those that we reproduced, while the copied from the original papers. UP-VLA achieves the best performance, demonstrating that our approach exhibits strong generalization capabilities in simulated environments.
Tasks completed in row
Method
Type
Task
3
4
5
Avg. Len 1
RT-1
other
ABCD->D
0.844
0.617
0.438
0.323
0.227
2.45
Robo-Flamingo
VLA
ABCD->
0.964
0.896
0.824
0.740
0.660
4.09
GR-1
Prediction
ABCD-D
0.949
0.896
0.844
0.789
0.731
4.21
UP-VLA
Prediction&VLA
ABCD-D
0.962
0.921
0.879
0.842
0.812
4.42
Table 2. Zero-shot long-houe han-horizonn evaluation on the Calvin ABCD-D benchmark. Results of baselines are copied from original papers.
image comprehension in the language prompts during out- performs
not encountered during training, shown in the middle of Figure 5. UP-VLA-RT-2 outperforms UP-VLA-phit w/o- mmu, suggesting that multi-modal understanding aids se- mantic generalization ability. UP-VLA demonstrates better visual-semantic generalization for these tasks, proving that our approach effectively aligns multi-modal understanding with objects and actions. The right side of Figure shows the performance on tasks requiring more precise operations (e.g., routing cables, grasping small objects, and picking up previously unseen pens). These tasks demand the model's enhanced spatial understanding and ability to perceive visual details. As opposed to new objects, excels at precise operations compared to UP-VLA-RT-2 UP-VLA performs best on these tasks, highlighting that the integration of future visual prediction enhances VLA's understanding of physical space and details.
put. As shown in Table 1, LP--LL-phi-w/o-e worse than UP-VLA, which indicates that injecting multi- modal understanding into the model during training helps improve its generalization ability in new scenarios.
5.3. Real Robot Evaluation
For real-world experimental results, we train RT-1 (Brohan et al., 2022), Diffusion Policy (Chi et al., 2023) on our datasets (using the open-source code and testing them on the same physical hardware). To ensure fair comparison, we reproduce RT-2 (Brohan et al., 2023) and GR-1 (Wu al., 2023) using our dataset and backbone, referred to respectively. We report the success rate of each task over 20 attempts during real-world roll-out.
as UP-VLA-RT-2 and WLL--ph-w/o-meee
Figure 6 presents the results of our evaluation on three types of world tasks. UP-VLA demonstrates significant improvement across all tasks. Specifically, for tasks seen during training, shown the left side of Figure 5(b), the three methods based on the UP-VLA backbone outper- form RT-1 and Diffusionpolicy, indicating that using LLM backbone exhibit superior multitasking capabilities.
5.4. Ablation Studies
In this section, we aim to understand each module UP- VLA. We compare the full UP-VLA with the following methods: UP-VLA-WO-MMA LLava tuning dataset for multi-modal understanding, UP- VLA-w/o-Brides Sridge-Pretraing which skips visual prediction pretraining on the bridge dataset; WL-V-L---/--Peeiieeee which bypasses visual prediction and directly learns actions;
which does not utilize the
For unseen tasks, we first test the ability to grasp new objects

---

A Unified Understanding and Prediction Model for Embodied Agent
UP-VLA-RT-2
Multi-modal Understanding As shown in Fig it can be observed that UP-VLA can identify the objects present in embodied scenes and estimate their approximate relative positions, which is critical for action learning. Therefore, ef- fectively integrating MMU capabilities with action learning is a promising approach to improving operational accuracy. Additionally, we observe that the model's identification of the specific objects sometimes inaccurate. This is likely due constraints in the scale of data and backbone, which is potential direction for future research.
Diffusion Polley
UP-VLA
UP-VLA-phii-e whi--wmm- (GR-1)
0.75
Rate
0.6
0.60
Q4
0.40
0.50
0.30
0.10
0.0
0.00
Seen tasks
Unseen [now objacts)
Unseen pprecise tasks)
Future Prediction Results of predicted images are shown in Fig 7. UP-VLA demonstrates the ability to accurately predict the positions robot arms and objects based on language instructions. However, we also identify some limitations. For instance, in the Calvin D environment, the predicted frames feature background colors that differ from the input frames. The model tends to use background colors from the training datasets (ABC). This issue is likely due to insufficient pretraining for visual generation, which limits the model's 's generalization capability in generation tasks.
Figure 6, Results on real-world manipulation tasks.
ABC-D
Real World
Method
Lent
Seen
Unseent
w/o MMU
3.89
0.85
0.20
w/o Bridge-Pretrait
2.74
0.65
0.30
w/o Prediction
1.44
0.65
0.35
w/o MMU-Condition
3.99
0.80
0.50
Full
4.08
0.80
0.58
Table Ablating components of VP-
6. Conclusion
which omits the mech- anism described in sec 4.3 that extends visual prediction prompts using MMU. Table 3 presents the performance of different ablation methods on simulated Calvin tasks and real-world robot tasks.
and UP-VLA-wwo-MMU-Condiin
In this paper, we introduce UP-VLA, vision-language action model that can understand, generate predicted future images, and plan actions in the embodied environment. We devise a novel VLA training paradigm that unifies policy learning with visual prediction and multi-modal understand- ing. Our results show that the use of future image prediction can significantly improve the precision and visual general- ization of policy. We also further enhance our model by introducing multi-modal understanding knowledge to vi- sual prediction- based policy learning, which demonstrates stronger generalization ability in both semantic grounding and spatial understanding.
How does the visual prediction mechanism affect the performance of UP-VLA? After removing the visual pre- diction task, performance of UP-VLA in Calvin drops from 4.08 to 1.44. This indicates that stronger visual supervi- sion contributes to the model's ability to generalize in new environments. Furthermore, omitting the visual prediction pretraining on the bridge dataset also led noticeable per- formance degradation, highlighting the importance of the pretraining step for the VLM to learn the dynamics of the physical world effectively.
Impact Statement
This paper presents a novel research advance robot ma- nipulation models with unifed training strategies. Given that robots operate in the physical world under certain human instructions, high-level semantic content and also low-level visual and spatial details are crucial for accurate robot con- trol. To mitigate this issue, our approach involves unified training paradigm force VLA capture both semantic information and learn dynamics of the physical world.
Does multi-modal understanding enhance the general- ization ability of the model? In real-world experiments, removing MMU tasks or MMU-condition mechanisms from training leads comparable higher performance on seen tasks but results in a decline in performance on unseen ob- jects. This observation indicates that joint training with MMU and using MMU to augment input prompts are cru- cial for semantic generalization and help prevent the model from overfitting to the dataset.
References
Balazadeh, V., Ataci, M., Cheong, H., Khasahmadi, A. H., and Krishnan, R. G. Synthetic vision: Training vision- language models to understand physics. arXiv preprint arXiv:2412.08619, 2024.
5.5. Quantitative Results
Figure 7 visualizes the performance of UP-VLA in multi- modal understanding question-answering answering (VQA) and future prediction across different types of data.
8

---

A Unified Understanding and Prediction Model for Embodied Agent
Instruction push block
Instruction up the carrot.
Instruction:Put stuffed elephant in stove Question Ploase describe this image detail
right.
Question Plesse describe image in detail
Input
Question.Pleese describo image in detail
UP-VLA
UP-VLA
UP-VLA
Answer: There are dask, including blue box, black bowl, and couple books. are also two chaire in the left side of desk end another on the right sido.
The toy eggolant
Answer:
Answer: The toy elephant postlioned middle stove, and appears be interacting with luy stove, In the backgrounc, there sink, which is the image.
green surface. possibly a table cutling board. There are two plaetic toys in the scene, a carrot located the center and yellow toy towards the right side.
Predicte
Pigure 7, Visualization of VQA results and predicted future images.
Black, K., Nakamoto, M., Atreya, P., Walke, H., Finn, C., Kumar, A., and Levine, S. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint arXiv:2310. 10639, 2023.
Chi, C.,, Feng, S., Du Y., Xu, Z., Cousineau, E.,, Burchfiel, B., and Song. S. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al. pi.0: ssiionlanguage-actionn flow model for general robot control. arXiv preprint arXiv:2410.24106 2024.
Cui, C., Ding, P., Song, W., Bai, S., Tong, X., Ge, Z., Suo, R., Zhou, W., Liu, Y., Jia, B., et al. Openhelix: A short survey, empirical analysis, and open-source dual- system vla model for robotic manipulation. arXiv preprint arXiv:2505.0300 2025.
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z.,, Voleti, V., Letts, A., al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 2023.
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li B., Fung, N., and Hoi, S. Instructblip: Towards general-purpoot vision- languale models with instruction tuning. Advances in Neural Information Processing Sys- tems, 36, 2024.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al. Rt-1: Robotics transformer for weal-world control at scale. arXiv preprint arXiv:2212.068/7 2022.
Ding, P., Zhao, H., Zhang, W., Song, W., Zhang, M., Huang, S., Yang, N., and Wang, D. Quar-vla: Vision-language- action model for quadruped robots. In European Confer- ence on Computer Vision, pp. 352-367. Springer, 2024.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.1588 15818 2023.
Ding, P., Ma, J., Tong, X., Zou, B., Luo, X., Fan, Y., Wang, T., Lu, H., Mo, P., Liu, J., et al. Humanoid-vlar Towards universal humanoid control with visual integration. arXiv preprint arXiv:2502. 14795. 2025.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn. D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929,, 2020.
Chen, B., Xu, Z.., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., and Xia, Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14455-14465, 2024a.
Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.. 033788, 2023.
Chen, X., Guo, J., He, T., Zhang, C., Zhang. P., Yang, D. C.,, Zhao, L., and Bian, J. Igor: Image-goal representations are the atomic control units for foundation models in embodied ai, 2024b. URL https: /araiiivv....... org/ abs/241 00785.
Du, Y., Yang, S., Dai, B., Dai, H., Nachum, O., Tenenbaum, J., Schuurmans, D., and Abbeel, P. Learning universal
9

---

A Unified Understanding and Prediction Model for Embodied Agent
policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024.
Li, Z.,, Wang, H., Liu, D., Zhang. C., Ma, A., Long, J., and Cai, W. Multimodal causal reasoning bench- mark: Challenging vision large language models to infer causal links between siamese images. arXiv preprint MrXiv:2408.0/11 2024.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for nigh-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Liu, H., Li, C.,, Wu, Q. and Lee, Y. Visual instruction tun- ing, Advances neural information processing systems, 36, 2024.
Ge, Y., Zhao, S., Zhu. J., Ge, Y., Yi, K., Song, L., Li, C., Ding, X., and Shan, Y. Seed-x: Multimodal models with unified multi- guantiigaaulrrity comprehension and generation. arXiv preprini arXiv:2404. 14396, 2024.
Mees, O., Hermann, L., Rosete- BeateeBBB,,, E., and Burgard, W. Calvin: A benchmark for aeguuag-connitionnel policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L), 7(3)7337-77334, 2022.
Ghaffari, S. and Krishnaswamy, N. Exploring failure cases in multimodal reasoning about physical dynamics. In Proceedings of the AAAI Symposium Series, volume 3,
pp. 105-114, 2024.
O' O'Neill, A., Rehman, A., Gupta, A., Maddukuri, A.,
Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:23 0.08864, 2023.
Guo, Y., Hu, Y., Zhang, J., Wang, Y.J.,, Chen, X., Lu, C., and Chen, J. Prediction with action: Visual pol- icy learning via joint denoising process. arXiv preprint arXiv:2411.18177 2024.
Radford, A., Kim, J. W., Hallacy, C.,, Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Jang. E., Irpan, A., Khansari, M., Kappler, D., Ebert, F., Lynch, C.. Levine, S., and Finn, C. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pp. 991-1002. PMLR, 2022.
Song, W., Chen, J., Ding, P., Zhao, H., Zhao, W., Zhong, Z., Ge, Z., Ma, J., and Li, H. Accelerating vision-language- action model integrated with action chunking via parallel decoding. arXiv preprint arXiv:2503.028 2025.
Ke, T.WW., Gkanatsios, N., and Fragkiadaki, K. 3d diffuser actor: Policy diffusion with 3d scene representations. arXiv preprint arXiv:2402.10885 10885, 2024.
Tan, M. and Le, Q. Efficientnet: Rethinking model seal- ing for convolutional neural networks. In International conference on machine learning, pp. 6105-6114. PMLR, 2019.
Kim, M., Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., Vuong, Q., Kollar, T., Burchfiel, B., Tedrake, R., Sadigh, D., Levine, S., Liang, P., and Finn, C. Open- vla: An open-source vision- vision-language-- language-actoo model. arXiv preprint wwxiv:2406.092444 2024.
Walke, H., Black, K., Lee, A., Kim, M. J., Du, M., Zheng, C., Zhao, T., Hansen-Estrucch, P., Vuong, Q., He, A.. Myers, V., Fang, K., Finn, C., and Levine, S. Bridgedata v2: dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023.
J., Li, D... Savarese, S., and Hoi, S. Blip-2: Bootstrapping language -eegugge-mmac pre- trainirinng with frozen image encoders and large language models. In International conference on machine learning, pp. 19730 19742. PMLR, 2023a.
Wang. W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., et al. Image as a foreign language: Beit pretraining for all vision and vision- lianun-language tasks. arXiv preprint arXiv:2208, 10442, 2022.
Li, X., Liu, M., Zhang, H., Yu, C., Xu, J., Wu, H., Cheang, C., Jing, Y., Zhang, W., Liu, H., et al. languae foundation models as effective robot imitators. arXiv preprint Xtx::23111.00313 2023b.
Wang, Y.J.,, Zhang, B., Chen, J., and Sreenath, K. Prompt robot to walk with large language models. arXiv preprint arXiv: 2309::2000009699 2023.
Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y. T. Textbooks are all you need ii: phi- 1.5 technical report. arXiv preprint arXiv:2309.05663 2023c.
Wen, C., Jayaraman, D., and Gao, Y. Can transformers capture spatial relations between objects? arXiv preprint arXiv:2403.00729, 2024.
10

---

A Unified Understanding and Prediction Model for Embodied Agent
Wu, H., Jing, Y., Cheang, C., Chen, G., Xu, Li, X., Liu, M., Li, H., and Kong, T. Unleashing large-scale video generative pre-training for visual robot manipula- tion. arXiv preprint arXiv:2312.13114 2023.
Xie, J., Mao, W. Bai, Z., Zhang, D. Wang, W., Lin, K. Q.. Gu, Y., Chen, Z.,, Yang, Z.,, and Shou, M. Z. Show-o: One single transformer unify multimodal understanding and generation. arXiv preprint arXiv:2408. /2528, 2024.
Yu, L., Cheng, Y., Sohn, K., Lezama, J., Zhang, H., Chang, H., Hauptmann, A. G., Yang, M.-H., Hao, Y., Essa, I., et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10459-10469, 2023.
Zhang, J., Guo, Y., Chen, X., Wang, Y-J., Hu, Y., Shi, C., and Chen, J. Hirt: Enhancing robotic control with hierarchical robot transformers. arXiv preprint arXiv:2410.05273, 2024.
Zhao, W., Ding, Zhang, M., Gong, Z., Bai, S., Zhao, H., and Wang, D. Vlas: sgiio-language-actin model with speech instructions for customized robot manipulation. arXiv preprint arXiv:2502.13508 2025.
Zhen, H., Qiu, X., Chen, P., Yang. J., Yan, X., Du, Y., Hong, Y., and Gan, C. 3d-vla: A 3d vision- lision-lagguage-cction gen- erative world model. arXiv preprini arXiv:2403.0900 1, 2024.
Zheng, B., Gu. Li, and Dong, C. Lm4ly: frozen large language model for low-level vision tasks. arXiv preprint arXiv:2405.1577 2024a.
Zheng, R., Liang, Y., Huang, S., Gao, J., Daumé III, H., Kolobov, A., Huang, F,, and Yang, Tracevla: Vi- sual trace prompting enhances seatoal-temporal aware- ness for generalist robotic policies. arXiv preprint 10X44:2412.10345, 2024b.

---

A Unified Understanding and Prediction Model for Embodied Agent
A. Implementation Details
use pretrained Showo-512x512(1 .3B) (Xie et al., 2024) as the backbone and CLIP-VIT (Radford et al., 2021), MagVIT (Yu et al.. 2023) (VQ-GAN (Esser et al., 2021)). In the pretrain stage, we train UP-VLA for 20k steps with batch size of 64 on future prediction and vision-language understanding tasks. We apply linear warmup the first Ik steps. In the action learning stage, we train UP-VLA with batch size of 64.
B. Manipulation Dataset Details
For the simulation environment data, we follow the setups of the CALVIN benchmark (using its training sets and evaluation sets).
In the real world, our training data is collected using both manual and scripted methods. Specifically, for tasks such as grasping two types of toy fruits (carrot and eggplant), opening drawers, and routing cable tasks, collect demonstrations manually using remote operation joystick, ensuring that the target objects are roughly evenly distributed in the field of view. For grasping blocks of different colors, we use scripted policies. We randomly initialize the robotic arm's 'ss position to collect trajectories for these tasks.
For real-world evaluation, our tests primarily focus on unseen settings to validate the semantic generalization capability of our method. We test whether the model can complete tasks despite the introduction of more distracting objects, broader range of positional variations, diverse backgrounds, and entirely arrow-shaped paper, unseen vegetables, toy pizza, and blocks with unseen color. We also design precise tasks to better test model's ability to handle visual details, like picking up small ball or pen.
objects, e.g. different shapes of vegetables, an
12


Reviews:
[{"review_id": "nUHsOWSolF", "reviewer": "Reviewer_dmnE", "summary": "This paper presents UP-VLA, a Unified Vision-and-Language Alignment (VLA) model trained with dual objectives: multimodal understanding and future prediction. Building on the foundation of the show-o framework, UP-VLA significantly improves both in-domain performance and generalization to unseen scenarios across real-world and simulation tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and efforts in reviewing our paper! Based on your review, we added a detailed discussion and additional experiments. \n\n---\n\n**Q1: predicting videos does not necessarily lead to a comprehensive understanding of physical dynamics**\n\nANS: Whether video models can truly understand physical dynamics remains an open research question, with arguments on both sides [1]. In our experiments, we found that training with a video prediction objective enables UP-VLA models to generate physically consistent future frames. Moreover, we think that better understanding can be reflected in better decision-making and control capabilities. As shown in Table [Q2], our ablation studies demonstrate that video prediction objective consistently improves robot control performance in all settings. We hope this clarification can solve your concern!\n\n[1] https://openai.com/index/video-generation-models-as-world-simulators\n\n\n---\n\n**Q2: Maybe performance improvements are more likely attributed to forcing the model to predict multiple future outcomes (future steps), as shown in VLAS, OpenVLA-OFT**\n\nANS: Thank you for your insightful question! We agree that multi-step prediction can benefit action learning, which we adopted as the default in our experiments. However, beyond multi-step prediction, our primary goal was to show that **predicting future images** (RGB modality) also improves action learning by facilitating information transfer from video datasets (image modality) to action learning (action modality).\n\nTo further illustrate the impact of multi-step action prediction and RGB image prediction, we conducted a detailed ablation study along these two axes. Our results show that increasing the number of predicted action steps and incorporating future image predictions both contribute to performance improvements. We hope this ablation further clarifies the effectiveness of UP-VLA, which integrates image prediction objectives into VLA model.\n\n| Length of predicted actions |   1   |   4   |   7   |  10  |\n| :-----------------------------------------: | :---: | :---: | :---: | :--: |\n|          w/o predict future image           | 1.44 | 1.94 | 2.17 | 2.25 |\n|           w/ predict future image           | 2.42 | 3.72 | 4.00 | 4.08 |\n\n\n---\n\n**Q3:The authors reproduced UP-VLA-RT-2 and phi-w/o-mmu, but one type is VLA and the other type is Prediction. Therefore, we should first determine whether the performance differences are related to the VLA backbone. To address this, results for both methods on other types should be implemented.**\n\nANS: Thank you for your comments. The Show-o model is fine-tuned from the Phi LLM on multimodal understanding datasets. Since we aimed to fully remove the influence of multi-model understanding tasks, we choose to start from the original Phi model in the phi-w/o-mmu ablation.\nFollowing your suggestion, we conducted a comprehensive comparison to better highlight the advantages of UP-VLA across different backbones. These experiments further demonstrate that both video prediction and multimodal understanding contribute to improved action learning.\n\n|   Method   | Backbone | Avg. Len ↑ |\n| :--------: | :------: | :--------: |\n|    VLA     |  Phi1.5  |    0.79    |\n| Prediction |  Phi1.5  |    3.13    |\n|    VLA     |  Show-o  |    1.44    |\n| Prediction |  Show-o  |    3.99    |\n\n---\n\n**Q4: Compare with Seer, VPP, and RoboVLMs. Both Seer and VPP also focus on Prediction and achieve excellent performance. RoboVLM falls under the VLA category and also performs very well.**\n\nANS: Thank you for your insightful comments! We would like to argue that UP-VLA has significant difference on learning paradigm with these previous works. Seer and VPP leverage internet video datasets to aid robot learning, while RoboVLM utilizes a pretrained VLM. Each of these methods relies on a single type of dataset—either video datasets or multimodal understanding datasets. To the best of our knowledge, UP-VLA is the first model to leverage both image prediction (video) and multimodal understanding (MMU) datasets for embodied decision-making. Furthermore, our ablation studies confirm that both types of data significantly contribute to the final performance.\n\nWe also note that Seer, VPP, RoboVLM, and UP-VLA all achieve an average task completion length exceeding 4.0 on the CALVIN benchmark. However, these three works adopt distinct architectures from UP-VLA and are concurrent with UP-VLA. To the best of our knowledge, UP-VLA achieves the best performance on CALVIN ABC benchmark among none-concurrent works. \n\n\n---\n\nWe hope our clarifications address your concerns and demonstrate the improved quality of our paper! Please feel free to reach out with any further questions.\nThank you again for your valuable time!", "claims_and_evidence": "The claim regarding the unified VLA paradigm is clear.\nHowever, I personally disagree with the authors' argument about \"limiting their ability to capture detailed spatial information and understand physical dynamics.\"\n\nFirstly, many existing works have already explored similar ideas, including GR-1 cited by the authors.\nSecondly, there is substantial evidence [1] showing that even predicting videos does not necessarily lead to a comprehensive understanding of physical dynamics.\nFinally, in my opinion, performance improvements are more likely attributed to forcing the model to predict future outcomes, which provides a prior for planning actions before execution. A more detailed discussion on this point is provided below.\n\n[1] How Far is Video Generation from World Model: A Physical Law Perspective", "methods_and_evaluation_criteria": "The proposed method makes sense. This is an attempt to adapt the unified MLLM to VLA, and it also yields some interesting conclusions.", "theoretical_claims": "I have checked the method, but since there is hardly much theoretical proof, I won't discuss it further here.", "experimental_designs_or_analyses": "I think the experimental section is relatively comprehensive, but there are still some issues with missing comparisons.\n\n1. The authors reproduced UP-VLA-RT-2* and phi-w/o-mmu*, but one type is VLA and the other type is Prediction. Therefore, we should first determine whether the performance differences are related to the VLA backbone. To address this, results for both methods on other types should be supplemented.\n\n2. In [1] and [2], they both reproduced VLA-RT-2 and found that simply allowing the model to predict multiple future steps could significantly improve the performance of the backbone model. So, is it possible to understand that as long as the model is tasked with predicting future steps, its performance can skyrocket? A similar conclusion comes from OpenVLA-OFT [3], where simple multi-step reasoning alone brought significant performance improvements. Additionally, the experiments on \"w/o prediction\" in the authors' Table 3 also demonstrate that predicting the future has a substantial impact on performance improvement. Based on the above, is the core reason for the performance improvement of UP-VLA the ability of the VLA model to predict the future, regardless of the specific prediction method? (This point relates to the Claims and Evidence section.)\n\n3. 3D-VLA is not a good baseline. First, its performance is not very strong to begin with. Additionally, its control mode requires reasoning over many steps at once, which prevents the model from dynamically adjusting its actions based on current observations. Furthermore, prediction errors from earlier steps in the point cloud can accumulate significantly. As a result, it is not an appropriate baseline to demonstrate the advantages of the proposed method.\n\n4. My suggestion is for the authors to compare their method with Seer [4], VPP [5], and RoboVLMs [6]. Both [4] and [5] also focus on Prediction and achieve excellent performance, while [6] falls under the VLA category and also performs very well. From this perspective, I do not see a clear necessity for using UP-VLA, which is crucial for me to understand the contribution of this work.\n\n\n\n[1]  VLAS: Vision-Language-Action Model With Speech Instructions For Customized Robot Manipulation, ICLR25\n[2] Accelerating Vision-Language-Action Model Integrated with Action Chunking via Parallel Decoding\n[3] Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success\n[4] Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation, ICLR 25\n[5] Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations\n[6] Towards Generalist Robot Policies: What Matters in Building Vision-Language-Action Models", "relation_to_broader_scientific_literature": "I think combining VLA with a world model has potential, but based on the author's current explanation, I don't see it yet.", "essential_references_not_discussed": "[0] How Far is Video Generation from World Model: A Physical Law Perspective;\n[1]  VLAS: Vision-Language-Action Model With Speech Instructions For Customized Robot Manipulation, ICLR25\n[2] Accelerating Vision-Language-Action Model Integrated with Action Chunking via Parallel Decoding\n[3] Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success\n[4] Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation, ICLR 25\n[5] Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations\n[6] Towards Generalist Robot Policies: What Matters in Building Vision-Language-Action Models", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "Please refer to the experiments.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ySOY5L0UG", "reviewer": "Reviewer_oZvA", "summary": "This paper introduces UP-VLA, a vision-language-action model that can understand, generate predicted future images, and plan actions in the embodied environment. It devises a novel VLA training paradigm that unifies policy learning with visual prediction and multi-modal understanding. The results show that the use of future image prediction can significantly improve the precision and visual generalization of policy.\n\n## update after rebuttal: \n\nAfter reading the results from the reviewer dmnE and FLHB, I was convinced and recognize the contribution \"pretraining with mixed multi-modal understanding and future prediction objectives helps the VLA models\". So I will change the score to \"Weekly accept\".", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and efforts in reviewing our paper! Based on your review, we added a detailed discussion and additional experiments. \n\n---\n\n**Q1: About stronger baselines.**\n\nANS: Thank you for your suggestions! We conduct more experiment on Pi0, OpenVLA, and Octo model. Since these methods only conduct experiments in real world which is hard to replicated, we utilize their open-source code to fine-tune and test them on the Calvin ABC-D (with settings consistent with UP-VLA). The results on the ABC-D task are shown in the table below, where an asterisk* indicates the results we reproduced using the open-source code.\n\n|  Method  |      Type      | Avg. Len ↑ |\n| :------: | :------------: | :--------: |\n| Openvla* |      VLA       |    1.60    |\n|  Octo*   |      VLA       |    0.59    |\n|   pi0*   |      VLA       |    3.63    |\n|  UP-VLA  | Prediction&VLA |    4.08    |\n\nFor the pi0-fast method, it primarily improves training efficiency by preprocessing actions. In terms of performance, it is almost identical to pi0. Therefore, we only present the reproduced results of pi0 here.\n\n---\n\n**Q2: I suggest the author to dig some extra benefits of your investigate, otherwise you have to compare with the sota results. For example, the predicted image is pretty good and could enforce some downstream tasks?**\n\nANS: Thanks for your insightful suggestions. We provide discussions from the following aspects:\n\n(1) Performance: As you noted, OpenVLA and Pi0-series models leverage powerful VLMs to enhance semantic understanding, similar to our UP-VLA, which employs Showo, a Unified VLM. However, we observed that these methods are good at semantic understanding but lack action precision. The results in Table [Q1] highlight their limitations on the Calvin benchmark. We notice the primary failure mode of these methods in CALVIN stems from imprecise actions(e.g., the robot arm correctly approaches the target object based on the command but fails to grasp it). In contrast, our approach mitigates these shortcomings by integrating visual prediction into the VLA model, leading to more precise actions and superior performance, as shown in Table 1 and Figure 6 of our paper. \n\n(2) Additional benefits of our proposed unified VLA training paradigm: Previous VLA models did not fully exploit the temporal and visual information encoded in video datasets. Our approach introduces a scalable training paradigm that integrates MMU datasets, video datasets, and robot datasets. This allows the robot to benefit from the language comprehension capabilities of VLMs while also excelling in fine-grained manipulation tasks with complex dynamics. As you mentioned, we indeed found that **precise visual prediction can leads to precise actions**, which is a key factor in our method's effectiveness. Some of these precise predictions are visualized in Figure 7. As a result, UP-VLA demonstrates strong performance on both the Calvin benchmark and real-world scenarios.\n\n---\n\n**Q3: I think the framework in this paper is hard to train.**\n\nANS: We completely agree that training with multiple objectives across multiple datasets can be more challenging than training with a single objective. Fortunately, several works in the generative modeling field have unified understanding and generation within a single model, such as TransFusion, Show-O, and others. These works provide a valuable foundation for our approach. In this paper, we build UP-VLA upon Show-O, and we have included all the source code in the supplementary material. Feel free to check it for more details on the training process!\n\n\n---\n\n**Q4: May have to carefully design the ablation study**\n\nANS: Yes! We have conducted comprehensive ablation studies by systematically removing different components respectively from our original framework to verify the effectiveness of each training objective. As reported in Table 3 of the original paper, eliminating multi-modal understanding pretraining, bridge image prediction pretraining, or image prediction fine-tuning all result in **a clear performance drop**.\n\n\nAlso, following Reviewer-dmnE, we conduct extra ablation study on multi-steps. Specifically, we change the length of action chunk with and without visual prediction to validate the effectiveness of our proposed visual prediction objective.\n\n| Ablating Effectiveness of Visual Prediction |   1   |   4   |   7   |  10  |\n| :-----------------------------------------: | :---: | :---: | :---: | :--: |\n|          w/o predict future image           | 1.44 | 1.94 | 2.17 | 2.25 |\n|           w/ predict future image           | 2.42 | 3.72 | 4.00 | 4.08 |\n\n---\n\nThank you again for your time and effort in reviewing our work! We hope our clarification can solve all your concerns, and we are always ready to answer any further questions!", "claims_and_evidence": "The claims made in the submission supported by clear and convincing evidence", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes, the experiment well designed with good analyses.", "relation_to_broader_scientific_literature": "This paper contributes for the direction of VLA foundation model training, which is a foundmental and important area. The proposed training strategy and model design are valuable for that community.", "essential_references_not_discussed": "All essential references are discussed.", "other_strengths_and_weaknesses": "The VLA is a big and important community, and the solution of this paper is reasonable.  \nExperiments are sufficient and solid.", "other_comments_or_suggestions": "I think this paper lying a competitive area which is highly care about the metric number.\nI think the baseline could be better and stronger e.g.\nopenvla/octo/openvla-oft\npi0 series\n - https://www.physicalintelligence.company/blog/pi0\n - https://www.physicalintelligence.company/research/fast\nVLA + reasoning\n - https://embodied-cot.github.io/\n\nI suggest the author to dig some extra benefits of your investigate, otherwise you have to compare with the sota results. For example, the predicted image is pretty good and could enforce some downstream tasks?\n\nBesides that, I think the framework in this paper is hard to train. May have to carefully desgin the ablation study which is not enough in this paper.", "questions_for_authors": "refer to the aboves.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FamgeYIlop", "reviewer": "Reviewer_dQTi", "summary": "The paper presents UP-VLA, a unified Vision-Language-Action (VLA) model designed for embodied agents. The model aims to enhance both high-level semantic comprehension and low-level spatial understanding by integrating multi-modal understanding and future prediction objectives while current VLMs focus on high-level semantics while neglecting fine-grained spatial and dynamic features.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and efforts in reviewing our paper! Based on your review, we added a detailed discussion and additional experiments. \n\n---\n\n**Q1: About model efficiency**\n\nANS:\nWe would like to clarify that UP-VLA operates at **almost the same control frequency as previous VLA methods**. We did not use autoregressive image generation, we **jointly predict future images and actions** with a single forward LLM pass during policy execution, resulting in almost similar computational cost to prior VLA models. The table below presents runtime efficiency based on the same backbone (Showo), comparing models with and without image prediction (measured as the reciprocal of the average inference time over 100 iterations on the same hardware):\n\n| Method        | Inference Speed ↑ |\n|---------------|-------------------|\n| UP-VLA-RT-2*  | 13.2Hz            |\n| UP-VLA        | 13.0Hz            |\n\nAdditionally, our method employs an action chunking size of 10. We observed that action chunk can further enhances performance and produces smoother trajectories.\n\n\n---\n\n**Q2: When pretraining on an internet video dataset, if robotic videos are used for future prediction, why not pretrain action prediction simultaneously?**\n\n\nANS: Thank you for your question. We did not include action prediction in pretraining stage based on these two reasons: (1) Different robot datasets have varying action spaces with distinct meanings, requiring additional design effort to standardize them under a unified output format. (2) We believe that robot trajectory videos provide more consistent representations across datasets and already contain valuable dynamics information.\n\nFollowing your suggestion, we conducted an additional experiment incorporating robot action learning during pretraining while keeping the architecture unchanged (by padding actions to the same dimensions). The results indicate that this modification provides no benefit. We hypothesize that this is because RGB images already contain sufficient prior information, and the differences in action spaces across robots make direct action prediction less effective.\n\n| CALVIN ABC-D        | Avg Len ↑ |\n|---------------|-------------------|\n| UP-VLA              | 4.08      |\n| UP-VLA with Action Pretraining        |  3.98            |\n\n\n---\n\n**Q3: About details: Which internet videos were used for pretraining? Why are human hands present in Figure 5? Why is there no execution video demonstration?**\n\nANS: We use the Bridge dataset for image-prediction pretraining, as mentioned in Section 5.2. The human hands in Figure 5 serve as disturbances during policy rollout. For your convenience, we have included additional execution details in the following anonymous video links: https://sites.google.com/view/upvla-rebuttal.\n\n---\n**Q4: About SOTA autoregressive-based VLA methods**\n\nANS: Following your suggestions, we conducted experiments on the influential models pi0, openvla, and octo. Since these methods do not have official test results, we utilize the open-source code to fine-tune and test them on the calvin data (with settings consistent with UP-VLA). The results on the ABC-D task are shown in the table below, where an asterisk indicates the results we reproduced using the open-source code.\n\n|  Method  |      Type      | Avg. Len ↑ |\n| :------: | :------------: | :--------: |\n| Openvla* |      VLA       |    1.60    |\n|  Octo*   |      VLA       |    0.59    |\n|   pi0*   |      VLA       |    3.63    |\n|  UP-VLA  | Prediction&VLA |    4.08    |\n\n---\n\nThank you again for your time and effort in reviewing our work! We hope our clarification can solve all your concerns and shows the improved quality of our paper!", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "Please refer to 'Other Strengths and Weaknesses.'", "theoretical_claims": "Application paper without making theoretical claims.", "experimental_designs_or_analyses": "Please refer to 'Other Strengths and Weaknesses.'", "relation_to_broader_scientific_literature": "Effectively constructing a VLA model for robotics.", "essential_references_not_discussed": "This paper lacks a comparison with the latest VLA models.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper proposes a unified training approach that merges semantic understanding and future prediction, incorporating three types of datasets —— Vision-Language datasets for high-level semantic understanding, Internet video datasets for low-level visual dynamics, and Robotic action datasets for embodied control. Through the constructed dataset and training strategy, UP-VLA explicitly enhances physical spatial comprehension. \n\n2. UP-VLA achieved promising results in the Calvin simulation benchmark, demonstrating strong long-horizon manipulation capabilities.\n\n\n\nWeaknesses:\n\n1. The primary concern is the model's efficiency. While incorporating future image prediction before action prediction can significantly enhance performance, it raises the question of whether this autoregressive next-token prediction paradigm could lead to a substantial decrease in inference speed. The author's exploration is interesting, but inference speed plays a critical role in determining the robot's control frequency, which is essential in robotics.\n\n2. When pretraining on an internet video dataset, if robotic videos are used for future prediction, why not pretrain action prediction simultaneously? For example, leveraging large-scale simulator data or real-world robotic data (e.g., Open X-Embodied or DROID).\n\n3. The paper's writing is incomplete and lacks many details. For example: Which internet videos were used for pretraining? Why are human hands present in Figure 5? Why is there no execution video demonstration?\n\n4. Lacks comparison with SOTA autoregressive-based VLA methods, such as OpenVLA.", "other_comments_or_suggestions": "Please refer to 'Other Strengths and Weaknesses.'", "questions_for_authors": "Please refer to 'Other Strengths and Weaknesses.'", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZQ5vOXbR1K", "reviewer": "Reviewer_FLHB", "summary": "This paper presents a new training paradigm for Vision-Language-Action (VLA) models by training with both multi-modal understanding and future prediction objectives. Multi-modal Understanding is in the form of question-answering given a paired image-text, and the future visual prediction aims to predict the target image given the current image and instruction pair. Prior research usually tackles one of these two, which the authors frame as \"high-level semantic content\" and \"low-level feature\". Experiments are conducted on the simulation CALVIN benchmark and a real-world manipulation environment to verify the effectiveness of our UP-VLA framework.\n\n## post rebuttal\nI appreciate the authors' rebuttal, especially about pretraining and extra experiment results. After reading opinions from other reviewers, I think the paper presents clear evidence that pretraining with mixed multi-modal understanding and future prediction objectives helps the VLA models, and will keep my original recommendation as accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and efforts in reviewing our paper! Based on your review, we added a detailed discussion. \n\n---\n\n**Q1: I'm curious about the quantitative performance of multi-modal and visual prediction after the pretraining stage, on top of the action performance currently presented in the paper.**\n\nANS: We would like to clarify that our primary focus is on robot decision-making. Due to limited computational resources, our pretraining stage uses the LLaVA instruction-tuning dataset for multi-modal understanding (MMU) training and the Bridge robot dataset for future visual prediction training. Our pretraining process adapts the general-purpose Show-o model to robotic domain-specific tasks, which may come at the cost of reduced performance in general settings, such as VQA, OCR, and text-to-image generation. In our approach, we expect the VLM to learn visual predictions that are consistent with language. Although this approach may lead to the loss of ability on general tasks, it will be more conducive to visual prediction and low-level action learning. Results on both simulation and real-world show that this adaptation significantly benefits robot control tasks.\n\n\n\n---\n\n**Q2: Another thing I find interesting, but currently missing in the paper is potentially the insights from the pretraining. Do the two objectives conflict with each other? Or is the pretraining simply just working?**\n\n\nANS: Thank you for your insightful comments! Pretraining models for both multi-modal understanding and generation is becoming increasingly popular. Many works in the generative modeling field have explored mixed-pretraining approaches, such as TransFusion [1] and Show-o [2]. Their original objective functions focus on multi-modal understanding and text-to-image generation.\n\nInspired by the success of these models, we designed an objective function tailored for embodied agents, combining MMU tasks with future image prediction. Since our model builds upon the Show-o framework, we observed that pretraining proceeds without conflicts, and the training remains stable without collapse. \n\n[1] Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model https://arxiv.org/abs/2408.11039 \n\n[2]Show-o: One Single Transformer to Unify Multimodal Understanding and Generation https://arxiv.org/abs/2408.12528\n\n---\n\nThank you again for your time and effort in reviewing our work! We hope this clarification can solve your concerns!", "claims_and_evidence": "I think the major claim of the paper \"combining both vision-language understanding and future prediction objectives, enabling the capture of both high-level semantic and low-level visual patterns essential for embodied agent\" is verified by the experimental results compared with the baselines on the CALVIN benchmark and real-world manipulation tasks. It is further verified by the ablative study, where a significant difference is observed when dropping either of them.", "methods_and_evaluation_criteria": "While visual prediction and language prediction are quite mature in model architecture on their own, and there are existing models that aim to make multi-modal predictions, I believe the proposed method, especially the unified prompting and attention mechanism, is reasonable for verifying the effectiveness of different pre-training objectives.", "theoretical_claims": "No major theoretical claims are made in the paper.", "experimental_designs_or_analyses": "The experiment results on the CALVIN benchmark demonstrate the proposed outperforms the listed baselines on the zero-shot evaluation. However, this cannot fully verify the proposed dual-objective pretraining as the pretrained data and model architecture vary across different methods. This is similar in real-world manipulation tasks. The results difference looks statistically significant in both settings. The ablation results more evidently demonstrate the effectiveness of the proposed pretraining paradigm.", "relation_to_broader_scientific_literature": "I think to improve the effectiveness and generalizability is a major focus of the VLA models in the era of embodied AI, and I think the major problem the paper aims to tackle is relevant and important.", "essential_references_not_discussed": "Not that I'm aware of.", "other_strengths_and_weaknesses": "[+] I find the motivation of this paper to introduce both multi-model and visual prediction as pretraining is valid, the architecture design is reasonable and the experiments verify its effectiveness.\n\n[-]  I'm curious about the quantitative performance of multi-modal and visual prediction after the pretraining stage, on top of the action performance currently presented in the paper. Could the model address some multi-model tasks directly? How is the performance of next-frame prediction, working as a world model? I think providing such evaluations will further help people understand how the model performs before and after the fine-tuning stage.\n\n[-] Another thing I find interesting, but currently missing in the paper is potentially the insights from the pretraining. Do the two objectives conflict with each other? Or is the pretraining simply just working?", "other_comments_or_suggestions": "Missing punctuation (periods or commas) in the equations.", "questions_for_authors": "See the strengths and weaknesses section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes UP-VLA, a unified VLA model that integrates multi-modal understanding and future image prediction to improve embodied decision-making. The authors present extensive empirical studies across simulation and real-world tasks, and provide additional ablations and baseline comparisons during the rebuttal phase.


Reviewer dmnE and FLHB value the direction of unifying prediction and understanding in VLA and believe UP-VLA presents a compelling method. While acknowledging the limited novelty, the reviewer finds the practical challenges of such integration non-trivial and supports acceptance due to the model’s inspiration and empirical value to the field.
Reviewer dQTi initially raised concerns about inference efficiency, lack of SOTA VLA baselines (e.g., OpenVLA, pi0), and missing implementation details. After the rebuttal, the reviewer was satisfied with the additional experimental support.
Reviewer oZvA raised concerns over missing stronger baselines, limited insights from the visual prediction component, and insufficient ablation. The reviewer decided to maintain the original score after rebuttal.

In summary, while all reviewers agree that the technical novelty is somewhat modest, as UP-VLA builds largely on the existing Show-o framework, and that the claim of being “unified” may be overstated given its reliance on the Bridge dataset, the paper nonetheless addresses a timely and important problem. VLA research is still in its early stages, and the exploration of unified pretraining strategies is a meaningful contribution that could inform future developments in the field.
Decision: Accept (poster)